Debug case: a racing monitor

The symptoms of a race

A race rarely looks like one. The usual signs are results that change between simulators, between versions of one simulator, or after edits that should not matter, such as adding a file or reordering always blocks.

Off-by-one-cycle mismatches are the most common form. The monitor records a value one beat early or late, or misses a transfer at the edge where the driver changed rvalid.

When a failure moves for reasons unrelated to the design, suspect a race before suspecting the DUT. Changing the simulator, the compile order or the random seed of an unrelated component are all good ways to provoke one on purpose.

What is racing

The driver resumes at @(posedge clk) and writes rdata = next;. The monitor, an always @(posedge clk) block, resumes at the same edge and reads rdata. Both are processes in the Active region of one time step.

The standard does not fix the order of those two processes. If the driver runs first, the monitor sees the new beat; if the monitor runs first, it sees the old one. Either is legal, so the testbench has no defined behaviour at all.

The DUT is exposed in the same way. Its always @(posedge clk) logic reads rvalid and rdata at that edge too, so even a perfect monitor cannot make this testbench correct.

Finding the racing signal

Look for testbench signals that change at exactly the timestamp of a sampling edge. In a waveform, zoom in on a rising edge and see which signals change at that same time rather than a little before or after it.

You can also count such changes in the simulation, as the example does: a small process notes every rising edge at which a bus signal changes. The time at which a write happens is fixed, even though the order of processes within that time step is not, so the count is reliable.

Fix both sides

Fix the driver first, because the DUT reads its signals too. Driving through a clocking block puts every change at the output skew, after the edge, so neither the DUT nor any monitor can see it at that edge.

Then fix the monitor: read through a clocking block with #1step inputs and wait on @(bus.mon_cb). A monitor built that way sees pre-edge values whatever any other process does, so it stays correct even if someone later adds a racy driver.

@(posedge clk);
bus.rvalid = 1;
bus.rdata  = next;

@(bus.drv_cb);
bus.drv_cb.rvalid <= 1;
bus.drv_cb.rdata  <= next;

Fixes that only move the race

Adding #1 before the driver's write moves it off the edge, but it ties correctness to the time unit and to the clock period. A later change to either brings the race back. Adding #0 moves the write into the Inactive region, which reorders it against some processes but not all.

Swapping the order of always blocks in the file, or changing a simulator option, may make the failure disappear on one tool without removing the race. A fix is real only when the standard defines the result.

A regression for the fix

Keep the change counter from the example in the testbench, and fail the test if it ever finds a bus change at a sampling edge. It costs a few lines and catches the next racy driver the day it is written.

Then run the test on every simulator you have. With the race gone, all of them must agree beat for beat, and a difference between them is worth investigating rather than explaining away.

Try it

The racy driver changed the bus at 4 rising edges, each one a point where the DUT or a raw monitor could see either value. The clean driver changed it at none, and the clocking-block monitor recorded the three beats in order.

In the racy driver, change @(posedge clk) to @(negedge clk) in both places and predict the first line again.

Check yourself

module tb;
  logic clk = 0;
  always #5 clk = ~clk;
  logic a = 0, b = 0;
  int n;
  time last = -1;
  always @(a or b)
    if ($time % 10 == 5 && $time != last) begin n++; last = $time; end
  initial begin
    @(posedge clk); a = 1;
    @(negedge clk); b = 1;
    @(posedge clk); a <= 0;
    #20 $display("%0d", n);
    $finish;
  end
endmodule

@(negedge clk);
bus.rvalid <= 1;
bus.rdata  <= 8'ha1 + i;

@(posedge clk);
bus.rvalid = 1;
bus.rdata  = 8'ha1 + i;

@(negedge clk);
bus.rvalid = 1;
bus.rdata  = 8'ha1 + i;

@(bus.drv_cb);
bus.drv_cb.rvalid <= 1;
bus.drv_cb.rdata  <= 8'ha1 + i;

Practice

• The monitor that saw the future Debug · Medium