Reading a Counterexample

Walk a formal trace backwards from the failing step, tell a real bug from a missing reset or assumption, and run covers before asserts.

Lesson
F1.4
Time
20 min
Runs on
SymbiYosys
Check
4 questions

A formal counterexample showed ack high with no request pending, and the designer spent a day in the RTL looking for the bug. The trace started with the pending flop already set in step 0, before any request: the bench had no reset assumption. The real bug was found the next morning, in a different trace, after the reset was assumed. Both traces failed on the same signal; the step-0 state told them apart.

Trace A: the ack after an abort
01234567clkfreerstfreereqfreeabortpendingpending_dbusy_cnt01230ackreqabort on the last busy cyclepending 0, pending_d 1ack with no pendingSBY reports FAIL

Measured: FAIL at reported step 7, 629 ms, depth 20, smtbmc yices. Reset is assumed in step 0.

Walk the Trace Backwards

  1. Step 6: the wrong signal

    ack is 1 while pending_q (the value of pending one step earlier) is 0. The check asks for a pending request behind every ack. There is none.

  2. Step 5: what made ack rise

    ack is registered, so look one step back. busy_cnt is 3 and pending_d is 1, but pending is already 0. The ack path read pending_d, a one-cycle-old copy.

  3. Step 4: the cause

    The shaded abort is 1 on the last busy cycle. The spec allows an abort at any time, so this input is legal. The design was right until here and an input turned it wrong: that is the cause. The fix is in the RTL, ack <= pending && busy_cnt == 3.

A counterexample reads backwards. Start at the marked step: which signal is wrong? Go one step back: which state bit or input made it so? Repeat until you reach a step where the design was right and a value turned it wrong. The shaded values are the solver's choices; if a shaded value is something the environment would never do, the fix is an assumption, not an RTL change.

Four Causes of a FAIL

Four Questions Before You Open the RTLArrows mean "if the answer is no, the fix is". Walk the trace from the failing step back to the start, then ask the questions left to right.
Four questions before you open the RTLArrows mean "if the answer is no, the fix is". Walk the trace from the failing step back to the start, then ask the questions left to right. component FAIL at step n walk back to thecause component Step 0 legal? could a reset produceit component Shaded inputslegal? would the spec allowthem component Check matchesthe spec? is the property toostrict component Model the reset assume rst in step 0 component Add an assumption on the input, from thespec component Fix the property the spec row, aswritten component RTL bug legal start, legalinputs, right check yesyesnononoyes
Read the diagram as text
  • FAIL at step n: walk back to the cause
  • Step 0 legal?: could a reset produce it
  • Shaded inputs legal?: would the spec allow them
  • Check matches the spec?: is the property too strict
  • Model the reset: assume rst in step 0
  • Add an assumption: on the input, from the spec
  • Fix the property: the spec row, as written
  • RTL bug: legal start, legal inputs, right check
  • FAIL at step n → Step 0 legal?
  • Step 0 legal? → Shaded inputs legal?: yes
  • Shaded inputs legal? → Check matches the spec?: yes
  • Step 0 legal? → Model the reset: no
  • Shaded inputs legal? → Add an assumption: no
  • Check matches the spec? → Fix the property: no
  • Check matches the spec? → RTL bug: yes

Only when the start state, the inputs and the check are all right is the counterexample a bug in the RTL. F3.3 turns this into a flow for protocol benches.

CauseThe tell in the traceOn this page
A bug in the RTLA legal start, legal inputs, a check that matches the spec row.Trace A: req, then abort on the last busy cycle, then a stale ack.
A missing assumptionA shaded input value the environment would never produce.Trace A again, if the spec forbade abort during busy. The diagnosis depends on the spec, not the trace.
A wrong propertyThe check is stricter than the spec.Asking for ack exactly one cycle after busy_cnt reaches 3 when the spec allows two.
Reset not modelledStep 0 holds a state no reset produces.Trace B: ack 1 and pending_q 0 at step 0, fail reported at 1.

Two traces that fail on the same signal can need opposite fixes. The first step and the shaded rows decide.

Trace B: the same check with no reset assumption
01clkfreerstfreereqfreeabortpendingpending_qackfree initial stateSBY reports FAIL

Measured: FAIL at reported step 1, 601 ms. The failing state is step 0 itself. F1.6 is about this.

Wiggle Before You Assert

Wiggling: the cover req ##[1:5] ack, before any assert
01234567clkfreerstfreereqfreeabortpendingbusy_cnt01230ackwin0123450reqack, 5 cycles laterreported step 7

Measured: Cover reached, reported step 7, 602 ms. The sequence looks like the spec: the environment is sane enough to assert against.

The habit for a new bench: write a cover for each behaviour you expect, run, and read the traces before you write a single assert. Short, odd-looking traces at this stage are normal; they show which flops the reset missed and which inputs still need an assumption. Here the cover shows a request, four busy cycles and an acknowledge, so the environment is fit to assert against. Only then run the assert and read its trace the same way. The $past in the property needs a value at the first tick (A1.2 Clocks and sampling); the engine's form keeps that value in pending_q, which starts at 0.

Asked in Interviews

QuestionAnswer
Is the counterexample a real bug?Walk it back. Check the start state against the reset, the shaded inputs against the spec, the check against the spec row; only then the RTL.
What is wiggling?Running covers first to see the design move under the tool's inputs before asserting anything.
What would make you add an assumption instead of fixing RTL?A shaded input value the real environment cannot produce.

Try It

Concurrent SVA

assert property (@(posedge clk) disable iff (rst) ack |-> $past(pending));
cover property (@(posedge clk) disable iff (rst) req ##[1:5] ack);

Every ack had a pending request one cycle earlier; a request can reach an ack within five cycles.

As this engine runs it

reg pending_q=0;
always @(posedge clk) begin
  pending_q <= pending;
  if (!rst && ack) assert(pending_q);
end
reg [2:0] win=0;   // cycles since req, 1..5
always @(posedge clk) begin
  win <= rst ? 0 : req ? 1 : (win && win < 5) ? win + 1 : 0;
  if (!rst) cover(ack && win != 0);
end

pending_q is the $past; win is the ##[1:5] window. F3.1 teaches the translation.

The task. Mode cover for the first step, mode bmc for the rest; the Mode selector sets it.
[options]
mode bmc
depth 20
[engines]
smtbmc yices
[script]
read_verilog -formal -sv design.sv
prep -top hs_ctrl
[files]
design.sv
  1. Wiggle

    Set the mode to cover and run. Read the trace: a request, busy_cnt 1, 2, 3, then ack. The cover is reported at step 7.

  2. The bug trace

    Set the mode to bmc and run: FAIL at reported step 7. Walk back from step 6 to the shaded abort at step 4. RTL, assumption, property or reset?

  3. The reset trace

    Fix the RTL (ack <= pending && busy_cnt == 2'd3), delete the two assume lines and every =0 initial value, and run. FAIL at reported step 1 with ack already high at step 0. RTL, assumption, property or reset?

  4. Both green

    Put the reset assumption back (or the initial values): BMC PASS to depth 20.

design.sv

props.sv

Use assert property (@(posedge clk) ...);. Leave blank for immediate assertions in the design.

Inputs are free unless constrained. Run to find a counterexample or check a proof.

Model: one positive-edge clock, binary data. Sampled history requires declaration-initialized variables; primary-input history is unsupported.

SBY summary and log

Check Yourself

Predict the outpuths_ctrl with the bug fixed, no reset assumption, no declared initial values, mode bmc, default depth 20, smtbmc yices, the check ack implies pending_q. SBY reports the failure at which step?
module hs_ctrl(input clk, rst, req, abort, output reg ack);
  reg pending;
  reg pending_d;
  reg [1:0] busy_cnt;
  always @(posedge clk) begin
    if (rst) begin pending<=0; pending_d<=0; busy_cnt<=0; ack<=0; end
    else begin
      pending_d <= pending;
      if (abort) pending <= 0;
      else if (req) pending <= 1;
      else if (busy_cnt == 2'd3) pending <= 0;
      busy_cnt <= pending ? busy_cnt + 2'd1 : 2'd0;
      ack <= pending && busy_cnt == 2'd3;
    end
  end
  reg pending_q;
  always @(posedge clk) begin
    pending_q <= pending;
    if (!rst && ack) assert(pending_q);
  end
endmodule
ConceptWhich diagnosis is wrong?
ConceptWhy read a counterexample backwards?
Read the waveformIn Trace A, which shaded input, at which step, caused the failure?
Trace A: the ack after an abort
01234567clkfreerstfreereqfreeabortpendingpending_dbusy_cnt01230ackreqabort on the last busy cyclepending 0, pending_d 1ack with no pendingSBY reports FAIL

Measured: FAIL at reported step 7, 629 ms, depth 20, smtbmc yices. Reset is assumed in step 0.

Remember

  • Read a counterexample backwards from the failing step to the first value that turned the design wrong.
  • Four causes of a FAIL: RTL bug, missing assumption, wrong property, reset not modelled. The step-0 state and the shaded inputs decide.
  • A shaded value the environment could never produce calls for an assumption, not an RTL change.
  • Wiggle first: covers show the design moving under free inputs before any assert can mislead you.