Formal for Post-Silicon Debug

Write a lab symptom as a cover and the lab notes as assumptions, read the shortest trace as a hypothesis, find which observation mattered, and prove the fix before the respin.

Lesson
F5.9
Time
15 min
Runs on
SymbiYosys
Check
4 questions

A chip hung once in a 72-hour lab run: the DMA said "waiting for grant" and the arbiter said "nothing pending". A week of simulation never reproduced it. The formal engineer wrote the two status bits as a cover and the lab notes as assumptions, and had a trace before lunch.

The abort lands in the one cycle the arbiter is gated off
0123456clkfreerstfreestartfreeabortclk_enreq_fsm012req_pendingarb_fsmstuckrequest raisedabort while clk_en is lowsymptom: stuckreported step 6

Measured with smtbmc yices, mode cover, depth 40, lab assumptions dma_en and cg_en. At step 4 the arbiter is gated off, so the abort clears req_pending before it was ever sampled; the requester keeps waiting.

The Symptom Is a Cover

From a Lab Symptom to a Proven FixPost-silicon formal asks "how could this have happened?" instead of "can anything bad happen?". Arrows mean "becomes".
call: who calls whom
From a lab symptom to a proven fixPost-silicon formal asks "how could this have happened?" instead of "can anything bad happen?". Arrows mean "becomes". component Lab symptom two status bits:waiting for grant,nothing pending component Lab notes DMA enabled, clockgating on component cover(stuck) the symptom as areachability question component assume(dma_en),assume(cg_en) the observations asenvironment component Shortest trace a hypothesis with awaveform component Fix, thenassert(!stuck)under prove the symptomimpossible from reset
Read the diagram as text
  • Lab symptom: two status bits: waiting for grant, nothing pending
  • Lab notes: DMA enabled, clock gating on
  • cover(stuck): the symptom as a reachability question
  • assume(dma_en), assume(cg_en): the observations as environment
  • Shortest trace: a hypothesis with a waveform
  • Fix, then assert(!stuck) under prove: the symptom impossible from reset
  • Lab symptom → cover(stuck)
  • Lab notes → assume(dma_en), assume(cg_en)
  • cover(stuck) → Shortest trace
  • assume(dma_en), assume(cg_en) → Shortest trace
  • Shortest trace → Fix, then assert(!stuck) under prove

You give the destination; the solver finds the journey. That is why formal fits a bug that random simulation missed.

Post-silicon formal turns the usual question around. The symptom (a register dump, two status bits, a stuck handshake) becomes a cover. What the lab observed becomes assume statements. The tool builds the shortest input sequence that matches.

On hs2 the cover is stuck = req_fsm == WAIT_GNT && arb_fsm == IDLE && !req_pending. Under the lab notes: Cover reached, reported step 6. The abort arrives in the one cycle when the request is pending, the arbiter has not sampled it, and its clock enable is low.

Which Observation Mattered

RunMeasured resultTime
Lab notes as assumptionsCover reached, reported step 6539 ms
No assumptionsCover reached, reported step 5520 ms
Clock gating forced offCover unreached within 40603 ms
assume(!abort): "the lab saw no abort"Cover unreached within 40586 ms
After the fix, coverCover unreached within 40601 ms
After the fix, assert(!stuck), abc pdrProve PASS (abc pdr)475 ms
After the fix, k = 4Prove PASS (k = 4)496 ms
After the fix, abc pdr, no lab assumptionsProve PASS (abc pdr)470 ms

Mode cover at depth 40 unless named, smtbmc yices, reset assumed in step 0.

Assumptions can only remove traces, so the run without them is never longer: 5 against 6. The solver then gated the clock itself. Forcing the gating off makes the symptom unreachable: the clock gating was essential, which tells the lab where to look.

The assume(!abort) row is the trap. The lab did not see an abort because it did not look; assuming one away removes the cause, and the symptom looks impossible (F3.2).

A Hypothesis, Then a Proof

The bug

R_WAIT: if (gnt) req_fsm <= R_ACT;
// an abort before the sample clears
// req_pending, but WAIT_GNT ignores it

Cover reached, reported step 6 under the lab notes.

The fix

R_WAIT: if (gnt) req_fsm <= R_ACT;
  else if (abort && arb_fsm == A_IDLE)
    req_fsm <= R_IDLE;

Cover unreached within 40; Prove PASS (abc pdr), with or without the lab notes.

A reached cover is one sequence that produces the symptom. The silicon may have failed another way, so replay the trace on the bench and prove the fix: the same assert, under the same environment, from reset.

This works only if the block's formal bench still exists when silicon returns, months after the RTL froze. Archive the bench, the suite version and the measured logs at tape-out.

Asked in Interviews

QuestionAnswer
Can formal reproduce a failure seen once in the lab?Yes, if a formal bench of the block exists: symptom as a cover, observations as assumptions, the trace as the hypothesis, the fix proven before the respin.
What must be in place for that?The block's formal environment, archived with the suite version and logs, surviving tape-out.
What is reactive formal?Using the bench to answer a question the lab or a bug report raises, as opposed to the planned proofs.

Try It

Symptom and fix check in SVA

cover property (@(posedge clk) !rst && stuck);
assert property (@(posedge clk) disable iff (rst) !stuck);

The same condition, first as a question, then as a claim.

As this engine runs the lab task

  always @(posedge clk) begin
    if ($initstate) begin assume(rst); end else begin assume(!rst); end
  end
  always @(*) begin assume(dma_en); assume(cg_en); end   // the lab notes
  always @(posedge clk) if (!rst) cover(stuck);

The lab notes are two combinational assumptions.

The lab task: mode cover, depth 40. The fix check uses mode prove with abc pdr.
[options]
mode cover
depth 40
[engines]
smtbmc yices
[script]
read_verilog -formal -sv design.sv
prep -top hs2
[files]
design.sv
  1. Write the symptom

    The lab saw "waiting for grant" from the requester and "idle, nothing pending" from the arbiter. Write the cover from those bits.

  2. Run under the lab notes

    Run as it is: Cover reached, reported step 6. Find the abort and the value of clk_en at that step.

  3. Drop the notes

    Delete the assumptions: Cover reached, reported step 5. Then assume !cg_en: Cover unreached within 40. Which observation mattered?

  4. Fix and prove

    Add the abort branch to WAIT_GNT, change the cover to assert(!stuck) and run abc pdr: Prove PASS (abc pdr).

design.sv

props.sv

Use assert property (@(posedge clk) ...);. Leave blank for immediate assertions in the design.

Inputs are free unless constrained. Run to find a counterexample or check a proof.

Model: one positive-edge clock, binary data. Sampled history requires declaration-initialized variables; primary-input history is unsupported.

SBY summary and log

Check Yourself

Predict the outpuths2 with the abort window, cover(stuck), mode cover, depth 40, smtbmc yices, reset assumed in step 0, assume(dma_en) and assume(cg_en). At which reported step is the cover reached?
module hs2(input clk, rst, start, abort, dma_en, cg_en, output gnt, output stuck);
  localparam R_IDLE=2'd0, R_REQ=2'd1, R_WAIT=2'd2, R_ACT=2'd3;
  localparam A_IDLE=2'd0, A_SAMPLE=2'd1, A_GRANT=2'd2;
  reg [1:0] req_fsm=R_IDLE, arb_fsm=A_IDLE;
  reg req_pending=0, clk_en=1;
  wire req = req_fsm == R_REQ;
  assign gnt = clk_en && arb_fsm == A_GRANT;
  always @(posedge clk)
    if (rst) begin req_fsm <= R_IDLE; arb_fsm <= A_IDLE; req_pending <= 0; clk_en <= 1; end
    else begin
      clk_en <= cg_en ? !clk_en : 1'b1;            // clock gating: the arbiter steps every other cycle
      case (req_fsm)
        R_IDLE: if (start && dma_en) req_fsm <= R_REQ;
        R_REQ:  req_fsm <= abort ? R_IDLE : R_WAIT;
        R_WAIT: if (gnt) req_fsm <= R_ACT;
        R_ACT:  req_fsm <= R_IDLE;
      endcase
      if (abort && arb_fsm == A_IDLE) req_pending <= 0;   // abort before the sample cancels the request
      else if (req) req_pending <= 1;
      else if (gnt) req_pending <= 0;
      if (clk_en) case (arb_fsm)
        A_IDLE:   if (req_pending) arb_fsm <= A_SAMPLE;
        A_SAMPLE: arb_fsm <= A_GRANT;
        default:  arb_fsm <= A_IDLE;
      endcase
    end
  assign stuck = req_fsm == R_WAIT && arb_fsm == A_IDLE && !req_pending;
`ifdef FORMAL
  always @(posedge clk) begin
    if ($initstate) begin assume(rst); end else begin assume(!rst); end
  end
  always @(*) begin assume(dma_en); assume(cg_en); end   // the lab notes
  always @(posedge clk) if (!rst) cover(stuck);
`endif
endmodule
ConceptWhich line in a post-silicon bench removes the cause of the symptom?
ConceptWhy is a reached cover a hypothesis rather than a diagnosis?
Read the waveformIn the lab trace, at which step does the abort fall inside the window, and what is clk_en then?
The abort lands in the one cycle the arbiter is gated off
0123456clkfreerstfreestartfreeabortclk_enreq_fsm012req_pendingarb_fsmstuckrequest raisedabort while clk_en is lowsymptom: stuckreported step 6

Measured with smtbmc yices, mode cover, depth 40, lab assumptions dma_en and cg_en. At step 4 the arbiter is gated off, so the abort clears req_pending before it was ever sampled; the requester keeps waiting.

Remember

  • Post-silicon formal writes the symptom as a cover and the lab observations as assumptions; the shortest trace is a hypothesis.
  • Dropping assumptions never lengthens the trace; forcing one off shows whether the bug needed it.
  • Never assume away what the lab did not observe; it may be the cause.
  • Prove the fix under the same environment, and keep the formal bench alive past tape-out.