Chasing a One-in-500 Failure

Where the interviewer goes next. Open a node to practise that follow-up.

WhyWhat IfProve ItBreak ItCompareScale It

1. Chasing a One-in-500 Failure
2. WhyWhy a Seed Stops Reproducing After a ChangeL2 Mid-LevelL3 Senior

   Why can the failing seed stop failing after an unrelated testbench change, before any fix?

   Reveal the Model Answer

   A seed fixes the root generator, and every thread and object derives its own seed from its parent in creation order. Add a component, reorder construction or change a loop count, and later objects get different seeds and different stimulus. The bug is still there; the seed just no longer reaches it.

   1. Run 1

      An object's generator is seeded when the object is created, from the generator of the process that creates it. In run 1 the burst generator is created first and takes the first seed.

      Follow the calls, values and waits

      1. data first seed: Run 1: creating process → Burst generator
   2. Run 2

      In run 2 a debug object is created first and takes that seed. The burst generator gets the next one, so its whole sequence of bursts changes.

      Follow the calls, values and waits

      1. data first seed: Run 2: creating process → Debug object
      2. data second seed: Run 2: creating process → Burst generator
   3. Seed it on purpose

      Calling srandom right after new, with a seed derived from the run's seed, fixes the generator's stream. Adding or reordering other objects no longer shifts it.

      Follow the calls, values and waits

      1. call run seed + 1: srandom(seed + 1) → Burst generator

   Read the diagram as text

   • Run 1: creating process: its generator hands out seeds in order
   • Burst generator: created first: first seed
   • Run 2: creating process: the same run seed
   • Debug object: created first: first seed
   • Burst generator: created second: second seed
   • srandom(seed + 1): called right after new

   • Run 1: creating process → Burst generator: new
   • Run 2: creating process → Debug object: new
   • Run 2: creating process → Burst generator: new
   • srandom(seed + 1) → Burst generator: sets the seed

   Calls on one object never disturb another object's stream; only the order of creation hands out different seeds.

   Self-Check

   Tick each key point you covered in your answer.

   Key point 1 A seed fixes the root generator, and every thread and object derives its own seed from its parent in creation order. Key point 2 Add a component, reorder construction or change a loop count, and later objects get different seeds and different stimulus. Key point 3 The bug is still there; the seed just no longer reaches it.

   Misconception

   "The edit fixed the bug, since the seed passes now."

   Without a cover point showing the corner was reached, a passing seed after a change is not evidence of a fix.

   Source Spear & Tumbush, SystemVerilog for Verification (3rd ed.), ch. 6.16.3 (PDF p258)

   • Lesson 5.6Random Stability and Callbacks
3. What IfA Failure Its Own Seed Cannot ReproduceL3 Senior

   Asked if the rerun passes

   What if the same seed on the same build passes when you rerun it?

   Reveal the Model Answer

   Then something besides the seed decides the outcome. Look first for races between testbench and DUT, which can flip with process order. Then check uninitialised values, code that depends on $time or the environment, and the rerun command. Running the seed on a second simulator or with different process ordering often makes a race show itself.

   Self-Check

   Tick each key point you covered in your answer.

   Key point 1 Then something besides the seed decides the outcome. Key point 2 Look first for races between testbench and DUT, which can flip with process order. Key point 3 Then check uninitialised values, code that depends on $time or the environment, and the rerun command. Key point 4 Running the seed on a second simulator or with different process ordering often makes a race show itself.

   What Interviewers Listen For (Senior)

   A good senior answer lists the other inputs that can change the result, most likely first. It starts with races at the boundary between testbench and DUT.

   Wrong Altitude

   "Mark it flaky and move on."

   A failure that depends on process order is a real race. It will show up again on another simulator or version, usually at a bad time.

   • Lesson 6.7Debug Case: A Racing Monitor
4. Break ItA Testbench That Drives on the Sampling EdgeL1 JuniorL2 Mid-Level

   The testbench does @(posedge clk) start = 1; and the DUT samples start in always_ff @(posedge clk). Sometimes the DUT reacts a cycle early. Why?

   // Testbench
   initial begin
     @(posedge clk) start = 1;
   end
   // DUT
   always_ff @(posedge clk) accepted <= start;

   Reveal the Model Answer

   The testbench process and the DUT flop wake on the same edge, in the same region, and their order is unspecified. If the testbench runs first, the DUT samples the new start on that same edge. That is a cycle earlier than real hardware would. Drive after the edge, through a clocking block or with <=, so the DUT always samples the pre-edge value.

   With a blocking write, q depends on which process runs first. With a nonblocking write, the DUT always sees the value from before the edge.

   Self-Check

   Tick each key point you covered in your answer.

   Key point 1 The testbench process and the DUT flop wake on the same edge, in the same region, and their order is unspecified. Key point 2 If the testbench runs first, the DUT samples the new start on that same edge. Key point 3 That is a cycle earlier than real hardware would. Key point 4 Drive after the edge, through a clocking block or with <=, so the DUT always samples the pre-edge value.

   Half-Truth

   "Put #0 before the drive and the race is gone."

   #0 moves the process to the Inactive region. That works only until another process also uses #0. Then their order is unspecified again, and the result can differ between simulators.

   Source IEEE 1800-2017 §4.4.2.3; Spear & Tumbush, SystemVerilog for Verification (3rd ed.) ch. 4 p141

   Tool as Truth

   "It passes on our simulator, so the testbench is correct."

   Passing on one tool only shows one legal order. The standard allows either order, so the test can fail after a tool update.

   Source IEEE 1800-2017 §4.7

   • Lesson 6.3Clocking Blocks and Sampled Views
   • Lesson 6.7Debug Case: A Racing Monitor
   • Problem 6.7.1The Monitor That Saw the Future
5. Prove ItSolve 6.7.1: The Monitor That Saw the FutureSimulator-graded

   Show it, don't just say it: this Silicon Lab problem is graded by the simulator.

   Open Problem 6.7.1

6. CompareKeep the Random Seed or Write a Directed TestL3 Senior

   Once the bug is fixed, do you add the failing seed to the regression, or write a directed test for the corner? Compare them.

   Reveal the Model Answer

   A saved seed is cheap but brittle: the next testbench change moves its stimulus, and it quietly stops testing the corner. A directed test, or a constrained sequence aimed at the corner, keeps hitting it whatever else changes. Add a cover point for the corner as well, so the regression report shows whether random tests reach it on their own.

   Self-Check

   Tick each key point you covered in your answer.

   Key point 1 A saved seed is cheap but brittle: the next testbench change moves its stimulus, and it quietly stops testing the corner. Key point 2 A directed test, or a constrained sequence aimed at the corner, keeps hitting it whatever else changes. Key point 3 Add a cover point for the corner as well, so the regression report shows whether random tests reach it on their own.

   What Interviewers Listen For (Senior)

   A good senior answer prefers a directed or targeted test, because it keeps working after the bench changes. It adds a cover point as proof that the case is still hit.

   Half-Truth

   "Add the seed to a golden list and rerun it every night."

   It reruns the same seed, not the same stimulus. After a few changes it tests something else while still passing.

   • Lesson 5.6Random Stability and Callbacks
   • Lesson 7.5Coverage and Tests for the Checker
7. Scale ItTriage When the Nightly Shows Hundreds of FailuresL4 Staff

   At subsystem level the nightly run shows 300 failing seeds. How do you organise triage so the team works on causes, not symptoms?

   Reveal the Model Answer

   Bucket failures automatically by first error signature, with addresses and IDs stripped, and pick the shortest reproducing seed per bucket. Assign each bucket an owner and track it to a cause, because one bug often produces dozens of signatures downstream. Rerun failing seeds automatically to separate reproducible failures from flaky ones, and treat the flaky list as bugs to fix, not noise.

   Self-Check

   Tick each key point you covered in your answer.

   Key point 1 Bucket failures automatically by first error signature, with addresses and IDs stripped, and pick the shortest reproducing seed per bucket. Key point 2 Assign each bucket an owner and track it to a cause, because one bug often produces dozens of signatures downstream. Key point 3 Rerun failing seeds automatically to separate reproducible failures from flaky ones, and treat the flaky list as bugs to fix, not noise.

   What Interviewers Listen For (Staff)

   A good staff answer designs the triage flow and its numbers: buckets, owners and how often failures reproduce. It refuses to let flaky failures become accepted background noise.

   Wrong Altitude

   Assigns engineers to failures in log order.

   Several engineers end up debugging the same root cause while other buckets wait.

   • Lesson 7.6Reset, Wrap and Reproducible Stress