Coverage and tests for the checker

Measure which situations your stimulus reached with covergroups, and prove that the checker can fail by running it against deliberately broken FIFOs.

Lesson: 7.5
Time: 30 min
Runs on: Coverage build
Check: 5 questions

A FIFO regression reached 100% functional coverage, passed every night for months, and still let a FIFO with a flipped data bit reach silicon. A refactor had moved the data comparison into a branch that never ran, so the scoreboard compared nothing. Coverage had measured what the stimulus did, and nothing had measured whether the checker could fail.

Coverage rises; checking never happens

One column per week of regressions. Every number looks healthy except the one nobody printed.

Two questions, two tools

A testbench has to answer two separate questions. Coverage asks which situations the stimulus produced: did a read and a write ever arrive together at full? Checking asks whether the FIFO behaved correctly in the situations that did occur.

Question	Answered by	Fooled by
Did the stimulus reach every situation in the plan?	covergroups and scenario counters	stimulus that never reaches a corner
Was the FIFO right whenever it was checked?	scoreboard and assertions	a checker that cannot fail
Can the checker fail?	running it against broken designs	testing only the correct design

Each question needs its own evidence.

The three questions depend on each other: checking means little where the stimulus never went, and coverage means little if the checks could not fail.

A green run with full coverage proves neither of the last two. The third question is the one most teams never ask, and it is the subject of the second half of this lesson.

A covergroup for the FIFO

fifo_coverage holds two covergroups. The first crosses the fill level before the edge (empty, in between or full) with the requests (idle, read, write or both). That gives 12 combinations, including the ones the test plan worries about most. The second records resets while empty and while holding data.

Bins are named for what they mean, such as empty and both, so a coverage report reads like the test plan. When a bin stays at zero, its name says which situation the stimulus never produced.

Both are sampled with the scoreboard's count from before the edge, so the bins describe the situation the FIFO faced, independently of anything the FIFO reported. get_coverage() on a covergroup returns the average of its coverpoints and crosses, so a cross with 12 bins pulls the total down until most of them are hit.

Test the checker with broken FIFOs

To show that a checker can fail, give it designs that are wrong on purpose. The FIFO in this lesson has a test-only variable fault that the testbench sets before each run. Fault 0 is the correct design, and each other number breaks one rule, with the same numbering as the faulty FIFOs in the problem bank.

The example runs the same seeded stimulus against every variant, with a fresh scoreboard each time. A broken FIFO must be caught, with at least one error; the correct one must be clean. An error on the correct design is a false alarm, and a broken design with no errors is missed.

Fault	What breaks	Measured result
0	nothing: the correct design	clean
1	bit 0 of read data flipped	caught, 25 errors
2	`read_valid` stuck at 1 after a read	caught, 34 errors
3	`read_valid` never rises	caught, 26 errors
5	read and write together return the new item	caught, 8 errors
6	write pointer wraps at 4 although the depth is 3	caught, 15 errors
8	`full` never rises	caught, 16 errors
9	reset keeps the stored items	caught, 8 errors

Results of the example: 60 random operations plus the directed cases, seed 7, depth 3.

The error counts say how often each bug became visible with this stimulus. Fault 5 shows up only when a read and a write meet with items stored, so a stimulus that rarely produces that situation would catch it rarely, or never.

When a fault is missed

A missed fault has one of two causes. Either the stimulus never produced the situation where the bug shows, which coverage can reveal, or the checker looked at the wrong thing, which coverage cannot.

The second kind is the opening story. A checker that compares read_data only when the FIFO raises read_valid misses a read_valid stuck at 1, because the stale data it then compares is still the item it expected. Running the checker against fault 2 exposes that hole in a second.

Run the broken designs whenever the scoreboard changes, not just once. A checker that caught every fault last month can lose that ability in a single refactor, which is exactly what happened in the opening story.

Keep the broken designs next to the correct one, in the same regression. The problem bank's Catch the faulty FIFOs asks you to write such a checker as a module and grades it against these faults.

Layer	Component	Job	Reuses	Status
Test	`fifo_test`, `fifo_config`	chooses depth, seed and length, runs the environment, gives the verdict	Module 4 classes	lesson 7.7
Generator	`fifo_generator`	creates `fifo_op` objects: directed, then seeded random	Modules 4 and 5	built in 7.3
Mailboxes	`mailbox #(fifo_op)`, `mailbox #(fifo_obs)`	carry operations and observations between processes	Module 3	built in 7.3
Driver	`fifo_driver`	drives one operation per clock through `drv_cb`	Module 6	built in 7.3
Monitor	`fifo_monitor`	samples the pins through `mon_cb` into a new `fifo_obs`	Module 6	built in 7.4
Scoreboard	`fifo_scoreboard`	predicts each edge with the queue `exp_q` and compares	Modules 2 and 4	built in 7.4
Assertions	`fifo_props`	checks timing rules on the pins at every edge	Module 6	built in 7.4
Coverage	`fifo_coverage`, `fifo_scenarios`	measures which situations the stimulus reached	Module 5 stimulus	this lesson
DUT	`fifo`	the design under test	lesson 7.1 rules	built in 7.2

The layered testbench after lesson 7.5.

Try it

The correct design is clean, every broken design is caught, and the stimulus reached all 12 level and request combinations plus a reset while holding data. All three questions now have an answer.

Change new(req, 60, 7) to new(req, 0, 7), so only the directed cases run, and predict which faults are still caught and what the coverage becomes.

RUNVerilator (coverage build) · simulator service

Edit the code, then run it

Expected: fault 0 correct design clean (0 errors) fault 1 read data bit 0 flipped caught (25 errors) fault 2 read_valid stuck high caught (34 errors) fault 3 read_valid never rises caught (26 errors) fault 5 read and write: new value caught (8 errors) fault 6 write pointer wraps at 4 caught (15 errors) fault 8 full never rises caught (16 errors) fault 9 reset keeps stored entries caught (8 errors) stimulus coverage: level x request 100.0%, reset 100.0%

Check yourself

Predict the outputThis covergroup is sampled three times. Predict the exact output.

module tb;
  bit [1:0] level, req;
  covergroup cg;
    cp_level: coverpoint level { bins empty = {0}; bins middle = {1}; bins full = {2}; }
    cp_req:   coverpoint req   { bins idle = {0}; bins read = {1}; bins write = {2}; bins both = {3}; }
    level_x_req: cross cp_level, cp_req;
  endgroup
  cg c = new();
  initial begin
    level = 0; req = 2; c.sample();   // write at empty
    level = 1; req = 3; c.sample();   // both in between
    level = 2; req = 1; c.sample();   // read at full
    $display("coverage=%0.1f", c.get_coverage());
  end
endmodule

Spot the bugEach option is step 1 of check(), the check of the previous edge's answer. The harness runs the correct FIFO and fault 2, read_valid stuck at 1. The checker must stay clean on the first and catch the second. Which option misses fault 2?

if (o.read_valid != due || (due && o.read_data != due_data)) error("response");

if (o.read_valid != due) error("read_valid");
else if (due && o.read_data != due_data) error("read_data");

if (o.read_valid && !due) error("extra answer");
if (!o.read_valid && due) error("missing answer");
if (due && o.read_data != due_data) error("read_data");

if (o.read_valid && o.read_data != due_data) error("read_data");

ConceptA regression reports 100% functional coverage and 0 errors. What does that establish?

That the stimulus reached every planned situation, and nothing moreThat the regression is complete and no further tests are neededThat the FIFO is correct in every situation the coverage plan listsThat the checker has been shown to fail on broken designs

ConceptWhy does the example run the correct FIFO, fault 0, along with the broken ones?

To make sure the seeded stimulus is the same for every designTo catch false alarms: a checker that flags every design would look perfectIt is not needed: catching every fault already proves the checkerTo get an error count that the faulty runs can be compared against

ConceptA broken FIFO that returns the newly written item when a read and a write meet with items stored is missed. Coverage shows the "both, in between" bin was never hit. What should you change?

Nothing: a fault that never shows up does not matterThe assertions, so they check data orderThe stimulus, so reads and writes meet while items are storedThe scoreboard, because it failed to compare the data it was given

Remember

Coverage tells you what the stimulus reached; checking tells you whether the FIFO was right there.
Sample covergroups from the model's own state, and remember that a large cross dominates the total.
Prove the checker can fail by running it against deliberately broken designs, along with the correct one.
A missed fault means the stimulus never reached it, or the checker looked at the wrong signal.
Never let a checker decide when to compare from a signal it is supposed to check.

Practice

Two coverpoints for the FIFO Implement · Easy
The cross that can never close Fix · Medium