Seeds, coverage and directed closure
Record what makes a random run repeatable, measure which situations the stimulus reached, and close the holes that randomness cannot reach with a few directed items.
- Lesson
- U16.2
- Time
- 25 min
- Runs on
- Verilator with UVM
- Check
- 5 questions
A FIFO regression runs 64 random items per test, and its coverage report has been stuck at 7 of 9 bins for a week. More items and more seeds close one hole now and then, but never the other, "reset with data": a reset while the FIFO holds values. The item class has constraint c_no_reset { reset == 0; }, so no seed can ever produce that reset.
Measured by the example below. More random items cannot hit a bin that the item's constraints forbid.
A seed names one run
Random stimulus looks different every time you change the seed, but it is not unpredictable. A simulator starts its random number generator from a seed. With the same seed, the same code and the same tool version, every random choice repeats exactly, so a failing run can be replayed.
In UVM the simulator seed flows into one value, uvm_global_random_seed, which the library reads from $urandom when it starts. Every seed UVM hands out later is derived from it. A regression therefore records, for each run: the test name, the seed, the source revision, the UVM version and the simulator version. Change any of them and the run may no longer reproduce.
Say a nightly regression fails on seed 81 of 200. Rerun seed 81 on the same build first, to confirm the failure repeats. After the fix, rerun seed 81 to show it passes, then let new seeds run. A seed that passes today proves nothing about the other 199.
UVM reseeds every item by name
A bench changes all the time. If adding a component shifted every random value after it, every old seed would become useless. UVM avoids that with random stability: it gives each object its own seed. That seed is built from the global seed, the object's type name, its full name, and a count of earlier seed requests for that type and name. Reseeding a reused object advances that count too.
start_item does this to the item. It calls the item's reseed(), which reseeds by default because use_uvm_seeding is on. So the item's stream depends on the sequence and the item's name, not on what other threads did first. Run the same test with an extra component elsewhere, and this sequence still produces the same items.
One consequence surprises people. Calling it.srandom(42) before start_item has no effect, because start_item reseeds the item right after it. If you really must pin one item's values, seed it between start_item and randomize(). Usually you should not: pin the run with the global seed instead.
Coverage: did the stimulus reach the cases that matter
A passing scoreboard says that every value it checked was right. It says nothing about which situations were tried. Functional coverage answers that: you list the situations the design must face, each one a bin, and count how often the stimulus reached each.
| Bins | Hit when the monitor sees |
|---|---|
| level 0 to level 4 | The FIFO holding that many values after an edge |
| write when full | A write offered while full, so the design refuses it |
| read when empty | A read offered while empty |
| write and read | A write and a read accepted at the same edge |
| reset with data | A reset while values are still stored |
The nine bins of the FIFO bench. Every bin comes from the design's specification, not from its code.
In the bench, a subscriber called fifo_coverage receives the monitor's samples on the same analysis port as the scoreboard. It keeps a hit count per bin and can list the holes, the bins with no hits. A covergroup does the same job in standard SystemVerilog, with a coverpoint per variable and a bin per range.
Write the bins from the specification, before looking at the results. A bin list copied from the code only confirms that the code does what it does.
Holes that randomness cannot reach
Two kinds of hole look the same in a report. An unlucky hole needs more items or other seeds. An unreachable hole cannot be hit by this stimulus at all, and the hook's reset bin is one: c_no_reset forbids it. Before adding items, ask whether the constraints even allow the bin.
The example has one hole of each kind. "Write when full" is unlucky. Random writes and reads keep the level near the middle, so four net writes in a row are rare. "Reset with data" is unreachable.
The fix is a short directed sequence aimed at the holes, a step called coverage closure. Five writes make the last one meet a full FIFO, whatever level the random stage left. A reset then finds the FIFO holding data. Six items close the two bins that 64 random items missed.
Keep the directed items few and aimed, and keep the random stages too. Random traffic finds the combinations nobody thought to write down. Directed items then cover the few cases that the specification names and the constraints rule out.
Asks for a reset the class forbids
start_item(it);
if (!it.randomize() with { reset == 1; })
`uvm_error("RAND", "reset item failed")
finish_item(it);The inline constraint conflicts with c_no_reset, so randomize() fails and the fields keep their old values. The item goes out without a reset.
Switches the class constraint off first
start_item(it);
it.c_no_reset.constraint_mode(0);
if (!it.randomize() with { reset == 1; })
`uvm_error("RAND", "reset item failed")
finish_item(it);Only this item may reset. The other fields are still random, and every other item keeps c_no_reset.
Always check what randomize() returns. A failed call leaves the fields unchanged, and the item is still sent; only the check turns a silent no-op into an error you can see.
Checking and coverage answer different questions
Coverage and checking answer two separate questions, and a run needs both. A hit bin says that the situation happened. It does not say the design handled it well; the scoreboard decides that, value by value.
So close coverage with a working scoreboard attached, as the example does: 9 of 9 bins, and zero mismatches across the same run. A bin hit while the scoreboard is switched off proves only that the stimulus can get there.
Try it
The bench, from the FIFO and its monitor to the coverage subscriber, is in fifo_bench.svh. The test sends 64 random items, then six directed ones, and prints the holes after each stage.
Delete the constraint_mode(0) line and run again: randomize() fails, a RAND error appears, and the last line still shows the reset hole. Then add uvm_global_random_seed = 7; before run_test in tb and compare the random stages.
record: test closure_test, seed 3271625496 after 32 random items: 6 of 9 bins; holes: level 4, write when full, reset with data after 64 random items: 7 of 9 bins; holes: write when full, reset with data after 6 directed items: 9 of 9 bins; holes: none scoreboard: 24 writes, 20 reads, 4 canceled, 0 mismatches UVM_ERROR : 0Read included file: fifo_bench.svh
// The four-entry FIFO and its UVM bench.
interface fifo_if (input bit clk);
bit rst = 1, wr = 0, rd = 0;
bit [7:0] wdata = 0;
wire [7:0] rdata;
wire full, empty;
endinterface
module fifo_dut (fifo_if bus);
bit [7:0] memory[4];
bit [1:0] wp = 0, rp = 0;
bit [2:0] count = 0;
wire push = bus.wr && !bus.full; // a write is refused when full
wire pop = bus.rd && !bus.empty; // a read is refused when empty
assign bus.full = (count == 4);
assign bus.empty = (count == 0);
assign bus.rdata = memory[rp];
always @(posedge bus.clk)
if (bus.rst) begin wp <= 0; rp <= 0; count <= 0; end
else begin
if (push) begin memory[wp] <= bus.wdata; wp <= wp + 1'b1; end
if (pop) rp <= rp + 1'b1;
count <= count + push - pop;
end
endmodule
// One bus cycle the sequence asks for.
class fifo_item extends uvm_sequence_item;
`uvm_object_utils(fifo_item)
rand bit reset, wr, rd;
rand bit [7:0] data;
constraint c_no_reset { reset == 0; }
function new(string name = "fifo_item"); super.new(name); endfunction
endclass
// What the monitor saw at one clock edge.
class fifo_sample extends uvm_sequence_item;
`uvm_object_utils(fifo_sample)
bit reset, push, pop, write_full, read_empty, full, empty;
bit [7:0] wdata, rdata;
function new(string name = "fifo_sample"); super.new(name); endfunction
endclass
class fifo_driver extends uvm_driver #(fifo_item);
`uvm_component_utils(fifo_driver)
virtual fifo_if vif;
int items;
function new(string name, uvm_component parent); super.new(name, parent); endfunction
task run_phase(uvm_phase phase);
forever begin
@(negedge vif.clk);
seq_item_port.try_next_item(req);
if (req == null) begin // no item: an idle cycle
vif.rst <= 0; vif.wr <= 0; vif.rd <= 0;
end else begin
vif.rst <= req.reset; vif.wr <= req.wr; vif.rd <= req.rd; vif.wdata <= req.data;
@(posedge vif.clk);
items++;
seq_item_port.item_done();
end
end
endtask
endclass
class fifo_monitor extends uvm_monitor;
`uvm_component_utils(fifo_monitor)
virtual fifo_if vif;
uvm_analysis_port #(fifo_sample) ap;
function new(string name, uvm_component parent); super.new(name, parent); ap = new("ap", this); endfunction
task run_phase(uvm_phase phase);
forever begin
fifo_sample s;
@(posedge vif.clk); // values just before the design updates
s = fifo_sample::type_id::create("s");
s.reset = vif.rst;
s.push = !vif.rst && vif.wr && !vif.full;
s.pop = !vif.rst && vif.rd && !vif.empty;
s.write_full = !vif.rst && vif.wr && vif.full;
s.read_empty = !vif.rst && vif.rd && vif.empty;
s.full = vif.full; s.empty = vif.empty;
s.wdata = vif.wdata; s.rdata = vif.rdata;
ap.write(s);
end
endtask
endclass
// Checks every accepted read against a queue model.
class fifo_scoreboard extends uvm_subscriber #(fifo_sample);
`uvm_component_utils(fifo_scoreboard)
bit [7:0] expected[$];
int writes, reads, canceled, mismatches;
function new(string name, uvm_component parent); super.new(name, parent); endfunction
function void write(fifo_sample t);
if (t.reset) begin canceled += expected.size(); expected.delete(); return; end
if (t.full != (expected.size() == 4) || t.empty != (expected.size() == 0)) begin
mismatches++;
`uvm_error("FLAGS", $sformatf("full=%0d empty=%0d with %0d values stored", t.full, t.empty, expected.size()))
end
if (t.pop) begin
bit [7:0] want = expected.pop_front();
reads++;
if (t.rdata != want) begin
mismatches++;
`uvm_error("DATA", $sformatf("read %0d: got %02h, expected %02h", reads, t.rdata, want))
end
end
if (t.push) begin expected.push_back(t.wdata); writes++; end
endfunction
function void check_phase(uvm_phase phase);
if (expected.size() != 0) `uvm_error("PENDING", $sformatf("%0d values never read", expected.size()))
endfunction
endclass
// Counts which situations the stimulus reached. Each name is one bin.
class fifo_coverage extends uvm_subscriber #(fifo_sample);
`uvm_component_utils(fifo_coverage)
string names[9] = '{"level 0", "level 1", "level 2", "level 3", "level 4",
"write when full", "read when empty", "write and read", "reset with data"};
int hits[9];
int level;
function new(string name, uvm_component parent); super.new(name, parent); endfunction
function void write(fifo_sample t);
if (t.reset) begin
if (level > 0) hits[8]++;
level = 0;
return;
end
if (t.write_full) hits[5]++;
if (t.read_empty) hits[6]++;
if (t.push && t.pop) hits[7]++;
level += t.push - t.pop;
hits[level]++;
endfunction
function int covered();
foreach (hits[i]) if (hits[i] > 0) covered++;
endfunction
function string holes();
foreach (hits[i]) if (hits[i] == 0) holes = {holes, holes == "" ? "" : ", ", names[i]};
if (holes == "") holes = "none";
endfunction
endclass
// Settings the test chooses for the environment.
class fifo_cfg extends uvm_object;
`uvm_object_utils(fifo_cfg)
virtual fifo_if vif;
uvm_active_passive_enum is_active = UVM_ACTIVE;
bit has_coverage = 1;
function new(string name = "fifo_cfg"); super.new(name); endfunction
endclass
class fifo_agent extends uvm_agent;
`uvm_component_utils(fifo_agent)
fifo_cfg cfg;
uvm_sequencer #(fifo_item) sqr;
fifo_driver drv;
fifo_monitor mon;
function new(string name, uvm_component parent); super.new(name, parent); endfunction
function void build_phase(uvm_phase phase);
if (!uvm_config_db #(fifo_cfg)::get(this, "", "cfg", cfg))
`uvm_fatal("NOCFG", "no fifo_cfg for the agent")
mon = fifo_monitor::type_id::create("mon", this);
mon.vif = cfg.vif;
if (cfg.is_active == UVM_ACTIVE) begin
sqr = uvm_sequencer #(fifo_item)::type_id::create("sqr", this);
drv = fifo_driver::type_id::create("drv", this);
drv.vif = cfg.vif;
end
endfunction
function void connect_phase(uvm_phase phase);
if (cfg.is_active == UVM_ACTIVE) drv.seq_item_port.connect(sqr.seq_item_export);
endfunction
endclass
class fifo_env extends uvm_env;
`uvm_component_utils(fifo_env)
fifo_cfg cfg;
fifo_agent agent;
fifo_scoreboard sb;
fifo_coverage cov;
function new(string name, uvm_component parent); super.new(name, parent); endfunction
function void build_phase(uvm_phase phase);
if (!uvm_config_db #(fifo_cfg)::get(this, "", "cfg", cfg))
`uvm_fatal("NOCFG", "no fifo_cfg for the environment")
uvm_config_db #(fifo_cfg)::set(this, "agent", "cfg", cfg);
agent = fifo_agent::type_id::create("agent", this);
sb = fifo_scoreboard::type_id::create("sb", this);
if (cfg.has_coverage) cov = fifo_coverage::type_id::create("cov", this);
endfunction
function void connect_phase(uvm_phase phase);
agent.mon.ap.connect(sb.analysis_export);
if (cfg.has_coverage) agent.mon.ap.connect(cov.analysis_export);
endfunction
endclass
// Builds the configuration and the environment. Tests extend this class.
class fifo_base_test extends uvm_test;
`uvm_component_utils(fifo_base_test)
fifo_cfg cfg;
fifo_env env;
function new(string name, uvm_component parent); super.new(name, parent); endfunction
function void build_phase(uvm_phase phase);
cfg = fifo_cfg::type_id::create("cfg");
if (!uvm_config_db #(virtual fifo_if)::get(this, "", "vif", cfg.vif))
`uvm_fatal("NOVIF", "no virtual interface")
configure(cfg);
uvm_config_db #(fifo_cfg)::set(this, "env", "cfg", cfg);
env = fifo_env::type_id::create("env", this);
endfunction
virtual function void configure(fifo_cfg cfg); endfunction // tests change settings here
endclass
// Random traffic: n items, each a write, a read, both or neither.
class random_seq extends uvm_sequence #(fifo_item);
`uvm_object_utils(random_seq)
int n = 16;
function new(string name = "random_seq"); super.new(name); endfunction
task body();
repeat (n) begin
fifo_item it = fifo_item::type_id::create("it");
start_item(it);
if (!it.randomize()) `uvm_error("RAND", "randomize failed")
finish_item(it);
end
endtask
endclass
Check yourself
Remember
- Record the test name, seed, source revision and tool versions; together they make a random run repeatable.
start_itemreseeds each item from its name, so a seed set beforestart_itemis discarded.- Coverage bins list the situations the specification requires; holes are bins with no hits.
- An unreachable hole needs a directed item, and sometimes
constraint_mode(0), not more seeds. - Check every
randomize()result: a failure leaves the old values, and the item is still sent.