Capstone: a coverage model that finds holes

A coverage model needs tests too

Module 7 tested a checker by running it against designs that were broken on purpose. A coverage model deserves the same treatment, with stimulus instead of designs. Run it against stimulus known to be complete, and against stimulus known to be missing something.

A model that passes both tests can be trusted to mean something when it reports 100%. A model that has only ever been run on good stimulus is like a checker that has only ever seen a correct design: nobody knows whether it can fail.

The packet model

The model in the example collects the work of this track. It has a kind coverpoint with automatic bins, a length coverpoint with three classes from the specification, and their cross with the impossible CONFIG cells ignored. It is sampled once per packet, through sample(packet p), and keeps per-instance coverage, so each stimulus set gets a fresh instance.

The coarse model has the same two coverpoints and no cross. It is the opening story: each field is covered on its own, and no combination is asked for.

Reading the results

Against the reference stimulus, both models reach 100%, which shows that neither asks for anything impossible. The real test is the deficient sets.

Without long writes, the full model reports 95.24%: one cell of its cross, WRITE/long, is empty. The coarse model reports 100%, because long reads fill the long bin and short writes fill the WRITE bin. It has failed its test, and in a real project it would have signed off the hole.

Without CONFIG packets, both models drop, because the kind coverpoint alone notices a missing kind. That set does not separate the models, so a test of a coverage model needs deficient stimulus aimed at each of its crosses, not only at single fields.

Deficient stimulus is easy to build with module 5's tools. An inline constraint that rules out one case, such as kind == WRITE -> len <= 4, removes exactly that case and keeps everything else random.

Each deficient set should differ from the reference in one way only. Then a model that misses it has a single clear explanation: it has no bin, or no cross, for that case. The same idea works for transitions: a sink that never stalls twice in a row is deficient stimulus for the two_stalls bin of lesson C4.

How the coverage problems are graded

The coverage problems in the problem bank use exactly this method. You write the coverage model for a stimulus generator, such as this packet generator or the handshake of lesson C4. The grader then runs your model against a reference stimulus set and against several deficient sets, each missing one specific case.

1. Reference run

   Every required item must reach 100%. A model that asks for impossible cases, or samples at the wrong moment, fails here.

2. Deficient runs

   Each deficient set leaves one known hole. Your model must report less than 100% for every one of them.

3. Result

   The problem is solved when the reference run is complete and every deficient run is caught, the same caught-or-missed matrix as the checker problems.

Once a model passes both tests, keep it with the test plan. Later changes to the model should go through the same two tests, just as changes to a checker go through its broken designs.

Everything in this track is a tool for passing both tests. Bins come from the specification (C3), transitions cover orderings (C4), and crosses cover combinations without impossible cells (C5). Guards and sampling time decide what is counted (C6), and closing holes starts from the list of holes rather than the total (C7).

Try it

Both models reach 100% on the reference stimulus. Without long writes, only the full model sees the hole. Without CONFIG packets, both see it, through the kind coverpoint.

Add a fourth stimulus set, kind != READ || len > 1, which never sends a single-beat READ. Predict both models' results before you run it.

Check yourself

typedef enum bit [1:0] {READ, WRITE, CONFIG} kind_e;

// One packet on the link. CONFIG packets carry a single beat.
class packet;
  rand kind_e    kind;
  rand bit [3:0] len;              // payload beats, 1 to 8
  constraint legal {
    kind inside {READ, WRITE, CONFIG};
    len inside {[1:8]};
    kind == CONFIG -> len == 1;
  }
endclass

// The coverage model under test: kind, length class and their legal cross.
covergroup pkt_model with function sample(packet p);
  option.per_instance = 1;
  cp_kind: coverpoint p.kind;
  cp_len:  coverpoint p.len { bins one = {1}; bins short_ = {[2:4]}; bins long_ = {[5:8]}; }
  kind_x_len: cross cp_kind, cp_len {
    ignore_bins config_multi = binsof(cp_kind) intersect {CONFIG} && !binsof(cp_len.one);
  }
endgroup
// A model that is too coarse: the same coverpoints, no cross.
covergroup pkt_model_coarse with function sample(packet p);
  option.per_instance = 1;
  cp_kind: coverpoint p.kind;
  cp_len:  coverpoint p.len { bins one = {1}; bins short_ = {[2:4]}; bins long_ = {[5:8]}; }
endgroup


module tb;
  pkt_model m = new();
  function automatic packet make(kind_e k, bit [3:0] len);
    packet p = new();
    p.kind = k; p.len = len;
    return p;
  endfunction
  initial begin
    m.sample(make(WRITE, 1));
    m.sample(make(WRITE, 3));
    m.sample(make(WRITE, 6));
    m.sample(make(CONFIG, 1));
    $display("coverage=%0.2f", m.get_inst_coverage());
  end
endmodule

cp_kind: coverpoint p.kind;
  cp_len:  coverpoint p.len { bins one = {1}; bins short_ = {[2:4]}; bins long_ = {[5:8]}; }
  kind_x_len: cross cp_kind, cp_len {
    ignore_bins config_short = binsof(cp_kind) intersect {CONFIG} && binsof(cp_len.short_);
    ignore_bins config_long  = binsof(cp_kind) intersect {CONFIG} && binsof(cp_len.long_);
  }

cp_kind: coverpoint p.kind;
  cp_len:  coverpoint p.len { bins one = {1}; bins short_ = {[2:4]}; bins long_ = {[5:8]}; }
  kind_x_len: cross cp_kind, cp_len {
    ignore_bins config_multi = binsof(cp_kind) intersect {CONFIG} && !binsof(cp_len.one);
  }

cp_kind: coverpoint p.kind;
  cp_len:  coverpoint p.len { bins each[] = {[1:8]}; }
  kind_x_len: cross cp_kind, cp_len {
    ignore_bins config_multi = binsof(cp_kind) intersect {CONFIG} && !binsof(cp_len) intersect {1};
  }

cp_kind: coverpoint p.kind;
  cp_len:  coverpoint p.len { bins one = {1}; bins short_ = {[2:4]}; bins long_ = {[5:8]}; }