Capstone: the generic link bug hunt

Run one bench against five broken links, read which detector catches which fault and why, and answer the ten questions this module prepared you for.

Lesson
P1.11
Time
40 min
Runs on
Your browser
Check
5 questions

A source dropped valid for one cycle in the middle of a stall, then offered the same beat again. The beat arrived a cycle late with the right value, the scoreboard matched every byte, and the regression stayed green for a month. The bug surfaced in silicon, against a sink from another vendor that had taken the first offer as a commitment and the gap as a cancel.

The gap nobody noticedA stalled offer vanishes for one cycle and comes back. The data still arrives; only a rule sees the gap.
validdatareadySourcefault 1Sink2-deep FIFO, full

The ring on the source names the state of its offer; the checker names the rule.

0123456clkvalidreadydatax12x12xhold_ok
  1. Cycle 2

    Beat 12 is offered while the sink is full. The offer waits, as it should.

  2. Cycle 3

    valid drops for one cycle. The checker reports hold at the next edge; the scoreboard sees nothing yet.

  3. Cycle 5

    The same beat is offered again and taken. The scoreboard matches 12 against 12 and is happy. Only the rule caught the gap.

Fault 1 of this lesson. A scoreboard checks what arrived; a rule checker checks how it arrived.

The bench on this page holds everything the module built. A source offers four beats into a 2-deep sink whose consumer wakes late, so the third beat stalls. The rule checker from P1.3 watches the wires. A scoreboard in your file records every beat taken on the link and every beat the consumer receives, and compares the two lists in order.

Both the source and the sink take a fault input. Fault 0 is correct behaviour; faults 1 to 5 each break one thing. The stimulus runs the same traffic six times, once per fault, with a reset in between, and prints one line per phase. The checker adds a line for every rule it sees broken, with the edge.

FaultWhat is brokenSymptom on the wiresCaught by
1 holdThe source drops valid for one cycle during a stallA one-cycle gap in a stalled offer; the beat arrives lateChecker: rule hold at edge 25
2 stableThe source changes data once while the offer waitsThe bus shows 13 where 12 was offeredChecker: rule stable at edge 43
3 resetThe source drives valid during resetvalid is 1 while rst_n is 0Checker: rule reset at edges 54 and 55
4 overflowThe sink keeps ready at 1 when full and overwrites its oldest beatEvery handshake is legal; two beats never reach the consumerScoreboard only: 4 taken, 2 received
5 repeatThe sink moves its read pointer late onceEvery handshake is legal; the consumer gets beat 10 twiceScoreboard only: 3 mismatches in order

The fault matrix, measured by the run below. Three faults break a rule of the link; two break only the data, and no rule on the wires can see them.

The lost beat, cycle by cycle

Fault 4 is the hardest, because every handshake on the wires is legal. The sink keeps ready high when it is full, so the source sees a transfer and moves on. Inside the sink the new beat lands on the slot of the oldest one, which is gone before the consumer ever reads it.

A full sink that still says readyTwo beats fill the sink. The third is taken anyway and overwrites the first.
validdatareadySourcefour beatsSinkfault 4: ready while full

Two slots. A third landing changes nothing on the outside and loses a beat on the inside.

012345clkvalidreadydatax10111213xfull
  1. Cycle 1

    Beat 10 is taken into the first slot.

  2. Cycle 2

    Beat 11 fills the second slot. The sink is now full, and a correct sink would drop ready.

  3. Cycle 3

    ready stays 1, so 12 is taken: a legal handshake. It lands on the slot of 10, which is lost.

  4. Cycle 4

    13 does the same to 11. The checker sees four clean transfers; the consumer will receive two beats.

The only witness is the scoreboard, which holds the four beats the link took and the two the consumer got.

Two detectors, two questions

A rule checker answers "did every transfer follow the rules?" from the wires alone. It needs no knowledge of what the data means, so it binds to any link and catches faults 1, 2 and 3 at the exact edge. It cannot see inside the sink.

A scoreboard answers "did every beat arrive once, in order, unchanged?" It needs both ends of the link and a model of what should come out. It catches faults 4 and 5, and it also catches a beat that arrives late with the right value, which is to say it does not catch fault 1 at all. The month-long bug in the opening story is the gap between the two questions.

Run the bench, then change the loop to run one fault at a time and read the waves. Then write a third detector in your file: a count of stalled edges per phase, printed beside the mismatches. Fault 4 is the only phase with no stall, and that number alone would have pointed at the sink.

The mock interview

Ten questions, drawn from the ten lessons before this one, in the words an interviewer uses. Answer each in two sentences before you open it.

Asked in interviews: the ten questions of this module
  1. What is a bus protocol, and what does an interconnect do?

    A protocol is the set of rules that let an initiator and a target move data without losing or duplicating it. An interconnect decodes addresses and routes requests and responses between many of each.

  2. Why are on-chip buses synchronous, and what does "sampled at the edge" mean?

    One clock gives both sides the same moment to agree on, so a signal driven after one edge is seen by everyone at the next. A value that changes exactly at the edge is the race every rule is written to avoid.

  3. When does a transfer happen on a valid/ready link, and why must valid never wait for ready?

    At a rising edge where both are 1, and at no other edge. If the sink may wait for valid, a source that waits for ready deadlocks the link on its first beat.

  4. What is the difference between a beat, a burst and a transaction?

    A beat is one handshake, a burst is a sequence of beats under one request, and a transaction is the request plus its data and its response. A monitor counts beats; a scoreboard expects transactions.

  5. Latency or throughput: which does pipelining improve?

    Throughput: overlapping the request of one transaction with the data of another moves more beats per cycle. Each transaction still waits its full latency.

  6. How do you size a FIFO between two blocks?

    From the longest stall the sink can cause and the rate at which beats arrive during it. A 16-deep FIFO behind a sink that can stall for 20 cycles at one beat per cycle overflows.

  7. When may responses complete out of order, and what must a scoreboard key on?

    When the requests carry different tags, so the initiator can match each response. Responses with the same tag stay in order, and the scoreboard keys on the tag.

  8. What is a byte enable and why does alignment matter?

    A byte enable says which lanes of a wide bus carry real bytes for this beat. An unaligned start lands bytes in lanes the receiver did not expect and can clobber its neighbours.

  9. Priority or round-robin, and what is starvation?

    Priority serves the most urgent initiator first; round-robin gives every initiator a turn. Starvation is a low-priority initiator that never gets served while a chatty one keeps requesting.

  10. How do you verify a protocol, and what is a vacuous pass?

    One property per must sentence of the spec, bound to every instance, plus a scoreboard for the data and a test that stalls. A vacuous pass is an attempt whose condition never came true, so it checked nothing.

Try it

Six phases, one per fault. Read each phase against the fault matrix above: the checker names a rule for faults 1 to 3, and the scoreboard line changes for faults 4 and 5.

Then run one fault alone, for example run_phase(5);, and open the waves: the handshake looks perfect, and out_data shows 10 twice. Everything you need to find any of the five lives in the two lists the scoreboard keeps.

RUNIcarus Verilog · in your browser
Edit the code, then run it
Expected: fault 0: taken 4, received 4, mismatches 0 RULE hold broken at edge 25: valid dropped before ready: the offer was withdrawn fault 1: taken 4, received 4, mismatches 0 RULE stable broken at edge 43: data changed while the offer waited fault 2: taken 4, received 4, mismatches 0 RULE reset broken at edge 54: valid is 1 during reset RULE reset broken at edge 55: valid is 1 during reset fault 3: taken 4, received 4, mismatches 0 fault 4: taken 4, received 2, mismatches 4 fault 5: taken 4, received 4, mismatches 3 checker: 24 transfers, 21 stalled edges, 4 rules broken
Read included file: link_sink.sv
// link_sink.sv: the target of the generic link (read-only in this lesson).
// A DEPTH-entry FIFO behind valid/ready. It takes a beat at every rising edge where valid and
// ready are both 1, and hands beats on to its consumer at every edge where drain and out_valid
// are both 1. ready is 1 while the FIFO has room: it never waits for valid.
module link_sink #(parameter int DEPTH = 4, parameter int WIDTH = 8) (
  input  logic             clk,
  input  logic             rst_n,
  input  logic             valid,
  input  logic [WIDTH-1:0] data,
  input  logic             last,
  output logic             ready,
  input  logic             drain,       // the consumer behind the FIFO takes a beat while 1
  output logic             out_valid,
  output logic [WIDTH-1:0] out_data,
  output logic             out_last,
  output int               count        // beats held right now
);
  logic [WIDTH:0] mem [DEPTH];          // {last, data}
  int wr = 0, rd = 0;
  assign ready     = (count < DEPTH);
  assign out_valid = (count > 0);
  assign {out_last, out_data} = mem[rd];
  wire push = valid && ready;
  wire pop  = drain && out_valid;
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      wr <= 0; rd <= 0; count <= 0;
    end else begin
      if (push) begin mem[wr] <= {last, data}; wr <= (wr + 1) % DEPTH; end
      if (pop) rd <= (rd + 1) % DEPTH;
      count <= count + (push ? 1 : 0) - (pop ? 1 : 0);
    end
  end
endmodule
Read included file: link_checker.sv
// link_checker.sv: the rules of the generic link, checked at every rising edge (read-only).
// Each broken rule is named with the edge where it was seen:
//   reset   valid is 0 while reset is active;
//   hold    once valid is 1 and ready is 0, valid is still 1 at the next edge (an offer is not withdrawn);
//   stable  while an offer waits, data and last do not change;
//   known   data is not x while valid is 1.
// The one rule no checker can see on the wires, "valid must not wait for ready", is the stimulus's
// own responsibility (lesson P1.3).
// It also counts transfers and stalled edges, and prints one summary line at the end of the run.
module link_checker #(parameter int WIDTH = 8) (
  input logic             clk,
  input logic             rst_n,
  input logic             valid,
  input logic             ready,
  input logic [WIDTH-1:0] data,
  input logic             last
);
  int cycle = -1, transfers = 0, stalls = 0, broken = 0;
  logic             pv = 0, pr = 0, pl = 0;
  logic [WIDTH-1:0] pd = 'x;
  always @(posedge clk) begin
    cycle++;
    if (!rst_n) begin
      if (valid === 1) report("reset", "valid is 1 during reset");
    end else begin
      if (pv && !pr && valid !== 1) report("hold", "valid dropped before ready: the offer was withdrawn");
      if (pv && !pr && valid === 1 && data !== pd) report("stable", "data changed while the offer waited");
      if (pv && !pr && valid === 1 && last !== pl) report("stable", "last changed while the offer waited");
      if (valid === 1 && $isunknown(data)) report("known", "data is x while valid is 1");
      if (valid === 1 && ready === 1) transfers++;
      if (valid === 1 && ready !== 1) stalls++;
    end
    pv = (valid === 1); pr = (ready === 1); pd = data; pl = last;
  end
  function void report(input string rule, input string what);
    broken++;
    $display("RULE %s broken at edge %0d: %s", rule, cycle, what);
  endfunction
  final $display("checker: %0d transfers, %0d stalled edges, %0d rules broken", transfers, stalls, broken);
endmodule
Read included file: link_tasks.svh
// link_tasks.svh: the source side of the generic link, as tasks for the stimulus (read-only).
// Every task returns one time unit after the rising edge where its last beat transferred, so the
// next call starts driving in the following cycle. `cycle` counts rising edges from 0.
int cycle = -1;
always @(posedge clk) cycle = cycle + 1;

// Offer one beat and hold it until the sink takes it.
task automatic send(input logic [7:0] d, input logic is_last = 0);
  valid = 1; data = d; last = is_last;
  do @(posedge clk); while (!ready);
  #1 $display("edge %0d: beat %02h taken", cycle, d);
  valid = 0; data = 'x; last = 0;
endtask

// A burst of n beats with consecutive values; last marks the final beat.
task automatic send_burst(input logic [7:0] first, input int n);
  for (int i = 0; i < n; i++) begin
    valid = 1; data = first + i[7:0]; last = (i == n - 1);
    do @(posedge clk); while (!ready);
    #1 $display("edge %0d: beat %02h taken%s", cycle, first + i[7:0], last ? " (last)" : "");
  end
  valid = 0; data = 'x; last = 0;
endtask

// Drive nothing for n cycles.
task automatic idle(input int n);
  repeat (n) @(posedge clk);
  #1;
endtask
Read included file: faulty_link.sv
// faulty_link.sv: a source and a sink for the generic link, each with selectable faults
// (read-only). fault 0 is correct behaviour.
//   1  hold      the source drops valid for one cycle during a stall, then offers the same beat again
//   2  stable    the source changes data once while an offer waits
//   3  reset     the source drives valid during reset
//   4  overflow  the sink keeps ready at 1 when it is full and overwrites its oldest beat
//   5  repeat    the sink moves its read pointer late once, so the consumer gets a beat twice
module faulty_source #(parameter int N = 4) (
  input  logic       clk,
  input  logic       rst_n,
  input  int         fault,
  input  logic       go,        // 1: offer N beats, then stop
  input  logic       ready,
  output logic       valid,
  output logic [7:0] data,
  output int         sent
);
  int i;
  logic misbehaved;
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      valid <= (fault == 3); data <= 8'h10; i <= 0; sent <= 0; misbehaved <= 0;
    end else if (!valid) begin
      if (go && i < N) begin valid <= 1; data <= 8'h10 + i[7:0]; end
    end else if (ready) begin                       // a transfer at this edge
      sent <= sent + 1; i <= i + 1;
      if (i + 1 < N) data <= 8'h10 + i[7:0] + 8'd1; else valid <= 0;
    end else if (!misbehaved) begin                 // stalled at this edge
      if (fault == 1) begin valid <= 0; misbehaved <= 1; end
      if (fault == 2) begin data <= data + 8'd1; misbehaved <= 1; end
    end
  end
endmodule

module faulty_sink #(parameter int DEPTH = 2) (
  input  logic       clk,
  input  logic       rst_n,
  input  int         fault,
  input  logic       valid,
  input  logic [7:0] data,
  output logic       ready,
  input  logic       drain,
  output logic       out_valid,
  output logic [7:0] out_data,
  output int         count
);
  logic [7:0] mem [DEPTH];
  int wr = 0, rd = 0;
  logic misbehaved;
  assign ready     = (fault == 4) ? 1'b1 : (count < DEPTH);
  assign out_valid = (count > 0);
  assign out_data  = mem[rd];
  wire push = valid && ready;
  wire pop  = drain && out_valid;
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      wr <= 0; rd <= 0; count <= 0; misbehaved <= 0;
    end else begin
      if (push) begin mem[wr] <= data; wr <= (wr + 1) % DEPTH; end
      if (pop) begin
        if (fault == 5 && !misbehaved) misbehaved <= 1;   // this pop forgets to move rd
        else rd <= (rd + 1) % DEPTH;
      end
      if (push && count == DEPTH) count <= count - (pop ? 1 : 0);   // full: the write lands on the oldest beat
      else count <= count + (push ? 1 : 0) - (pop ? 1 : 0);
    end
  end
endmodule

Check yourself

Predict the outputOnly the source and the sink are here, with fault 4 (the sink says ready while full), and a counter of beats the consumer receives. What line is printed?
`timescale 1ns/1ps
module tb;
  logic clk = 0, rst_n = 0;
  always #5 clk = ~clk;
  int         fault = 4;                // the sink says ready while full
  logic       go = 0, drain = 0;
  logic       valid, ready, out_valid;
  logic [7:0] data, out_data;
  int         sent, count, n_got = 0;
  faulty_source src (.clk(clk), .rst_n(rst_n), .fault(fault), .go(go), .ready(ready), .valid(valid), .data(data), .sent(sent));
  faulty_sink  snk (.clk(clk), .rst_n(rst_n), .fault(fault), .valid(valid), .data(data), .ready(ready), .drain(drain),
                    .out_valid(out_valid), .out_data(out_data), .count(count));
  always @(posedge clk) if (out_valid && drain) n_got++;
  initial begin
    repeat (2) @(posedge clk); #1 rst_n = 1;
    @(posedge clk); #1 go = 1;            // four beats offered; the consumer is asleep
    repeat (6) @(posedge clk); #1 drain = 1;
    wait (sent == 4);
    repeat (6) @(posedge clk);
    $display("sent %0d, received %0d", sent, n_got);
    $finish;
  end
endmodule
Read included file: faulty_link.sv
// faulty_link.sv: a source and a sink for the generic link, each with selectable faults
// (read-only). fault 0 is correct behaviour.
//   1  hold      the source drops valid for one cycle during a stall, then offers the same beat again
//   2  stable    the source changes data once while an offer waits
//   3  reset     the source drives valid during reset
//   4  overflow  the sink keeps ready at 1 when it is full and overwrites its oldest beat
//   5  repeat    the sink moves its read pointer late once, so the consumer gets a beat twice
module faulty_source #(parameter int N = 4) (
  input  logic       clk,
  input  logic       rst_n,
  input  int         fault,
  input  logic       go,        // 1: offer N beats, then stop
  input  logic       ready,
  output logic       valid,
  output logic [7:0] data,
  output int         sent
);
  int i;
  logic misbehaved;
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      valid <= (fault == 3); data <= 8'h10; i <= 0; sent <= 0; misbehaved <= 0;
    end else if (!valid) begin
      if (go && i < N) begin valid <= 1; data <= 8'h10 + i[7:0]; end
    end else if (ready) begin                       // a transfer at this edge
      sent <= sent + 1; i <= i + 1;
      if (i + 1 < N) data <= 8'h10 + i[7:0] + 8'd1; else valid <= 0;
    end else if (!misbehaved) begin                 // stalled at this edge
      if (fault == 1) begin valid <= 0; misbehaved <= 1; end
      if (fault == 2) begin data <= data + 8'd1; misbehaved <= 1; end
    end
  end
endmodule

module faulty_sink #(parameter int DEPTH = 2) (
  input  logic       clk,
  input  logic       rst_n,
  input  int         fault,
  input  logic       valid,
  input  logic [7:0] data,
  output logic       ready,
  input  logic       drain,
  output logic       out_valid,
  output logic [7:0] out_data,
  output int         count
);
  logic [7:0] mem [DEPTH];
  int wr = 0, rd = 0;
  logic misbehaved;
  assign ready     = (fault == 4) ? 1'b1 : (count < DEPTH);
  assign out_valid = (count > 0);
  assign out_data  = mem[rd];
  wire push = valid && ready;
  wire pop  = drain && out_valid;
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      wr <= 0; rd <= 0; count <= 0; misbehaved <= 0;
    end else begin
      if (push) begin mem[wr] <= data; wr <= (wr + 1) % DEPTH; end
      if (pop) begin
        if (fault == 5 && !misbehaved) misbehaved <= 1;   // this pop forgets to move rd
        else rd <= (rd + 1) % DEPTH;
      end
      if (push && count == DEPTH) count <= count - (pop ? 1 : 0);   // full: the write lands on the oldest beat
      else count <= count + (push ? 1 : 0) - (pop ? 1 : 0);
    end
  end
endmodule
Spot the bugFault 5 makes the consumer receive beat 10 twice, and the fourth beat never leaves the sink. The scoreboard holds the list of beats taken on the link and the list received. Each option is the comparison that sets caught. Which one misses the fault?
Read included file: faulty_link.sv
// faulty_link.sv: a source and a sink for the generic link, each with selectable faults
// (read-only). fault 0 is correct behaviour.
//   1  hold      the source drops valid for one cycle during a stall, then offers the same beat again
//   2  stable    the source changes data once while an offer waits
//   3  reset     the source drives valid during reset
//   4  overflow  the sink keeps ready at 1 when it is full and overwrites its oldest beat
//   5  repeat    the sink moves its read pointer late once, so the consumer gets a beat twice
module faulty_source #(parameter int N = 4) (
  input  logic       clk,
  input  logic       rst_n,
  input  int         fault,
  input  logic       go,        // 1: offer N beats, then stop
  input  logic       ready,
  output logic       valid,
  output logic [7:0] data,
  output int         sent
);
  int i;
  logic misbehaved;
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      valid <= (fault == 3); data <= 8'h10; i <= 0; sent <= 0; misbehaved <= 0;
    end else if (!valid) begin
      if (go && i < N) begin valid <= 1; data <= 8'h10 + i[7:0]; end
    end else if (ready) begin                       // a transfer at this edge
      sent <= sent + 1; i <= i + 1;
      if (i + 1 < N) data <= 8'h10 + i[7:0] + 8'd1; else valid <= 0;
    end else if (!misbehaved) begin                 // stalled at this edge
      if (fault == 1) begin valid <= 0; misbehaved <= 1; end
      if (fault == 2) begin data <= data + 8'd1; misbehaved <= 1; end
    end
  end
endmodule

module faulty_sink #(parameter int DEPTH = 2) (
  input  logic       clk,
  input  logic       rst_n,
  input  int         fault,
  input  logic       valid,
  input  logic [7:0] data,
  output logic       ready,
  input  logic       drain,
  output logic       out_valid,
  output logic [7:0] out_data,
  output int         count
);
  logic [7:0] mem [DEPTH];
  int wr = 0, rd = 0;
  logic misbehaved;
  assign ready     = (fault == 4) ? 1'b1 : (count < DEPTH);
  assign out_valid = (count > 0);
  assign out_data  = mem[rd];
  wire push = valid && ready;
  wire pop  = drain && out_valid;
  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      wr <= 0; rd <= 0; count <= 0; misbehaved <= 0;
    end else begin
      if (push) begin mem[wr] <= data; wr <= (wr + 1) % DEPTH; end
      if (pop) begin
        if (fault == 5 && !misbehaved) misbehaved <= 1;   // this pop forgets to move rd
        else rd <= (rd + 1) % DEPTH;
      end
      if (push && count == DEPTH) count <= count - (pop ? 1 : 0);   // full: the write lands on the oldest beat
      else count <= count + (push ? 1 : 0) - (pop ? 1 : 0);
    end
  end
endmodule
ConceptWhich of the five faults can a rule checker on the wires never catch, whatever properties it holds?
ConceptThe bench resets the link between phases. What goes wrong without the reset?
Read the waveformThe checker reports hold at one edge of this trace. Which cycle shows the broken rule?
A stalled offer with a gap
01234validreadydata12x12

One value per cycle, sampled at the edge that ends the cycle.

Remember

  • A rule checker on the wires catches every handshake fault at its edge, with no knowledge of the data, and binds to any link.
  • A scoreboard with both ends of the link catches beats lost, repeated or corrupted inside a block, which no rule on the wires can see.
  • A beat that arrives late with the right value satisfies the scoreboard and breaks the hold rule: the two detectors answer different questions, and a bench needs both.
  • Reset between phases, and the reset rule between tests: state that carries over turns one fault into a chain of them.
  • The ten interview questions of this module all come down to one picture: who moves first, when a beat moves, and what must stay in order.

Practice