Ordering, tags and out-of-order completion

Explain why responses come back in a different order from the requests, what a tag is for, which order a bus still promises, and what a scoreboard must key on.

Lesson
P1.7
Time
30 min
Runs on
Your browser
Check
5 questions

A chip had a fast on-die SRAM and a slow path to external memory, both behind one bus. The scoreboard kept the expected read data in one queue, in request order, and compared each response with the head of that queue. On the first test that mixed the two paths it reported hundreds of data errors. Every one of them was the scoreboard's own fault: the bus had returned the SRAM reads first, exactly as its rules allow.

Two tagged requests, two targets, responses back in a new orderRequests go out with a tag; each response carries the tag of the request it answers.
requestresponserequestresponseInitiatorissues tags 1, 2, 7, 7Fast targetanswers in 2 cyclesSlow targetanswers in 4 cycles

A pill going right is a request, going left a response; the slots are requests still inside a target.

01234567clkreq_tag127xreq_fastreq_slowrsp_tagx217xrsp_fastrsp_slow
  1. Cycle 0

    Tag 1 goes to the slow target. It will sit there for four cycles.

  2. Cycle 1

    Tag 2 goes to the fast target, one cycle later.

  3. Cycle 3

    Tag 2 is answered first. Different tags, different targets: this reorder is allowed.

  4. Cycle 4

    Tag 1 comes back. A scoreboard that expected it first was wrong; the bus was right.

  5. Cycle 5

    Tag 7 was sent to the slow target in cycle 2 and to the fast one in cycle 3. The fast copy answers first: the same tag has overtaken itself.

  6. Cycle 6

    The second tag 7 arrives. The initiator cannot tell which answer belongs to which request. This reorder is forbidden.

Two reorders, one legal and one illegal. The tag is what makes the difference visible.

Why responses change order

Targets differ in speed. In this lesson the fast target answers two cycles after it accepts a request and the slow one takes four. If the bus insisted on answering in request order, a fast read behind a slow one would wait two extra cycles for nothing. Every initiator queued behind it would wait too.

So most buses let a response return as soon as its target has it. The initiator must then recognise which request each response answers, because position in the stream no longer says so. That is the job of the tag.

A tag names the request

A tag is a small number the initiator attaches to a request. The target, or the interconnect on its behalf, copies the tag onto the response. The initiator matches the two by tag and never by arrival order. AXI calls it an ID, PCIe calls it a tag in the packet header, and TileLink calls it a source.

A tag is a name for a stream of requests, not for one request. An initiator that must keep ten reads in flight needs enough tags to tell them apart. Otherwise it reuses a tag and relies on the rule below to keep that tag's answers in order.

BusWhat the tag is calledCarried on
AXIID, one per channel pairthe address, data and response channels
PCIeTag, with the requester IDthe request and completion packets
TileLinksource on A, sink on Devery message of a transaction
Wishbone, APBnone: one request at a timenothing; order is position

Buses without tags allow one outstanding transaction, so order and position are the same thing.

What must stay in order

The rule has two halves. Responses that carry different tags may return in any order. Responses that carry the same tag must return in the order of their requests. A target that answers in order keeps the second half by itself; the interconnect must keep it when one tag is sent to two targets of different speed.

In the figure, tag 7 went to the slow target and then to the fast one, and the fast answer came back first. An interconnect that allows this has a bug: it must hold the second request until the first has been answered, as the bench does when its SAME_TAG_IN_ORDER parameter is 1. Holding costs cycles, which is why an initiator that wants speed spreads its requests over several tags.

  1. Allowed

    Tags 1 and 2 return as 2, 1. Different tags, so any order is correct.

  2. Forbidden

    Two requests with tag 7 return in the wrong order. The initiator would pair the data with the wrong request.

  3. Who fixes it

    The interconnect, by holding a request whose tag is still outstanding at another target. The initiator may also wait for the first response itself.

What a scoreboard keys on

A scoreboard for a reordering bus keys its expectations by tag, with one small queue per tag in request order. A response is compared with the head of its own tag's queue. One global queue is wrong on every legal reorder, and the opening story shows how loudly.

One queue for everything

// at a response
exp = order_q[head++];
if (rsp.data !== exp.data) error();

Complains on the legal reorder of tags 1 and 2. The checker, not the design, is wrong.

One queue per tag

// at a response
exp = exp_q[rsp.tag][head[rsp.tag]++];
if (rsp.data !== exp) error();

Silent on the legal reorder. Complains only when tag 7 overtakes itself, which is the bug.

The per-tag queue relies on the ordering rule: because a tag's answers come back in request order, the head of its queue is always the next expectation. Lesson U8.2 builds the same idea into a UVM scoreboard.

A read after a write

Reordering also reaches across request types. If an initiator writes a location and then reads it through a different path, the read may overtake the write and return stale data. The initiator must wait for the write's response before issuing the read, or send both on the same tag through the same target, where order is kept.

Asked in interviews: ordering and tags
  1. When is out-of-order completion allowed?

    Between responses with different tags. It lets a fast target answer without waiting behind a slow one.

  2. What must stay in order?

    Responses with the same tag, in request order. The interconnect holds a request whose tag is still outstanding at another target.

  3. What is a tag for?

    To match a response with its request when arrival order no longer says so. It names a stream, and reusing it promises in-order answers within that stream.

  4. What does a scoreboard key on?

    On the tag, with one queue per tag in request order. One queue for everything reports legal reorders as errors.

  5. How do you keep a read after a write safe?

    Wait for the write's response before issuing the read, or send both on one tag through one target. A read through another path may overtake the write.

Try it

Two scoreboards watch the same responses. The one that expects request order complains about tags 1 and 2, which is its own mistake. The one keyed by tag complains only about tag 7, which the interconnect answered out of order.

Now set SAME_TAG_IN_ORDER(1) in the instantiation and run again. The second tag 7 request is held until edge 14, its answer arrives at edge 16, and the tag-keyed scoreboard is silent. Then give the fourth request tag 8 instead and watch both runs.

RUNIcarus Verilog · in your browser
Edit the code, then run it
Expected: edge 3: request tag 1 to the slow target edge 4: request tag 2 to the fast target edge 6: response tag 2 data 11 scoreboard in order: expected tag 1, got tag 2 edge 7: response tag 1 data 81 scoreboard in order: expected tag 2, got tag 1 edge 9: request tag 7 to the slow target edge 10: request tag 7 to the fast target edge 12: response tag 7 data 21 scoreboard by tag: tag 7 expected 91, got 21 edge 13: response tag 7 data 91 scoreboard by tag: tag 7 expected 21, got 91 4 responses, 2 mismatches by tag, 2 complaints in request order
Read included file: two_targets.sv
// two_targets.sv: a tagged request/response link with two targets (read-only in this lesson).
// A request carries a tag and an address. Addresses below 128 go to the fast target, which answers
// two edges after accepting; addresses from 128 go to the slow target, which answers four edges
// after accepting. Every response carries its request's tag and data = address + 1. The two
// response streams are merged into one: when both targets have a response waiting, the slow one
// leaves first and the fast one waits a cycle. With SAME_TAG_IN_ORDER = 1 the interconnect holds
// a request whose tag is still outstanding at the other target (req_ready stays 0), so responses
// with one tag always return in request order.
module two_targets #(parameter bit SAME_TAG_IN_ORDER = 0) (
  input  logic       clk,
  input  logic       rst_n,
  input  logic       req_valid,
  input  logic [3:0] req_tag,
  input  logic [7:0] req_addr,
  output logic       req_ready,
  output logic       rsp_valid,
  output logic [3:0] rsp_tag,
  output logic [7:0] rsp_data
);
  wire to_slow = req_addr[7];
  wire [3:0] tag_i = req_valid ? req_tag : 4'd0;      // a safe index while nothing is offered
  // Outstanding requests per tag (at most 15), and which target holds them.
  logic [3:0] outstanding [16];
  logic       dest [16];
  wire blocked = SAME_TAG_IN_ORDER && req_valid && outstanding[tag_i] != 4'd0 && dest[tag_i] != to_slow;
  assign req_ready = !blocked;
  wire accept = req_valid && req_ready;

  // The fast target: one pipeline stage. The slow target: three.
  logic       fv;        logic [3:0] ft;        logic [7:0] fd;
  logic       sv [3];    logic [3:0] st [3];    logic [7:0] sd [3];
  // Each target's responses wait in a small queue; the merged output shows the slow queue's head
  // when it has one, else the fast queue's head, and the initiator takes a response every edge.
  logic [11:0] fq [4], sq [4];
  int fw = 0, fr = 0, fn = 0, sw = 0, sr = 0, sn = 0;
  wire take_slow = sn > 0, take_fast = !take_slow && fn > 0;
  assign rsp_valid = take_slow || take_fast;
  assign {rsp_tag, rsp_data} = take_slow ? sq[sr] : take_fast ? fq[fr] : 12'bx;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      fv <= 0; sv[0] <= 0; sv[1] <= 0; sv[2] <= 0;
      fw <= 0; fr <= 0; fn <= 0; sw <= 0; sr <= 0; sn <= 0;
      for (int i = 0; i < 16; i++) begin outstanding[i] <= 0; dest[i] <= 0; end
    end else begin
      // Accept a request into one of the pipelines.
      fv <= accept && !to_slow; ft <= req_tag; fd <= req_addr + 8'd1;
      sv[0] <= accept && to_slow; st[0] <= req_tag; sd[0] <= req_addr + 8'd1;
      sv[1] <= sv[0]; st[1] <= st[0]; sd[1] <= sd[0];
      sv[2] <= sv[1]; st[2] <= st[1]; sd[2] <= sd[1];
      if (accept) begin outstanding[tag_i] <= outstanding[tag_i] + 4'd1; dest[tag_i] <= to_slow; end
      // Responses leaving the pipelines wait in their target's queue.
      if (fv) begin fq[fw] <= {ft, fd}; fw <= (fw + 1) % 4; end
      if (sv[2]) begin sq[sw] <= {st[2], sd[2]}; sw <= (sw + 1) % 4; end
      // The response at the output is taken at this edge.
      if (take_slow) sr <= (sr + 1) % 4;
      else if (take_fast) fr <= (fr + 1) % 4;
      fn <= fn + (fv ? 1 : 0) - (take_fast ? 1 : 0);
      sn <= sn + (sv[2] ? 1 : 0) - (take_slow ? 1 : 0);
      if (rsp_valid) outstanding[rsp_tag] <= outstanding[rsp_tag] - 4'd1 + ((accept && tag_i == rsp_tag) ? 4'd1 : 4'd0);
    end
  end
endmodule
Read included file: tag_tasks.svh
// tag_tasks.svh: the initiator's tasks, the response monitor and two scoreboards (read-only).
// `cycle` counts rising edges from 0. A request is accepted at an edge where req_valid and
// req_ready are both 1; a response arrives at an edge where rsp_valid is 1.
int cycle = -1;
always @(posedge clk) cycle = cycle + 1;

// Scoreboard A keys by tag: one small queue of expected data per tag, in request order.
logic [7:0] exp_q [16][8];
int exp_w [16], exp_r [16];
// Scoreboard B assumes responses return in request order: one queue for everything.
logic [3:0] order_q [32];
int order_w = 0, order_r = 0;
int responses = 0, tag_mismatches = 0, order_complaints = 0;
bit quiet = 0;                       // 1: print the summary only
initial for (int t = 0; t < 16; t++) begin exp_w[t] = 0; exp_r[t] = 0; end

// Issue one request and hold it until the interconnect accepts it.
task automatic request(input logic [3:0] tag, input logic [7:0] addr);
  req_valid = 1; req_tag = tag; req_addr = addr;
  exp_q[tag][exp_w[tag]] = addr + 8'd1; exp_w[tag] = (exp_w[tag] + 1) % 8;
  order_q[order_w] = tag; order_w = (order_w + 1) % 32;
  do @(posedge clk); while (!req_ready);
  #1 if (!quiet) $display("edge %0d: request tag %0d to the %s target", cycle, tag, addr[7] ? "slow" : "fast");
  req_valid = 0; req_tag = 'x; req_addr = 'x;
endtask

// Wait n edges.
task automatic idle(input int n);
  repeat (n) @(posedge clk);
  #1;
endtask

// Every response is printed and checked by both scoreboards.
always @(posedge clk) begin
  logic v; logic [3:0] t; logic [7:0] d;
  v = rsp_valid; t = rsp_tag; d = rsp_data;
  #1 if (v) begin
    responses++;
    if (!quiet) $display("edge %0d: response tag %0d data %02h", cycle, t, d);
    if (d !== exp_q[t][exp_r[t]]) begin
      tag_mismatches++;
      if (!quiet) $display("  scoreboard by tag: tag %0d expected %02h, got %02h", t, exp_q[t][exp_r[t]], d);
    end
    exp_r[t] = (exp_r[t] + 1) % 8;
    if (t !== order_q[order_r]) begin
      order_complaints++;
      if (!quiet) $display("  scoreboard in order: expected tag %0d, got tag %0d", order_q[order_r], t);
    end
    order_r = (order_r + 1) % 32;
  end
end
final $display("%0d responses, %0d mismatches by tag, %0d complaints in request order", responses, tag_mismatches, order_complaints);

Check yourself

Predict the outputThree requests: tag 3 to the fast target, tag 4 to the slow one, tag 5 to the fast one. Only the summary line is printed. What is it?
`timescale 1ns/1ps
module tb;
  logic clk = 0, rst_n = 0;
  always #5 clk = ~clk;                 // rising edges at 5, 15, 25, ...
  logic       req_valid = 0, req_ready, rsp_valid;
  logic [3:0] req_tag = 'x, rsp_tag;
  logic [7:0] req_addr = 'x, rsp_data;
  // SAME_TAG_IN_ORDER = 1 makes the interconnect hold a request whose tag is still out at the other target.
  two_targets #(.SAME_TAG_IN_ORDER(0)) dut (.clk(clk), .rst_n(rst_n), .req_valid(req_valid), .req_tag(req_tag),
    .req_addr(req_addr), .req_ready(req_ready), .rsp_valid(rsp_valid), .rsp_tag(rsp_tag), .rsp_data(rsp_data));
  `include "tag_tasks.svh"
  initial quiet = 1;           // only the summary line is printed
  initial begin
    repeat (2) @(posedge clk); #1 rst_n = 1;   // reset released after edge 1
    @(posedge clk); #1;                        // the first request may start after edge 2
    request(4'd3, 8'h40);        // fast
    request(4'd4, 8'hc0);        // slow
    request(4'd5, 8'h44);        // fast
    repeat (8) @(posedge clk);
    $finish;
  end
endmodule
Read included file: two_targets.sv
// two_targets.sv: a tagged request/response link with two targets (read-only in this lesson).
// A request carries a tag and an address. Addresses below 128 go to the fast target, which answers
// two edges after accepting; addresses from 128 go to the slow target, which answers four edges
// after accepting. Every response carries its request's tag and data = address + 1. The two
// response streams are merged into one: when both targets have a response waiting, the slow one
// leaves first and the fast one waits a cycle. With SAME_TAG_IN_ORDER = 1 the interconnect holds
// a request whose tag is still outstanding at the other target (req_ready stays 0), so responses
// with one tag always return in request order.
module two_targets #(parameter bit SAME_TAG_IN_ORDER = 0) (
  input  logic       clk,
  input  logic       rst_n,
  input  logic       req_valid,
  input  logic [3:0] req_tag,
  input  logic [7:0] req_addr,
  output logic       req_ready,
  output logic       rsp_valid,
  output logic [3:0] rsp_tag,
  output logic [7:0] rsp_data
);
  wire to_slow = req_addr[7];
  wire [3:0] tag_i = req_valid ? req_tag : 4'd0;      // a safe index while nothing is offered
  // Outstanding requests per tag (at most 15), and which target holds them.
  logic [3:0] outstanding [16];
  logic       dest [16];
  wire blocked = SAME_TAG_IN_ORDER && req_valid && outstanding[tag_i] != 4'd0 && dest[tag_i] != to_slow;
  assign req_ready = !blocked;
  wire accept = req_valid && req_ready;

  // The fast target: one pipeline stage. The slow target: three.
  logic       fv;        logic [3:0] ft;        logic [7:0] fd;
  logic       sv [3];    logic [3:0] st [3];    logic [7:0] sd [3];
  // Each target's responses wait in a small queue; the merged output shows the slow queue's head
  // when it has one, else the fast queue's head, and the initiator takes a response every edge.
  logic [11:0] fq [4], sq [4];
  int fw = 0, fr = 0, fn = 0, sw = 0, sr = 0, sn = 0;
  wire take_slow = sn > 0, take_fast = !take_slow && fn > 0;
  assign rsp_valid = take_slow || take_fast;
  assign {rsp_tag, rsp_data} = take_slow ? sq[sr] : take_fast ? fq[fr] : 12'bx;

  always_ff @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      fv <= 0; sv[0] <= 0; sv[1] <= 0; sv[2] <= 0;
      fw <= 0; fr <= 0; fn <= 0; sw <= 0; sr <= 0; sn <= 0;
      for (int i = 0; i < 16; i++) begin outstanding[i] <= 0; dest[i] <= 0; end
    end else begin
      // Accept a request into one of the pipelines.
      fv <= accept && !to_slow; ft <= req_tag; fd <= req_addr + 8'd1;
      sv[0] <= accept && to_slow; st[0] <= req_tag; sd[0] <= req_addr + 8'd1;
      sv[1] <= sv[0]; st[1] <= st[0]; sd[1] <= sd[0];
      sv[2] <= sv[1]; st[2] <= st[1]; sd[2] <= sd[1];
      if (accept) begin outstanding[tag_i] <= outstanding[tag_i] + 4'd1; dest[tag_i] <= to_slow; end
      // Responses leaving the pipelines wait in their target's queue.
      if (fv) begin fq[fw] <= {ft, fd}; fw <= (fw + 1) % 4; end
      if (sv[2]) begin sq[sw] <= {st[2], sd[2]}; sw <= (sw + 1) % 4; end
      // The response at the output is taken at this edge.
      if (take_slow) sr <= (sr + 1) % 4;
      else if (take_fast) fr <= (fr + 1) % 4;
      fn <= fn + (fv ? 1 : 0) - (take_fast ? 1 : 0);
      sn <= sn + (sv[2] ? 1 : 0) - (take_slow ? 1 : 0);
      if (rsp_valid) outstanding[rsp_tag] <= outstanding[rsp_tag] - 4'd1 + ((accept && tag_i == rsp_tag) ? 4'd1 : 4'd0);
    end
  end
endmodule
Read included file: tag_tasks.svh
// tag_tasks.svh: the initiator's tasks, the response monitor and two scoreboards (read-only).
// `cycle` counts rising edges from 0. A request is accepted at an edge where req_valid and
// req_ready are both 1; a response arrives at an edge where rsp_valid is 1.
int cycle = -1;
always @(posedge clk) cycle = cycle + 1;

// Scoreboard A keys by tag: one small queue of expected data per tag, in request order.
logic [7:0] exp_q [16][8];
int exp_w [16], exp_r [16];
// Scoreboard B assumes responses return in request order: one queue for everything.
logic [3:0] order_q [32];
int order_w = 0, order_r = 0;
int responses = 0, tag_mismatches = 0, order_complaints = 0;
bit quiet = 0;                       // 1: print the summary only
initial for (int t = 0; t < 16; t++) begin exp_w[t] = 0; exp_r[t] = 0; end

// Issue one request and hold it until the interconnect accepts it.
task automatic request(input logic [3:0] tag, input logic [7:0] addr);
  req_valid = 1; req_tag = tag; req_addr = addr;
  exp_q[tag][exp_w[tag]] = addr + 8'd1; exp_w[tag] = (exp_w[tag] + 1) % 8;
  order_q[order_w] = tag; order_w = (order_w + 1) % 32;
  do @(posedge clk); while (!req_ready);
  #1 if (!quiet) $display("edge %0d: request tag %0d to the %s target", cycle, tag, addr[7] ? "slow" : "fast");
  req_valid = 0; req_tag = 'x; req_addr = 'x;
endtask

// Wait n edges.
task automatic idle(input int n);
  repeat (n) @(posedge clk);
  #1;
endtask

// Every response is printed and checked by both scoreboards.
always @(posedge clk) begin
  logic v; logic [3:0] t; logic [7:0] d;
  v = rsp_valid; t = rsp_tag; d = rsp_data;
  #1 if (v) begin
    responses++;
    if (!quiet) $display("edge %0d: response tag %0d data %02h", cycle, t, d);
    if (d !== exp_q[t][exp_r[t]]) begin
      tag_mismatches++;
      if (!quiet) $display("  scoreboard by tag: tag %0d expected %02h, got %02h", t, exp_q[t][exp_r[t]], d);
    end
    exp_r[t] = (exp_r[t] + 1) % 8;
    if (t !== order_q[order_r]) begin
      order_complaints++;
      if (!quiet) $display("  scoreboard in order: expected tag %0d, got tag %0d", order_q[order_r], t);
    end
    order_r = (order_r + 1) % 32;
  end
end
final $display("%0d responses, %0d mismatches by tag, %0d complaints in request order", responses, tag_mismatches, order_complaints);
ConceptA target behind an interconnect answers reads in the order it receives them. Two reads with different tags go to two different targets. Which statement is correct?
ConceptWhat does reusing one tag for a second request promise the initiator?
Read the waveformRequests and responses on a tagged link. Which response breaks the ordering rule?
Tags out, tags back
0123456req_tag565xreq_addr801020xrsp_tagx65xrsp_datax112181x

Addresses from 80 go to the slow target; data is the address plus one.

ConceptAn initiator writes a status word to external memory and immediately reads it back through a faster path with a different tag. The read returns the old value. Who is at fault?

Remember

  • Responses may return in a different order from their requests, because targets differ in speed and nobody should wait behind a slow one.
  • A tag travels with the request and comes back on the response; the initiator matches by tag, never by position.
  • Responses with different tags may return in any order. Responses with the same tag return in request order, and the interconnect must enforce that across targets.
  • A scoreboard keys by tag with one queue per tag. One global queue reports legal reorders as errors.
  • A read after a write on different tags or paths may overtake the write; wait for the write response first.

Practice