Ordered Sets, Alignment and Lane Deskew
Explain how a receiver finds symbols, then blocks, then lines up several lanes.
You Will Be Able To
- Explain what an ordered set is and name the alignments a receiver performs.
- Predict the latency a deskew adds for a table of lane skews.
- Choose a FIFO depth that absorbs a given skew.
- Explain how SKP ordered sets and an elastic buffer absorb a clock difference.
- Lesson
- P10.12
- Time
- 30 min
- Runs on
- Your browser
- Check
- 5 questions
Four lanes carry one byte stream. On the board, one lane is five symbols longer than another. A receiver that reads the lanes as they arrive mixes bytes from different moments. A bench run of 2,000 symbol times gave 5,805 wrong symbols.
- Control / FSM
- Structure
- Bit lock
The CDR of P10.6 centres the sampling clock on the bits. USB 3.2 sends D10.2, alternating ones and zeros, for this (§6.4.1.1).
- Symbol lock
The receiver finds where each symbol starts: from a comma in 8b/10b, or from the sync header in a block code.
- Lane deskew
With several lanes, each one has its own delay. A marker sent on every lane at once shows the receiver how to line them up.
- Clock compensation
The two ends run on different clocks. SKP ordered sets give the receiver symbols it may drop or repeat.
Read the diagram as text
- Bit lock: CDR finds the bit centre
- Symbol lock: comma or block header
- Lane deskew: a marker on every lane
- Clock compensation: SKP and the elastic buffer
- Bit lock → Symbol lock
- Symbol lock → Lane deskew
- Symbol lock → Clock compensation: runs all the time
An ordered set is a fixed group of symbols with a meaning: TS1, TS2, SKP, a marker.
Ordered Sets
An ordered set is a fixed group of symbols with a meaning that the link layer, not the data, understands. Training sets, skip sets and markers are ordered sets.
At Gen 1, USB 3.2 trains with TS1 and TS2 sets, which are always 8b/10b encoded and never scrambled (§6.4.1.1). At Gen 2 they are 128b/132b control blocks, and most of their symbols are scrambled (§6.3.2.3).
A SKP set at Gen 1 is two K28.1 symbols (§6.4.3.1).
The comma inside them is unique: it never appears across data symbols of either polarity, unless a bit is wrong (§6.4.1.1). P10.3 measured why.
Ordered sets come after bit lock. USB 3.2 first sends D10.2, alternating ones and zeros, so that the CDR can centre its clock (§6.4.1.1).
Lane Deskew
A wide link splits its data across lanes. Each lane has its own wire length and its own circuits, so each arrives with its own delay. That difference is lane-to-lane skew.
The transmitter sends a marker on every lane at the same moment. Each receiving lane writes into its own FIFO from its marker on. When every lane has its marker, the receiver reads one symbol from each FIFO per step.
- Cycle 1
Lane 0 has no skew. Its marker arrives first, and it starts filling its FIFO.
- Cycle 2
Lane 2 is one symbol late. Its marker arrives one step later.
- Cycle 4
Lane 1, three symbols late, starts filling. Lane 3 is still waiting.
- Cycle 6
Lane 3, the latest, finally has its marker. Every lane is locked.
- Cycle 7
Reading starts: one symbol from each FIFO per step. The fill levels stop changing. Each FIFO now holds the skew it absorbs.
Bench run, depth 8. The deepest FIFO holds 6 symbols, and the deskew adds 5 symbol times of latency: the largest skew minus the smallest.
| Skews (symbol times) | FIFO depth | Wrong symbols of 4 lanes × about 1,900 | Added latency |
|---|---|---|---|
| 0, 3, 1, 5 | 8 | 0 | 5 |
| 0, 3, 1, 5 | 4 | 3,860 | 5 |
| 0, 3, 1, 70 | 16 | 5,505 | 70 |
| 0, 3, 1, 70 | 128 | 0 | 70 |
0, 3, 1, 5, ds.read_on_first_lock = 1 | 8 | 5,805 | 0 |
Bench runs, 2,000 symbol times each. A FIFO must hold at least the largest skew minus the smallest, plus one symbol.
Deskew adds latency: every lane waits for the slowest. It never removes it.
USB4 sends a de-skew ordered set on every lane at the same time and in the same place in the symbol. If the skew is larger than the buffer, the router may treat it as an error (§4.4.4).
AMD's transceiver wizard calls this channel bonding, configured by one or two bonding sequences (PG182).
Striping Bytes Across Lanes
A wide link deals its bytes out to the lanes in turn. USB4 alternates them: byte 0 on lane 0, byte 1 on lane 1, byte 2 on lane 0 (§4.3.1.1).
The receiver must reassemble them in the same order. The bench striped 4,000 bytes over four lanes. In order, 0 came out wrong; with lanes 0 and 1 swapped, 1,992.
A receiver must therefore learn which physical lane is which, for example from lane numbers sent during training. Only then can it undo a swap and reassemble the bytes.
Clock Compensation
Deskew lines up lanes. A different problem remains: the transmitter and receiver run on different clocks. A faster transmitter delivers more symbols than the receiver reads.
USB 3.2 allows each reference clock ±300 ppm, and spread spectrum adds up to 5000 ppm. The total difference ranges from -5300 to +300 ppm (§6.4.3).
The fix is a symbol the receiver may drop or repeat without harm: SKP. An elastic buffer sits between the recovered clock and the local clock. It drops a SKP when it is too full and repeats one when it is too empty.
| Transmitter | SKPs removed | SKPs added | Overflows | Underflows | Data symbols wrong |
|---|---|---|---|---|---|
| 5000 ppm faster | 145 | 0 | 0 | 0 | 0 |
| 5000 ppm slower | 0 | 146 | 0 | 0 | 0 |
| 300 ppm faster | 4 | 0 | 0 | 0 | 0 |
5000 ppm faster, eb.no_skip_removal = 1 | 0 | 0 | 143 | 0 | 27,675 |
5000 ppm slower, eb.no_skip_insertion = 1 | 0 | 0 | 0 | 142 | 0 |
Bench runs of 30,000 symbol times. Bench values: depth 16, SKPs dropped above 12 entries and repeated below 4, a one-symbol SKP every 40 symbols.
At 5000 ppm faster the buffer removed 145 SKPs in 30,000 symbol times. That is the 150 extra symbols the transmitter sent, less the few the buffer could still hold.
Without removal, the buffer overflowed 143 times and lost data. Without insertion it ran empty 142 times. The bench hands out a SKP when it is empty, so no data went wrong here. A buffer that reads stale entries would pass garbage.
USB 3.2 Gen 1 sends a two-symbol SKP set every 354 symbols on average, and the receiver buffers 8 symbols. Gen 2 sends one every 40 blocks, with an 11-symbol buffer (§6.4.3).
A check of our own: two symbols every 354 is 2 / 354, about 5,650 ppm. That covers the 5,300 ppm worst case with a little room.
PIPE describes two buffer models (§8.14). A nominal half full buffer adds or removes SKPs and reports each change on RxStatus. A nominal empty buffer removes every SKP and flags the gaps with RxDataValid.
Sizing the Elastic Buffer
Two numbers size an elastic buffer: how fast the clocks drift apart, and the longest wait for a usable SKP.
The first is a rate rule. The SKPs must arrive at least as fast as the drift: SKP symbols per interval ≥ interval × ppm. If not, no depth is enough; the buffer only fills or empties more slowly.
The second is a depth rule. Between two chances to use a SKP, the buffer must hold the drift of that gap, plus its watermark margins.
| SKPs at ±5300 ppm | Drift per gap | Smallest clean depth |
|---|---|---|
| one every 40 symbols | 0.21 symbols | 8 |
| one every 150 symbols | 0.80 symbols | 8 |
| one every 354 symbols | 1.88 symbols | none up to 64 |
| 4 together, every 600 symbols | 3.18 symbols | 20 |
| 8 together, every 1,200 symbols | 6.36 symbols | 32 |
| 16 together, every 2,400 symbols | 12.72 symbols | 56 |
Bench runs of hs_ebuf, 10,000 symbol times each way. Watermarks sit at a quarter and three quarters, SKPs are one symbol, depths go in steps of 4.
One SKP every 354 symbols fails at every depth tried: 1.88 symbols of drift per gap against one SKP. A deeper buffer only fails later.
USB 3.2 Gen 1 sends two SKP symbols each time, which passes the rate rule: 2 / 354 is about 5,650 ppm.
Clumping SKPs keeps the rate but lengthens the gap. The depth grows with it: about four times the drift of a gap, plus a few symbols.
SKPs may not interrupt a packet, so USB 3.2 Gen 1 lets a transmitter hold up to four SKP sets. Its receiver has an 8-symbol buffer (§6.4.3).
Try It
Each lane locks at step 64 plus its skew. The first column is read one step after the last lock.
The deepest FIFO is lane 0, the earliest: it holds 6 symbols, the skew of 5 plus the one being written.
Change the depth from 8 to 4 and run again. The early lanes overwrite their oldest symbols, and the wrong count jumps.
markers locked at steps 64 67 65 69 first column read at step 70: added latency 5, deepest FIFO 6 1930 columns read, 0 wrong symbolsRead included file: hs_pkg.sv
`timescale 1ns/1ns
// hs_pkg.sv: shared pure functions, and the bench random generator (read-only).
package hs_pkg;
// xorshift32 (Marsaglia 2003): a linear generator over GF(2), period 2^32 - 1. A zero state is
// replaced by 1, because zero maps to itself.
function automatic [31:0] xs32(input [31:0] s);
reg [31:0] x;
x = (s == 32'd0) ? 32'd1 : s;
x = x ^ (x << 13); x = x ^ (x >> 17); x = x ^ (x << 5);
return x;
endfunction
// A Fibonacci PRBS generator x^n + x^m + 1 (PRBS7: 7,6; PRBS9: 9,5; PRBS15: 15,14; PRBS23: 23,18;
// PRBS31: 31,28). The output bit is s[n-1]; the next state shifts the feedback into bit 0.
function automatic [31:0] prbs_next(input [31:0] s, input int n, input int m);
reg fb;
fb = s[n-1] ^ s[m-1];
return ((s << 1) | {31'd0, fb}) & ((n == 32) ? 32'hffffffff : ((32'd1 << n) - 32'd1));
endfunction
function automatic int ones(input [63:0] v);
int n; n = 0;
for (int i = 0; i < 64; i++) n += v[i];
return n;
endfunction
// GF(16) with x^4 + x + 1, and GF(1024) with x^10 + x^3 + 1: shift-and-add multiply, no tables.
function automatic [3:0] gf16_mul(input [3:0] a, input [3:0] b);
reg [3:0] p, x; p = 4'd0; x = a;
for (int i = 0; i < 4; i++) begin
if (b[i]) p = p ^ x;
x = x[3] ? ({x[2:0], 1'b0} ^ 4'b0011) : {x[2:0], 1'b0};
end
return p;
endfunction
function automatic [3:0] gf16_pow(input int e); // alpha^e, alpha = x
reg [3:0] p; int k; p = 4'd1; k = e % 15; if (k < 0) k += 15;
for (int i = 0; i < k; i++) p = gf16_mul(p, 4'd2);
return p;
endfunction
function automatic int gf16_log(input [3:0] a); // -1 for 0
reg [3:0] p; p = 4'd1;
if (a == 4'd0) return -1;
for (int i = 0; i < 15; i++) begin if (p == a) return i; p = gf16_mul(p, 4'd2); end
return -1;
endfunction
function automatic [9:0] gf1024_mul(input [9:0] a, input [9:0] b);
reg [9:0] p, x; p = 10'd0; x = a;
for (int i = 0; i < 10; i++) begin
if (b[i]) p = p ^ x;
x = x[9] ? ({x[8:0], 1'b0} ^ 10'b0000001001) : {x[8:0], 1'b0};
end
return p;
endfunction
function automatic [9:0] gf1024_pow(input int e);
reg [9:0] p; int k; p = 10'd1; k = e % 1023; if (k < 0) k += 1023;
for (int i = 0; i < k; i++) p = gf1024_mul(p, 10'd2);
return p;
endfunction
endpackage
// hs_rng: the bench's random numbers. Same seed, same numbers, on every engine.
module hs_rng;
int unsigned s = 32'h2545f491;
function automatic void seed(input int unsigned v); s = (v == 0) ? 32'd1 : v; endfunction
function automatic int unsigned next(); s = hs_pkg::xs32(s); return s; endfunction
function automatic int below(input int n); s = hs_pkg::xs32(s); return (n <= 0) ? 0 : int'(s % n); endfunction
function automatic bit bit1(); s = hs_pkg::xs32(s); return s[16]; endfunction
function automatic bit chance_ppm(input int ppm); s = hs_pkg::xs32(s); return (s % 1000000) < ppm; endfunction
function automatic [7:0] byte8(); s = hs_pkg::xs32(s); return s[23:16]; endfunction
endmodule
Read included file: hs_deskew.sv
`timescale 1ns/1ns
// hs_deskew.sv: lane deskew and clock compensation (read-only).
// hs_deskew: up to 8 lanes of 9-bit symbols ({k, d}). Each lane locks on its first marker (default
// K28.5, 9'h1bc) and writes from the marker on into its own FIFO of depth symbols. Once every lane
// has locked, one column (one symbol per lane) is read each symbol time. Per symbol time the bench
// calls put(lane, sym) for each lane that has a symbol, then step(); step() reads first (if every lane
// locked on an earlier step), then writes. col(lane) and col_valid are the column read.
// Counters: reads, max_fill, overflows (a write into a full FIFO overwrites the oldest unread entry),
// first_read (the step of the first read), marker_at[lane] (the step at which each lane locked),
// added_latency (first_read - 1 - the earliest lock: the wait the deskew adds; it equals the largest skew
// minus the smallest).
// hs_ebuf: an elastic buffer between a write clock and a read clock (depth up to 64). eb_write(sym, is_skp) on the
// write clock, eb_read() on the read clock (returns {is_skp, sym}). Above hi entries an arriving SKP
// is dropped; below lo entries a SKP just read is sent twice. The buffer starts at start_fill
// entries of SKP. Counters: removed, added, overflows, underflows, min_fill, max_fill.
// Bench values: lanes up to 8, depth up to 128 (restart clamps larger values and says so); hs_ebuf's
// depth 16, lo 4, hi 12, start_fill 8 and its one-symbol SKP. USB 3.2 Gen 1 sends a two-symbol SKP
// ordered set about every 354 symbols, Gen 2 one SKP block every 40 blocks (§6.4.3).
// Faults: read_on_first_lock (hs_deskew starts reading when the first lane locks),
// no_skip_removal (hs_ebuf never drops a SKP), no_skip_insertion (hs_ebuf never adds one).
// The FIFO depth is a setting, not a fault.
module hs_deskew;
int lanes = 4, depth = 8; bit [8:0] marker = 9'h1bc;
bit read_on_first_lock = 0, trace = 0;
bit [8:0] fifo [0:1023]; int wr [0:7]; bit locked [0:7]; int marker_at [0:7]; bit [8:0] inq [0:7]; bit inv [0:7];
bit [8:0] colv [0:7]; bit col_valid = 0; int rd = 0, steps = 0, reads = 0, max_fill = 0, overflows = 0, first_read = -1, added_latency = -1;
function automatic void restart(input int n, input int d);
lanes = n; depth = d;
if (lanes > 8) begin $display("hs_deskew: %0d lanes clamped to 8", lanes); lanes = 8; end
if (depth > 128) begin $display("hs_deskew: depth %0d clamped to 128", depth); depth = 128; end
if (depth < 1) depth = 1;
rd = 0; steps = 0; reads = 0; max_fill = 0; overflows = 0; first_read = -1; added_latency = -1; col_valid = 0;
for (int l = 0; l < 8; l++) begin wr[l] = 0; locked[l] = 0; marker_at[l] = -1; inv[l] = 0; end
for (int i = 0; i < 1024; i++) fifo[i] = 9'd0;
endfunction
function automatic void put(input int l, input [8:0] sym); inq[l] = sym; inv[l] = 1; endfunction
function automatic [8:0] col(input int l); return colv[l]; endfunction
function automatic bit step();
reg go; int occ; go = !read_on_first_lock; steps++; col_valid = 0;
for (int l = 0; l < lanes; l++) if (read_on_first_lock) go = go | locked[l]; else go = go & locked[l];
if (go) begin
if (first_read < 0) begin
first_read = steps; added_latency = 0;
for (int l = 0; l < lanes; l++) if (locked[l] && first_read - 1 - marker_at[l] > added_latency) added_latency = first_read - 1 - marker_at[l];
end
for (int l = 0; l < lanes; l++) begin
occ = wr[l] - rd; if (occ > max_fill) max_fill = occ;
colv[l] = fifo[l * 128 + (rd % depth)];
end
rd++; reads++; col_valid = 1;
end
for (int l = 0; l < lanes; l++) if (inv[l]) begin
if (!locked[l] && inq[l] == marker) begin locked[l] = 1; marker_at[l] = steps; if (trace) $display("s%0d L%0d LOCK", steps, l); end
if (locked[l]) begin
if (wr[l] - rd >= depth) overflows++;
fifo[l * 128 + (wr[l] % depth)] = inq[l]; wr[l]++;
end
inv[l] = 0;
end
return col_valid;
endfunction
endmodule
module hs_ebuf;
int depth = 16, lo = 4, hi = 12, start_fill = 8; bit [8:0] skp = 9'h13c; // SKP: K28.1
bit no_skip_removal = 0, no_skip_insertion = 0, trace = 0;
bit [9:0] q [0:63]; int head = 0, count = 0;
int removed = 0, added = 0, overflows = 0, underflows = 0, min_fill = 0, max_fill = 0;
bit [9:0] last = 10'd0; bit repeat_skp = 0;
function automatic void eb_reset();
if (depth > 64) begin $display("hs_ebuf: depth %0d clamped to 64", depth); depth = 64; end
head = 0; count = 0; removed = 0; added = 0; overflows = 0; underflows = 0; repeat_skp = 0;
for (int i = 0; i < start_fill; i++) begin q[i] = {1'b1, skp}; count++; end
min_fill = count; max_fill = count;
endfunction
function automatic void eb_write(input [8:0] sym, input bit is_skp);
if (is_skp && count > hi && !no_skip_removal) begin removed++; if (trace) $display("EB removed SKP fill=%0d", count); return; end
if (count == depth) begin overflows++; return; end
q[(head + count) % 64] = {is_skp, sym}; count++; if (count > max_fill) max_fill = count;
endfunction
function automatic [9:0] eb_read();
if (repeat_skp) begin repeat_skp = 0; added++; if (trace) $display("EB added SKP fill=%0d", count); return {1'b1, skp}; end
if (count == 0) begin underflows++; return {1'b1, skp}; end
last = q[head]; head = (head + 1) % 64; count--; if (count < min_fill) min_fill = count;
if (last[9] && count < lo && !no_skip_insertion) repeat_skp = 1;
return last;
endfunction
endmodule
Check Yourself
Common Mistakes
Wrong Start reading the lanes when the first one sees its marker.
The other lanes have not locked, so their FIFOs hold nothing for that column: 5,805 wrong symbols on the bench.
Fix Read only when every lane has locked on its marker.
go = 0;
for (int l = 0; l < LANES; l++) go |= locked[l];go = 1;
for (int l = 0; l < LANES; l++) go &= locked[l];Wrong Size the deskew FIFO for the typical skew.
A FIFO shorter than the worst skew overwrites symbols; a skew of 70 in FIFOs of 16 gave 5,505 wrong symbols.
Fix Size it for the worst skew the standard allows plus one symbol, and flag skew beyond it as an error.
Wrong Expect the elastic buffer to fix lane skew.
The elastic buffer handles a rate difference between two clocks; skew is a fixed delay difference between lanes.
Fix Use deskew FIFOs and a marker for skew, and SKP with an elastic buffer for the clock difference.
Wrong Treat a SKP as data, or count it in the payload.
SKPs are added and removed on the way, so their number at the receiver differs from what was sent.
Fix Drop SKPs before the data path, and never let them change scrambler or sequence state.
Remember
- An ordered set is a fixed group of symbols with a link meaning: training, skip, marker.
- A receiver aligns bits, then symbols, then lanes.
- Deskew FIFOs wait for the latest lane. They must hold the largest skew minus the smallest, plus one.
- Deskew adds latency equal to the skew it absorbs. It never removes latency.
- SKP ordered sets and an elastic buffer absorb a clock difference by dropping or repeating SKPs.