Why Serial at Speed

Explain why fast links use a few serial lanes with an embedded clock, and find the rate where a parallel bus breaks.

You Will Be Able To

  • Predict the rate at which a parallel bus with a given skew fails.
  • Explain why a forwarded clock does not remove skew between wires.
  • Explain what an embedded clock asks of the data.
  • Choose the right word: lane, link, unit interval, baud or bit rate.
Lesson
P10.1
Time
25 min
Runs on
Your browser
Check
5 questions

An eight-bit bus works at 2 Gb/s on each wire. The team raises the clock to 2.5 Gb/s, and 22 of 40 words come out wrong. Every wire still has clean edges. The clock edge no longer lands where all eight wires are stable.

Rate on each wireBit timeShared windowWrong wordsWrong words, one wire 400 ps longer
100 Mb/s10000 ps9710 ps0 of 400 of 40
200 Mb/s5000 ps4710 ps0 of 400 of 40
500 Mb/s2000 ps1710 ps0 of 400 of 40
1 Gb/s1000 ps710 ps0 of 400 of 40
2 Gb/s500 ps210 ps0 of 4020 of 40
2.5 Gb/s400 ps110 ps22 of 4031 of 40
5 Gb/s200 ps-90 ps31 of 4037 of 40
10 Gb/s100 ps-190 ps33 of 4035 of 40

Bench run: eight data wires and a clock wire, 40 random words per rate. Trace delays from 450 to 700 ps, setup and hold 20 ps each. All of these are bench values.

Skew Eats the Bit Time

Two Ways to Move a ByteTop: eight wires and a clock wire. Bottom: one lane. Arrows carry bits from left to right.
  • Storage
  • Channel / handshake
  • Clock / reset
  • Datapath
  • Data
  • Clock
Two ways to move a byteTop: eight wires and a clock wire. Bottom: one lane. Arrows carry bits from left to right. storage Transmit register 8 bits at once channel 8 data wires each with its owndelay clock reset Clock wire one more delay storage 8 flip-flops one clock edge for all datapath Serializer 8 bits, one afteranother channel One lane one differential pair clock reset Clock recovery clock from the dataedges arrive at 8 differenttimessamples all 8data and timingtogether
  1. storage Transmit register8 bits at once
    • 8 data wires
    • Clock wire
  2. channel 8 data wireseach with its own delay
    • arrive at 8 different times → 8 flip-flops
  3. clock reset Clock wireone more delay
    • samples all 8 → 8 flip-flops
  4. storage 8 flip-flopsone clock edge for all
  5. datapath Serializer8 bits, one after another
    • One lane
  6. channel One laneone differential pair
    • data and timing together → Clock recovery
  7. clock reset Clock recoveryclock from the data edges
  1. Parallel: launch

    All eight bits leave on one clock edge. Each wire has a slightly different length, so the bits arrive at eight different times.

  2. Parallel: capture

    One clock edge must catch all eight wires. It has to fall after the slowest wire settles and before the fastest wire changes again.

  3. Serial: one wire pair

    The serializer sends the same bits one after another on one lane. No second data wire exists to be early or late.

  4. Serial: clock from the data

    The receiver finds the bit timing from the edges of the data itself. The lane delay moves data and timing together.

Read the diagram as text
  • Transmit register: 8 bits at once
  • 8 data wires: each with its own delay
  • Clock wire: one more delay
  • 8 flip-flops: one clock edge for all
  • Serializer: 8 bits, one after another
  • One lane: one differential pair
  • Clock recovery: clock from the data edges
  • Transmit register → 8 data wires
  • 8 data wires → 8 flip-flops: arrive at 8 different times
  • Transmit register → Clock wire
  • Clock wire → 8 flip-flops: samples all 8
  • Serializer → One lane
  • One lane → Clock recovery: data and timing together

The time difference between wires is called skew. The serial lane has no skew against itself.

A flip-flop needs its input to be stable for a short time before the clock edge (setup) and after it (hold).

On a bus, one clock edge samples every wire. So there must be a moment in each bit when all the wires are stable together. Call it the shared window.

TransientThe Shared Window Shrinks Faster Than the Bit TimeThe shared window is the time in each bit when all eight wires are valid and settled together.
  • bit time
  • shared window
-2 ns02 ns4 ns6 ns8 ns10 ns12 nsbit time, shared window100100010000Rate on each wire, Mb/sno time left-2 ns02 ns4 ns6 ns8 ns10 ns12 nsbit time, shared window100100010000Rate on each wire, Mb/sno time left
Read the plot as a table
Rate on each wire, Mb/sbit timeshared window
10010 ns9.71 ns
10001 ns710 ps
10000100 ps-190 ps

Bench run. The gap between the curves is constant: 250 ps of skew plus 40 ps of setup and hold. The window reaches zero while the bit time is still 290 ps.

The skew does not shrink when the clock gets faster. The traces are as long as they were. So skew takes a growing share of every bit.

The Clock Wire Has Skew Too

Sending the clock along with the data is called source-synchronous clocking. It removes the delay that all wires share. It cannot remove the differences between them.

WireTrace delaySetup margin at 2.5 Gb/sHold margin at 2.5 Gb/s
0500 ps180 ps180 ps
1520 ps160 ps200 ps
2480 ps200 ps160 ps
3610 ps70 ps290 ps
4450 ps230 ps130 ps
5530 ps150 ps210 ps
6700 ps-20 ps380 ps
7490 ps190 ps170 ps

Bench run. The clock wire has a 500 ps delay and its edge is sent in the middle of the bit. Wire 6 is the slowest, so it settles last. The first wrong word had the wire pattern 01000000, with wire 7 on the left.

The clock wire is one more trace with its own delay. Here it sits early in the window, so the slowest wire runs out of setup time first. In 100 Mb/s steps the bus first fails at 2.3 Gb/s. Wire 6 then has a setup margin of -3 ps.

At that rate the shared window is still open. The clock edge is simply not in the middle of it. The twin in the first table makes one trace 400 ps longer, and the bus already fails at 2 Gb/s.

One Lane with the Clock Inside the Data

A serial lane sends the bits one after another on one differential pair. There is no clock wire. The receiver recovers the clock from the edges in the data. This is an embedded clock.

The bench sent 80 bits at 10 Gb/s down one lane with trace delays of 450, 700, 900 ps. No bit was wrong in any run. The delay moves the data and its timing together, so it cancels.

The embedded clock has a price. The receiver needs edges, so the data must change often enough. Raw data makes no such promise. Line codes and scramblers exist to keep it (TI SLYY057B, p.3).

To go faster still, a link uses several lanes. Each lane recovers its own clock. The lanes are lined up later, on whole symbols, where a skew of many bit times is easy to absorb.

The Counter-Example: Wide Buses That Train Each Wire

Wide parallel buses did not disappear. DDR5, GDDR7 and HBM memories still send data on many wires beside a strobe, at several Gb/s per wire. HBM4 has 2,048 data wires at up to 10 Gb/s each (Rambus ISCA 2025 DRAM tutorial, slide 14).

They survive skew by measuring it. At start-up the controller trains each wire (slides 43 and 44). It sweeps that wire's delay, finds where the data is caught right, and parks the delay in the middle.

WireTrace delayPassing extra delay at 5 Gb/sTrim chosenDelay after training
0500 ps220 to 380 ps300 ps800 ps
1520 ps200 to 360 ps280 ps800 ps
2480 ps240 to 400 ps320 ps800 ps
3610 ps110 to 270 ps190 ps800 ps
4450 ps270 to 400 ps335 ps785 ps
5530 ps190 to 350 ps270 ps800 ps
6700 ps20 to 180 ps100 ps800 ps
7490 ps230 to 390 ps310 ps800 ps

Bench run on the same eight wires, with the clock wire moved to 800 ps. Each extra delay was swept from 0 to 400 ps in 10 ps steps. The sweep cut off wire 4's window.

Rate per wireUntrained: words wrong of 40Trained: words wrong of 40
2,500 Mb/s220
5,000 Mb/s310
10,000 Mb/s330
12,500 Mb/snot run0
16,000 Mb/snot run21
20,000 Mb/snot run21

Bench runs. Untrained, the bus fails from 2.5 Gb/s. Trained, it runs clean to 12.5 Gb/s.

After training the skew is gone except the 15 ps that the 10 ps steps leave. The limit is now the setup and hold times, and in real silicon jitter and noise.

The training has a cost. It must run at every power-up and track temperature, and each wire needs its own delay line. DDR5 adds dedicated training modes and patterns for this (Kingston DDR5 overview).

So the rule is not that parallel cannot go fast. It is that every wire must be measured and trimmed, while a serial lane carries its own timing for free.

TermMeaning
Differential pairTwo wires that carry one signal and its inverse. The receiver looks at the difference.
LaneOne differential pair in each direction.
LinkOne or more lanes between two ports, used together.
Unit interval (UI)The time of one symbol on the wire: 100 ps at 10 Gb/s with two levels.
BaudSymbols per second on the wire. It equals the line rate only for two-level signalling.
Line rateBits per second on the wire, line-code overhead included. USB Gen 1's 5 Gb/s is a line rate.
Data rateData bits per second after the line code: the payload rate.
Embedded clockNo clock wire. The receiver recovers the clock from the data edges.

USB 3.2 calls its lane "dual-simplex": four wires, one pair for each direction (Table 3-1).

The two wires of one pair must still match each other closely. TI asks for 5 ps inside a pair in one of its layout guides (SPRABC1 §3.2). That is one short pair to match, not a whole bus.

Try It

The bench sends 40 random words at two rates and prints the margins of each wire at the second rate.

A negative setup margin means the wire changes less than 20 ps before the clock edge. The receiver then captures an unknown value.

Change clk_delay_ps to 575, the middle of the data delays. Wire 6 gets its setup margin back, and both rates pass.

RUNIcarus Verilog · in your browser
Edit the code, then run it
Expected: 2000 Mb/s: 0 of 40 words wrong 2500 Mb/s: 22 of 40 words wrong wire 0: delay 500 ps, setup margin 180 ps, hold margin 180 ps wire 1: delay 520 ps, setup margin 160 ps, hold margin 200 ps wire 2: delay 480 ps, setup margin 200 ps, hold margin 160 ps wire 3: delay 610 ps, setup margin 70 ps, hold margin 290 ps wire 4: delay 450 ps, setup margin 230 ps, hold margin 130 ps wire 5: delay 530 ps, setup margin 150 ps, hold margin 210 ps wire 6: delay 700 ps, setup margin -20 ps, hold margin 380 ps wire 7: delay 490 ps, setup margin 190 ps, hold margin 170 ps first wrong word 6, wrong wires 01000000 (wire 7 on the left)
Read included file: hs_pkg.sv
`timescale 1ns/1ns
// hs_pkg.sv: shared pure functions, and the bench random generator (read-only).
package hs_pkg;
  // xorshift32 (Marsaglia 2003): a linear generator over GF(2), period 2^32 - 1. A zero state is
  // replaced by 1, because zero maps to itself.
  function automatic [31:0] xs32(input [31:0] s);
    reg [31:0] x;
    x = (s == 32'd0) ? 32'd1 : s;
    x = x ^ (x << 13); x = x ^ (x >> 17); x = x ^ (x << 5);
    return x;
  endfunction
  // A Fibonacci PRBS generator x^n + x^m + 1 (PRBS7: 7,6; PRBS9: 9,5; PRBS15: 15,14; PRBS23: 23,18;
  // PRBS31: 31,28). The output bit is s[n-1]; the next state shifts the feedback into bit 0.
  function automatic [31:0] prbs_next(input [31:0] s, input int n, input int m);
    reg fb;
    fb = s[n-1] ^ s[m-1];
    return ((s << 1) | {31'd0, fb}) & ((n == 32) ? 32'hffffffff : ((32'd1 << n) - 32'd1));
  endfunction
  function automatic int ones(input [63:0] v);
    int n; n = 0;
    for (int i = 0; i < 64; i++) n += v[i];
    return n;
  endfunction
  // GF(16) with x^4 + x + 1, and GF(1024) with x^10 + x^3 + 1: shift-and-add multiply, no tables.
  function automatic [3:0] gf16_mul(input [3:0] a, input [3:0] b);
    reg [3:0] p, x; p = 4'd0; x = a;
    for (int i = 0; i < 4; i++) begin
      if (b[i]) p = p ^ x;
      x = x[3] ? ({x[2:0], 1'b0} ^ 4'b0011) : {x[2:0], 1'b0};
    end
    return p;
  endfunction
  function automatic [3:0] gf16_pow(input int e);   // alpha^e, alpha = x
    reg [3:0] p; int k; p = 4'd1; k = e % 15; if (k < 0) k += 15;
    for (int i = 0; i < k; i++) p = gf16_mul(p, 4'd2);
    return p;
  endfunction
  function automatic int gf16_log(input [3:0] a);   // -1 for 0
    reg [3:0] p; p = 4'd1;
    if (a == 4'd0) return -1;
    for (int i = 0; i < 15; i++) begin if (p == a) return i; p = gf16_mul(p, 4'd2); end
    return -1;
  endfunction
  function automatic [9:0] gf1024_mul(input [9:0] a, input [9:0] b);
    reg [9:0] p, x; p = 10'd0; x = a;
    for (int i = 0; i < 10; i++) begin
      if (b[i]) p = p ^ x;
      x = x[9] ? ({x[8:0], 1'b0} ^ 10'b0000001001) : {x[8:0], 1'b0};
    end
    return p;
  endfunction
  function automatic [9:0] gf1024_pow(input int e);
    reg [9:0] p; int k; p = 10'd1; k = e % 1023; if (k < 0) k += 1023;
    for (int i = 0; i < k; i++) p = gf1024_mul(p, 10'd2);
    return p;
  endfunction
endpackage

// hs_rng: the bench's random numbers. Same seed, same numbers, on every engine.
module hs_rng;
  int unsigned s = 32'h2545f491;
  function automatic void seed(input int unsigned v); s = (v == 0) ? 32'd1 : v; endfunction
  function automatic int unsigned next(); s = hs_pkg::xs32(s); return s; endfunction
  function automatic int below(input int n); s = hs_pkg::xs32(s); return (n <= 0) ? 0 : int'(s % n); endfunction
  function automatic bit bit1(); s = hs_pkg::xs32(s); return s[16]; endfunction
  function automatic bit chance_ppm(input int ppm); s = hs_pkg::xs32(s); return (s % 1000000) < ppm; endfunction
  function automatic [7:0] byte8(); s = hs_pkg::xs32(s); return s[23:16]; endfunction
endmodule

Check Yourself

Predict the outputSeven wires and the clock wire have a 500 ps delay. One wire has 620 ps. Setup and hold are 20 ps each, and the clock edge is sent mid-bit. The program raises the rate in steps of 1,000 Mb/s. What does it print?
`timescale 1ps/1ps
module tb;
  hs_rng rng();
  localparam int WIRES = 8, SETUP = 20, HOLD = 20;      // ps, bench values
  int delay_ps [0:WIRES-1];                             // one trace delay per data wire, ps
  int clk_delay_ps = 500;                               // the clock wire's own delay, ps
  reg [WIRES-1:0] pins = 0;                             // the data wires at the receiver
  reg clk_rx = 0;                                       // the clock wire at the receiver
  reg [7:0] sent [0:63]; int n_sent, n_got, wrong, first_wrong; reg [7:0] bad_wires;
  int bit_ps;
  // The receiver: one flip-flop per wire, all on the same clock wire. A wire that changes less than
  // SETUP before the clock edge, or less than HOLD after it, is captured as unknown (x).
  reg [WIRES-1:0] cap, now_v = 0, before_v = 0; int t_change [0:WIRES-1]; int t_edge;
  genvar g;
  generate for (g = 0; g < WIRES; g++) begin : watch
    always @(pins[g]) begin before_v[g] = now_v[g]; now_v[g] = pins[g]; t_change[g] = $time; end
  end endgenerate
  always @(posedge clk_rx) begin
    t_edge = $time;
    #(HOLD + 1);                                        // judge the edge once the hold time has passed
    for (int w = 0; w < WIRES; w++) begin
      if (t_change[w] <= t_edge - SETUP) cap[w] = now_v[w];          // settled in time
      else if (t_change[w] >= t_edge + HOLD) cap[w] = before_v[w];   // changed after the hold time
      else cap[w] = 1'bx;                                            // changed inside the window
    end
    if (cap !== sent[n_got]) begin
      if (wrong == 0) begin
        first_wrong = n_got;
        for (int w = 0; w < WIRES; w++) bad_wires[w] = (cap[w] !== sent[n_got][w]);
      end
      wrong++;
    end
    n_got++;
  end
  // The margins: how long each wire has been stable at the clock edge, and for how long it stays.
  function automatic int setup_margin(input int w); return bit_ps / 2 + clk_delay_ps - delay_ps[w] - SETUP; endfunction
  function automatic int hold_margin(input int w);  return bit_ps / 2 - clk_delay_ps + delay_ps[w] - HOLD;  endfunction
  function automatic int window();                      // the time when all eight wires are valid and settled together
    int lo, hi; lo = delay_ps[0]; hi = delay_ps[0];
    for (int w = 1; w < WIRES; w++) begin if (delay_ps[w] < lo) lo = delay_ps[w]; if (delay_ps[w] > hi) hi = delay_ps[w]; end
    return bit_ps - (hi - lo) - SETUP - HOLD;
  endfunction
  task automatic send_words(input int mbps, input int words);
    reg [7:0] v;
    bit_ps = 1000000 / mbps; n_sent = 0; n_got = 0; wrong = 0; rng.seed(32'd11);
    for (int i = 0; i < words; i++) begin
      v = rng.byte8(); sent[n_sent] = v; n_sent++;
      for (int w = 0; w < WIRES; w++) pins[w] <= #(delay_ps[w]) v[w];       // each wire has its own delay
      clk_rx <= #(clk_delay_ps + bit_ps / 2) 1'b1;                          // the clock edge, mid-bit
      clk_rx <= #(clk_delay_ps + bit_ps) 1'b0;
      #(bit_ps);
    end
    #(3000);
  endtask
  task automatic load_delays();
    delay_ps[0] = 500;     delay_ps[1] = 500;     delay_ps[2] = 500;     delay_ps[3] = 500;     delay_ps[4] = 500;
    delay_ps[5] = 500;     delay_ps[6] = 500;     delay_ps[7] = 620;
  endtask
  initial begin
    load_delays();
    for (int mbps = 1000; mbps <= 8000; mbps += 1000) begin
      send_words(mbps, 40);
      if (wrong != 0) begin $display("first failing rate %0d Mb/s", mbps); $finish; end
    end
    $display("no failure");
    $finish;
  end
endmodule
Read included file: hs_pkg.sv
`timescale 1ns/1ns
// hs_pkg.sv: shared pure functions, and the bench random generator (read-only).
package hs_pkg;
  // xorshift32 (Marsaglia 2003): a linear generator over GF(2), period 2^32 - 1. A zero state is
  // replaced by 1, because zero maps to itself.
  function automatic [31:0] xs32(input [31:0] s);
    reg [31:0] x;
    x = (s == 32'd0) ? 32'd1 : s;
    x = x ^ (x << 13); x = x ^ (x >> 17); x = x ^ (x << 5);
    return x;
  endfunction
  // A Fibonacci PRBS generator x^n + x^m + 1 (PRBS7: 7,6; PRBS9: 9,5; PRBS15: 15,14; PRBS23: 23,18;
  // PRBS31: 31,28). The output bit is s[n-1]; the next state shifts the feedback into bit 0.
  function automatic [31:0] prbs_next(input [31:0] s, input int n, input int m);
    reg fb;
    fb = s[n-1] ^ s[m-1];
    return ((s << 1) | {31'd0, fb}) & ((n == 32) ? 32'hffffffff : ((32'd1 << n) - 32'd1));
  endfunction
  function automatic int ones(input [63:0] v);
    int n; n = 0;
    for (int i = 0; i < 64; i++) n += v[i];
    return n;
  endfunction
  // GF(16) with x^4 + x + 1, and GF(1024) with x^10 + x^3 + 1: shift-and-add multiply, no tables.
  function automatic [3:0] gf16_mul(input [3:0] a, input [3:0] b);
    reg [3:0] p, x; p = 4'd0; x = a;
    for (int i = 0; i < 4; i++) begin
      if (b[i]) p = p ^ x;
      x = x[3] ? ({x[2:0], 1'b0} ^ 4'b0011) : {x[2:0], 1'b0};
    end
    return p;
  endfunction
  function automatic [3:0] gf16_pow(input int e);   // alpha^e, alpha = x
    reg [3:0] p; int k; p = 4'd1; k = e % 15; if (k < 0) k += 15;
    for (int i = 0; i < k; i++) p = gf16_mul(p, 4'd2);
    return p;
  endfunction
  function automatic int gf16_log(input [3:0] a);   // -1 for 0
    reg [3:0] p; p = 4'd1;
    if (a == 4'd0) return -1;
    for (int i = 0; i < 15; i++) begin if (p == a) return i; p = gf16_mul(p, 4'd2); end
    return -1;
  endfunction
  function automatic [9:0] gf1024_mul(input [9:0] a, input [9:0] b);
    reg [9:0] p, x; p = 10'd0; x = a;
    for (int i = 0; i < 10; i++) begin
      if (b[i]) p = p ^ x;
      x = x[9] ? ({x[8:0], 1'b0} ^ 10'b0000001001) : {x[8:0], 1'b0};
    end
    return p;
  endfunction
  function automatic [9:0] gf1024_pow(input int e);
    reg [9:0] p; int k; p = 10'd1; k = e % 1023; if (k < 0) k += 1023;
    for (int i = 0; i < k; i++) p = gf1024_mul(p, 10'd2);
    return p;
  endfunction
endpackage

// hs_rng: the bench's random numbers. Same seed, same numbers, on every engine.
module hs_rng;
  int unsigned s = 32'h2545f491;
  function automatic void seed(input int unsigned v); s = (v == 0) ? 32'd1 : v; endfunction
  function automatic int unsigned next(); s = hs_pkg::xs32(s); return s; endfunction
  function automatic int below(input int n); s = hs_pkg::xs32(s); return (n <= 0) ? 0 : int'(s % n); endfunction
  function automatic bit bit1(); s = hs_pkg::xs32(s); return s[16]; endfunction
  function automatic bit chance_ppm(input int ppm); s = hs_pkg::xs32(s); return (s % 1000000) < ppm; endfunction
  function automatic [7:0] byte8(); s = hs_pkg::xs32(s); return s[23:16]; endfunction
endmodule
ConceptA parallel bus forwards its clock on a ninth wire. Why does skew between the data wires still limit the rate?
ConceptWhat does an embedded clock ask of the data?
ConceptA lane runs at 5 gigabaud with two levels and 8b/10b coding. What is its data rate?
ConceptA link needs four times the throughput of one lane. What does a serial link do?

Common Mistakes

Wrong Say that serial is slower because it sends one bit at a time.

A lane has no skew against another wire, so the shared window does not cap its rate.

Fix Compare total throughput: lanes times line rate times the share of line bits that are data.

Wrong Count a differential pair as two data streams.

The two wires carry one signal and its inverse, and the receiver uses only their difference.

Fix One pair is one stream in one direction. A lane has two pairs, one for each direction.

Wrong Use baud and bit rate as the same number.

Baud counts symbols on the wire, and a symbol may carry more or less than one data bit.

Fix State both: a 5 Gb/s line with 8b/10b coding carries 4 Gb/s of data.

Wrong Expect a forwarded clock to remove every timing problem.

The clock wire has its own delay, and it does nothing about the spread between data wires.

Fix Budget skew, setup and hold against the bit time, and place the clock edge in the middle of the shared window.

Remember

  • On a parallel bus one clock edge samples every wire, so all wires must be stable together.
  • The shared window is the bit time minus skew, setup and hold. Skew stays the same as the bit time shrinks.
  • A forwarded clock is one more wire with its own delay. It cannot remove skew between data wires.
  • A serial lane carries its clock in the data edges, so its own delay cancels. The data must then have enough transitions.
  • Baud counts symbols on the wire, and bit rate counts data bits. Line coding makes them differ.

Practice

Report a problem with this page