P11.34.2 A Restart Goes Wrong

MediumDebugIcarus Verilog · serverReview the topic

An 800G link runs clean until block 527. Then the far end is reset, and block 513 arrives where block 521 was due. Five bugs would give this log. Find the one that did.

Hardware context: An Ethernet MAC and physical coding sublayer: frame builder and checker, reconciliation sublayer and block encoder. Build, check and pace Ethernet frames, and code them for the physical layer as IEEE 802.3 specifies.

The design below is the two ends of an 800GBASE-R PCS, as in P11.34.1. It is written anew and has another bug.

The near end deals 66-bit blocks to two flows and says where each flow puts its marker group.

The far end sorts 32 wires into two sets of 16 lanes. Then it weaves the two decoders' blocks into one stream.

The bench sends 700 numbered blocks. The far end listens from the first block on, and every block arrives in order.

At cycle 560 the far end is reset. The near end runs on. The far end reads its markers and collects again.

After that the log says: ERROR: the far end got block 513 where block 521 was due.

Step 1: find the first cycle where a signal differs from a correct design, and name the signal.

Cycles 1 to 32 read the markers. From cycle 33 on, cycle N sends block N − 33.

Step 2: choose the cause from five bugs. Each is real. On this bench each one gives the same log, line for line.

The two flows are written in two different ways, so neither is a pattern for the other.

Check each cause against the waveform. Some can only be told apart at certain cycles.

Step 3 is optional: fix the design and run the tests.

A 257-bit block is kept here as its four 66-bit blocks. The bench stands in for the decoders.

A decoder keeps its place through the restart. Only the wires and the collector start again: a choice of this bench.

Signals in the Timeline

SignalWidthRole
test_number8input of each check
Full specification
The API (keep it)
function automatic void tx_reset(input int gap);
function automatic void tx_block(input [65:0] blk);            // one 66-bit block from the encoder
function automatic void rx_reset();
function automatic void rx_marker(input int wire_no, input [119:0] m);   // the marker found on one of the 32 wires
function automatic void rx_units(input bit v0, input bit am0, input [263:0] u0, input bit v1, input bit am1, input [263:0] u1);
ItemRule
Block distributionthe first 66-bit block after tx_reset goes to flow 0, the next to flow 1, and so on in turn (172.2.4.3). tx_flow is the flow of the block of this call
257-bit blockfour 66-bit blocks of one flow in a row. tx_v is 1 in the call that brings the fourth. tx_unit then holds the four, the first in bits 65:0
Marker groupeach flow sends a group of its own, eight 257-bit blocks long. tx_am = 1 with tx_v means: the group of tx_flow goes out just before this 257-bit block
Where the groups goat the same point of the stream in both flows (172.2.4.6). Say the group of flow 0 is before the 257-bit block that holds 66-bit block k. Then the group of flow 1 is before the one that holds k + 1
How oftena course choice of interface: before the first 257-bit block of each flow after tx_reset, and then after every gap blocks of that flow. The standard's number is 163,832 (119.2.4.4.2)
rx_markerrx_l is the PCS lane of the marker, from am.lane_of, or −1. Calls may come in any order, and a wire may be read again
A wire read againthe new reading replaces the old one. A wire that now shows no lane has dropped its old lane
Lanes to flowsPCS lanes 0 to 15 belong to flow 0 and lanes 16 to 31 to flow 1 (172.2.5.2). rx_f is the flow. rx_s is the place among the 16 lanes of that flow. Both are −1 with no lane
rx_readya course choice: 1 when, as the wires stand now, every one of the 32 PCS lanes is found on a wire
rx_unitsv0 = 1: decoder 0 hands over a 257-bit block, u0. am0 = 1: it is the first after a marker group. The same for decoder 1
Block collectionthe first 66-bit block after the group of flow 0 is followed by the first after the group of flow 1 (172.2.5.8). Then they alternate
Where it startsa course choice: after rx_reset, a flow's blocks are dropped until one arrives with its am set. That one is kept. Later groups change nothing
rx_go0, rx_go1a course choice: 1 while the flow is being collected, from its first kept block until rx_reset
rx_w0, rx_w1a course choice: how many 257-bit blocks of the flow wait for their partner at the end of the call
rx_v, rx_outas soon as both flows have a block waiting, the oldest of each leave together. rx_out holds their eight 66-bit blocks in stream order, the first in bits 65:0
Waitinga course choice of the tests: they never leave more than four 257-bit blocks of one flow waiting for the other
Left outtranscoding, scrambling, the content of the marker group, FEC and the lanes' symbols. Deskew has lined the lanes up

The fixture runs each end beside a private model on random streams, wire orders, pipeline delays and restarts.

Some streams are over 1,200 blocks long, and one is over 5,000. Scripted runs check both ends together.

Keep the work of one call small: the tests make about 20,000 calls.

Diagnose the Failing Run

The test below runs the buggy design. Read its log, then find where it first goes wrong in the waveform.

Log

Waveform

Click a signal at a cycle to select it. Or focus the waveform and use the arrow keys: Left and Right move one cycle, Up and Down one signal.

Step 1 · The first bad cycle and signal
Step 2 · The cause

Step 3 is optional: fix the code in the editor, then run the tests.

Examples

Example 1

Sixty-Four Blocks, a Group Every 3
Input
`tx_reset(3)`, then 66-bit blocks 0 to 63
Output
marker_groups_of_flow_0 = 3, marker_groups_of_flow_1 = 3, last_group_of_flow_0_is_before_block = 48, last_group_of_flow_1_is_before_block = 49, blocks_of_257_bits_sent = 16, wrong_calls = 0

Each flow sends eight 257-bit blocks and three marker groups. The last group of flow 0 is before the block that starts with 66-bit block 48. That of flow 1 is before block 49.

Example 2

Thirty-Two Wires, Lanes in a Mixed Order
Input
wire w carries PCS lane (7w + 3) mod 32
Output
flow_of_wire_0 = 0, place_of_wire_0 = 3, flow_of_wire_31 = 1, place_of_wire_31 = 12, wires_sent_to_flow_0 = 16, wires_sent_to_flow_1 = 16, ready_after_32_markers = 1, ready_after_16_markers = 0, sum_over_all_wires = 0xa660, ready_at_the_end = 1

Wire 0 carries lane 3: flow 0, place 3. Wire 31 carries lane 28: flow 1, place 12. Ready rises only with the last marker.

Example 3

Both Ends and a Link
Input
a group every 4, a start at 257-bit block 2, 160 blocks
Output
ready = 1, decoder_0_decodes_flow = 0, decoder_1_decodes_flow = 1, blocks_out = 112, first_block_out = 32, blocks_out_of_order = 0

The first group at or after block 2 is before block 4. So the first 66-bit block out is 32, and 112 blocks leave in order.

Constraints and Contract

  • Keep the five functions with their arguments.
  • Keep the names tx_flow, tx_v, tx_am, tx_unit, rx_l, rx_f, rx_s, rx_ready, rx_go0, rx_go1, rx_w0, rx_w1, rx_v and rx_out: the fixture reads them.
  • tx_reset and rx_reset must clear all state of their end.
  • After a reset only rx_ready is checked before the next call. It must be 0 after rx_reset.
  • You may call am.lane_of, am.am and the other functions of am.
  • The contract is that of P11.34.1, with four more names. A design that passes there fits here once it has them.
  • Off limits: the names ref_p, rng, tb, root and the modules of the bench.
  • The fixture's own names are off limits too. They begin with fx_.

Hints

Hint 1

The log is clean up to the restart. Find cycle 560 in the waveform. What is the collector handed after it?

Hint 2

For each flow, note the first block that arrives with its am flag set. in_b0 and in_b1 show the block numbers.

Hint 3

Decide which end is at fault. Then follow one signal of that end back through the whole run, not only its last part.

Hint 4

Each flow owes a marker group before every fifth of its 257-bit blocks. List every cycle in which tx_am is 1 for flow 1.

What the grader checks: The failing run is simulated live with the faulty design. The first bad cycle and each option's symptom come from runs of each version on the same bench. The fix is checked on 18 scripted runs, with expected values from a JavaScript model. It is also checked on 45 random runs, seeded for each submission, in four mixed parts. Forty are of four kinds, about ten of each. The kinds are streams through the near end, wire orders, links, and links with a collector reset. They compare every output with a private model after every call. Four more are streams of over 1,200 blocks. One is a stream of over 5,000 blocks with a gap of 130 to 137, checked against the rule. All other gaps are from 1 to 11. A fault that first shows after more than 5,000 blocks is not reached. The course choices are those of P11.34.1, and the status outputs rx_go0, rx_go1, rx_w0 and rx_w1. Transcoding, scrambling, the marker group's content, FEC, symbol distribution, deskew and marker lock are not graded here.

22 checks on Icarus Verilog · server. Run the examples first, then check the full regression. Full checks use the server judge. Downloads contain your source only; samples require the simulation service. A passing result establishes the tested cases, not a proof for all possible inputs.

See all 2 steps and the final capstone

Report a problem with this page

Your code SystemVerilogSaved locally
Ln 1, Col 1Tab indents · Ctrl/⌘ + Enter runs samples
Not run yet

Sixty-Four Blocks, a Group Every 3

Input
`tx_reset(3)`, then 66-bit blocks 0 to 63
Expected output
marker_groups_of_flow_0 = 3, marker_groups_of_flow_1 = 3, last_group_of_flow_0_is_before_block = 48, last_group_of_flow_1_is_before_block = 49, blocks_of_257_bits_sent = 16, wrong_calls = 0

Each flow sends eight 257-bit blocks and three marker groups. The last group of flow 0 is before the block that starts with 66-bit block 48. That of flow 1 is before block 49.

Run samples executes all 3 public checks. Check all tests runs 22 checks on the server.

Icarus Verilog · server