How problems are graded
Every problem is judged by running your code on a real simulator against hidden tests. This page explains what you submit in each format, how it is judged, and what you see.
Running a problem
Each problem names its simulator. Icarus Verilog runs inside your browser. Verilator problems are compiled and run on the simulator server, and only the results come back.
Run samples runs the few tests shown in the examples. Submit runs the full hidden suite. A submission is accepted when every test passes, and, for the few problems that check how the code is written, every compile check passes too.
After a run, a failing test shows a field-by-field table of expected and actual values, with the wrong fields highlighted. With Capture waves on, you also get a waveform of every test with the mismatch marked, and you can download it.
Every published problem has three hints, from a nudge to a near-answer, and its reference solution can be loaded into the editor. Undo restores your own draft.
The five formats
The format is shown on every problem, and the starter code always matches it.
| Format | You submit | How it is judged | What you see |
|---|---|---|---|
| Implement | Code for an empty skeleton | Every hidden test; each output must match exactly | Each test, a field-by-field diff and a waveform |
| Fix | Working code with one realistic defect, repaired | Hidden tests, as for Implement | As for Implement |
| Verify | A checker for a given design | Runs on the correct design and every faulty variant, with one stimulus | The fault matrix |
| Break it | Test inputs for a correct design, within stated rules | Your inputs run on the design and on hidden buggy copies | The kill matrix |
| Debug | The first wrong cycle and signal, then the cause | Against the first divergence from the correct design | Whether to look earlier or later, and what a wrong cause would have looked like |
Verify: the fault matrix
A Verify problem grades your checker on several designs. On the correct design it must stay silent; a report there is a false alarm. On each faulty design it must report at least one error, or that design is missed.
| Design | Your checker | Result |
|---|---|---|
| Correct FIFO | no error reported | Clean |
| Read data bit 0 flipped | error reported | Caught |
read_valid stuck high | error reported | Caught |
| Read response one edge late | no error reported | Missed |
Part of a fault matrix for a FIFO checker. This submission is not accepted: one fault was missed.
A submission is accepted only when the first row is clean and every other row is caught. A checker that reports errors on everything catches every fault and is still rejected, because the first row shows a false alarm.
Break it: the kill matrix
In a Break it problem you write the test inputs for a correct design, usually with a limit on how many. The first row checks that your inputs follow the rules; each other row is one hidden bug.
| Row | Result |
|---|---|
| Your inputs | Legal: 4 inputs, all fully known |
| Hidden bug 1 | Caught |
| Hidden bug 2 | Caught |
| Hidden bug 3 | Missed |
The layout of a kill matrix. Each bug is named, with its symptom, only after your run.
A copy is caught when its output differs from the correct design on at least one of your inputs.
Difficulty and depth
Difficulty says how hard a problem is to solve: Easy, Medium, Hard or Expert. Depth says what you must already know: Foundations, Core or Advanced, with links to the lessons it builds on. A problem can be Easy but Advanced.
The starting difficulty comes from a rubric. Each of five dimensions is scored from 1 to 5, and the total sets the band.
| Dimension | Scores 1 | Scores 5 |
|---|---|---|
| Concepts combined | one idea | many ideas across topics |
| Code to write | up to 5 lines | more than 80 lines, or several files |
| Edge cases to defeat | one obvious case | many subtle fault classes |
| Debugging needed | none | reasoning over a long waveform or a race |
| Spec to absorb | one paragraph | a protocol section |
Totals of 5 to 9 are Easy, 10 to 14 Medium, 15 to 19 Hard and 20 to 25 Expert.
Playtests then move the band: down if more than 80% of testers pass on the first try, up if fewer than 20% solve it within twice the median time. Playtests have not started, so today every label comes from the rubric alone, and no time estimates are shown.
What "qualified" means
A problem is published only after a script runs it on its own simulator and every condition below holds.
- The reference solution passes every hidden test, and the starter code fails.
- Every named mutant, a realistic wrong answer, fails at least one test.
- Verify: at least three wrong checkers, including one false alarm and one that misses a fault, each failing for exactly the reason it declares.
- Break it: the reference stimulus is legal and catches every bug, and at least two legal but weak stimulus sets each miss the bugs they declare.
- Debug: the answer and every option's symptom are recomputed from real waveforms, and no two options share one.
- The statement, examples, hints and labels meet the written style guide.
Honest limits
- One simulator per problem. Icarus is four-state and runs in your browser; Verilator is two-state and handles classes, assertions and randomization. The problem names which one judges it, and Simulators and known differences lists where each departs from the standard.
- Hidden tests ship with the page today. The full suite is downloaded when you submit, so a determined reader can see it. The reference solution is in the page too. Both move to the server when the hosted judge launches.
- Verilator problems need the simulator server. Until the hosted judge launches, they run only on a local copy of the site.
- Passing proves the tests passed. If wrong code is accepted, please report it.