Exceptions, batches and recovery

Turn a Python exception into a status the testbench must read, keep one call per packet, and prove that the next good packet still works after a failure.

Lesson: D2.2
Time: 25 min
Runs on: Verilator with C/C++
Check: 5 questions

A regression finished green, and the checksum model had been returning zero since Tuesday. Someone had widened the packet buffer to 65 bytes, and the model's size check raised a ValueError on every call. The bridge caught it and returned status 4, but the testbench never looked at the status: it compared total with the hardware's zero and called it a match. This lesson turns a Python failure into a result the testbench cannot ignore, and shows that the next good packet still works.

Call	Status	total	Hardware sum	What the testbench said
64-byte packet, before the change	0	197	197	match
65-byte packet, after the change	4	0	0	match
65-byte packet, with the status read	4	0	0	model failed: packet exceeds 64 bytes

The same zero, three readings. Only the third one is true.

Two ways out of one call

When checksum raises, PyObject_CallFunctionObjArgs returns null and leaves the exception pending inside the interpreter. The bridge takes it with PyErr_GetRaisedException, which also clears the pending state, prints its type and message on one line, releases it, and returns status 4. total was set to zero at the top of the call and stays zero. An exception left pending would surface in the next call as a failure that belongs to someone else.

A normal return and a raised oneArrows are calls and returns. The upper path returns a number; the lower path returns an exception, which the bridge turns into a status before SystemVerilog sees anything.

A legal packet
Three bytes go in. The model returns the integer 4; the bridge converts it, checks it fits a byte, and writes total.
Follow the calls, values and waits
1. call checksum(bytes): main.sv → bridge.cpp
2. call checksum(bytes): bridge.cpp → model.py
3. return status 0, total 4: model.py → bridge.cpp
4. return status 0, total 4: bridge.cpp → main.sv
A packet of 65 bytes
The model raises. The C call returns null with the exception pending; the bridge takes it, prints one line, and returns status 4 with total still zero.
Follow the calls, values and waits
1. call checksum(bytes): bridge.cpp → model.py
2. data python ValueError: ...: model.py → ValueError
3. return status 4, total 0: bridge.cpp → main.sv
The testbench reads the status first
Status 0 is the only value under which total means anything. Every other status says the model gave no answer.

Read the diagram as text

main.sv: status = py_sum(packet, n, total)
bridge.cpp: total = 0 first; status last
model.py: checksum(data)
integer 4: 255 + 2 + 3, low byte
ValueError: packet exceeds 64 bytes

main.sv → bridge.cpp: DPI call
bridge.cpp → model.py: checksum(bytes)
model.py → integer 4: return
model.py → ValueError: raise

Both paths end in SystemVerilog with a status and a total. Only one of them carries a checksum.

A status for every way to fail

The bridge returns one small integer, and each value names one cause. The testbench compares the status with zero before it trusts total; a report can print the number, or the one-line message the bridge already printed.

Status	Cause	Where it is decided
0	The model returned a value from 0 to 255	C, after the conversion and the range check
1	The bridge is closed: no kept function	C, before anything touches Python
2	A bad array: wrong shape, negative count, count past the array, or over 1024 bytes	C, before the copy
3	The `bytes` object could not be created	C, from the interpreter's allocator
4	The model raised: here `ValueError` for a packet over 64 bytes	Python
5	The result is no unsigned integer: a negative number, a string	C, converting the result
6	The result is above 255	C, after the conversion

Seven answers, one of them good. Statuses 1 to 3 never reach Python; 4 comes from the model; 5 and 6 reject what it returned.

Two limits look alike and do different jobs. The C transport refuses more than 1024 bytes, which keeps the sum of the copied bytes inside an unsigned 32-bit count. The model refuses more than 64 bytes, because that is the longest packet the protocol allows. Change the protocol and only model.py changes.

Empty, full and too full

An empty packet is legal: count is zero, the loop copies nothing, and the model returns the checksum of no bytes, which is 0. Status 0 with total 0 is a real answer here, and the testbench must be able to tell it from status 4 with total 0.

count says how many bytes of the array are valid, and it may be smaller than the array. It may never be larger: a count past the array is refused with status 2 before any element is read, so no stale memory reaches the model. Walking from the array's left bound works for ascending and descending ranges alike, and for a sum the order does not matter.

Send one packet per call. A call per byte would pay the crossing, the bytes allocation and a Python call for every byte, and the model could no longer see a packet as a whole. Transaction boundaries, not bits, are the right places to cross.

Prove the recovery

The example sends the oversized packet, reads status 4, and then sends the same three legal bytes as before. The second answer must again be 4. If it were anything else, the error had left something behind: a pending exception, a released function, a half-built object. That one extra call is the cheapest regression test a bridge can have.

The same idea covers the bridge's own refusals. Calling py_sum before py_open gives status 1, and opening afterwards still works. A status is a report, never a change of state.

Fail on purpose
A packet of 65 bytes makes the model raise; the bridge reports it and returns.
length 65
status / total 4 / 0
Recover
The next legal packet gets its real checksum, with nothing inherited from the error.
bytes 255, 2, 3
status / total 0 / 4

length	`65`
status / total	`4 / 0`

bytes	`255, 2, 3`
status / total	`0 / 4`

Try it

Six calls, six status lines. The one-line python ValueError report comes from the bridge, printed before py_sum returns status 4; the testbench prints the status line after it.

Change the model so that checksum returns -1, and run again: the conversion fails, the bridge prints an OverflowError line and returns status 5 with total zero. Return sum(data) without the mask and send {200, 100}: status 6.

RUNVerilator + C/C++ over DPI · simulator service

Edit the code, then run it

Expected: closed bridge: status=1 total=0 255+2+3: status=0 total=4 empty: status=0 total=0 python ValueError: packet exceeds 64 bytes 65 bytes: status=4 total=0 after the error: status=0 total=4

Read included file: bridge.cpp

#include <Python.h>      // first: it sets the feature macros the C library headers read
#include "Vtb__Dpi.h"
#include "svdpi.h"
#include <cstdio>

// One interpreter, opened and closed from SystemVerilog, on the simulator's thread. Between calls
// the bridge holds exactly one Python object: the checksum function.
static PyObject* checksum = nullptr;

// Take the pending Python exception, print it as one line, and return the status code.
static int py_error(int code) {
  PyObject* exc = PyErr_GetRaisedException();   // clears the error indicator; the object is ours
  if (exc) {
    PyObject* text = PyObject_Str(exc);         // new reference
    std::printf("python %s: %s\n", Py_TYPE(exc)->tp_name, text ? PyUnicode_AsUTF8(text) : "?");
    Py_XDECREF(text);
    Py_DECREF(exc);
  }
  return code;
}

extern "C" int py_open() {
  if (checksum) return 0;              // already open: nothing to do
  if (Py_IsInitialized()) return 1;    // some other code runs this interpreter; leave it alone
  PyConfig config;
  PyConfig_InitIsolatedConfig(&config);   // ignore environment variables and the user site directory
  config.install_signal_handlers = 0;     // Ctrl-C stays with the simulator
  PyStatus status = Py_InitializeFromConfig(&config);
  PyConfig_Clear(&config);
  if (PyStatus_Exception(status)) return 2;
  PyObject* sys_path = PySys_GetObject("path");   // borrowed reference: never Py_DECREF
  PyObject* here = PyUnicode_FromString(".");     // new reference: ours to release
  if (!here || PyList_Insert(sys_path, 0, here) < 0) { Py_XDECREF(here); py_error(3); Py_FinalizeEx(); return 3; }
  Py_DECREF(here);
  PyObject* module = PyImport_ImportModule("model");   // new reference
  if (module) { checksum = PyObject_GetAttrString(module, "checksum"); Py_DECREF(module); }   // new reference, kept
  if (!checksum || !PyCallable_Check(checksum)) {
    Py_XDECREF(checksum); checksum = nullptr; py_error(4); Py_FinalizeEx(); return 4;
  }
  return 0;
}

extern "C" int py_sum(const svOpenArrayHandle data, int count, unsigned int* total) {
  *total = 0;                                   // the output is set on every path
  if (!checksum) return 1;                      // closed bridge
  if (svDimensions(data) != 1 || count < 0 || count > svSize(data, 1) || count > 1024) return 2;
  const int left = svLeft(data, 1), step = left <= svRight(data, 1) ? 1 : -1;
  PyObject* bytes = PyBytes_FromStringAndSize(nullptr, count);   // new reference
  if (!bytes) return py_error(3);
  char* target = PyBytes_AsString(bytes);
  for (int k = 0; k < count; ++k) {
    const unsigned char* p = static_cast<const unsigned char*>(svGetArrElemPtr1(data, left + k * step));
    if (!p) { Py_DECREF(bytes); return 2; }
    target[k] = static_cast<char>(*p);          // a copy: Python never keeps a pointer into the simulator
  }
  PyObject* result = PyObject_CallFunctionObjArgs(checksum, bytes, nullptr);   // new reference, or null
  Py_DECREF(bytes);
  if (!result) return py_error(4);              // the model raised
  const unsigned long value = PyLong_AsUnsignedLong(result);
  Py_DECREF(result);
  if (PyErr_Occurred()) return py_error(5);     // negative, or no integer at all
  if (value > 255) return 6;                    // a checksum is one byte
  *total = static_cast<unsigned int>(value);
  return 0;
}

extern "C" int py_close() {
  Py_XDECREF(checksum);
  checksum = nullptr;
  return Py_IsInitialized() ? Py_FinalizeEx() : 0;
}

Read included file: model.py

"""The reference calculation only. SystemVerilog keeps the clock, the stimulus and the check."""


def checksum(data):
    if not isinstance(data, bytes):
        raise TypeError("expected bytes")
    if len(data) > 64:
        raise ValueError("packet exceeds 64 bytes")
    return sum(data) & 0xff

Check yourself

Predict the outputThe packet is {128, 128}. What line does this program print?

`timescale 1ns/1ps
module tb;
  import "DPI-C" function int py_open();
  import "DPI-C" function int py_sum(input byte unsigned data[], input int count, output int unsigned total);
  import "DPI-C" function int py_close();
  byte unsigned data[2] = '{128, 128};
  int status;
  int unsigned total;
  initial begin
    void'(py_open());
    status = py_sum(data, 2, total);
    void'(py_close());
    $display("status=%0d total=%0d", status, total);
    $finish;
  end
endmodule

Read included file: bridge.cpp

#include <Python.h>      // first: it sets the feature macros the C library headers read
#include "Vtb__Dpi.h"
#include "svdpi.h"
#include <cstdio>

// One interpreter, opened and closed from SystemVerilog, on the simulator's thread. Between calls
// the bridge holds exactly one Python object: the checksum function.
static PyObject* checksum = nullptr;

// Take the pending Python exception, print it as one line, and return the status code.
static int py_error(int code) {
  PyObject* exc = PyErr_GetRaisedException();   // clears the error indicator; the object is ours
  if (exc) {
    PyObject* text = PyObject_Str(exc);         // new reference
    std::printf("python %s: %s\n", Py_TYPE(exc)->tp_name, text ? PyUnicode_AsUTF8(text) : "?");
    Py_XDECREF(text);
    Py_DECREF(exc);
  }
  return code;
}

extern "C" int py_open() {
  if (checksum) return 0;              // already open: nothing to do
  if (Py_IsInitialized()) return 1;    // some other code runs this interpreter; leave it alone
  PyConfig config;
  PyConfig_InitIsolatedConfig(&config);   // ignore environment variables and the user site directory
  config.install_signal_handlers = 0;     // Ctrl-C stays with the simulator
  PyStatus status = Py_InitializeFromConfig(&config);
  PyConfig_Clear(&config);
  if (PyStatus_Exception(status)) return 2;
  PyObject* sys_path = PySys_GetObject("path");   // borrowed reference: never Py_DECREF
  PyObject* here = PyUnicode_FromString(".");     // new reference: ours to release
  if (!here || PyList_Insert(sys_path, 0, here) < 0) { Py_XDECREF(here); py_error(3); Py_FinalizeEx(); return 3; }
  Py_DECREF(here);
  PyObject* module = PyImport_ImportModule("model");   // new reference
  if (module) { checksum = PyObject_GetAttrString(module, "checksum"); Py_DECREF(module); }   // new reference, kept
  if (!checksum || !PyCallable_Check(checksum)) {
    Py_XDECREF(checksum); checksum = nullptr; py_error(4); Py_FinalizeEx(); return 4;
  }
  return 0;
}

extern "C" int py_sum(const svOpenArrayHandle data, int count, unsigned int* total) {
  *total = 0;                                   // the output is set on every path
  if (!checksum) return 1;                      // closed bridge
  if (svDimensions(data) != 1 || count < 0 || count > svSize(data, 1) || count > 1024) return 2;
  const int left = svLeft(data, 1), step = left <= svRight(data, 1) ? 1 : -1;
  PyObject* bytes = PyBytes_FromStringAndSize(nullptr, count);   // new reference
  if (!bytes) return py_error(3);
  char* target = PyBytes_AsString(bytes);
  for (int k = 0; k < count; ++k) {
    const unsigned char* p = static_cast<const unsigned char*>(svGetArrElemPtr1(data, left + k * step));
    if (!p) { Py_DECREF(bytes); return 2; }
    target[k] = static_cast<char>(*p);          // a copy: Python never keeps a pointer into the simulator
  }
  PyObject* result = PyObject_CallFunctionObjArgs(checksum, bytes, nullptr);   // new reference, or null
  Py_DECREF(bytes);
  if (!result) return py_error(4);              // the model raised
  const unsigned long value = PyLong_AsUnsignedLong(result);
  Py_DECREF(result);
  if (PyErr_Occurred()) return py_error(5);     // negative, or no integer at all
  if (value > 255) return 6;                    // a checksum is one byte
  *total = static_cast<unsigned int>(value);
  return 0;
}

extern "C" int py_close() {
  Py_XDECREF(checksum);
  checksum = nullptr;
  return Py_IsInitialized() ? Py_FinalizeEx() : 0;
}

Read included file: model.py

"""The reference calculation only. SystemVerilog keeps the clock, the stimulus and the check."""


def checksum(data):
    if not isinstance(data, bytes):
        raise TypeError("expected bytes")
    if len(data) > 64:
        raise ValueError("packet exceeds 64 bytes")
    return sum(data) & 0xff

Spot the bugThe bridge is open. Each option prepares a packet and calls py_sum; the check expects status 0 and total 4. Which option prints FAIL?

byte unsigned p[3] = '{255, 2, 3};
      status = py_sum(p, 3, total);

byte unsigned p[5] = '{255, 2, 3, 9, 9};
      status = py_sum(p, 3, total);

byte unsigned p[3] = '{255, 2, 3};
      status = py_sum(p, 4, total);

byte unsigned p[3:1] = '{255, 2, 3};
      status = py_sum(p, 3, total);

Read included file: bridge.cpp

#include <Python.h>      // first: it sets the feature macros the C library headers read
#include "Vtb__Dpi.h"
#include "svdpi.h"
#include <cstdio>

// One interpreter, opened and closed from SystemVerilog, on the simulator's thread. Between calls
// the bridge holds exactly one Python object: the checksum function.
static PyObject* checksum = nullptr;

// Take the pending Python exception, print it as one line, and return the status code.
static int py_error(int code) {
  PyObject* exc = PyErr_GetRaisedException();   // clears the error indicator; the object is ours
  if (exc) {
    PyObject* text = PyObject_Str(exc);         // new reference
    std::printf("python %s: %s\n", Py_TYPE(exc)->tp_name, text ? PyUnicode_AsUTF8(text) : "?");
    Py_XDECREF(text);
    Py_DECREF(exc);
  }
  return code;
}

extern "C" int py_open() {
  if (checksum) return 0;              // already open: nothing to do
  if (Py_IsInitialized()) return 1;    // some other code runs this interpreter; leave it alone
  PyConfig config;
  PyConfig_InitIsolatedConfig(&config);   // ignore environment variables and the user site directory
  config.install_signal_handlers = 0;     // Ctrl-C stays with the simulator
  PyStatus status = Py_InitializeFromConfig(&config);
  PyConfig_Clear(&config);
  if (PyStatus_Exception(status)) return 2;
  PyObject* sys_path = PySys_GetObject("path");   // borrowed reference: never Py_DECREF
  PyObject* here = PyUnicode_FromString(".");     // new reference: ours to release
  if (!here || PyList_Insert(sys_path, 0, here) < 0) { Py_XDECREF(here); py_error(3); Py_FinalizeEx(); return 3; }
  Py_DECREF(here);
  PyObject* module = PyImport_ImportModule("model");   // new reference
  if (module) { checksum = PyObject_GetAttrString(module, "checksum"); Py_DECREF(module); }   // new reference, kept
  if (!checksum || !PyCallable_Check(checksum)) {
    Py_XDECREF(checksum); checksum = nullptr; py_error(4); Py_FinalizeEx(); return 4;
  }
  return 0;
}

extern "C" int py_sum(const svOpenArrayHandle data, int count, unsigned int* total) {
  *total = 0;                                   // the output is set on every path
  if (!checksum) return 1;                      // closed bridge
  if (svDimensions(data) != 1 || count < 0 || count > svSize(data, 1) || count > 1024) return 2;
  const int left = svLeft(data, 1), step = left <= svRight(data, 1) ? 1 : -1;
  PyObject* bytes = PyBytes_FromStringAndSize(nullptr, count);   // new reference
  if (!bytes) return py_error(3);
  char* target = PyBytes_AsString(bytes);
  for (int k = 0; k < count; ++k) {
    const unsigned char* p = static_cast<const unsigned char*>(svGetArrElemPtr1(data, left + k * step));
    if (!p) { Py_DECREF(bytes); return 2; }
    target[k] = static_cast<char>(*p);          // a copy: Python never keeps a pointer into the simulator
  }
  PyObject* result = PyObject_CallFunctionObjArgs(checksum, bytes, nullptr);   // new reference, or null
  Py_DECREF(bytes);
  if (!result) return py_error(4);              // the model raised
  const unsigned long value = PyLong_AsUnsignedLong(result);
  Py_DECREF(result);
  if (PyErr_Occurred()) return py_error(5);     // negative, or no integer at all
  if (value > 255) return 6;                    // a checksum is one byte
  *total = static_cast<unsigned int>(value);
  return 0;
}

extern "C" int py_close() {
  Py_XDECREF(checksum);
  checksum = nullptr;
  return Py_IsInitialized() ? Py_FinalizeEx() : 0;
}

Read included file: model.py

"""The reference calculation only. SystemVerilog keeps the clock, the stimulus and the check."""


def checksum(data):
    if not isinstance(data, bytes):
        raise TypeError("expected bytes")
    if len(data) > 64:
        raise ValueError("packet exceeds 64 bytes")
    return sum(data) & 0xff

ConceptWhy must the testbench compare status with 0 before it uses total?

Python exceptions stop the simulation unless the status is read in the same statementThe status carries the high bits of the checksum, which total cannot holdThe bridge writes total only on success, so after a failure the variable still holds the previous checksumEvery failed call leaves total at 0, and 0 is also a legal checksum, so the value alone says nothing

ConceptWhy does the bridge send a whole packet in one call rather than one byte per call?

One call per packet pays the crossing once, and the model sees the whole packet it was written forThe 64-byte limit in the model counts calls rather than bytes, so per-byte calls reach it soonerThe DPI open-array type cannot carry a single byte, so a per-byte call would fail to buildPython would run out of references after a few hundred single-byte calls and raise MemoryError

ConceptSuppose the bridge returned 4 without taking the exception. What would the next legal call see?

The failed packet again, because the interpreter retries pending work before anything newThe stale error can make the next call fail or report the old exception as if it were newA clean interpreter, because every DPI call starts from a fresh Python error stateStatus 1, because Python finalises itself after an exception nobody handled

Remember

A model exception becomes a status: the bridge takes the exception, prints one line, clears the pending state and returns 4 with total at zero.
Read the status before the total. Zero is a legal checksum and also the value every failed call leaves behind.
Each status names one cause: closed bridge, bad array or count, allocation, model exception, bad result type, result out of range.
A count may be smaller than the array, never larger; empty packets are legal; send one packet per call.
After a failure, send a known good packet and check its answer. Recovery is part of the test.