Externally verifying Linux’s real-time deadline scheduling capabilities

<p>A number of industrial applications now demand hard real-time scheduling capabilities from the kernel of a Linux-based operating system, but scheduling measurements from the system itself cannot be completely trusted as they are referenced to the same clock as the kernel-under-test. Yet, if the system can output signals to hardware as it runs, their timing can be analysed by an external microcontroller, and a second "external" measurement obtained to compare with the system's own report.</p> <p>Codethink wrote embedded Rust firmware to run on a Raspberry Pi Pico which analyses the timings of characters received on a UART serial port. On the other end of the serial port is our "Rusty Worker": a Rust program on a Linux system which uses the <code>sched_setattr</code> syscall to request that Linux schedules it with specified parameters, and then measures its own scheduling period and runtime. The single-threaded and interrupt-based architecture of this firmware allowed accurate external measurements of the Rusty Worker’s scheduling parameters at microsecond precision, and meant it was easy to extend to monitor the "petting" behaviour of a watchdog.</p> <p>Rust was a natural choice for the verification firmware and Rusty Worker. Higher level or interpreted languages would increase non-determinism and reduce our confidence in the accuracy of the collected timing data, whereas C or C++ programs risk undefined behaviour unacceptable in a safety-related context. Yet, Rust really came into its own with the relative simplicity of the cargo toolchain for embedded targets, so reproducibly building the firmware with a self-built toolchain (and without access to the internet) was just as straightforward as building the Rusty Worker for a Linux target.</p> <p>Having equipped our suite of bare-metal CI runners with KiCad-designed custom PCBs that feature a Raspberry Pi Pico and Debug Probe, we are able to run “soak” tests to collect thousands of self- and externally-measured deadline scheduling parameters for each iteration of our <a href="https://www.codethink.co.uk/ctrl-os.html">Codethink Trustable Reproducible Linux (CTRL OS)</a>, as soon as engineers push a new commit. We then use the open source <a href="https://gitlab.com/CodethinkLabs/trustable/trustable">Eclipse Trustable Software Framework (TSF)</a> to facilitate automatic “real time” aggregation and statistical analysis of the external scheduling measurements for each commit, clearly communicating the results and trends to senior stakeholders and junior engineers alike.</p> <p>TSF challenges us both to robustly, systematically, and continuously evidence our claims about Linux’s capabilities as a real-time operating system, and to scrutinise the software used for testing as strictly as the software under test. We are excited to share how we work towards these goals with external scheduling measurements and embedded Rust.</p>