HPC, Big Data & Data Science
Zero‑Touch HPC Nodes: NetBox, Tofu and Packer for a Self‑Configuring SLURM Cluster
February 1, 2026
12:30 PM – 12:55 PM Add to calendar
12:30 PM – 12:55 PM Add to calendar
H.1308 (Rolin)
<p>Over the last five years, we ran an HPC system for life sciences on top of OpenStack, with a deployment pipeline built from Ansible, manual steps (see <a href="https://archive.fosdem.org/2020/schedule/event/hpc_openstack/">FOSDEM 2020 talk</a>). It worked—but it wasn’t something we could easily rebuild from scratch or apply consistently to other parts of our infrastructure.</p>
<p>As we designed our new HPC system (coming online in early 2026), we set ourselves a goal: treat the cluster as something we can declare and then recreate, not pet and nurture. The result is a “zero‑touch” style pipeline where a new node can go from “just racked” to “in SLURM and running jobs” with no manual intervention.</p>
<p>In this talk, we walk through the end‑to‑end workflow:</p>
<ul>
<li>NetBox as DCIM and source of truth: racking a server and adding it to NetBox is the trigger; MACs, serials and IPs are automatically imported from vendor tools and IPAM/DNS into our automation.</li>
<li>Using Tofu/Terragrunt (instead of Openstack's Heat orchestration service) to provision OpenStack/Ironic, SLURM infrastructure and network fabric across three environments (dev plus two interchangeable prod clusters for blue/green rollouts).</li>
<li>Image‑based deployment with Packer and Ansible: we split roles into “install” and “configure”. Packages and heavy setup are baked into images, while an ansible-init service runs locally on first boot to apply configuration and join the cluster.</li>
<li>Making nodes self‑sufficient, including fetching the secrets they need via short‑lived credentials and a minimal external dependency chain.</li>
</ul>
<p>Come and see how we built a reproducible HPC/Big-Data cluster on open‑source tooling, reusing as much of the stack as possible for the rest of our infrastructure.</p>
Additional information
| Live Stream | https://live.fosdem.org/watch/h1308 |
|---|---|
| Type | devroom |
| Language | English |
More sessions
| 2/1/26 |
<p>Scientific models are today limited by compute resources, forcing approximations driven by feasibility rather than theory. They consequently miss important physical processes and decision-relevant regional details. Advances in AI-driven supercomputing — specialized tensor accelerators, AI compiler stacks, and novel distributed systems — offer unprecedented computational power. Yet, scientific applications such as ocean models, often written in Fortran, C++, or Julia and built for ...
|
| 2/1/26 |
<p>ROCm™ has been AMD’s software foundation for both high-performance computing (HPC) and AI workloads and continues to support the distinct needs of each domain. As these domains increasingly converge, ROCm™ is evolving into a more modular and flexible platform. Soon, the distribution model shifts to a core SDK with domain-specific add-ons—such as HPC—allowing users to select only the components they need. This reduces unnecessary overhead while maintaining a cohesive and ...
|
| 2/1/26 |
<p>Wherever research software is developed and used, it needs to be installed, tested in various ways, benchmarked, and set up within complex workflows. Typically, in order to perform such tasks, either individual solutions are implemented - imposing significant restrictions due to the lack of portability - or the necessary steps are performed manually by developers or users, a time-consuming process, highly susceptible to errors. Furthermore, particularly in the field of high-performance ...
|
| 2/1/26 |
<h2>Content</h2> <p>High-frequency wave simulations in 3D (with e.g. Finite Elements) involve systems with hundreds of millions unknowns (up to 600M in our runs), prompting the use of massively parallel algorithms. In the harmonic regime, we favor Domain Decomposition Methods (DDMs) where local problems are solved in smaller regions (subdomains) and the full solution of the PDE is recovered iteratively. This requires each rank to own a portion of the mesh and to have a view on neighboring ...
|
| 2/1/26 |
<p>As the computing needs of the world have grown, the need for parallel systems has grown to match. However, the programming languages used to target those systems have not had the same growth. General parallel programming targeting distributed CPUs and GPUs is frequently locked behind low-level and unfriendly programming languages and frameworks. Programmers must choose between parallel performance with low-level programming or productivity with high-level languages.</p> <p><a ...
|
| 2/1/26 |
<p>With the rapid acceleration of ML/AI research in the last couple of years, the already energy-hungry HPC platforms have become even more demanding. A major part of this energy consumption is due to users’ workloads and it is only by the participation of end users that it is possible to reduce the overall energy consumption of the platforms. However, most of the HPC platforms do not provide any sort of metrics related to energy consumption, nor the performance metrics out of the box, which ...
|
| 2/1/26 |
<p><a href="www.ecmwf.int">ECMWF</a> manages petabytes of meteorological data critical for weather and climate research. But traditional storage formats pose challenges for machine learning, big-data analytics, and on-demand workflows. </p> <p>We propose a solution which introduces a Zarr store implementation for creating virtual views of ECMWF’s Fields Database (FDB), enabling users to access GRIB data as if it were a native Zarr dataset. Unlike existing approaches such as VirtualiZarr or ...
|
