Software & Infrastructure

Build Anything with Warpforge -- Reproducibly, Decentralized, With Friends

Warpforge is a "Build Anything" tool for the modern era with a deep emphasis on the ability to "factory reset" the build environment at any time -- and not just reset one environment to a checkpoint, but also to be able to effortlessly reproduce all the prior builds and computations that produce the "factory floor" itself, too: on demand, and without fuss. How much time have you lost to debugging library versioning mismatches? How often has a system update left your machine in a broken state? How often have you installed something on your machine just to try to "make it work"? If you've moved to containers -- how much time have you spent trying to build "the perfect base image", and trying to maintain and wrangle your container build scripts? How much work would still remain in front of you if you needed to reproduce a container image you snapshotted months ago? If the answer to any of these questions troubles you -- we hope Warpforge will present a viable way to help fix all that. In this talk, we'll introduce Warpforge, describe its objectives, give an overview of its declarative API, briefly cover how the guts work (spoiler: LOTS of hashes)... and ultimately, demonstrate it working, live, because seeing is believing. Also in this talk, we'll discuss some conventions in package management and dynamic library linking -- and how we can do these differently, to produce systems of more reusable components. We've started pursuing this together with building Warpforge, in a quest to make the most transparent and the most reusable systems possible. Comparisons to Bazel/Blaze, Nix, Guix, and even to Docker all abound -- we hope you'll find some of the best parts of each visible in Warpforge, plus a few key innovations and user experience improvements of our own, together all in one bundle.
Warpforge's approach to becoming a powerful "Build Anything" tool is to orient itself around providing the user with "computation-addressable" systems -- meaning: the same idea as content-addressable systems, but now extended to also apply to data processing and software compilation. The use of cryptographic hashes to identify data has valuable properties for both security and for simplifying organization; in Warpforge, we apply that same idea to hashing the descriptions of environments and the computations we want to run within them, for similar victories in both security and simplification. What Git did for source code version control, Warpforge wants to do for build instructions and data processing: decentralized, snapshottable, portable, and utterly agnostic to whatever you put in it. Warpforge is granular: although it uses containers for hermeticity, Warpforge lets you compose the filesystem from as many pieces as you want. Warpforge features a freeform filesystem assembly syntax, which lets you state any content you want shall appear at any path you want. This stands in stark contrast to most other container systems which limit you to monolithic "images" -- which invariably devolve into balls-of-mud, becoming difficult to maintain, difficult to introspect, and almost impossible to compose. Warpforge is a hackable tool. Everything works via a JSON API. There are both "high" and "low" level variants of this API: the "high" level lets you describe pipelines of computations, wired together with human-readable names of your choosing; the "low" level API always uses only content-addressable hashes for all data input, and always hashes things immediate upon output. The interaction of these two APIs is the secret sauce: it makes a system that's both usable and also highly precise and highly auditable and reproducible. Warpforge aims to be a developer productivity tool, but at the same time, shift the culture: much like Git taught the world about hash trees, Warpforge wants to teach the world about reproducible builds. Because every output is hashed, computation that fails to reproduce the same output becomes noticed immediately. Warpforge is a perfect tool for those interested in SBOMs (Software Bill Of Materials). Because Warpforge identifies all inputs explicitly, it can very naturally produce an SBOM. In fact, it does so in standard operation -- and unlike many other sytems, Warpforge doesn't just produce a descriptive, after-the-fact SBOM: Warpforge instructions are actually a load-bearing bill-of-materials by nature. Warpforge stops slightly short of doing package management. However, Warpforge does offer some suggestions on ways to build package management systems... with more merkle trees, so that things remain reproducible, introspectable, and auditable even as they grow in systemic complexity. (That's enough spoilers; come the talk to see how!) You can find out more about Warpforge at http://warpforge.io/ and https://github.com/warpfork/warpforge/ .

Additional information

Live Stream https://streaming.media.ccc.de/gpn20/blauersalon
Type Vortrag
Language English

More sessions

5/19/22
Software & Infrastructure
dysphoricUnicorn
Vortragssaal
Are you the kind of person who, when given a personalized link with some number in it, will always try out different numbers to see if things are secured? Do you enjoy exposing security mishaps and edge cases that weren’t thought off by developers, not for profit but for the simple fun of it? If that describes you, I’ve got another thing you could try to feed your curiosity: putting emoji where they don’t belong. It‘s gonna break more stuff than you’d think.🏳️‍⚧️
5/19/22
Software & Infrastructure
Leyrer
Medientheater
Die letzten Jahre, und insbesondere das Agieren der Bundes- und Landesregierungen wenn IT Infrastruktur im Umgang mit der Pandemie bzw. deren Auswikrungen benötigt worden wären, haben zu zahlreichen "interessanten" Meldungen in den Medien geführt. Ich möchte in diesem Talk einige der Ereignisse nochmal Revue passieren lassen und aus den Ereignissen und Vorkommnissen auch Handlungsempfehlungen und Grundlagen für weitere Diskussionen zu dem Thema ableiten.
5/20/22
Software & Infrastructure
Medientheater
In diesem Vortrag geht es um die Geschichte der digitalen Bildungsplattform in Baden-Württemberg und wie es von der ersten Idee 2011 bis heute noch nicht so ganz will.
5/20/22
Software & Infrastructure
Promasu
Blauer Salon
Das Team vom GPN NOC stellt sich allen Fragen über die Arbeit im NOC. Die ideale Gelegenheit um einen Einblick in die Arbeit im NOC zu bekommen oder zu erfahren, wie man zukünftig selber mitwirken kann.
5/20/22
Software & Infrastructure
Cedi
Medientheater
Have you ever considered that your incident from last night might actually be something very positive? No? Then you should watch this talk! I'm going to introduce you to some concepts in the domain of resiliency engineering and then have a look into how you can build an alerting strategy that doesn't page you unnecessarily at 3am. We will have a look into various topics inside the monitoring and alerting domain and how to connect them to make your monitoring and alerting solution better. We will ...
5/20/22
Software & Infrastructure
Leyrer
Medientheater
Glaubt mensch der öffentlichen und politischen Kommunikation rund um das "EU Convid Certificate" aka. den "Green Pass", wie er in der Politik gerne genannt wird, so könnte mensch den Eindruck gewinnen, dass es sich dabei um schwarze Magie handelt, die niemand verstehen kann. In diesem Talk möchte ich aufzeigen, wie elegant das "EU Digital COVID Certificate" eigentlich entworfen wurde, dass die Technologie dahinter eigentlich sehr einfach zu verstehen ist, die Limitation (zusätzlicher Ausweis ...
5/20/22
Software & Infrastructure
Michael Stapelberg
Vortragssaal
I wrote my own rsync implementation in Go! Why? Check out my talk and find out :D