Databases
From Disks to Distributed: Our Journey of Database Evolution in the Cloud
<p>Our database had reached a point where failure scenarios were becoming increasingly complex and time consuming. A single node could take up to 15 minutes to recover. It was expensive to run and operate, and it simply couldn’t scale to meet the customer demand we were facing. It became clear that we needed a new design.
By leveraging a modern architecture and the latest open-source technologies, we rebuilt our database for the cloud era. Recoveries that once took 15 minutes now complete in seconds. Operational costs dropped by 50%, and query latencies improved by 200%. These gains weren’t the result of any single change, but of a holistic redesign powered by technologies like Vortex, DataFusion, Delta Lake, and Rust. </p>
<p>In this talk, Thor will walk you through the end-to-end journey of this evolution
the failure patterns and scaling limits that forced a rethink,</p>
<p>the architectural principles that guided the redesign,</p>
<p>the trade-offs and dead ends along the way,</p>
<p>how modern open-source components were evaluated and integrated, and</p>
<p>the concrete performance and reliability improvements unlocked by the new design.</p>
<p>You’ll leave with a blueprint for modernizing a legacy data system: how to identify when your architecture is holding you back, and how to apply today’s open-source ecosystem to build a cloud-native database that’s fast, resilient, and ready for the future.</p>
Weitere Infos
| Live Stream | https://live.fosdem.org/watch/ub2252a |
|---|---|
| Format | devroom |
| Sprache | Englisch |
Weitere Sessions
| 31.01.26 |
<p>In this session, four seasoned database administrators with sound knowledge of both PostgreSQL and MySQL present an unbiased comparison of the two technologies. Attendees will learn about the architectural and DX differences between the world's two most popular databases.</p> <p>Pep Pla, with his peculiar sense of humour, will open the session with a deep dive into the MVCC architectures between the two. The audience will learn why we need MVCC. Postgres and MySQL take very different ...
|
| 31.01.26 |
<p>The success of open source databases like PostgreSQL and MySQL/MariaDB has created an ecosystem of derivatives claiming "drop-in compatibility." But as the distance between upstream and these derivatives grows, user confusion and brand dilution can follow.</p> <p>To address this, we explore the challenge of compatibility with de facto standards from two distinct angles: a governance perspective on defining the compatibility criteria, and a systems engineering case study on implementing ...
|
| 31.01.26 |
<p>As analytics ecosystems grow more diverse, organisations increasingly need to query data across warehouses, data lakes, and operational systems without excessive movement or duplication. Query federation has become essential by enabling unified SQL access and intelligent pushdown into heterogeneous sources. This talk introduces the core principles of federation and why it matters for modern OLAP workloads and how it is different to Trino.</p> <p>Using StarRocks as a model system, we highlight ...
|
| 31.01.26 |
<p>We all write SQL, but how many of us have looked under the hood of a relational database like PostgreSQL? This talk is a deep dive into the guts of the database engine, tracking a simple SELECT statement from the moment you hit "Enter" to the final result set.</p> <p>We'll lift the veil on the core components: the parser, the planner (and the optimizer's black magic!), and the executor, and see how they transform a text string into a low-level, high-performance operation. Using a live, ...
|
| 31.01.26 |
<p>While optimizing a new heap storage engine across both MySQL and a PostgreSQL-based database we encountered a puzzling result: while on MySQL the throughput stalled below 500k tpmC, on the other database it achieved over 1 million tpmC. The mystery deepened when three different TPC-C benchmarks each told a conflicting story about MySQL’s speed.</p> <p>This talk details the systematic investigation to resolve these contradictions and reclaim the lost performance. We’ll walk through the ...
|
| 31.01.26 |
<p>RonDB is a high-performance, MySQL-compatible distributed database engineered for real-time, latency-critical workloads. Built on decades of development in the MySQL NDB Cluster—led by the original founder of the NDB product—RonDB extends the NDB storage engine with new capabilities, cloud-native automation, and modern APIs tailored for large-scale AI and online services.</p> <p>This talk will describe how RonDB consistently delivers 1–4 ms latency even for large batched operations ...
|
| 31.01.26 |
<p>DuckDB has traditionally been seen as a last-mile analytics powerhouse, the fastest way to run a SQL query on your laptop. But DuckDB offers more than just fast SQL, of course; it supports full database semantics and ACID transactions, behaving like a fully fledged, in-process OLAP database. The in-process component has sometimes been viewed as a limitation when considering DuckDB as a data warehouse.</p> <p>However, DuckDB now supports reading and writing to most Open Table Formats (OTFs), ...
|
