Ducks to the rescue - ETL using Python and DuckDB

<p><em>Summary:</em></p> <p>ETL stands for "extract, transform, load" and is a synonym for moving data around. This has traditionally often required managing complex systems in the cloud or large data centers. The talk will demonstrate how all this can be greatly simplified by applying modern tools for the task: Python and DuckDB, both open source and readily available to run on most systems - even your notebook.</p> <p><em>Description:</em></p> <p><strong>ETL</strong> stands for "extract, transform, load" and is a synonym for moving data from one system to another. </p> <p>Traditionally, ETL was done in exactly that order: first you extract the data you want to process, then you transform it and then you load it into the target system. More modern approaches based on data lakes, swap the T and L, since transformation is more efficiently done in a database system, especially when it comes to large volumes of data.</p> <p>In order to make all this work, the usual approach is to have a workflow system, taking care of managing all the intermediate steps, a large data lake database and distributed storage systems. This results in lots of complexity, need for system/cluster administration and maintenance.</p> <p>Now, with today's computers, most data sizes used in ETL no longer need all this complexity. Even notebooks or single VMs can handle the load, when used with external object storage, so all you really just need is the right software stack to manage your ETL - without all the overhead:</p> <ul> <li> <p><strong>Python</strong> has grown to be the number one programming language on the planet and is especially well suited for integration work due to its many readily available connectors to plenty of backend systems. It often comes preinstalled on Linux machines and is easy to install on most other systems.</p> </li> <li> <p><strong>DuckDB</strong> has emerged as one of the most capable embedded OLAP database systems and supports data lakes with the DuckLake extension, right out of the box. Installation is just a <code>uv add duckdb</code> away.</p> </li> </ul> <p>Both can be run on the same machine and are very resource friendly.</p> <p>The talk will give an overview of the typical steps involved in ETL processes, give a short intro to DuckDB and showcase how DuckDB can be put to good use when implementing ETL processes. If time permits, I can also cover a few advanced topics addressing optimization strategies.</p> <p><em>Resources:</em></p> <ul> <li> <p><a href="https://www.python.org/">Python.org</a></p> </li> <li> <p><a href="https://duckdb.org/">DuckDB – An in-process SQL OLAP database management system</a></p> </li> </ul>