Databases
AMENDMENT LumoSQL - Experiments with SQLite, LMDB and more
SQLite is justly famous, but also has well-known limitations
LumoSQL is an experimental fork of SQLite, the embeddable database library founding everything from Android to iOS to Firefox. As a replacement for fopen(), SQLite is a good choice for singer-writer applications and disconnected, slow and small devices. Modern IoT and application use cases are increasingly multi-writer, fast, high-capacity and internet-connected, and LumoSQL aims to address these very different modern needs. LumoSQL initially aims to improving speed and reliability, by replacing the internal key-value store with LMDB, by updating and fixing a prototype from 2013, and allowing multiple storage backends. Next up we are designing the architecture for replacing the write-ahead log system (as used by all other open and closed source databases) with a single-level store, drawing on LMDB as an example of a single-level store in production at scale. Challenges so far involve code archeology, understanding and updating benchmarking, designing a system for keeping parity with upstream code changes, file format migration and identifying bugs in both SQLite and LMDB.
Please do join us in testing and improving at https://github.com/LumoSQL/LumoSQL .
In this talk we welcome questions and contributions. This conference has many SQLite users and developers. What do you want to see?
LumoSQL is a combination of two embedded data storage C language libraries: SQLite and LMDB. LumoSQL is an updated version of Howard Chu's 2013 proof of concept combining the codebases. Howard's LMDB library has become ubiquitous on the basis of performance and reliability, so the 2013 claims of it greatly increasing the performance of SQLite seem credible. D Richard Hipp's SQLite is relied on by many millions of people on a daily basis (every Android and Firefox user, as just two projects of the thousands that use SQLite) so an improved version of SQLite would benefit billions of people.
The original code changes btree.c in SQLite 3.7.17 to use LMDB 0.9.9 . It takes some work to replicate the original results because not only has much changed since, but as a proof of concept there was no project established to package code or make it accessible. LumoSQL revives the original code and shows how it is still relevant in 2019. The premise seems sound. Some bugs have been fixed in LMDB and the prototype SQLightning work.
There needs to be multiple backends, initially the original SQLite on-disk format and LMDB and initially for compatibilit and conversion purposes. However the ability to have more backends is very attractive and already there are draft designs for where that could lead.
The design taking shape for tracking SQLite upstream may be useful to other projects, where an automated process and can handle most changes that do not change some of the basic APIs.
Write-Ahead Logs are in every single widely-used database today, a concurrency model developed in the 1990s and now the only option in both closed and open source SQL databases. There are pros and cons for WALs, but the merge-back model of WALs is a lack of atomicity that becomes obvious in corruption and reliability issues at speed and scale. Databases go to very expensive efforts to avoid this, but combined with a lack of real-time integrity checking in almost all databases, this is a fundamental problem and especially for modern SQLite-type use cases.
Additional information
| Type | maintrack |
|---|
More sessions
| 2/1/20 |
Free and Open Source software has revolutionized the Software Industry and nearly all other areas of human endeavor, but until now its reach into actual governance at the municipal citizen level has not been very deep. Initiatives like Code for America have encountered challenges driving acceptance for FOSS alternatives to proprietary software for citizen governance. At the same time the gap between citizen need and cities’ capabilities as widened. But several new projects are aiming to change ...
|
| 2/1/20 |
From ten years of LibreOffice, how can you apply what we learned to your project ? What is going on in LibreOffice today, and where is it going ? and How can you re-use or contribute to the story.
|
| 2/1/20 |
It has become very popular in the last several years to think of free and open source as a community forward activity, indeed the modern approach is to try and form a community or foundation first and do code second. There is also much talk about maintainer burn out and community exploitation. However, the same people who talk about this still paraphrase the most famous quote from the Cathedral and the Bazaar "Scratching your own itch". They forget this is your own itch not everyone else's ...
|
| 2/1/20 |
Over the past twenty years, the automation landscape has changed dramatically. As our hunger for complex technical infrastructure increased, and our inability to keep up with these demands faltered, we've outsourced a lot of the work to third-parties and cloud providers. We'll step backwards and show where we came from, and where we're going. If we don't understand this future, and step up to the challenge, then we eventually won't control our own computers anymore. We'll discuss this timeline ...
|
| 2/1/20 |
Internet of Things (IoT) devices are part of the future we were promised. Armed with our mobile devices, we can control everything from our cars to our toasters to the doors of our homes. Along with convenience, IoT devices bring us ethical quandaries, as designers and users. We need to consider the ethical implicates of the technologies we are building and ask ourselves not just about the ways they are being used, for both good and evil, but the potential use cases we might encounter in the ...
|
| 2/1/20 |
The presentation is going to be audiovisual and entertaining; based on a number of short videos I want to tell the story of Blender. Starting in late 90s, how Blender became open source, going over the big milestones for Blender, end ending with the fast growth of our project and the interest of the film and game industry. Blender now is a more mature project now, which involves a different dynamics than it used to be. How are we going to tackle the challenges of the industry, while not losing ...
|
| 2/1/20 |
Despite the number of working groups, advisory committees, and coordination roundtables, there is little progress towards creating more ethical and safe AI systems. AI systems are deployed in increasingly fragile contexts. From law enforcement to humanitarian aid, several organizations use AI powered systems to make or inform critical decisions with increasingly outsized side effects. What is a rights-based approach for designing minimally safe and transparent guidelines for AI systems? In this ...
|
