Databases
SWIM - Protocol to Build a Cluster
SWIM gossip protocol, its implementation, and improvements
SWIM - is a relatively new protocol to discover and monitor cluster nodes, to disseminate events and data between them. The protocol is extremely lightweight, decentralised, and its speed and load per node do not depend on cluster size.
The protocol solves several tasks at once. First - build and keep up to date topology of a cluster without explicit configuration. The task is quite intricate because:
new just started nodes know nothing about others, and they should somehow discover them;
already working nodes can fail, and it should be detected so as to change a master, or evict an unrecoverable node from the cluster, or restart it.
According to the protocol, cluster nodes broadcast packets and send p2p ping requests. Broadcast helps to discover new nodes, p2p pings help to detect failure of a known node.
A second task - events dissemination in a cluster. Event is a node failure; UUID change; IP address update; new node appearance - anything that affects cluster state. Sometimes users define their own event types. When a node learns about an event, it needs to disseminate the event to other nodes. SWIM protocol describes an algorithm how to detect and disseminate events, and gives the following guarantees:
it takes a constant time to learn about an event on at least one node in the cluster;
it takes logarithmic from cluster size time to disseminate that event to each node of the cluster.
In the talk I tell about how SWIM works, how and with which essential improvements it was implemented, how to use SWIM, and what are the practical performance results.
Implementation is a part of Tarantool DBMS. Tarantool is the biggest Russian Open-Source DBMS. Tarantool currently goes toward better scalability, improvements in horizontal scaling, in cluster-wide calculations, and better cluster management. In scope of that roadmap SWIM protocol implementation was recently released.
Additional information
| Type | maintrack |
|---|
More sessions
| 2/1/20 |
Free and Open Source software has revolutionized the Software Industry and nearly all other areas of human endeavor, but until now its reach into actual governance at the municipal citizen level has not been very deep. Initiatives like Code for America have encountered challenges driving acceptance for FOSS alternatives to proprietary software for citizen governance. At the same time the gap between citizen need and cities’ capabilities as widened. But several new projects are aiming to change ...
|
| 2/1/20 |
From ten years of LibreOffice, how can you apply what we learned to your project ? What is going on in LibreOffice today, and where is it going ? and How can you re-use or contribute to the story.
|
| 2/1/20 |
It has become very popular in the last several years to think of free and open source as a community forward activity, indeed the modern approach is to try and form a community or foundation first and do code second. There is also much talk about maintainer burn out and community exploitation. However, the same people who talk about this still paraphrase the most famous quote from the Cathedral and the Bazaar "Scratching your own itch". They forget this is your own itch not everyone else's ...
|
| 2/1/20 |
Over the past twenty years, the automation landscape has changed dramatically. As our hunger for complex technical infrastructure increased, and our inability to keep up with these demands faltered, we've outsourced a lot of the work to third-parties and cloud providers. We'll step backwards and show where we came from, and where we're going. If we don't understand this future, and step up to the challenge, then we eventually won't control our own computers anymore. We'll discuss this timeline ...
|
| 2/1/20 |
Internet of Things (IoT) devices are part of the future we were promised. Armed with our mobile devices, we can control everything from our cars to our toasters to the doors of our homes. Along with convenience, IoT devices bring us ethical quandaries, as designers and users. We need to consider the ethical implicates of the technologies we are building and ask ourselves not just about the ways they are being used, for both good and evil, but the potential use cases we might encounter in the ...
|
| 2/1/20 |
The presentation is going to be audiovisual and entertaining; based on a number of short videos I want to tell the story of Blender. Starting in late 90s, how Blender became open source, going over the big milestones for Blender, end ending with the fast growth of our project and the interest of the film and game industry. Blender now is a more mature project now, which involves a different dynamics than it used to be. How are we going to tackle the challenges of the industry, while not losing ...
|
| 2/1/20 |
Despite the number of working groups, advisory committees, and coordination roundtables, there is little progress towards creating more ethical and safe AI systems. AI systems are deployed in increasingly fragile contexts. From law enforcement to humanitarian aid, several organizations use AI powered systems to make or inform critical decisions with increasingly outsized side effects. What is a rights-based approach for designing minimally safe and transparent guidelines for AI systems? In this ...
|
