Continuous Integration and Continuous Deployment

Multi-master scaling for MariaDB Buildbot

D.cicd
Vlad Bogolin
<p>Continuous Integration is a very important part in order to ensure a high quality software product. In order to be able to test the MariaDB server across multiple platforms, operating systems and configurations, the MariaDB Foundation has developed a continuous integration framework based on Buildbot. This talk will focus on describing the design decisions in order to be able to scale and test the MariaDB server on over 100 different configurations. We use a multi-master setup and combine Docker latent workers with LibVirt workers and classical Buildbot workers in order to achieve this.</p>
In order to ensure that MariaDB runs smoothly, it needs to be tested on multiple platforms and configurations. In order to ensure this, the MariaDB Foundation has developed a new continuous integration framework, based on Buildbot, that thoroughly tests the server across multiple platforms, operating systems and configurations. In order to achieve this, we use a mixture of builders, varying from Docker latent workers, to LibVirt workers but also the classical “always on” buildbot workers. For easier management, we have a separate master process that deals with each type of worker. In this talk, I will talk about our design decisions and some limitations that stand behind these. For example, the testing phase of the MariaDB server generates a very high amount of logs. Since Buildbot can deal with around 10k lines of logs per second, having multiple builds that run at the same time may saturate the master process. In order to deal with this, we use a master locking mechanism together with a multi-master setup.

Additional information

Type devroom

More sessions

2/6/22
Continuous Integration and Continuous Deployment
D.cicd
<p>Continuous Delivery is all about getting your software released, and as software/system complexity continues to increase, so does the need to have many different tools, services and even people involved in building, verifying, deploying and monitoring your software. So, how do we make all these tools and services work together to complement each other without creating a massive spider web of explicit interconnectedness? CDEvents!</p> <p>In this talk, Andrea Frittoli and Erik Sternerson will ...
2/6/22
Continuous Integration and Continuous Deployment
Noé Gaumont
D.cicd
<p>Logilab has been using heptapod, a GitLab fork with mercurial support, for 2 years now. We are maintaining the open source software CubicWeb and its components called cubes. Thus, the code is split in dozens of repositories, depending on one another. Over the years, it has become hard to maintain code quality and good practices in the whole codebase.</p> <p>In this talk, we will present the tools that helped us. Some of them are mercurial specific, but most could be used in GitLab.</p> <ul> ...
2/6/22
Continuous Integration and Continuous Deployment
Adam Souzis
D.cicd
<p>Unfurl is a brand-new open-source command line tool for managing your DevOps infrastructure without depending on any server infrastructure. Unfurl lets you easily track configuration, secrets, software and code dependencies, and deployment history all in git. It integrates with the DevOps tools you are already using -- like Terraform, Ansible, and Helm -- allowing you to encapsulate your DevOps processes into reusable building blocks and describe your cloud infrastructure in simple, ...
2/6/22
Continuous Integration and Continuous Deployment
Harshit Chopra
D.cicd
<p>Jenkins a self-contained, open-source automation server used in automation of all sorts of tasks related to building, testing, and delivering or deploying software. With high extensibility through plugins to provide much of the user-needed functionality. One such plugin is the Git plugin which deals with git operations for Jenkins project but lacked authenticated access to git repositories for Pipeline users. The Git Credentials binding project provides an out of the box solution by ...
2/6/22
Continuous Integration and Continuous Deployment
Bram Vogelaar
D.cicd
<p>Things like Infrastructure as Code, Service Discovery and Config Management can and have helped us to quickly build and rebuild infrastructure but we haven't nearly spend enough time to train our self to review, monitor and respond to outages. Does our platform degrade in a graceful way or what does a high cpu load really mean? What can we learn from level 1 outages to be able to run our platforms more reliably.</p> <p>This talk will focus on on setting up a CICD pipeline using Jenkins. We ...
2/6/22
Continuous Integration and Continuous Deployment
Markus Lippert
D.cicd
<p>How can you leverage Kubernetes-native CI/CD solutions for your Windows applications? What's needed to create Windows container images in your pipelines? This hands-on talk will get developers and operators started with Windows container support of Argo Workflows and Tekton. You will learn how you can use these solutions to build and test your Windows applications and create Windows container images for them. All work presented will be shared with you.</p>
2/6/22
Continuous Integration and Continuous Deployment
Alexander Jung
D.cicd
<p>Unikernels are bespoke, single-address space, lightweight Virtual Machines which do one thing and one thing only: fully facilitate the runtime of one application. Unikernels represent a compile-time specialization strategy and leverage the joining of user and kernel space via a library Operating System in order to achieve this, offering as a result incredible runtime performance whilst increasing application security via isolated hardware virtualization. As Unikraft, an open-source unikernel ...