Python
Follow Your Celery Tasks
All Python developer who want to run asynchronous tasks should know Celery. If you have already used it, you know how great it is ! But you also discovered how it can be complicated to follow the state of a complex workflow. Celery Director is a tool
we created at OVH to fix this problem : using some concepts of Event Sourcing, Celery Director helps us to follow the whole lifecycle of our workflows. It allows us to check when a problem occurred and relaunch the whole DAG (or just a subpart if tasks are not completely idempotent). During this talk we will introduce you the different concepts of Celery Director then we'll make a demonstration of it.
All Python developer who want to run asynchronous tasks should know Celery. If you have already used it, you know how great it is ! But you also discovered how it can be complicated to follow the state of a complex workflow. Celery Director is a tool we created at OVH to fix this problem : using some concepts of Event Sourcing, Celery Director helps us to follow the whole lifecycle of our workflows. It allows us to check when a problem occurred and relaunch the whole DAG (or just a subpart if tasks are not completely idempotent). During this talk we will introduce you the different concepts of Celery Director then we'll make a demonstration of it.
Additional information
| Type | devroom |
|---|
More sessions
| 2/1/20 |
From my talk you will learn about some lesser-known features of sudo, and how you can make your security more flexible by extending sudo using Python.
|
| 2/1/20 |
While working on the Mercurial version control system, we hit our heads against the limits of Python's performance. In this talk we will see how Python and Rust can cohabit to play off of each other's strenghts to improve a big open-source project, and what advances have been made in bridging the two languages.
|
| 2/1/20 |
This is a hands-on short tutorial on how to write your own estimator or transformer which can be used in a scikit-learn pipeline, and works seamlessly with the other meta-estimators of the library. It also includes how they can be conveniently tested with a simple set of tests.
|
| 2/1/20 |
For almost 20 years, we relied on a CGI based protocol called WSGI to use Python to handle HTTP requests and responses software. Because Python is singled threaded we relied on a couple of hacks such as Gunicorn or uWSGI to share a socket through multiple processes. However the cost of all these multiple processes was a bit heavy and error prone. Through Django Channels Andrew Godwin paved the way for a better way of creating web services with Python. This work landed in Django 3.0. Let's ...
|
| 2/1/20 |
How does one manage and document change in Python projects, be that new features or deprecation or removal of a feature? Let's explore some of the tools a Python developer can keep in their toolbox for just this purpose.
|
| 2/1/20 |
We, as developer, aim to provide code that, almost matches our team code style, looks better and behaves right. Static code analysis (SCA) tools are one of the way to achieves that. But, with multi-programming languages projects and all kinds of code related needs, It's difficult to address all thoses usecases without dealing with a vast majority of SCA tools. Coala is a — language agnostic — static code analysis framework that provides a common command-line interface for linting and fixing ...
|
| 2/1/20 |
SortingHat is an open source Python tool that helps to manage the different contributor identities within an open source project. Under the hood SortingHat relies on a relational database, which can be queried via SQL, command line or directly via its Python interface. However, these ways of interacting with SortingHat hinder its integration with external tools, web interfaces and new web technologies (e.g., Django, REST services). To overcome these obstacles, we have evolved SortingHat's ...
|
