Oban, the job processing framework from Elixir, has finally come to Python. I spent some time exploring it, and here is how it works.
Setting the Stage#
I’ve used Oban in Elixir for almost as long as I’ve been writing software in Elixir, and it has always been an essential tool for processing jobs. I always knew Oban was cool, but I never dug deeper. This article is a collection of my notes and observations on how the Python implementation of Oban works and what I’ve learned while exploring its codebase. I’ll also try to compare it with the Elixir version and talk about concurrency in general.
Surface Level#
Oban allows you to insert and process jobs using only your database. You can insert the job to send a confirmation email in the same database transaction where you create the user. If one thing fails, everything is rolled back.
Additionally, like most job processing frameworks, Oban has queues with local and global queue limits. But unlike others, it stores your completed jobs and can even keep their results if needed. It has built-in cron scheduling and many more features to control how your jobs are processed.
Oban comes in two versions - Open Source Oban-py and commercial Oban-py-pro.
OSS Oban has a few limitations, which are automatically lifted in the Pro version:
Single-threaded asyncio execution - concurrent but not truly parallel, so CPU-bound jobs block the event loop.
- concurrent but not truly parallel, so CPU-bound jobs block the event loop. No bulk inserts - each job is inserted individually.
... continue reading