I’ve been using CI/CD pipelines for years now - GitLab CI, GitHub Actions, Jenkins, you name it. Like most developers, I treated them as configuration files that “just work.” You write a YAML file, push it, and somehow your code gets built, tested, and deployed. But I never really understood what was happening behind the scenes.
That changed when I needed to set up CI/CD for an air-gapped environment at work - no access to GitHub Actions or GitLab’s hosted runners. I needed to understand how these tools actually work under the hood so I could build something custom. That’s when I realized: pipeline runners are just orchestration tools that execute jobs in isolated environments.
In this post, I’ll show you how to build a complete CI/CD pipeline runner from scratch in Python. We’ll implement the core features you use every day: stages, parallel execution, job dependencies, and artifact passing. By the end, you’ll understand exactly how GitLab Runner and GitHub Actions work internally.
What Actually IS a CI/CD Pipeline?
Before we start coding, let’s understand what a CI/CD pipeline actually does.
A CI/CD pipeline is an automated workflow that takes your code from commit to deployment. It’s defined as a series of jobs organized into stages:
stages : - build - test - deploy build-job : stage : build script : - python -m build test-job : stage : test script : - pytest tests/ deploy-job : stage : deploy script : - ./deploy.sh
When you push code, a pipeline runner (like GitLab Runner or GitHub Actions) does this:
Parses the pipeline configuration file Creates a dependency graph of jobs Executes jobs in isolated environments (usually containers) Streams logs back to you in real-time Passes artifacts between jobs Reports success or failure
The key insight: a pipeline runner is just a job orchestrator. It figures out what to run, in what order, and handles the execution.
... continue reading