Making PyPI's test suite 81% faster – The Trail of Bits Blog

Trail of Bits has collaborated with PyPI for several years to add features and improve security defaults across the Python packaging ecosystem. Our previous posts have focused on features like digital attestations and Trusted Publishing, but today we’ll look at a equally critical aspect of holistic software security: test suite performance. A robust testing suite is essential to the security and reliability of a complex codebase. However, as test coverage grows, so does execution time, creating friction in the development process and disincentivizing frequent and meaningful (i.e., deep) testing. In this post, we’ll detail how we methodically optimized the test suite for Warehouse (the back end that powers PyPI), reducing execution time from 163 seconds to 30 seconds while the test count grew from 3,900 to over 4,700. Figure 1: Warehouse test execution time over a 12-month period (March 2024 to April 2025). We achieved a 81% performance improvement through several steps: Paralleliz ... Read full article.