GoKawiilIncident Report: File Hosting Errors
Executive summary
For about two weeks in August 2026, some PyPI users hit intermittent 502 and 503 errors downloading files from files.pythonhosted.org , triggering failures during installation from PyPI. Thanks to our users filing reports in the support tracker; one report in particular narrowed the problem to a single cache node.
Two separate problems were uncovered. A canary deployment inside Fastly's network had triggered a misconfiguration at one cache node, causing Fastly's routing layer to return 502 responses for traffic reaching the affected cache node. Separately, I found and fixed several bugs in our own Fastly configuration around origin fallback and range request behavior. Those had been there for a while, and only surfaced while digging into these reports.
Both intermittent problems are now fixed, and downloads are back to full function since August 28.
Background: How PyPI's file hosting cache works
Most people run pip install (or your installer of choice) and it just works: a request goes out, the desired file comes back. Behind the scenes, files.pythonhosted.org is a Fastly CDN service in front of three origins (aka 'backends').
When a file is uploaded, PyPI writes it to Amazon S3 first as the main, durable copy. A background job syncs it to Backblaze B2, because Fastly and Backblaze have a zero-cost egress agreement: serving files out of B2 through Fastly doesn't cost anything beyond storage fees.
On the reader (installer) path, Fastly tries B2 first. If B2 doesn't answer, or answers with something we don't expect, Fastly falls back to S3. PyPI uses the Amazon S3 Glacier Instant Retrieval storage class to balance storage and retrieval costs. When Fastly calls S3 as a fallback, this costs more than from B2, but will continue to work for consumers as a stop-gap.
Once cached, Fastly no longer has to check B2 or S3 - the file should never change, and the Cache-Control header sets max-age=365000000, immutable, public - about 11.5 years.
... continue reading