Cloudflare R2 service outage caused by password rotation error

Cloudflare announced that its R2 object storage and dependent services experienced an outage lasting 1 hour and 7 minutes, causing 100% write and 35% read failures globally. Cloudflare R2 is a scalable, S3-compatible object storage service with free data retrieval, multi-region replication, and tight Cloudflare integration. The incident, which lasted between 21:38 UTC and 22:45 UTC, was reportedly caused by a credential rotation that caused the R2 Gateway (API frontend) to lose authentication access to the backend storage. Specifically, new credentials were mistakenly deployed to a development environment instead of production, and when the old credentials were deleted, the production service was left with no valid credentials. The issue stemmed from omitting a single command-line flag, '--env production,' which causes the new credentials to be deployed to the production R2 Gateway Worker rather than the production worker. R2 Gateway Worker authentication diagram Source: Cloudfla ... Read full article.