GoKawiilOn Jan 13th, 2025, we noticed strange 404 errors in our backend logs, originating from Bunny Storage. We investigated and found that files which were uploaded successfully to Bunny via their API simply vanished, with no deletion from our side, and no recorded write operation of any kind in Bunny's own logs.
Bunny's own support confirmed it the next day (January 14th, 2025) saying: files were found in the replication region but not in the main region.
Timeline
Jan 13, 2025 - Ticket opened after dozens of missing files.
- Ticket opened after dozens of missing files. Jan 13 - Support asks if the files were "deleted then re-uploaded." We explain they were never deleted by us and there are no logs showing deletions on Bunny's end either.
- Support asks if the files were "deleted then re-uploaded." We explain they were never deleted by us and there are no logs showing deletions on Bunny's end either. Jan 14 - Support escalated to their Storage team. They confirm the files were found in some replication regions but not in the main region, which is odd. Engineering will "try to recover files."
- Support escalated to their Storage team. They confirm the files were found in some replication regions but not in the main region, which is odd. Engineering will "try to recover files." Jan 15 - More 404s.
- More 404s. Jan 17 - 200+ instances in 7 days. Four different agents have touched the ticket. None have a resolution.
- 200+ instances in 7 days. Four different agents have touched the ticket. None have a resolution. Jan 19 - We ask to speak with someone directly. Refused. All communication must go through the ticket system.
Feb 2-6 - Still not fixed. Every response: "forwarded to the team, no update yet."
... continue reading