The Cost of Being Crawled: LLM Bots and Vercel Image API Pricing

On Friday, Feb 7, 2025 we had an incident with our Next.js web app hosted on Vercel that could've cost us $7,000 if we didn't notice it in time. We had a spike in LLM bot traffic coming from Amazonbot, Claudebot, Meta and an unknown bot. Together they sent 66.5k requests to our site within a single day. Bots scraped thousands of images that used Vercel's Image Optimization API, which cost us $5 per 1k images. The misconfiguration on our side combined with the aggressive bot traffic created an economically risky situation for our tiny bootstrapped startup. Metacast is a podcast tech startup. Our main product is a podcast app for iOS and Android. For every podcast episode on the platform, our web app has a web page. Our platform has ~1.4M episodes, which means we have 1.4M web pages that are discoverable by crawlers. These pages are generated server-side at request time, then cached. First, we received a cost alert from Vercel saying that we've hit 50% of the budget for resources me ... Read full article.