GoKawiilIn a nutshell: Amazon's growing use of AI across its operations is starting to expose a practical problem: the technology can become expensive quickly, and in some cases, no one notices until the bill is already high. Internal discussions reviewed by the Financial Times show that several AI-driven projects at the company have run over budget, sometimes by a wide margin.
In one case, Amazon spent $1.8 million on a project that used Anthropic's Claude Sonnet model to match author information with product listings. The system ultimately failed, and spending exceeded the original budget by 860%. The issue went undetected for five months.
Engineers say the problem isn't limited to one project. As more teams shift from traditional software to AI models, routine mistakes are becoming far more costly. Tasks that once required minimal computing resources now depend on systems that charge based on usage, often measured in tokens. When those systems are misconfigured or left unchecked, costs can climb quickly.
During a recent internal meeting, senior engineers described these errors as "catastrophically expensive," noting that similar mistakes in conventional systems were "trivially cheap." The difference comes down to how AI workloads are priced and executed. Instead of predictable infrastructure costs, teams are now dealing with variable expenses that depend on how often models are called and how they are used.
Other projects have run into similar issues. One effort to build a financial auditing tool generated about $541,000 in unexpected costs. Another project aimed at improving delivery speeds in Amazon's logistics network incurred $134,000 in additional spending, and it took more than two weeks to catch the problem.
According to people familiar with the situation, engineers are now working on safeguards to prevent similar incidents from happening again. These include automated controls to limit how AI systems are used and improved real-time spending tracking.
Part of the challenge is visibility. "It's difficult to figure out how much anything [AI related] costs," one senior Amazon employee told the Financial Times. Unlike traditional software systems, where costs are easier to estimate, AI introduces more moving parts. Model calls, prompt chains, and autonomous agents can all drive usage in ways that are difficult to track until after the fact.
Amazon, for its part, says the company is still in a learning phase. "As with any new technology, we're experimenting, learning and improving how we use it, including how we drive cost efficiencies," the company said in an internal presentation.
It also pushed back on the idea that these incidents reflect broader problems, adding that "Cherry-picking small, isolated examples where teams are learning from one another and portraying them as business as usual doesn't reflect how teams across Amazon are using AI."
Still, the examples point to a broader shift happening across the tech industry. As companies move away from fixed pricing models and toward usage-based billing, AI costs are becoming harder to predict. Systems that rely on autonomous agents can generate large volumes of activity, sometimes far beyond what teams initially expect.
... continue reading