GoKawiilThis is a continuation of the discussion in How I’m thinking about the value of LLMs. I’m arguing elsewhere that LLMs will never be geniuses. This is not part 2 of The Ontology Argument.
In How I’m thinking I said I wasn’t ready to take a stance on LLM value creation. That changes in this post. Here is the stance I’m taking:
On average, how we’re using LLMs is likely destroying value.
My stance originates from stumbling on Faros.ai - a software development telemetry firm. They have products that pipe into common development tools like Jira, Github, and CI/CD pipelines to directly measure major operational metrics for software development teams.
Faros published a report in March that directly compares transaction level data between teams using AI in their software development process vs those that are not across their customer base. 22,000 developers, 4000 teams in the sample. This is, by far, the best data I’ve been able to locate that directly measures the operational impact of use of LLMs in the software development process.
It’s bad. Really, really bad.
The Faros Data
The whole report is worth a read, but I’m going to cover just three major headline conclusions.
First - developer level productivity has improved
I think this supports what I said in How I’m thinking - there is clearly an individual productivity speedup that happens with LLMs. Although, I will say - it doesn’t look like it’s 10x from here. It’s a much more modest improvement than what the optimistic AI case would tell you.
... continue reading