GoKawiilArs: What do you say to the notion that models are improving so fast that building anything with any assumptions at all makes no sense? That’s the argument for a super lean harness or not doing any of this, is just that you don’t know where it’s gonna be in six months or twelve months or whatever as the exponential goes up.
Perneti: There’s a lot of truth to that, but I think you have to think about it from a couple of different perspectives.
At Augment, the way we’ve been thinking about this is there’s two ingredients that you need for higher quality outcomes, intelligence and context. When models are getting really good, intelligence is going to get exponentially better, no doubt about that. But just because they’re more intelligent does not mean they have the context.
Now, a person can get the context that they want by spending the tokens on it. And that’s where the second dimension comes, which is what all of the engineering leaders are asking right now, is cost. How much of your token budget is going towards context gathering and producing the right outcomes? Are you using the model the right way to get the highest quality outcomes?
And that’s where I think harness design and context matter. So to me, it’s a combo of intelligence and context, and then it’s a systems engineering problem at that point: how do you put the right energy into each of these verticals so that you get the most optimal outcome with the least cost?
Ars: We have a fair number of readers who are skeptical of AI in software development. I see two objections that are especially common. One of them is that they still feel that they just can’t trust these systems enough, that they’re going to create technical debt that’s not worth it. The other is that these agentic workflows are too expensive. What do you think?
Perneti: I’ll actually touch on both. Let me preface for both of them. I think the fundamental truth I hope everybody agrees with is models are continuing to improve at an exponential rate. So whatever you’re doing, you want to ride the exponential so that you’re understanding where things are going and you’re orienting yourself to benefit from that.
So, going back to your question around trust and verification and tech debt, it is true if you approach this problem as, ‘I’m handing this off to agents and I’m walking away.’ That’s not how it’s gonna work, and that’s why I point out that it’s teams of humans working with teams of agents, where there are many points where humans are still better suited for judgment, right? Spec reviews is an example. We find that agents are actually not good at writing specs. So you have to really work with an agent to guide the agent to write a good, high-quality spec. But once you have the spec we are at a place, they’re really good at executing. So it makes no sense to not take advantage of that.