7 ms·

I think this sounds like a true yet short sighted take. Keep in mind these features are immature but they exist to obtain a flywheel and corner the market. I do

I think this sounds like a true yet short sighted take. Keep in mind these features are immature but they exist to obtain a flywheel and corner the market. I don’t know why but people seem to consistently miss two points and their implications - performance is continuing to increase incredibly quickly, even if you rightfully don’t trust a particular evaluation. Scaling laws like chinchilla and RL scaling laws (both training and test time) - coding is a verifiable domain The second one is most important. Agent quality is NOT limited by human code in the training set, this code is simply used for efficiency: it gets you to a good starting point for RL. Claiming that things will not reach superhuman performance, INCLUDING all end to end tasks: understanding a vague business objective poorly articulated, architecting a system, building it out, testing it, maintaining it, fixing bugs, adding features, refactoring, etc. is what requires the burden of proof because we literally can predict performance (albeit it has a complicated relationship with benchmarks and real world performance). Yes definitely, error rates are too high so far for this to be totally trusted end to end but the error rates are improving consistently, and this is what explains the METR time horizon benchmark.

nprateem 7mo ago
But the issue isn't coding, it's doing the right thing. I don't see anywhere in your plan some way of staying aligned to core business strategy, forethought, etc. The number of devs will reduce but there will still be large activities that can't be farmed out without an overall strategy
- aspenmartin 7mo ago
  Why do you think this is a problem? Reasoning is constantly improving, it has ample access to humans to gather more business context, it has access to the same industry data and other signals that humans do, and it can get any data necessary. It has Zoom meeting notes, I mean why do people think there's somehow a fundamental limit beyond coding? The other thing you're missing here is generalizability. Better coding performance (which is verifiable and not limited by human data quality) generalizes performance on other benchmarks. This is a long known phenomenon.
  - skydhash 7mo ago
    > Why do you think this is a problem? Because it cannot do it? Every investment has a date where there should be a return on that investment. If there’s no date, it’s a donation of resources (or a waste depending on perspective). You may be OK with continuing to try to make things work. But others aren’t and have decided to invest their finite resources somewhere else.
    - aspenmartin 7mo ago
      > Because it cannot do it? Ah ok so you didn't really read my comment, what is your counter argument? Models are just fundamentally incapable of understanding business context? They are demonstrably already capable of this to a large extent. > Every investment has a date where there should be a return on that investment. If there’s no date, it’s a donation of resources (or a waste depending on perspective). what are you implying here? This convo now turns into the "AI is not profitable and this is a house of cards" theme? That's ok, we can ignore every other business model like say Uber running at a loss to capture what is ultimately an absolutely insane TAM. Little ol' Uber accumuluated ~33B in losses over 14 years, and you're right they tanked and collapsed like a dying star...oh wait...hmm interesting I just looked at their market cap and it's 141 Billion. > You may be OK with continuing to try to make things work. But others aren’t and have decided to invest their finite resources somewhere else. I truly love that. If you want to code as a hobby that is fantastic, and we can go ahead and see in 2 years how your comment ages.
      - skydhash 7mo ago
        > They are demonstrably already capable of this to a large extent. I’d very like to see such demonstration. Where someone hands over a department to an agent and let it makes decisions. > This convo now turns into the "AI is not profitable and this is a house of cards" theme? Where did I say that? I didn’t even mention money, just the broader resource term. A lot of business are mostly running experiments if the current set of tooling can match the marketing (or the hype). They’re not building datacenters or running AI labs. Such experiments can’t run forever.
embedding-shape 7mo ago
> - coding is a verifiable domain You're missing the point though. "1 + 1" vs "one.add(1)" might both be "passable" and correct, but it's missing the forest for the trees, how do you know which one is "long-term the right choice, given what we know?", which is the engineering part of building software, and less about "coding" which tends to be the easy part. How do you evaluate, score and/or benchmark something like that? Currently, I don't think we have any methodologies for this, probably because it's pretty subjective in the end. That's where the "creative" parts of software engineering becomes more important, and it's also way harder to verify.
- aspenmartin 7mo ago
  because it has business context and better reasoning, and can ask humans for clarification and take direction. You don't need to benchmark this, although it's important. We have clear scaling laws on true statistical performance that is monotonically related to any notion of what performance means. I do benchmarks for a living and can attest: benchmarks are bad, but it doesn't matter for the point I'm trying to make.
  - troupo 7mo ago
    > because it has business context It doesn't because it doesn't learn. Every time you run it, it's a new dawn with no knowledge of your business or your business context > better reasoning It doesn't have better reasoning beyond very localized decisions. > and can ask humans for clarification and take direction. And yet it doesn't, no matter how many .md file you throw at it, at crucial places in code. > We have clear scaling laws on true statistical performance that is monotonically related to any notion of what performance means. This is just a bunch of words stringed together, isn't it?
    - skydhash 7mo ago
      Almost every task that people are tackling agents on, it’s either not worth doing, can be done better with scripts and software, or require human oversight (that negates all the advantages.
sobellian 7mo ago
Scaling laws vs combinatorial explosion, who wins? In personal experience claude does exceedingly well on mundane code (do a migration, add a field, wire up this UI) and quite poorly on code that has likely never been written (even if it is logically simple for a human). The question is whether this is a quantitative or qualitative barrier. Of course it's still valuable. A real app has plenty of mundane code despite our field's best efforts.
- aspenmartin 7mo ago
  Combinatorial explosion? What do you mean? Again, your experiences are true, but they are improving with each release. The error rate on tasks continues to go down, even novel tasks (as far as we can measure them). Again this is where verifiable domains come in -- whatever problems you can specify the model will improve on them, and this improvement will result in better generalization, and improvements on unseen tasks. This is what I mean by taking your observations of today, ignoring the rate of progress that got us here and the known scaling laws, and then just asserting there will be some fundamental limitation. My point is while this idea may be common, it is not at all supported by literature and the mathematics.
  - sobellian 7mo ago
    The space of programs is incomprehensibly massive. Searching for a program that does what you need is a particularly difficult search problem. In the general case you can't solve search, there's no free lunch. Even scaling laws must bow to NFL. But depending on the type of search problem some heuristics can do well. We know human brains have a heuristic that can program (maybe not particularly well, but passably). To evaluate these agents we can only look at it experimentally, there is no sense in which they are mathematically destined to eventually program well. How good are these types of algorithms at generalization? Are they learning how to code; or are they learning how to code migrations, then learning how to code caches, then learning how to code a command line arg parser, etc? Verifiable domains are interesting. It is unquestionably why agents have come first for coding. But if you've played with claude you may have experienced it short-circuiting failing tests, cheating tests with code that does not generalize, writing meaningless tests, and at long last if you turn it away from all of these it may say something like "honest answer - this feature is really difficult and we should consider a compromise."

nprateem 7mo ago

But the issue isn't coding, it's doing the right thing. I don't see anywhere in your plan some way of staying aligned to core business strategy, forethought, etc. The number of devs will reduce but there will still be large activities that can't be farmed out without an overall strategy

aspenmartin 7mo ago
Why do you think this is a problem? Reasoning is constantly improving, it has ample access to humans to gather more business context, it has access to the same industry data and other signals that humans do, and it can get any data necessary. It has Zoom meeting notes, I mean why do people think there's somehow a fundamental limit beyond coding? The other thing you're missing here is generalizability. Better coding performance (which is verifiable and not limited by human data quality) generalizes performance on other benchmarks. This is a long known phenomenon.
- skydhash 7mo ago
  > Why do you think this is a problem? Because it cannot do it? Every investment has a date where there should be a return on that investment. If there’s no date, it’s a donation of resources (or a waste depending on perspective). You may be OK with continuing to try to make things work. But others aren’t and have decided to invest their finite resources somewhere else.
  - aspenmartin 7mo ago
    > Because it cannot do it? Ah ok so you didn't really read my comment, what is your counter argument? Models are just fundamentally incapable of understanding business context? They are demonstrably already capable of this to a large extent. > Every investment has a date where there should be a return on that investment. If there’s no date, it’s a donation of resources (or a waste depending on perspective). what are you implying here? This convo now turns into the "AI is not profitable and this is a house of cards" theme? That's ok, we can ignore every other business model like say Uber running at a loss to capture what is ultimately an absolutely insane TAM. Little ol' Uber accumuluated ~33B in losses over 14 years, and you're right they tanked and collapsed like a dying star...oh wait...hmm interesting I just looked at their market cap and it's 141 Billion. > You may be OK with continuing to try to make things work. But others aren’t and have decided to invest their finite resources somewhere else. I truly love that. If you want to code as a hobby that is fantastic, and we can go ahead and see in 2 years how your comment ages.
    - skydhash 7mo ago
      > They are demonstrably already capable of this to a large extent. I’d very like to see such demonstration. Where someone hands over a department to an agent and let it makes decisions. > This convo now turns into the "AI is not profitable and this is a house of cards" theme? Where did I say that? I didn’t even mention money, just the broader resource term. A lot of business are mostly running experiments if the current set of tooling can match the marketing (or the hype). They’re not building datacenters or running AI labs. Such experiments can’t run forever.

embedding-shape 7mo ago

> - coding is a verifiable domain You're missing the point though. "1 + 1" vs "one.add(1)" might both be "passable" and correct, but it's missing the forest for the trees, how do you know which one is "long-term the right choice, given what we know?", which is the engineering part of building software, and less about "coding" which tends to be the easy part. How do you evaluate, score and/or benchmark something like that? Currently, I don't think we have any methodologies for this, probably because it's pretty subjective in the end. That's where the "creative" parts of software engineering becomes more important, and it's also way harder to verify.

aspenmartin 7mo ago
because it has business context and better reasoning, and can ask humans for clarification and take direction. You don't need to benchmark this, although it's important. We have clear scaling laws on true statistical performance that is monotonically related to any notion of what performance means. I do benchmarks for a living and can attest: benchmarks are bad, but it doesn't matter for the point I'm trying to make.
- troupo 7mo ago
  > because it has business context It doesn't because it doesn't learn. Every time you run it, it's a new dawn with no knowledge of your business or your business context > better reasoning It doesn't have better reasoning beyond very localized decisions. > and can ask humans for clarification and take direction. And yet it doesn't, no matter how many .md file you throw at it, at crucial places in code. > We have clear scaling laws on true statistical performance that is monotonically related to any notion of what performance means. This is just a bunch of words stringed together, isn't it?
  - skydhash 7mo ago
    Almost every task that people are tackling agents on, it’s either not worth doing, can be done better with scripts and software, or require human oversight (that negates all the advantages.

sobellian 7mo ago

Scaling laws vs combinatorial explosion, who wins? In personal experience claude does exceedingly well on mundane code (do a migration, add a field, wire up this UI) and quite poorly on code that has likely never been written (even if it is logically simple for a human). The question is whether this is a quantitative or qualitative barrier. Of course it's still valuable. A real app has plenty of mundane code despite our field's best efforts.

aspenmartin 7mo ago
Combinatorial explosion? What do you mean? Again, your experiences are true, but they are improving with each release. The error rate on tasks continues to go down, even novel tasks (as far as we can measure them). Again this is where verifiable domains come in -- whatever problems you can specify the model will improve on them, and this improvement will result in better generalization, and improvements on unseen tasks. This is what I mean by taking your observations of today, ignoring the rate of progress that got us here and the known scaling laws, and then just asserting there will be some fundamental limitation. My point is while this idea may be common, it is not at all supported by literature and the mathematics.
- sobellian 7mo ago
  The space of programs is incomprehensibly massive. Searching for a program that does what you need is a particularly difficult search problem. In the general case you can't solve search, there's no free lunch. Even scaling laws must bow to NFL. But depending on the type of search problem some heuristics can do well. We know human brains have a heuristic that can program (maybe not particularly well, but passably). To evaluate these agents we can only look at it experimentally, there is no sense in which they are mathematically destined to eventually program well. How good are these types of algorithms at generalization? Are they learning how to code; or are they learning how to code migrations, then learning how to code caches, then learning how to code a command line arg parser, etc? Verifiable domains are interesting. It is unquestionably why agents have come first for coding. But if you've played with claude you may have experienced it short-circuiting failing tests, cheating tests with code that does not generalize, writing meaningless tests, and at long last if you turn it away from all of these it may say something like "honest answer - this feature is really difficult and we should consider a compromise."