4 ms·

You're right. It's been pretty incredible. It's also frustrating as hell though when people extrapolate from this progress Just because we're here doesn't mean

You're right. It's been pretty incredible. It's also frustrating as hell though when people extrapolate from this progress Just because we're here doesn't mean we're getting to AGI or software developers begging for jobs at Starbucks

simianwords 8mo ago
Sure then make your prediction? It’s always easy to hand wave and dismiss other people’s predictions. But make yours: what do you think llms can do in 2 years?
- AstroBen 8mo ago
  You're asking me to do the thing I just said was frustrating haha. I have no idea. It's a new technology and we have nothing to draw from to make predictions. But for the sake of fun.. New code generation / modification I think we're hitting a point of diminishing returns and they're not going to improve much here The limitation is fundamentally that they can only be as good as the detail in the specs given, or the test harnesses provided to them. Any detail left out they're going to make up, and hopefully it's what you want (often it's not!). If you make the specs detailed enough so that there's no misunderstanding possible: you've just written code, what we already do today Code optimization I think they'll get quite a bit better. If you give them GCC it's probable they'll be able to improve upon it
  - simianwords 8mo ago
    Hmm. It’s not clear what specific task it can’t handle. Can you come up with a concrete example?
    - AstroBen 8mo ago
      Are you saying you've never had them fail at a task? I wanted to refactor a bunch of tests in a TypeScript project the other day into a format similar to table driven tests that are common in Golang, but seemingly not so much in TypeScript. Vitest has specific syntax affordances for it, though It utterly failed at the task. Tried many times with increasing specificity in my prompt, did one myself and used it as an example. I ended up giving up and just doing it manually
      - simianwords 8mo ago
        I see. Did you use Claude code? With access to compiling and running.
        AstroBen 8mo ago
        Codex on high, yeah it had access to compiling/running
        simianwords 8mo ago
        thanks for the data point
  - codethief 8mo ago
    > If you make the specs detailed enough so that there's no misunderstanding possible: you've just written code, what we already do today This was my opinion for a very long time. Having build a few applications from scratch using AI, though, nowadays I think: Sometimes not everything needs to be spelled out. Like in math papers some details can be left to the ~~reader~~LLM and it'll be fine. I mean, in many cases it doesn't really matter what exactly the code looks like, as long as it ends up doing the right thing. For a given Turing machine, the equivalence class of equivalent implementations is infinite. If a short spec written in English leads the LLM to identify the correct equivalence class, that's all we need and, in fact, a very impressive compression result.
    - AstroBen 8mo ago
      Sometimes, yeah. I don't think we're disagreeing What I'd also add: Because of the unspecified behaviour, you're always going to need someone technical that understands the output to verify it. Tests aren't enough I'm not even sure if this is a net productivity benefit. I think it is? Some cases it's a clear win.. but definitely not always. You're reducing time coding and now putting extra into spec writing + review + verification
      - bwfan123 8mo ago
        > Sometimes, yeah. I don't think we're disagreeing I would disagree. Formalism and precision have a critical role to play which is often underestimated. More so with the advent of llms. Fuzziness of natural languages is both a strength and weakness. We have adopted precise but unnatural languages (math/C/C++) for describing machine models of the physical world or of the computing world. Such precision was a real human breakthrough which is often overlooked in these debates.
- kmaitreys 8mo ago
  Something that looks and sounds impressive but in the end not of much substance. This will be true for next 2 years, 4 years, next decade, few decades. Until the state of the art ML paradigm remains language models.

simianwords 8mo ago

Sure then make your prediction? It’s always easy to hand wave and dismiss other people’s predictions. But make yours: what do you think llms can do in 2 years?

AstroBen 8mo ago
You're asking me to do the thing I just said was frustrating haha. I have no idea. It's a new technology and we have nothing to draw from to make predictions. But for the sake of fun.. New code generation / modification I think we're hitting a point of diminishing returns and they're not going to improve much here The limitation is fundamentally that they can only be as good as the detail in the specs given, or the test harnesses provided to them. Any detail left out they're going to make up, and hopefully it's what you want (often it's not!). If you make the specs detailed enough so that there's no misunderstanding possible: you've just written code, what we already do today Code optimization I think they'll get quite a bit better. If you give them GCC it's probable they'll be able to improve upon it
- simianwords 8mo ago
  Hmm. It’s not clear what specific task it can’t handle. Can you come up with a concrete example?
  - AstroBen 8mo ago
    Are you saying you've never had them fail at a task? I wanted to refactor a bunch of tests in a TypeScript project the other day into a format similar to table driven tests that are common in Golang, but seemingly not so much in TypeScript. Vitest has specific syntax affordances for it, though It utterly failed at the task. Tried many times with increasing specificity in my prompt, did one myself and used it as an example. I ended up giving up and just doing it manually
    - simianwords 8mo ago
      I see. Did you use Claude code? With access to compiling and running.
      - AstroBen 8mo ago
        Codex on high, yeah it had access to compiling/running
        simianwords 8mo ago
        thanks for the data point
- codethief 8mo ago
  > If you make the specs detailed enough so that there's no misunderstanding possible: you've just written code, what we already do today This was my opinion for a very long time. Having build a few applications from scratch using AI, though, nowadays I think: Sometimes not everything needs to be spelled out. Like in math papers some details can be left to the ~~reader~~LLM and it'll be fine. I mean, in many cases it doesn't really matter what exactly the code looks like, as long as it ends up doing the right thing. For a given Turing machine, the equivalence class of equivalent implementations is infinite. If a short spec written in English leads the LLM to identify the correct equivalence class, that's all we need and, in fact, a very impressive compression result.
  - AstroBen 8mo ago
    Sometimes, yeah. I don't think we're disagreeing What I'd also add: Because of the unspecified behaviour, you're always going to need someone technical that understands the output to verify it. Tests aren't enough I'm not even sure if this is a net productivity benefit. I think it is? Some cases it's a clear win.. but definitely not always. You're reducing time coding and now putting extra into spec writing + review + verification
    - bwfan123 8mo ago
      > Sometimes, yeah. I don't think we're disagreeing I would disagree. Formalism and precision have a critical role to play which is often underestimated. More so with the advent of llms. Fuzziness of natural languages is both a strength and weakness. We have adopted precise but unnatural languages (math/C/C++) for describing machine models of the physical world or of the computing world. Such precision was a real human breakthrough which is often overlooked in these debates.
kmaitreys 8mo ago
Something that looks and sounds impressive but in the end not of much substance. This will be true for next 2 years, 4 years, next decade, few decades. Until the state of the art ML paradigm remains language models.