4 ms·

So far everything I tried indicates that it is superb at coding. Maybe the best yet (though I understand that some of the published benchmarks dispute that). H

So far everything I tried indicates that it is superb at coding. Maybe the best yet (though I understand that some of the published benchmarks dispute that). Here it one-shotted a fully functional LISP interpreter for me: https://everything.intellectronica.net/p/the-little-lisper https://everything.intellectronica.net/p/the-little-lisper

arresin 2y ago
hmm but how many examples of that exist on the internet practically verbatim?
- intellectronica 2y ago
  Not sure that's relevant. Obviously an LLM has to learn from something, but it's not a database. I could also program this myself and I don't think that it's an argument against my coding abilities that I have read the source code to many existing interpreters. I can only do it because I not only read but also understood and internalised.
  - jgalt212 2y ago
    But it's almost a database. See the full glass of wine conundrum.
    - sebzim4500 2y ago
      That seems to only have affected standalone diffusion models. E.g. gpt-4o native image generation is easily able to generate a full glass of wine.
      - jgalt212 2y ago
        not a satisfactory version, from what I saw posted here yesterday. https://news.ycombinator.com/item?id=43475314 https://news.ycombinator.com/item?id=43475314
  - wat10000 2y ago
    It’s not a database, it’s a prediction engine. It’s going to be very good at predicting things it was trained on.
- hadlock 2y ago
  Good point. Turns out OpenAI's llms are pretty decent at coding x86_64 bios bootloaders in assembly, but as soon as you go off script from the two main examples online, it falls apart really quickly, as it's crystal clear it has no idea what is actually going on or the limitations of how bootloaders (and 2, and 3 stage bootloaders) work.
  - hatefulmoron 2y ago
    A while back I wanted to learn more about how direct threaded Forth implementations work. Just explanations, not code. I went back and forth with Claude for a while, and I noticed that its ability to give me a coherent explanation was terrible. However, it was able to give me perfectly fine x86 assembler to implement portions of it -- I knew it was perfectly fine because it was reproducing code from jonesforth, which I had open in another tab.

arresin 2y ago

hmm but how many examples of that exist on the internet practically verbatim?

intellectronica 2y ago
Not sure that's relevant. Obviously an LLM has to learn from something, but it's not a database. I could also program this myself and I don't think that it's an argument against my coding abilities that I have read the source code to many existing interpreters. I can only do it because I not only read but also understood and internalised.
- jgalt212 2y ago
  But it's almost a database. See the full glass of wine conundrum.
  - sebzim4500 2y ago
    That seems to only have affected standalone diffusion models. E.g. gpt-4o native image generation is easily able to generate a full glass of wine.
    - jgalt212 2y ago
      not a satisfactory version, from what I saw posted here yesterday. https://news.ycombinator.com/item?id=43475314 https://news.ycombinator.com/item?id=43475314
- wat10000 2y ago
  It’s not a database, it’s a prediction engine. It’s going to be very good at predicting things it was trained on.
hadlock 2y ago
Good point. Turns out OpenAI's llms are pretty decent at coding x86_64 bios bootloaders in assembly, but as soon as you go off script from the two main examples online, it falls apart really quickly, as it's crystal clear it has no idea what is actually going on or the limitations of how bootloaders (and 2, and 3 stage bootloaders) work.
- hatefulmoron 2y ago
  A while back I wanted to learn more about how direct threaded Forth implementations work. Just explanations, not code. I went back and forth with Claude for a while, and I noticed that its ability to give me a coherent explanation was terrible. However, it was able to give me perfectly fine x86 assembler to implement portions of it -- I knew it was perfectly fine because it was reproducing code from jonesforth, which I had open in another tab.