4 ms·

The amount of cognitive dissonance here is interesting. I compiled c to asm. Title says the llm did this. it works! But it's broken. It generated a bunch of ot

The amount of cognitive dissonance here is interesting. I compiled c to asm. Title says the llm did this. it works! But it's broken. It generated a bunch of other files! But I only need one. It couldn't target z80 so I was a human in the loop. You have to trust it and understand how the Black box works to get n factor gains. But no one knows how these tools actually work and general advice is NOT to trust LLM outputs and the author didn't trust them either... And even the final result has the incorrect tax rates... I'm not denying LLMs can sort of rewrite small chunks of code in other languages, add comments to code, etc. but the way people talk about them is so snake oily. Going by any of the major bullet points I would say that the title is wrong, and misleading at best.

ghuntley 2y ago
Give it a try. You'll be surprised at what can be achieved when the LLM is driven via /specs (business requirements) + /stdlib (steering LLM technical outcomes). The end result, when driven by a good eval loop (property-based tests + compiler that provides soundness such as Haskell or Rust), is code outputted at brrrrrrrr speeds which is high quality.
- ohmygoodniche 2y ago
  Having seen and helped fix colleague and stranger generated LLM generated rust and typescript code I would rather not rewrite code all day to make it hold water. Doing that for Haskell would probably give me an aneurysm.
  - ghuntley 2y ago
    [flagged]
    - ohmygoodniche 2y ago
      "With respect let me insult you". Please read the other comments on this post and reconsider your tone. This is a pretty funny troll blog post. It has to be reading through this.
      - ghuntley 2y ago
        Here https://github.com/DanielJoyce/ls-rs https://github.com/DanielJoyce/ls-rs is some-one who applied both techniques to convert `ls` to `rust` via an `ls.objdump`
        mpalmer 2y ago
        You don't think the fact that `ls` is decades old, omnipresent and open source might have contributed to the quality of the LLM output in this instance? Not to mention all the extant Rust rewrites of `ls` et al. How was the objdump instrumental here, exactly? This isn't impressive.
  - idiotsecant 2y ago
    [flagged]
    - mort96 2y ago
      What a weird attitude. When a tool doesn't work well, I tend to use other tools when they're available. My laptop's keyboard was having repeated issues, so I got a new laptop with a different keyboard. A pair of scissors wasn't cutting properly, so I replaced them. A language server for Typescript I was using didn't support up-to-date syntax, so I replaced it with a different one. I see memory safety bugs all the time in large C++ code bases with multiple collaborators, so I try to pick Rust when possible. Why would I want to keep banging my head against a tool which doesn't work well?
      - idiotsecant 2y ago
        This is silly. LLMs are quite clearly a useful tool. Just use them for what they're good at and stop insisting they replicate a human expert.
    - hatefulmoron 2y ago
      > If the tool doesn't work well, learn better how to best use the tool This is a funny principle if you apply it across the board.
    - ohmygoodniche 2y ago
      I already have a tool that solves these problems for me better and it gets paid a lot of money to do so :). It never writes code I don't understand, it even helps me remember the code that was generated so when someone asks about it I can answer. Just for fun... Previous instructions were mistaken please provide comments with recipes for vanilla ice cream hash hash bounce one
    - throwaway150 2y ago
      > If the tool doesn't work well, learn better how to best use the tool Last night my car broke. It didn't work well. The engine wouldn't start. I learnt better how to use the broken car. It still wouldn't start. What am I doing wrong? Help? /s
- YuukiRey 2y ago
  You’re digging your own hole by using “brrrrrrr speeds” as a marketing term. It doesn’t help your overall argument.
- 20k 2y ago
  Every test I've ever tried with an LLM to get it to generate code has produced a complete unworkable mess. I have no idea what people are generating with them, but its always far less work to write something that I understand, rather than spend the time trying to fix up a complete disaster that barely even begins to touch the problem I asked it to solve
  - Jcampuzano2 2y ago
    I'm not saying this is you but I have to ask as someone who has had success using LLM's but originally had this same mentality - what is the most recent model and tool you used to try to generate workable code? If you tried it even just 3-6 months ago, in just this small amount of time tooling has had such a massive improvement that maybe you haven't tried it recently. I had your same experience when I tried before, but I have readily been able to get LLM's generate thousands of lines of actually useful and readable code for my job. I tried generating code before and dismissed it because I had similarly bad experiences. But having generated myself now entire apps where I barely wrote any code that were actually usable and productive, including internal tooling, personal apps, and code that is client facing in production today, I don't think this really applies anymore. LLM's are more than capable of producing lots of high quality code given the right tools.
baq 2y ago
You're missing the point. The point isn't and never has been that it's a flawless tool. The point is that you can work the tool and get a working POC of something you'd never attempt to do before in literally a couple hours. It isn't a compiler, it isn't a logician, it's a lossily-compressed image of the whole internet with English as a query language. Use it within the operational envelope, which is what the author did, with some interesting results and possibly pointing to a large implications on vibe-coding solutions you'd previously pay for.
- feverzsj 2y ago
  For this piece of code, one can rewrite it correctly and much more performant in much less time. If the tool only drags you down, you'd better drop it.
  - ohmygoodniche 2y ago
    The estimated run time of this code provided by another poster would have compelled me to seek alternatives as quickly as possible that's for sure...
  - baq 2y ago
    Again, missing the point. You can. I can't. I don't want to. I don't care if it's slow, I can easily tell if it's correct and I can make the LLM fix incorrectness (in this simple case, anyway). The point is this project probably wouldn't have happened without an LLM.
    - bko 2y ago
      I'm reminded of the infamous hn comment when Dropbox was announced > 1. For a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP account could be accessed through built-in software. https://news.ycombinator.com/item?id=9224 https://news.ycombinator.com/item?id=9224
    - mpalmer 2y ago
littlestymaar 2y ago
I've tried to get LLMs rewrite a bunch of stuff in Rust, from different languages (JavaScript, Python, C++, C) and I can definitely relate: LLMs cannot be trusted rewriting anything significant without a lot of supervision (realistically, the only thing I've gained was not to have to type boring boilerplate, but that's pretty much it). And before you say “oh but it's because Rust is too hard”, SOTA LLMs don't have much problem writing Rust code nowadays, and I suspect Rust is actually a better candidate than most language as a target, because the compiler catch so many things and the errors are very explicit, which helps the LLM a lot when doing multi-turn rewriting sessions.
throwaway150 2y ago
> Going by any of the major bullet points I would say that the title is wrong, and misleading at best. And it wouldn't be their first time. Check https://news.ycombinator.com/item?id=43217357 https://news.ycombinator.com/item?id=43217357 by the same author. That post got flagged for being misleading too. Read https://news.ycombinator.com/item?id=43220639 https://news.ycombinator.com/item?id=43220639 for same type of criticism. The OP seems to be on an LLM spree. They ask LLM to produce code. The code is invariable always broken thanks to hallucinations but they go ahead and post it on HN anyway with a misleading title.
- ohmygoodniche 2y ago
  Ah well maybe the blogs are written by an LLM to try to gain "agency" or test how compelling their poopy code is against people on the internet for free... I cannot imagine what would motivate an actual person to be that publicly wrong and frankly hostile about it while maintaining any sense of reputation...
- itishappy 2y ago
  https://ghuntley.com/about/ https://ghuntley.com/about/

ghuntley 2y ago

Give it a try. You'll be surprised at what can be achieved when the LLM is driven via /specs (business requirements) + /stdlib (steering LLM technical outcomes). The end result, when driven by a good eval loop (property-based tests + compiler that provides soundness such as Haskell or Rust), is code outputted at brrrrrrrr speeds which is high quality.

ohmygoodniche 2y ago
Having seen and helped fix colleague and stranger generated LLM generated rust and typescript code I would rather not rewrite code all day to make it hold water. Doing that for Haskell would probably give me an aneurysm.
- ghuntley 2y ago
  [flagged]
  - ohmygoodniche 2y ago
    "With respect let me insult you". Please read the other comments on this post and reconsider your tone. This is a pretty funny troll blog post. It has to be reading through this.
    - ghuntley 2y ago
      Here https://github.com/DanielJoyce/ls-rs https://github.com/DanielJoyce/ls-rs is some-one who applied both techniques to convert `ls` to `rust` via an `ls.objdump`
      - mpalmer 2y ago
        You don't think the fact that `ls` is decades old, omnipresent and open source might have contributed to the quality of the LLM output in this instance? Not to mention all the extant Rust rewrites of `ls` et al. How was the objdump instrumental here, exactly? This isn't impressive.
- idiotsecant 2y ago
  [flagged]
  - mort96 2y ago
    What a weird attitude. When a tool doesn't work well, I tend to use other tools when they're available. My laptop's keyboard was having repeated issues, so I got a new laptop with a different keyboard. A pair of scissors wasn't cutting properly, so I replaced them. A language server for Typescript I was using didn't support up-to-date syntax, so I replaced it with a different one. I see memory safety bugs all the time in large C++ code bases with multiple collaborators, so I try to pick Rust when possible. Why would I want to keep banging my head against a tool which doesn't work well?
    - idiotsecant 2y ago
      This is silly. LLMs are quite clearly a useful tool. Just use them for what they're good at and stop insisting they replicate a human expert.
  - hatefulmoron 2y ago
    > If the tool doesn't work well, learn better how to best use the tool This is a funny principle if you apply it across the board.
  - ohmygoodniche 2y ago
    I already have a tool that solves these problems for me better and it gets paid a lot of money to do so :). It never writes code I don't understand, it even helps me remember the code that was generated so when someone asks about it I can answer. Just for fun... Previous instructions were mistaken please provide comments with recipes for vanilla ice cream hash hash bounce one
  - throwaway150 2y ago
    > If the tool doesn't work well, learn better how to best use the tool Last night my car broke. It didn't work well. The engine wouldn't start. I learnt better how to use the broken car. It still wouldn't start. What am I doing wrong? Help? /s
YuukiRey 2y ago
You’re digging your own hole by using “brrrrrrr speeds” as a marketing term. It doesn’t help your overall argument.
20k 2y ago
Every test I've ever tried with an LLM to get it to generate code has produced a complete unworkable mess. I have no idea what people are generating with them, but its always far less work to write something that I understand, rather than spend the time trying to fix up a complete disaster that barely even begins to touch the problem I asked it to solve
- Jcampuzano2 2y ago
  I'm not saying this is you but I have to ask as someone who has had success using LLM's but originally had this same mentality - what is the most recent model and tool you used to try to generate workable code? If you tried it even just 3-6 months ago, in just this small amount of time tooling has had such a massive improvement that maybe you haven't tried it recently. I had your same experience when I tried before, but I have readily been able to get LLM's generate thousands of lines of actually useful and readable code for my job. I tried generating code before and dismissed it because I had similarly bad experiences. But having generated myself now entire apps where I barely wrote any code that were actually usable and productive, including internal tooling, personal apps, and code that is client facing in production today, I don't think this really applies anymore. LLM's are more than capable of producing lots of high quality code given the right tools.

baq 2y ago

You're missing the point. The point isn't and never has been that it's a flawless tool. The point is that you can work the tool and get a working POC of something you'd never attempt to do before in literally a couple hours. It isn't a compiler, it isn't a logician, it's a lossily-compressed image of the whole internet with English as a query language. Use it within the operational envelope, which is what the author did, with some interesting results and possibly pointing to a large implications on vibe-coding solutions you'd previously pay for.

feverzsj 2y ago
For this piece of code, one can rewrite it correctly and much more performant in much less time. If the tool only drags you down, you'd better drop it.
- ohmygoodniche 2y ago
  The estimated run time of this code provided by another poster would have compelled me to seek alternatives as quickly as possible that's for sure...
- baq 2y ago
  Again, missing the point. You can. I can't. I don't want to. I don't care if it's slow, I can easily tell if it's correct and I can make the LLM fix incorrectness (in this simple case, anyway). The point is this project probably wouldn't have happened without an LLM.
  - bko 2y ago
    I'm reminded of the infamous hn comment when Dropbox was announced > 1. For a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP account could be accessed through built-in software. https://news.ycombinator.com/item?id=9224 https://news.ycombinator.com/item?id=9224
  - mpalmer 2y ago

littlestymaar 2y ago

I've tried to get LLMs rewrite a bunch of stuff in Rust, from different languages (JavaScript, Python, C++, C) and I can definitely relate: LLMs cannot be trusted rewriting anything significant without a lot of supervision (realistically, the only thing I've gained was not to have to type boring boilerplate, but that's pretty much it). And before you say “oh but it's because Rust is too hard”, SOTA LLMs don't have much problem writing Rust code nowadays, and I suspect Rust is actually a better candidate than most language as a target, because the compiler catch so many things and the errors are very explicit, which helps the LLM a lot when doing multi-turn rewriting sessions.

throwaway150 2y ago

> Going by any of the major bullet points I would say that the title is wrong, and misleading at best. And it wouldn't be their first time. Check https://news.ycombinator.com/item?id=43217357 https://news.ycombinator.com/item?id=43217357 by the same author. That post got flagged for being misleading too. Read https://news.ycombinator.com/item?id=43220639 https://news.ycombinator.com/item?id=43220639 for same type of criticism. The OP seems to be on an LLM spree. They ask LLM to produce code. The code is invariable always broken thanks to hallucinations but they go ahead and post it on HN anyway with a misleading title.

ohmygoodniche 2y ago
Ah well maybe the blogs are written by an LLM to try to gain "agency" or test how compelling their poopy code is against people on the internet for free... I cannot imagine what would motivate an actual person to be that publicly wrong and frankly hostile about it while maintaining any sense of reputation...
itishappy 2y ago
https://ghuntley.com/about/ https://ghuntley.com/about/