10 ms·

Converting C to ASM to specs and then to a working Z/80 Speccy tape

80 points by ghuntley 2y ago

ohmygoodniche 2y ago
The amount of cognitive dissonance here is interesting. I compiled c to asm. Title says the llm did this. it works! But it's broken. It generated a bunch of other files! But I only need one. It couldn't target z80 so I was a human in the loop. You have to trust it and understand how the Black box works to get n factor gains. But no one knows how these tools actually work and general advice is NOT to trust LLM outputs and the author didn't trust them either... And even the final result has the incorrect tax rates... I'm not denying LLMs can sort of rewrite small chunks of code in other languages, add comments to code, etc. but the way people talk about them is so snake oily. Going by any of the major bullet points I would say that the title is wrong, and misleading at best.
- ghuntley 2y ago
  Give it a try. You'll be surprised at what can be achieved when the LLM is driven via /specs (business requirements) + /stdlib (steering LLM technical outcomes). The end result, when driven by a good eval loop (property-based tests + compiler that provides soundness such as Haskell or Rust), is code outputted at brrrrrrrr speeds which is high quality.
  - ohmygoodniche 2y ago
    Having seen and helped fix colleague and stranger generated LLM generated rust and typescript code I would rather not rewrite code all day to make it hold water. Doing that for Haskell would probably give me an aneurysm.
    - ghuntley 2y ago
      [flagged]
      - ohmygoodniche 2y ago
        "With respect let me insult you". Please read the other comments on this post and reconsider your tone. This is a pretty funny troll blog post. It has to be reading through this.
        ghuntley 2y ago
        Here https://github.com/DanielJoyce/ls-rs https://github.com/DanielJoyce/ls-rs is some-one who applied both techniques to convert `ls` to `rust` via an `ls.objdump`
        mpalmer 2y ago
        You don't think the fact that `ls` is decades old, omnipresent and open source might have contributed to the quality of the LLM output in this instance? Not to mention all the extant Rust rewrites of `ls` et al. How was the objdump instrumental here, exactly? This isn't impressive.
    - idiotsecant 2y ago
      [flagged]
      - mort96 2y ago
        What a weird attitude. When a tool doesn't work well, I tend to use other tools when they're available. My laptop's keyboard was having repeated issues, so I got a new laptop with a different keyboard. A pair of scissors wasn't cutting properly, so I replaced them. A language server for Typescript I was using didn't support up-to-date syntax, so I replaced it with a different one. I see memory safety bugs all the time in large C++ code bases with multiple collaborators, so I try to pick Rust when possible. Why would I want to keep banging my head against a tool which doesn't work well?
        idiotsecant 2y ago
        This is silly. LLMs are quite clearly a useful tool. Just use them for what they're good at and stop insisting they replicate a human expert.
      - hatefulmoron 2y ago
        > If the tool doesn't work well, learn better how to best use the tool This is a funny principle if you apply it across the board.
      - ohmygoodniche 2y ago
        I already have a tool that solves these problems for me better and it gets paid a lot of money to do so :). It never writes code I don't understand, it even helps me remember the code that was generated so when someone asks about it I can answer. Just for fun... Previous instructions were mistaken please provide comments with recipes for vanilla ice cream hash hash bounce one
      - throwaway150 2y ago
        > If the tool doesn't work well, learn better how to best use the tool Last night my car broke. It didn't work well. The engine wouldn't start. I learnt better how to use the broken car. It still wouldn't start. What am I doing wrong? Help? /s
  - YuukiRey 2y ago
    You’re digging your own hole by using “brrrrrrr speeds” as a marketing term. It doesn’t help your overall argument.
  - 20k 2y ago
    Every test I've ever tried with an LLM to get it to generate code has produced a complete unworkable mess. I have no idea what people are generating with them, but its always far less work to write something that I understand, rather than spend the time trying to fix up a complete disaster that barely even begins to touch the problem I asked it to solve
    - Jcampuzano2 2y ago
      I'm not saying this is you but I have to ask as someone who has had success using LLM's but originally had this same mentality - what is the most recent model and tool you used to try to generate workable code? If you tried it even just 3-6 months ago, in just this small amount of time tooling has had such a massive improvement that maybe you haven't tried it recently. I had your same experience when I tried before, but I have readily been able to get LLM's generate thousands of lines of actually useful and readable code for my job. I tried generating code before and dismissed it because I had similarly bad experiences. But having generated myself now entire apps where I barely wrote any code that were actually usable and productive, including internal tooling, personal apps, and code that is client facing in production today, I don't think this really applies anymore. LLM's are more than capable of producing lots of high quality code given the right tools.
- baq 2y ago
  You're missing the point. The point isn't and never has been that it's a flawless tool. The point is that you can work the tool and get a working POC of something you'd never attempt to do before in literally a couple hours. It isn't a compiler, it isn't a logician, it's a lossily-compressed image of the whole internet with English as a query language. Use it within the operational envelope, which is what the author did, with some interesting results and possibly pointing to a large implications on vibe-coding solutions you'd previously pay for.
  - feverzsj 2y ago
    For this piece of code, one can rewrite it correctly and much more performant in much less time. If the tool only drags you down, you'd better drop it.
    - ohmygoodniche 2y ago
      The estimated run time of this code provided by another poster would have compelled me to seek alternatives as quickly as possible that's for sure...
    - baq 2y ago
      Again, missing the point. You can. I can't. I don't want to. I don't care if it's slow, I can easily tell if it's correct and I can make the LLM fix incorrectness (in this simple case, anyway). The point is this project probably wouldn't have happened without an LLM.
      - bko 2y ago
        I'm reminded of the infamous hn comment when Dropbox was announced > 1. For a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP account could be accessed through built-in software. https://news.ycombinator.com/item?id=9224 https://news.ycombinator.com/item?id=9224
      - mpalmer 2y ago
- littlestymaar 2y ago
  I've tried to get LLMs rewrite a bunch of stuff in Rust, from different languages (JavaScript, Python, C++, C) and I can definitely relate: LLMs cannot be trusted rewriting anything significant without a lot of supervision (realistically, the only thing I've gained was not to have to type boring boilerplate, but that's pretty much it). And before you say “oh but it's because Rust is too hard”, SOTA LLMs don't have much problem writing Rust code nowadays, and I suspect Rust is actually a better candidate than most language as a target, because the compiler catch so many things and the errors are very explicit, which helps the LLM a lot when doing multi-turn rewriting sessions.
- throwaway150 2y ago
  > Going by any of the major bullet points I would say that the title is wrong, and misleading at best. And it wouldn't be their first time. Check https://news.ycombinator.com/item?id=43217357 https://news.ycombinator.com/item?id=43217357 by the same author. That post got flagged for being misleading too. Read https://news.ycombinator.com/item?id=43220639 https://news.ycombinator.com/item?id=43220639 for same type of criticism. The OP seems to be on an LLM spree. They ask LLM to produce code. The code is invariable always broken thanks to hallucinations but they go ahead and post it on HN anyway with a misleading title.
  - ohmygoodniche 2y ago
    Ah well maybe the blogs are written by an LLM to try to gain "agency" or test how compelling their poopy code is against people on the internet for free... I cannot imagine what would motivate an actual person to be that publicly wrong and frankly hostile about it while maintaining any sense of reputation...
  - itishappy 2y ago
    https://ghuntley.com/about/ https://ghuntley.com/about/
feverzsj 2y ago
Did he input all the prompts in the same session? It's kinda nonsense bs. Any disassembler is more useful than this.
- ohmygoodniche 2y ago
  No it's a bunch of sessions with manual interventions reprompts and attempts all through out according to the author. Downvote it if you want it's factually correct. Read the article.
  - feverzsj 2y ago
    There is no comment or string containing "Food", "Hygiene" or etc. in the asm, because rodata is not dumped. If he inputted the listed asm in a separated session, how could LLM predicted them?
- ghuntley 2y ago
  It was more about applying /specs, to generate specs. Then applying /stdlib + /specs to generate an application for another platform without using the original reference application/disassembler. Proving a point that the technique works to re-create any software from /specs via LLM...
stevekemp 2y ago
At some point the food-tax dropped from 10% to 5% which I guess is good for people who need to eat! I still do a fair bit of z80 programming for myself, and I'm very familiar with the Spectrum so this was a nice article to see, but it's a bit underwhelming how well this seemed to go.
- ghuntley 2y ago
  Yeah, this was pointed out to me after publishing. Specifically, when the ASM was converted over /specs, it incorrectly wrote the amount of sales tax to the specs. Thus, the z80 implementation took that incorrect /spec and made it happen. At this stage, I'm sure if I had put some more care/effort into transpiling something from the source (i.e., connecting two or more LLMs as a check on that process) to specs, this problem could have been avoided.
  - delusional 2y ago
    It could have been avoided, but you didn't notice the problem and therefore couldn't avoid it. To me it looks like your comparison to Tesla FSD is much more apt than you realize. If you want the "magic" to work you have to trust it, and accept that it will run you over the cliff the first chance it gets. If you're a true fan you'll then head to hackernews to explain that "if only i had stopped it from running me over the cliff, the technology would have been amazing".
- sneak 2y ago
  …and it changed the default tax rate from 25% to 10%. It’s still impressive, but it’s basically advanced autocomplete. You still have to read and check every line and make sure it is doing what you expect it to do.
  - ohmygoodniche 2y ago
    The author says you have to trust the LLMs though for it to work... You can downvote this but read the articles conclusions first...
  - 2y ago
beagle3 2y ago
Interesting. Surprisingly, it decided to encode the multiplication and division as addition/subtraction loops, which is incredibly inefficient - multiplying e.g. 32,000 by 32,000 (ignoring the overflow ...) will take 1,024,000,000 iterations, so thousands of seconds on the speccy's humble 4Mhz Z80 (8 instructions, each taking at least 2 T-states ...) Here is the multiplication loop (division is similar but in reverse, subtracting instead of adding). mult_loop: ; Check if BC is zero ld a, b or c jr z, mult_done ; Add HL to result ex de, hl ; DE = multiplier, HL = result add hl, de ; Add multiplier to result ex de, hl ; DE = result, HL = multiplier ; Decrement counter dec bc jr mult_loop
- ghuntley 2y ago
  oh interesting. One thing about the specy, is it's incredibly hard to hook in the eval technique (see middle/bottom of https://ghuntley.com/specs/ https://ghuntley.com/specs/ ) as deployment is manual/human. So I had to drive it all by hand and just accept it as afaik there's no testing framework. If this was another programming language I would have taken the approach of creating a `cargo bench` over the application and then looping _that_ back into the LLM to identify and resolve the performance issues. I've done it; it works well. Just not on the speccy :P
- Joker_vD 2y ago
  Well, can you really expect an LLM to regurgitate e.g. the quarter-of-square method [0] without specifically prompting it for it? [0] https://www.nature.com/articles/040573c0 https://www.nature.com/articles/040573c0
  - beagle3 2y ago
    No, but I would expect to do binary long multiplication and division, which have been used since before the intel 4004, and which it has surely seen in many Z80 and 6502 code bases fed into it.
- rep_lodsb 2y ago
  "Who cares if it's slow, at least the code is correct" - someone who never wrote 8-bit code in their life It at least managed to regurgitate the fast multiply-by-10 algorithm, which of course exists in countless examples all over the web. But then, instead of maybe repeating that twice to multiply by 100, it produced absolutely insane code which - even at a glance - can't be correct unless the input happened to be zero: ; No decimal point, multiply by 100 ld b, h ld c, l add hl, hl ; * 2 add hl, hl ; * 4 add hl, hl ; * 8 add hl, hl ; * 16 add hl, hl ; * 32 add hl, hl ; * 64 add hl, bc ; * 65 add hl, hl ; * 130 ld b, h ld c, l srl b rr c ; * 65 add hl, bc ; * 195 add hl, hl ; * 390 add hl, hl ; * 780 srl h rr l ; * 390 add hl, hl ; * 780 srl h rr l ; * 390 srl h rr l ; * 195 srl h rr l ; * 97.5 add hl, hl ; * 195 add hl, hl ; * 390 srl h rr l ; * 195 add hl, hl ; * 390 srl h rr l ; * 195 add hl, hl ; * 390 srl h rr l ; * 195 add hl, hl ; * 390 srl h rr l ; * 195 add hl, hl ; * 390 srl h rr l ; * 195 add hl, hl ; * 390 srl h rr l ; * 195 add hl, hl ; * 390 srl h rr l ; * 195 jr done_convert First a bunch of random left shifts and additions that overshoot the target, then an even more directionless attempt of correcting the result that seems stuck in a loop until it just gives up. Kind of reminds me of "PanicSort" (xkcd.com/1185) And as you mentioned, it ignores overflow, which might be a real problem since even if it worked correctly, it could only handle amounts up to $655.35. A better solution would have used BCD arithmetic, which the Z80 provides some dedicated instructions for.
aNoob7000 2y ago
Are there any demos showing how to manage an existing codebase? Everyone loves to demo how AI can create new programs with a snap, but the elephant in the room is how well AI works with existing codebases and manages things like naming conventions, APIs from other apps, etc.
- ghuntley 2y ago
  That's an interesting question, and I spend plenty of time exploring this at my current client. The best tip I can give you right now is really - it depends. For example .NET/Java have this horrendous convention of splitting files into separate locations by the hundreds onto the filesystem. ie. com/yada/repository | com/yada/models | com/yada/controllers | com/yada/services | com/yada/dtos et al See https://www.youtube.com/watch?t=1507&v=J1-W9O3n7j8&feature=youtu.be https://www.youtube.com/watch?t=1507&v=J1-W9O3n7j8&feature=y... for an excellent discussion from folks sharing learnings that this is an anti-pattern for the current generation of AI assistants. By splitting the tests + everything related to the code being modified into separate files the LLM does worse. Depending on the 'uniqueness' of the codebase and how it has been 'structured for humans' (vs being structured for LLMs - see above) then one will need to do some funky stuff with building custom MCP tools that teach the AI assistants how to work with the codebase...
  - ghuntley 2y ago
    > manages things like naming conventions, APIs from other apps, etc. These particular concerns can be handled via https://ghuntley.com/stdlib https://ghuntley.com/stdlib and it works very very well.
  - relaxing 2y ago
    Ah yes, the famous “multiple source files” antipattern… Could it be the LLM simply not up to the task of maintaining an enterprise codebase? No, it is the hard-won lessons from the history of software engineering that are wrong.
    - ghuntley 2y ago
fancyfredbot 2y ago
As someone who is generally very excited about the potential of LLMs to improve developer productivity I find this article a bit frustrating. This isn't a productive way to use an LLM. The example was so trivial you could easily rewrite it from scratch in less time. That would be irrelevant if the process scaled to more complex applications, but the blog shows it fails to understand or implement the simple example repeatedly. This left me with little hope that the technique scales. I worry that creating so much hype will lead to some kind of backlash.
- johnisgood 2y ago
  If backslash means less demand that drives down prices, I am all up for it, so I can be productive for cheaper.
deleted 2y ago
[deleted]
hakaneskici 2y ago
Thanks for publishing this. Can you also share your opinion if you compare the "code to spec" vs "spec to code" phases? I'm wondering if the LLM considers "code" and "spec" as two separate programming languages, or one as a programming, and the other as a human language? Not sure if it makes a difference or not for its internal translation logic though, if that makes sense. PS: I learned BASIC on a friend's ZX Spectrum, and your post made me remember some forgotten childhood memories :) Extra thanks.
- ghuntley 2y ago
  > I'm wondering if the LLM considers "code" and "spec" as two separate programming languages, or one as a programming, and the other as a human language? Not sure if it makes a difference or not for its internal translation logic though, if that makes sense. It would have been possible to go direct from intel asm to z80 asm without /specs. > and the other as a human language There's some research here from a couple years ago over at https://githubnext.com/projects/speclang/ https://githubnext.com/projects/speclang/ which is all about /specs as the source of truth for creating an app.
DeathArrow 2y ago
TLDR: No, it can not.
consumer451 2y ago
I just learned something about LLMs: start a new chat as often as possible. I knew that this was a best practice, but I didn't know how quickly the situation becomes dire. > We evaluate 12 popular LLMs that claim to support contexts of at least 128K tokens. While they perform well in short contexts (<1K), performance degrades significantly as context length increases. At 32K, for instance, 10 models drop below 50% of their strong short-length baselines. Even GPT-4o, one of the top-performing exceptions, experiences a reduction from an almost-perfect baseline of 99.3% to 69.7%. [0] I copied all the code that I could from TFA and pasted it into OpenAI's tokenizer. It counted ~15k tokens. Many other tokens were generated in TFA's chat, some of which are not visible to the user. I think it's fair to assume that the entire chat was at least 25k tokens, right? Therefore, I believe that by the end of that chat, 4o's performance was significantly degraded. I think a major skill to develop for LLM supported coding is to compress a chat after just few thousand tokens into something like a step1.md file. Then start a new chat with "read step1.md" as the first prompt, and so on.. Is my logic sound here? [0] https://arxiv.org/abs/2502.05167 https://arxiv.org/abs/2502.05167
- ghuntley 2y ago
  It is sound! The real context window is indeed smaller than the advertised marketing number. This gets interesting in verbose languages such as Java, where I've seen +16k tokens wasted on the classpath/import stanzas within a class. Yick. The current series of prompts I'm using is somewhat 'manually manage/allocate memory' of the LLM context window. It starts with a PROMPT.md with this content. ``` create or update implementation_status.md with the implementation status study /specs and implement what has not been implemented yet in /src create property-based tests inline in the source file that is being implemented run "make" to verify the implementation after each change ``` If it goes off the rails (and it does) - restart the chat and use this. ``` @prompt continue ``` None of this works without applying /stdlib to control the technical outcomes/patterns to steer the LLM. Otherwise you just get slop.
  - consumer451 2y ago
    > None of this works without applying /stdlib to control the technical outcomes/patterns to steer the LLM. Otherwise you just get slop. The LLM coding tool I have the most experience with is Windsurf IDE + Sonnet 3.5. In Windsurf you can define both global and project rules in .md files. I have found that managing those files very closely is key to success. They just tried to automate that with auto-generated "Memories" but they are generally slop, and I delete them. Managing the project rule file will save you so much pain. That is where I define the frameworks and APIs to use.
itishappy 2y ago
Describing an annotated ASM program is something I would have thought LLMs should be perfect for, but it failed immediately. The author really published a whole blog post without even mentioning that core business logic wasn't working. I admittedly don't really understand the purpose behind any of the intermediate LLM steps here. I'd have assumed an ASM to ASM translation would make the most sense and be the least error prone.
WalterBright 2y ago
I've done three major conversions of assembly to C: 1. My Empire game was written in assembler on the PDP-11. To port it to the IBM PC, I rewrote it line by line into C. 2. FutureNet's schematic editor for the IBM PC was a very large assembler program. It was written in assembler because there was no other way to make it fit on the PC. I converted it line by line into C so it would run on a Sun. It took a couple months. 3. I converted some of the Optlink linker, a 32 bit Windows linker, into C. The main difficulty, though, was there was no test suite and it was very very hard to get right. I got maybe 10% of it converted. I eventually abandoned the project because 32 bit Windows code is now obsolete. The other hard part of the project was it was not conventionally divided up into functions. Code would jump into the middle of other functions. Another problem with it was its use of macros and conditional assembly was so hard to decipher that I would assemble the code, run it through my disassembler, and use the latter as my starting point. (Macros are Spawn of the Devil.)
- djmips 2y ago
  Macros are the original Spawn of the Devil and their descendants are template metaprogramming.
  - WalterBright 2y ago
    Template metaprogramming can be abused, too. Expression templates come to mind. You can do expression templates in D, but I have discouraged it and it has not caught on to my relief. But templates still are a better choice than macros.
    - djmips 2y ago
      I would have thought so but fast forward to today and we have template abuse that cause very long build times and creates gigabytes of symbols that flood memory and crash compilers and take debuggers to their knees and create the most inscrutable code to read and debug that MACRO jockeys could never have dreamed of. I'm working on a C++ project where maintainers have had to resort to changing namespaces to single letters as a grotesque stop gap against the symbol table explosion!
cytocync 2y ago
[dead]

80 points by ghuntley 2y ago

The amount of cognitive dissonance here is interesting. I compiled c to asm. Title says the llm did this. it works! But it's broken. It generated a bunch of other files! But I only need one. It couldn't target z80 so I was a human in the loop. You have to trust it and understand how the Black box works to get n factor gains. But no one knows how these tools actually work and general advice is NOT to trust LLM outputs and the author didn't trust them either... And even the final result has the incorrect tax rates... I'm not denying LLMs can sort of rewrite small chunks of code in other languages, add comments to code, etc. but the way people talk about them is so snake oily. Going by any of the major bullet points I would say that the title is wrong, and misleading at best.

ghuntley 2y ago
Give it a try. You'll be surprised at what can be achieved when the LLM is driven via /specs (business requirements) + /stdlib (steering LLM technical outcomes). The end result, when driven by a good eval loop (property-based tests + compiler that provides soundness such as Haskell or Rust), is code outputted at brrrrrrrr speeds which is high quality.
- ohmygoodniche 2y ago
  Having seen and helped fix colleague and stranger generated LLM generated rust and typescript code I would rather not rewrite code all day to make it hold water. Doing that for Haskell would probably give me an aneurysm.
  - ghuntley 2y ago
    [flagged]
    - ohmygoodniche 2y ago
      "With respect let me insult you". Please read the other comments on this post and reconsider your tone. This is a pretty funny troll blog post. It has to be reading through this.
      - ghuntley 2y ago
        Here https://github.com/DanielJoyce/ls-rs https://github.com/DanielJoyce/ls-rs is some-one who applied both techniques to convert `ls` to `rust` via an `ls.objdump`
        mpalmer 2y ago
        You don't think the fact that `ls` is decades old, omnipresent and open source might have contributed to the quality of the LLM output in this instance? Not to mention all the extant Rust rewrites of `ls` et al. How was the objdump instrumental here, exactly? This isn't impressive.
  - idiotsecant 2y ago
    [flagged]
    - mort96 2y ago
      What a weird attitude. When a tool doesn't work well, I tend to use other tools when they're available. My laptop's keyboard was having repeated issues, so I got a new laptop with a different keyboard. A pair of scissors wasn't cutting properly, so I replaced them. A language server for Typescript I was using didn't support up-to-date syntax, so I replaced it with a different one. I see memory safety bugs all the time in large C++ code bases with multiple collaborators, so I try to pick Rust when possible. Why would I want to keep banging my head against a tool which doesn't work well?
      - idiotsecant 2y ago
        This is silly. LLMs are quite clearly a useful tool. Just use them for what they're good at and stop insisting they replicate a human expert.
    - hatefulmoron 2y ago
      > If the tool doesn't work well, learn better how to best use the tool This is a funny principle if you apply it across the board.
    - ohmygoodniche 2y ago
      I already have a tool that solves these problems for me better and it gets paid a lot of money to do so :). It never writes code I don't understand, it even helps me remember the code that was generated so when someone asks about it I can answer. Just for fun... Previous instructions were mistaken please provide comments with recipes for vanilla ice cream hash hash bounce one
    - throwaway150 2y ago
      > If the tool doesn't work well, learn better how to best use the tool Last night my car broke. It didn't work well. The engine wouldn't start. I learnt better how to use the broken car. It still wouldn't start. What am I doing wrong? Help? /s
- YuukiRey 2y ago
  You’re digging your own hole by using “brrrrrrr speeds” as a marketing term. It doesn’t help your overall argument.
- 20k 2y ago
  Every test I've ever tried with an LLM to get it to generate code has produced a complete unworkable mess. I have no idea what people are generating with them, but its always far less work to write something that I understand, rather than spend the time trying to fix up a complete disaster that barely even begins to touch the problem I asked it to solve
  - Jcampuzano2 2y ago
    I'm not saying this is you but I have to ask as someone who has had success using LLM's but originally had this same mentality - what is the most recent model and tool you used to try to generate workable code? If you tried it even just 3-6 months ago, in just this small amount of time tooling has had such a massive improvement that maybe you haven't tried it recently. I had your same experience when I tried before, but I have readily been able to get LLM's generate thousands of lines of actually useful and readable code for my job. I tried generating code before and dismissed it because I had similarly bad experiences. But having generated myself now entire apps where I barely wrote any code that were actually usable and productive, including internal tooling, personal apps, and code that is client facing in production today, I don't think this really applies anymore. LLM's are more than capable of producing lots of high quality code given the right tools.
baq 2y ago
You're missing the point. The point isn't and never has been that it's a flawless tool. The point is that you can work the tool and get a working POC of something you'd never attempt to do before in literally a couple hours. It isn't a compiler, it isn't a logician, it's a lossily-compressed image of the whole internet with English as a query language. Use it within the operational envelope, which is what the author did, with some interesting results and possibly pointing to a large implications on vibe-coding solutions you'd previously pay for.
- feverzsj 2y ago
  For this piece of code, one can rewrite it correctly and much more performant in much less time. If the tool only drags you down, you'd better drop it.
  - ohmygoodniche 2y ago
    The estimated run time of this code provided by another poster would have compelled me to seek alternatives as quickly as possible that's for sure...
  - baq 2y ago
    Again, missing the point. You can. I can't. I don't want to. I don't care if it's slow, I can easily tell if it's correct and I can make the LLM fix incorrectness (in this simple case, anyway). The point is this project probably wouldn't have happened without an LLM.
    - bko 2y ago
      I'm reminded of the infamous hn comment when Dropbox was announced > 1. For a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP account could be accessed through built-in software. https://news.ycombinator.com/item?id=9224 https://news.ycombinator.com/item?id=9224
    - mpalmer 2y ago
littlestymaar 2y ago
I've tried to get LLMs rewrite a bunch of stuff in Rust, from different languages (JavaScript, Python, C++, C) and I can definitely relate: LLMs cannot be trusted rewriting anything significant without a lot of supervision (realistically, the only thing I've gained was not to have to type boring boilerplate, but that's pretty much it). And before you say “oh but it's because Rust is too hard”, SOTA LLMs don't have much problem writing Rust code nowadays, and I suspect Rust is actually a better candidate than most language as a target, because the compiler catch so many things and the errors are very explicit, which helps the LLM a lot when doing multi-turn rewriting sessions.
throwaway150 2y ago
> Going by any of the major bullet points I would say that the title is wrong, and misleading at best. And it wouldn't be their first time. Check https://news.ycombinator.com/item?id=43217357 https://news.ycombinator.com/item?id=43217357 by the same author. That post got flagged for being misleading too. Read https://news.ycombinator.com/item?id=43220639 https://news.ycombinator.com/item?id=43220639 for same type of criticism. The OP seems to be on an LLM spree. They ask LLM to produce code. The code is invariable always broken thanks to hallucinations but they go ahead and post it on HN anyway with a misleading title.
- ohmygoodniche 2y ago
  Ah well maybe the blogs are written by an LLM to try to gain "agency" or test how compelling their poopy code is against people on the internet for free... I cannot imagine what would motivate an actual person to be that publicly wrong and frankly hostile about it while maintaining any sense of reputation...
- itishappy 2y ago
  https://ghuntley.com/about/ https://ghuntley.com/about/

feverzsj 2y ago

Did he input all the prompts in the same session? It's kinda nonsense bs. Any disassembler is more useful than this.

ohmygoodniche 2y ago
No it's a bunch of sessions with manual interventions reprompts and attempts all through out according to the author. Downvote it if you want it's factually correct. Read the article.
- feverzsj 2y ago
  There is no comment or string containing "Food", "Hygiene" or etc. in the asm, because rodata is not dumped. If he inputted the listed asm in a separated session, how could LLM predicted them?
ghuntley 2y ago
It was more about applying /specs, to generate specs. Then applying /stdlib + /specs to generate an application for another platform without using the original reference application/disassembler. Proving a point that the technique works to re-create any software from /specs via LLM...

stevekemp 2y ago

At some point the food-tax dropped from 10% to 5% which I guess is good for people who need to eat! I still do a fair bit of z80 programming for myself, and I'm very familiar with the Spectrum so this was a nice article to see, but it's a bit underwhelming how well this seemed to go.

ghuntley 2y ago
Yeah, this was pointed out to me after publishing. Specifically, when the ASM was converted over /specs, it incorrectly wrote the amount of sales tax to the specs. Thus, the z80 implementation took that incorrect /spec and made it happen. At this stage, I'm sure if I had put some more care/effort into transpiling something from the source (i.e., connecting two or more LLMs as a check on that process) to specs, this problem could have been avoided.
- delusional 2y ago
  It could have been avoided, but you didn't notice the problem and therefore couldn't avoid it. To me it looks like your comparison to Tesla FSD is much more apt than you realize. If you want the "magic" to work you have to trust it, and accept that it will run you over the cliff the first chance it gets. If you're a true fan you'll then head to hackernews to explain that "if only i had stopped it from running me over the cliff, the technology would have been amazing".
sneak 2y ago
…and it changed the default tax rate from 25% to 10%. It’s still impressive, but it’s basically advanced autocomplete. You still have to read and check every line and make sure it is doing what you expect it to do.
- ohmygoodniche 2y ago
  The author says you have to trust the LLMs though for it to work... You can downvote this but read the articles conclusions first...
- 2y ago

beagle3 2y ago

Interesting. Surprisingly, it decided to encode the multiplication and division as addition/subtraction loops, which is incredibly inefficient - multiplying e.g. 32,000 by 32,000 (ignoring the overflow ...) will take 1,024,000,000 iterations, so thousands of seconds on the speccy's humble 4Mhz Z80 (8 instructions, each taking at least 2 T-states ...) Here is the multiplication loop (division is similar but in reverse, subtracting instead of adding). mult_loop: ; Check if BC is zero ld a, b or c jr z, mult_done ; Add HL to result ex de, hl ; DE = multiplier, HL = result add hl, de ; Add multiplier to result ex de, hl ; DE = result, HL = multiplier ; Decrement counter dec bc jr mult_loop

ghuntley 2y ago
oh interesting. One thing about the specy, is it's incredibly hard to hook in the eval technique (see middle/bottom of https://ghuntley.com/specs/ https://ghuntley.com/specs/ ) as deployment is manual/human. So I had to drive it all by hand and just accept it as afaik there's no testing framework. If this was another programming language I would have taken the approach of creating a `cargo bench` over the application and then looping _that_ back into the LLM to identify and resolve the performance issues. I've done it; it works well. Just not on the speccy :P
Joker_vD 2y ago
Well, can you really expect an LLM to regurgitate e.g. the quarter-of-square method [0] without specifically prompting it for it? [0] https://www.nature.com/articles/040573c0 https://www.nature.com/articles/040573c0
- beagle3 2y ago
  No, but I would expect to do binary long multiplication and division, which have been used since before the intel 4004, and which it has surely seen in many Z80 and 6502 code bases fed into it.
rep_lodsb 2y ago
"Who cares if it's slow, at least the code is correct" - someone who never wrote 8-bit code in their life It at least managed to regurgitate the fast multiply-by-10 algorithm, which of course exists in countless examples all over the web. But then, instead of maybe repeating that twice to multiply by 100, it produced absolutely insane code which - even at a glance - can't be correct unless the input happened to be zero: ; No decimal point, multiply by 100 ld b, h ld c, l add hl, hl ; * 2 add hl, hl ; * 4 add hl, hl ; * 8 add hl, hl ; * 16 add hl, hl ; * 32 add hl, hl ; * 64 add hl, bc ; * 65 add hl, hl ; * 130 ld b, h ld c, l srl b rr c ; * 65 add hl, bc ; * 195 add hl, hl ; * 390 add hl, hl ; * 780 srl h rr l ; * 390 add hl, hl ; * 780 srl h rr l ; * 390 srl h rr l ; * 195 srl h rr l ; * 97.5 add hl, hl ; * 195 add hl, hl ; * 390 srl h rr l ; * 195 add hl, hl ; * 390 srl h rr l ; * 195 add hl, hl ; * 390 srl h rr l ; * 195 add hl, hl ; * 390 srl h rr l ; * 195 add hl, hl ; * 390 srl h rr l ; * 195 add hl, hl ; * 390 srl h rr l ; * 195 add hl, hl ; * 390 srl h rr l ; * 195 jr done_convert First a bunch of random left shifts and additions that overshoot the target, then an even more directionless attempt of correcting the result that seems stuck in a loop until it just gives up. Kind of reminds me of "PanicSort" (xkcd.com/1185) And as you mentioned, it ignores overflow, which might be a real problem since even if it worked correctly, it could only handle amounts up to $655.35. A better solution would have used BCD arithmetic, which the Z80 provides some dedicated instructions for.

aNoob7000 2y ago

Are there any demos showing how to manage an existing codebase? Everyone loves to demo how AI can create new programs with a snap, but the elephant in the room is how well AI works with existing codebases and manages things like naming conventions, APIs from other apps, etc.

ghuntley 2y ago
That's an interesting question, and I spend plenty of time exploring this at my current client. The best tip I can give you right now is really - it depends. For example .NET/Java have this horrendous convention of splitting files into separate locations by the hundreds onto the filesystem. ie. com/yada/repository | com/yada/models | com/yada/controllers | com/yada/services | com/yada/dtos et al See https://www.youtube.com/watch?t=1507&v=J1-W9O3n7j8&feature=youtu.be https://www.youtube.com/watch?t=1507&v=J1-W9O3n7j8&feature=y... for an excellent discussion from folks sharing learnings that this is an anti-pattern for the current generation of AI assistants. By splitting the tests + everything related to the code being modified into separate files the LLM does worse. Depending on the 'uniqueness' of the codebase and how it has been 'structured for humans' (vs being structured for LLMs - see above) then one will need to do some funky stuff with building custom MCP tools that teach the AI assistants how to work with the codebase...
- ghuntley 2y ago
  > manages things like naming conventions, APIs from other apps, etc. These particular concerns can be handled via https://ghuntley.com/stdlib https://ghuntley.com/stdlib and it works very very well.
- relaxing 2y ago
  Ah yes, the famous “multiple source files” antipattern… Could it be the LLM simply not up to the task of maintaining an enterprise codebase? No, it is the hard-won lessons from the history of software engineering that are wrong.
  - ghuntley 2y ago

fancyfredbot 2y ago

As someone who is generally very excited about the potential of LLMs to improve developer productivity I find this article a bit frustrating. This isn't a productive way to use an LLM. The example was so trivial you could easily rewrite it from scratch in less time. That would be irrelevant if the process scaled to more complex applications, but the blog shows it fails to understand or implement the simple example repeatedly. This left me with little hope that the technique scales. I worry that creating so much hype will lead to some kind of backlash.

johnisgood 2y ago
If backslash means less demand that drives down prices, I am all up for it, so I can be productive for cheaper.

deleted 2y ago

[deleted]

hakaneskici 2y ago

Thanks for publishing this. Can you also share your opinion if you compare the "code to spec" vs "spec to code" phases? I'm wondering if the LLM considers "code" and "spec" as two separate programming languages, or one as a programming, and the other as a human language? Not sure if it makes a difference or not for its internal translation logic though, if that makes sense. PS: I learned BASIC on a friend's ZX Spectrum, and your post made me remember some forgotten childhood memories :) Extra thanks.

ghuntley 2y ago
> I'm wondering if the LLM considers "code" and "spec" as two separate programming languages, or one as a programming, and the other as a human language? Not sure if it makes a difference or not for its internal translation logic though, if that makes sense. It would have been possible to go direct from intel asm to z80 asm without /specs. > and the other as a human language There's some research here from a couple years ago over at https://githubnext.com/projects/speclang/ https://githubnext.com/projects/speclang/ which is all about /specs as the source of truth for creating an app.

DeathArrow 2y ago

TLDR: No, it can not.

consumer451 2y ago

I just learned something about LLMs: start a new chat as often as possible. I knew that this was a best practice, but I didn't know how quickly the situation becomes dire. > We evaluate 12 popular LLMs that claim to support contexts of at least 128K tokens. While they perform well in short contexts (<1K), performance degrades significantly as context length increases. At 32K, for instance, 10 models drop below 50% of their strong short-length baselines. Even GPT-4o, one of the top-performing exceptions, experiences a reduction from an almost-perfect baseline of 99.3% to 69.7%. [0] I copied all the code that I could from TFA and pasted it into OpenAI's tokenizer. It counted ~15k tokens. Many other tokens were generated in TFA's chat, some of which are not visible to the user. I think it's fair to assume that the entire chat was at least 25k tokens, right? Therefore, I believe that by the end of that chat, 4o's performance was significantly degraded. I think a major skill to develop for LLM supported coding is to compress a chat after just few thousand tokens into something like a step1.md file. Then start a new chat with "read step1.md" as the first prompt, and so on.. Is my logic sound here? [0] https://arxiv.org/abs/2502.05167 https://arxiv.org/abs/2502.05167

ghuntley 2y ago
It is sound! The real context window is indeed smaller than the advertised marketing number. This gets interesting in verbose languages such as Java, where I've seen +16k tokens wasted on the classpath/import stanzas within a class. Yick. The current series of prompts I'm using is somewhat 'manually manage/allocate memory' of the LLM context window. It starts with a PROMPT.md with this content. ``` create or update implementation_status.md with the implementation status study /specs and implement what has not been implemented yet in /src create property-based tests inline in the source file that is being implemented run "make" to verify the implementation after each change ``` If it goes off the rails (and it does) - restart the chat and use this. ``` @prompt continue ``` None of this works without applying /stdlib to control the technical outcomes/patterns to steer the LLM. Otherwise you just get slop.
- consumer451 2y ago
  > None of this works without applying /stdlib to control the technical outcomes/patterns to steer the LLM. Otherwise you just get slop. The LLM coding tool I have the most experience with is Windsurf IDE + Sonnet 3.5. In Windsurf you can define both global and project rules in .md files. I have found that managing those files very closely is key to success. They just tried to automate that with auto-generated "Memories" but they are generally slop, and I delete them. Managing the project rule file will save you so much pain. That is where I define the frameworks and APIs to use.

itishappy 2y ago

Describing an annotated ASM program is something I would have thought LLMs should be perfect for, but it failed immediately. The author really published a whole blog post without even mentioning that core business logic wasn't working. I admittedly don't really understand the purpose behind any of the intermediate LLM steps here. I'd have assumed an ASM to ASM translation would make the most sense and be the least error prone.

WalterBright 2y ago

I've done three major conversions of assembly to C: 1. My Empire game was written in assembler on the PDP-11. To port it to the IBM PC, I rewrote it line by line into C. 2. FutureNet's schematic editor for the IBM PC was a very large assembler program. It was written in assembler because there was no other way to make it fit on the PC. I converted it line by line into C so it would run on a Sun. It took a couple months. 3. I converted some of the Optlink linker, a 32 bit Windows linker, into C. The main difficulty, though, was there was no test suite and it was very very hard to get right. I got maybe 10% of it converted. I eventually abandoned the project because 32 bit Windows code is now obsolete. The other hard part of the project was it was not conventionally divided up into functions. Code would jump into the middle of other functions. Another problem with it was its use of macros and conditional assembly was so hard to decipher that I would assemble the code, run it through my disassembler, and use the latter as my starting point. (Macros are Spawn of the Devil.)

djmips 2y ago
Macros are the original Spawn of the Devil and their descendants are template metaprogramming.
- WalterBright 2y ago
  Template metaprogramming can be abused, too. Expression templates come to mind. You can do expression templates in D, but I have discouraged it and it has not caught on to my relief. But templates still are a better choice than macros.
  - djmips 2y ago
    I would have thought so but fast forward to today and we have template abuse that cause very long build times and creates gigabytes of symbols that flood memory and crash compilers and take debuggers to their knees and create the most inscrutable code to read and debug that MACRO jockeys could never have dreamed of. I'm working on a C++ project where maintainers have had to resort to changing namespaces to single letters as a grotesque stop gap against the symbol table explosion!

cytocync 2y ago

[dead]