3 ms·

> - The knobs you'd need to get serious performance are nearly unreachable in languages which LMs are good at (even Rust requires a discipline that the default

> - The knobs you'd need to get serious performance are nearly unreachable in languages which LMs are good at (even Rust requires a discipline that the default language doesn't enforce). When you drop into the lower realms, you're trading consumption context for access to these levers. The levers are also "soft": you find yourself writing a bunch of skills, and tools to try and enforce the discipline. Hasn’t been my experience at all. The latest LLMs can knock out assembly optimized subroutines and benchmark 100 different variations faster than I ever could dream of.

mccoyb 2mo ago
That's fair for a well-scoped subroutine: what I meant is that if you ask an agent to write a compiler and let it rip for a few days, you are going to be spending a few more days correcting the default behaviors in the distribution, which often do not tend towards hardware-oriented design. To correct those behaviors, you're going to write tools and skills, and that's going to help, but it is still clear that you are fighting the distribution (today).
- deleted 2mo ago
  [deleted]
- embedding-shape 2mo ago
  Yes, say "Build a compiler" will require you to clean up stuff if you leave the agent for days, but not because of the LLM or the quality of the tool, but because you hardly specified anything, so of course it's gonna make assumptions you need to correct. If you instead spend a day writing a proper specification, then ask the agent to spend a week implementing that, you'll need zero tools and skills afterwards to clean it up, because there won't be any misunderstandings, assumptions or other things, just code fulfilling what the specification says. Granted, this does require you to not use obviously dumb models, like anything you can run locally today, and at least within reasonable range of SOTA models. But they been able to do this for 6 months or more at this point.
  - n4r9 2mo ago
    You're making assumptions; OP made no mention of how detailed their spec was.
    - embedding-shape 2mo ago
      Indeed, you're right, I did do assumptions, based on what I could understand from it, and also based on my own experience using these tools too. May be their specification is very detailed and exactly as they wish it to be, yet the agent creates something completely else, then probably something else than the specification is wrong, and my comment is pointless :)
  - mccoyb 2mo ago
    Bro, what the fuck do you think I’m doing? Do you think I don’t know about spec driven development? What is the most complicated thing you’ve built with LM agents? Have you done it with a single spec? How novel was it? This comment is so laughably “you’re holding it wrong” I can’t respond to you seriously. > If you instead spend a day writing a proper specification, then ask the agent to spend a week implementing that, you'll need zero tools and skills afterwards to clean it up, because there won't be any misunderstandings, assumptions or other things The set of software that has followed this process is measure zero.
    - embedding-shape 2mo ago
      > Bro, what the fuck do you think I’m doing? Do you think I don’t know about spec driven development? Bro, I don't know you, and have no idea what you do, except for the parts you share in your comment, like talking to a LLM :) If you say "if you ask an agent to write a compiler" and don't expand on that, I'll take your word for it. I'm glad you got a laugh, and didn't take it so serious :) Not entirely sure why you got so defensive, but it's all good, again I don't know you, sorry if it felt like I called you stupid or something. At least some other people found it helpful I suppose, so in the end it sucks you're unable to get value from a process I've found working for me, hope you'll find something that works out for you in the end, regardless of what it is!
senderista 2mo ago
Agreed, IME Fable can churn out decent SIMD kernels optimized for whatever tradeoffs you give it.

mccoyb 2mo ago

That's fair for a well-scoped subroutine: what I meant is that if you ask an agent to write a compiler and let it rip for a few days, you are going to be spending a few more days correcting the default behaviors in the distribution, which often do not tend towards hardware-oriented design. To correct those behaviors, you're going to write tools and skills, and that's going to help, but it is still clear that you are fighting the distribution (today).

deleted 2mo ago
[deleted]
embedding-shape 2mo ago
Yes, say "Build a compiler" will require you to clean up stuff if you leave the agent for days, but not because of the LLM or the quality of the tool, but because you hardly specified anything, so of course it's gonna make assumptions you need to correct. If you instead spend a day writing a proper specification, then ask the agent to spend a week implementing that, you'll need zero tools and skills afterwards to clean it up, because there won't be any misunderstandings, assumptions or other things, just code fulfilling what the specification says. Granted, this does require you to not use obviously dumb models, like anything you can run locally today, and at least within reasonable range of SOTA models. But they been able to do this for 6 months or more at this point.
- n4r9 2mo ago
  You're making assumptions; OP made no mention of how detailed their spec was.
  - embedding-shape 2mo ago
    Indeed, you're right, I did do assumptions, based on what I could understand from it, and also based on my own experience using these tools too. May be their specification is very detailed and exactly as they wish it to be, yet the agent creates something completely else, then probably something else than the specification is wrong, and my comment is pointless :)
- mccoyb 2mo ago
  Bro, what the fuck do you think I’m doing? Do you think I don’t know about spec driven development? What is the most complicated thing you’ve built with LM agents? Have you done it with a single spec? How novel was it? This comment is so laughably “you’re holding it wrong” I can’t respond to you seriously. > If you instead spend a day writing a proper specification, then ask the agent to spend a week implementing that, you'll need zero tools and skills afterwards to clean it up, because there won't be any misunderstandings, assumptions or other things The set of software that has followed this process is measure zero.
  - embedding-shape 2mo ago
    > Bro, what the fuck do you think I’m doing? Do you think I don’t know about spec driven development? Bro, I don't know you, and have no idea what you do, except for the parts you share in your comment, like talking to a LLM :) If you say "if you ask an agent to write a compiler" and don't expand on that, I'll take your word for it. I'm glad you got a laugh, and didn't take it so serious :) Not entirely sure why you got so defensive, but it's all good, again I don't know you, sorry if it felt like I called you stupid or something. At least some other people found it helpful I suppose, so in the end it sucks you're unable to get value from a process I've found working for me, hope you'll find something that works out for you in the end, regardless of what it is!

senderista 2mo ago

Agreed, IME Fable can churn out decent SIMD kernels optimized for whatever tradeoffs you give it.