3 ms·

One-shotting is a bit of a red herring, imo. Let's say I wanted to build a "Super Dario" meme platformer. I would do some research, find a platformer on Codepen

by dvt 3mo ago

One-shotting is a bit of a red herring, imo. Let's say I wanted to build a "Super Dario" meme platformer. I would do some research, find a platformer on Codepen or Github that "feels" good, tweak it to my liking, change the sprites, and voila. I constantly have to tell agents to just "look it up online" and "don't hallucinate your own components" because people have already done this a million times. Ironically, being more lazy could make these models more useful.

bogwog 3mo ago
So you're saying that it's not fair to rate a coding model on its ability to code, and instead the best way to use it is to tell it to find existing human-written code online rather than generate actual code on its own?
- darkwater 3mo ago
  Isn't part of the human coding that Super Dario tells us LLM are going to replace also made of what you are saying GP is saying?
- dvt 3mo ago
  If you're testing how good LLMs are at compressing information, then I think that's a fair test. Personally, I don't really think that's where their strength comes from (especially considering how much more useful local models that are orders of magnitude smaller than Claude/OpenAI-tier models have gotten). In other words, we already have a "super-intelligence"—it's called the internet, so just use the darn thing.
- deleted 3mo ago
  [deleted]
sefrost 3mo ago
Yes for any slightly non-standard web UI components I either tell the LLM to search the web and find examples or I provide them myself. Usually the ones I provide myself are better than the ones they find themselves. I'm not super clear how they search the web or what they do and don't have access to. It seems to be quite fast.
jimkleiber 3mo ago
TBH, I wonder how much one-shotting posts are just lying about one-shotting it.
- uncivilized 3mo ago
  Many for sure.
port11 2mo ago
Strange, I use M3 and Sonnet a lot, they never hallucinated. Gemini used to make up some function names, but it was in long context windows. My AGENTS.md does specific they should use context7 when unsure about a library, but still…

by dvt 3mo ago

bogwog 3mo ago

So you're saying that it's not fair to rate a coding model on its ability to code, and instead the best way to use it is to tell it to find existing human-written code online rather than generate actual code on its own?

darkwater 3mo ago
Isn't part of the human coding that Super Dario tells us LLM are going to replace also made of what you are saying GP is saying?
dvt 3mo ago
If you're testing how good LLMs are at compressing information, then I think that's a fair test. Personally, I don't really think that's where their strength comes from (especially considering how much more useful local models that are orders of magnitude smaller than Claude/OpenAI-tier models have gotten). In other words, we already have a "super-intelligence"—it's called the internet, so just use the darn thing.
deleted 3mo ago
[deleted]

sefrost 3mo ago

Yes for any slightly non-standard web UI components I either tell the LLM to search the web and find examples or I provide them myself. Usually the ones I provide myself are better than the ones they find themselves. I'm not super clear how they search the web or what they do and don't have access to. It seems to be quite fast.

jimkleiber 3mo ago

TBH, I wonder how much one-shotting posts are just lying about one-shotting it.

uncivilized 3mo ago
Many for sure.

port11 2mo ago

Strange, I use M3 and Sonnet a lot, they never hallucinated. Gemini used to make up some function names, but it was in long context windows. My AGENTS.md does specific they should use context7 when unsure about a library, but still…