3 ms·

That's not how it works. It's not on everyone else to prove claims false, it's on you (or the people who argue any of this had a measurable impact) to prove it

• tokioyoyo 8mo ago
  Do you actively use LLMs to do semi-complex coding work? Because if not, it will sound mumbo-jumbo to you. Everyone else can nod along and read on, as they’ve experienced all of it first hand.
  • scuff3d 8mo ago
    You've missed the point. This isn't engineering, it's gambling. You could take the exact same documents, prompts, and whatever other bullshit, run it on the exact same agent backed by the exact same model, and get different results every single time. Just like you can roll dice the exact same way on the exact same table and you'll get two totally different results. People are doing their best to constrain that behavior by layering stuff on top, but the foundational tech is flawed (or at least ill suited for this use case). That's not to say that AI isn't helpful. It certainly is. But when you are basically begging your tools to please do what you want with magic incantations, we've lost the fucking plot somewhere.
    • gf000 8mo ago
      > You could take the exact same documents, prompts, and whatever other bullshit, run it on the exact same agent backed by the exact same model, and get different results every single time This is more of an implementation detail/done this way to get better results. A neural network with fixed weights (and deterministic floating point operations) returning a probability distribution, where you use a pseudorandom generator with a fixed seed called recursively will always return the same output for the same input.
    • geoelectric 8mo ago
      I think that's a pretty bold claim, that it'd be different every time. I'd think the output would converge on a small set of functionally equivalent designs, given sufficiently rigorous requirements. And even a human engineer might not solve a problem the same way twice in a row, based on changes in recent inspirations or tech obsessions. What's the difference, as long as it passes review and does the job?
• guiambros 8mo ago
  If you read the transformer paper, or get any book on NLP, you will see that this is not magic incantation; it's purely the attention mechanism at work. Or you can just ask Gemini or Claude why these prompts work. But I get the impression from your comment that you have a fixed idea, and you're not really interested in understanding how or why it works. If you think like a hammer, everything will look like a nail.
  • scuff3d 8mo ago
    I know why it works, to varying and unmeasurable degrees of success. Just like if I poke a bull with a sharp stick, I know it's gonna get it's attention. It might choose to run away from me in one of any number of directions, or it might decide to turn around and gore me to death. I can't answer that question with any certainty then you can. The system is inherently non-deterministic. Just because you can guide it a bit, doesn't mean you can predict outcomes.
    • winrid 8mo ago
      But we can predict the outcomes, though. That's what we're saying, and it's true. Maybe not 100% of the time, but maybe it helps a significant amount of the time and that's what matters. Is it engineering? Maybe not. But neither is knowing how to talk to junior developers so they're productive and don't feel bad. The engineering is at other levels.
      • imiric 8mo ago
        > But we can predict the outcomes [...] Maybe not 100% of the time So 60% of the time, it works every time. ... This fucking industry.
        • winrid 8mo ago
          Again, it's called management. You're managing something unpredictable: the LLM. This is nothing new, at all. Do you have a strategy that makes other engineers do what you want exactly 100% of the time?
• yaku_brang_ja 8mo ago
  These coding agents are literally Language Models. The way you structure your prompting language affect the actual output.