3 ms·

As an aside, one of the worst aspects of the rise of LLMs, for me, has been the wholesale replacement of engineering with trial-and-error hand-waving. Try this,

As an aside, one of the worst aspects of the rise of LLMs, for me, has been the wholesale replacement of engineering with trial-and-error hand-waving. Try this, or maybe that, and maybe you'll see a +5% improvement. Why? Who knows. It's just not how I like to work.

pclmulqdq 2y ago
Software engineering has involved a lot of people doing trial-and-error hand-waving for at least a decade. We are now codifying the trend.
zoogeny 2y ago
I think trial-and-error hand-waving isn't all that far from experimentation. As an aside, I was working in the games industry when multi-core was brand new. Maybe Xbox-360 and PS3? I'm hazy on the exact consoles but there was one generation where the major platforms all went multi-core. No one knew how to best use the multi-core systems for gaming. I attended numerous tech talks by teams that had tried different approaches and were give similar "maybe do this and maybe see x% improvement?". There was a lot of experimentation. It took a few years before things settled and best practices became even somewhat standardized. Some people found that era frustrating and didn't like to work in that way. Others loved the fact it was a wide open field of study where they could discover things.
- jorvi 2y ago
  Yes, it was the generation of the X360 and PS3. X360 was 3 core and the PS3 was 1+7 core (sort of a big.little setup). Although it took many, many more years until games started to actually use multi-core properly. With rendering being on a 16.67ms / 8.33ms budget and rendering tied to world state, it was just really hard to not tie everything into eachother. Even today you'll usually only see 2-4 cores actually getting significant load.
- hackernewds 2y ago
  there probably was still a structured way to test this through cross hatching but yeah like blind guessing might take longer and arrive at the same solution
- Nullabillity 2y ago
  Performance optimization is different, because there's still some kind of a baseline truth. Every knows what a FPS is, and +5% FPS is +5% FPS. Even the tricky cases have some kind of boundary (+5% FPS on this hardware but -10% on this other hardware, +2% on scenes meeting these conditions but -3% otherwise, etc). Meanwhile, nobody can agree on what a "good" LLM in, let alone how to measure it.
brokencode 2y ago
Out of curiosity, what do you work on where you don’t have to experiment with different solutions to see what works best?
- FridgeSeal 2y ago
  Usually when we’re doing it in practice there’s _somewhat_ more awareness of the mechanics than just throwing random obstructions in and hoping for the best.
  - RussianCow 2y ago
    LLMs are still very young. We'll get there in time. I don't see how it's any different than optimizing for new CPU/GPU architectures other than the fact that the latter is now a decades-old practice.
    - girvo 2y ago
      > I don't see how it's any different than optimizing for new CPU/GPU architectures I mean that seems wild to say to me. Those architectures have documentation and aren't magic black boxes that we chuck inputs at and hope for the best: we do pretty much that with LLMs. If that's how you optimise, I'm genuinely shocked.
      - swyx 2y ago
        i bet if we talked to a real low level hardware systems/chip engineer they'd laugh and take another shot at how we put them on a pedestal
        girvo 2y ago
        Not really, in my experience. There's still fundamental differences between designed systems and trained LLMs.
    - th0ma5 2y ago
kitsunemax 2y ago
I feel like this a common pattern with people who work in STEM. As someone who is used to working with formal proofs, equations, math, having a startup taught me how to rewire myself to work with the unknowns, imperfect solutions, messy details. I'm going on a tangent, but just wanted to share.
muzani 2y ago
One of the major advantages and disadvantages of LLMs is they act a bit more like humans. I feel like most "prompt advice" out there is very similar to how you would teach a person as well. Teachers and parents have some advantages here.
barrkel 2y ago
The disadvantage is that LLMs are probabilistic, mercurial, unreliable. The advantage is that humans are probabilistic, mercurial and unreliable, and LLMs are a way to bridge the gap between humans and machines that, while not wholly reliable, makes the gap much smaller than it used to be. If you're not making software that interacts with humans or their fuzzy outputs (text, images, voice etc.), and have the luxury of well defined schema, you're not going to see the advantage side.
moffkalast 2y ago
Yeah this is why I don't like statistical and ML solutions in general. Monte Carlo sampling is already kinda throwing bullshit at the wall and hoping something works with absolutely zero guarantees and it's perfectly explainable. But unfortunately for us, clean and logical classical methods suck ass in comparison so we have no other choice but to deal with the uncertainty.
make3 2y ago
prompt tuning is a temporary necessity

pclmulqdq 2y ago

Software engineering has involved a lot of people doing trial-and-error hand-waving for at least a decade. We are now codifying the trend.

zoogeny 2y ago

I think trial-and-error hand-waving isn't all that far from experimentation. As an aside, I was working in the games industry when multi-core was brand new. Maybe Xbox-360 and PS3? I'm hazy on the exact consoles but there was one generation where the major platforms all went multi-core. No one knew how to best use the multi-core systems for gaming. I attended numerous tech talks by teams that had tried different approaches and were give similar "maybe do this and maybe see x% improvement?". There was a lot of experimentation. It took a few years before things settled and best practices became even somewhat standardized. Some people found that era frustrating and didn't like to work in that way. Others loved the fact it was a wide open field of study where they could discover things.

jorvi 2y ago
Yes, it was the generation of the X360 and PS3. X360 was 3 core and the PS3 was 1+7 core (sort of a big.little setup). Although it took many, many more years until games started to actually use multi-core properly. With rendering being on a 16.67ms / 8.33ms budget and rendering tied to world state, it was just really hard to not tie everything into eachother. Even today you'll usually only see 2-4 cores actually getting significant load.
hackernewds 2y ago
there probably was still a structured way to test this through cross hatching but yeah like blind guessing might take longer and arrive at the same solution
Nullabillity 2y ago
Performance optimization is different, because there's still some kind of a baseline truth. Every knows what a FPS is, and +5% FPS is +5% FPS. Even the tricky cases have some kind of boundary (+5% FPS on this hardware but -10% on this other hardware, +2% on scenes meeting these conditions but -3% otherwise, etc). Meanwhile, nobody can agree on what a "good" LLM in, let alone how to measure it.

brokencode 2y ago

Out of curiosity, what do you work on where you don’t have to experiment with different solutions to see what works best?

FridgeSeal 2y ago
Usually when we’re doing it in practice there’s _somewhat_ more awareness of the mechanics than just throwing random obstructions in and hoping for the best.
- RussianCow 2y ago
  LLMs are still very young. We'll get there in time. I don't see how it's any different than optimizing for new CPU/GPU architectures other than the fact that the latter is now a decades-old practice.
  - girvo 2y ago
    > I don't see how it's any different than optimizing for new CPU/GPU architectures I mean that seems wild to say to me. Those architectures have documentation and aren't magic black boxes that we chuck inputs at and hope for the best: we do pretty much that with LLMs. If that's how you optimise, I'm genuinely shocked.
    - swyx 2y ago
      i bet if we talked to a real low level hardware systems/chip engineer they'd laugh and take another shot at how we put them on a pedestal
      - girvo 2y ago
        Not really, in my experience. There's still fundamental differences between designed systems and trained LLMs.
  - th0ma5 2y ago

kitsunemax 2y ago

I feel like this a common pattern with people who work in STEM. As someone who is used to working with formal proofs, equations, math, having a startup taught me how to rewire myself to work with the unknowns, imperfect solutions, messy details. I'm going on a tangent, but just wanted to share.

muzani 2y ago

One of the major advantages and disadvantages of LLMs is they act a bit more like humans. I feel like most "prompt advice" out there is very similar to how you would teach a person as well. Teachers and parents have some advantages here.

barrkel 2y ago

The disadvantage is that LLMs are probabilistic, mercurial, unreliable. The advantage is that humans are probabilistic, mercurial and unreliable, and LLMs are a way to bridge the gap between humans and machines that, while not wholly reliable, makes the gap much smaller than it used to be. If you're not making software that interacts with humans or their fuzzy outputs (text, images, voice etc.), and have the luxury of well defined schema, you're not going to see the advantage side.

moffkalast 2y ago

Yeah this is why I don't like statistical and ML solutions in general. Monte Carlo sampling is already kinda throwing bullshit at the wall and hoping something works with absolutely zero guarantees and it's perfectly explainable. But unfortunately for us, clean and logical classical methods suck ass in comparison so we have no other choice but to deal with the uncertainty.

make3 2y ago

prompt tuning is a temporary necessity