8 ms·

I can't assess this, but I do worry that overnight some algorithmic advance will enhance LLMs by orders of magnitude and the next big model to get trained is su

I can't assess this, but I do worry that overnight some algorithmic advance will enhance LLMs by orders of magnitude and the next big model to get trained is suddenly 10,000x better than GPT-4 and nobody's ready for it.

DeathArrow 2y ago
>some algorithmic advance will enhance LLMs by orders of magnitude I would worry if I'd own Nvidia shares.
- krasin 2y ago
  Actually, that would be fantastic for NVIDIA shares; 1. A new architecture would make all/most of these upcoming Transformer accelerators obsolete => back to GPUs. 2. Higher performance LLMs on GPUs => we can speed up LLMs with 1T+ parameters. So, LLMs become more useful, so more of GPUs would be purchased.
  - mindcrime 2y ago
    1. A new architecture would make all/most of these upcoming Transformer accelerators obsolete => back to GPUs. There's no guarantee that that is what would happen. The right (or wrong, depending on your POV) algorithmic breakthrough might make GPU's obsolete for AI, by making CPU's (or analog computing units, or DSP's, or "other") the preferred platform to run AI.
    - edmara 2y ago
      Assuming there is a development that makes GPUs obsolete, I think it's safe to assume that what will replace them at scale will still take the form dedicated AI card/rack 1. Tight integration necessary for fundamental compute constraints like memory latency. 2. Economies of scale 3. Opportunity cost to AI orgs. Meta, OpenAI etc want 50k h100s to arrive in shipping container and plug in so they can focus on their value-add. Everyone will have to readjust to this paradigm. Even if next get AI runs better on CPU, Intel won't suddenly be signing contracts to sell 1,000,000 xeons and 1,000,000 motherboards etc Also, Nvidia have 25bn cash in hand and almost 10 billion yearly r&d spend. They've been an AI-first company for over a decade now, they're more prepared to pivot than anyone else Edit: nearly forgot - Nvidia can issue 5% new stocks and raise 100B like it's nothing.
6mian 2y ago
What to be worried about? Technical progress will happen, sometimes by sudden jumps. Some company will become a leader, competitors will catch up after a while.
- cess11 2y ago
  "Technical progress" has been destroying our habitat for centuries, causing lots of other species to go extinct. Pretty much the entire planet surface has been 'technically progressed', spreading plastics, climate change and whatnot over the entirety of it. Are you assuming that this particular "progress" would be relatively innocent?
  - 6mian 2y ago
    On the other hand, the same "technical progress" (if we're putting machine learning, deforestation, and mining in the same bag) gave you medicine, which turns many otherwise deadly diseases into inconveniences and allows you to work less than 12 hrs/7 days per week to not die from hunger in a large portion of the world. A few hundred years ago, unless you were born into the lucky 0.01% of the ruling population, working from dawn to sunset was the norm for a lot more people than now. I'm not assuming that something 10k x better than GPT-4 will be good or bad; I don't know. I was just curious what exactly to be worried about. I think in the current state, LLMs are already advanced enough for bad uses like article generation for SEO, spam, scams, etc., and I wonder if an order of magnitude better model would allow for something worse.
    - cess11 2y ago
      Where did you learn that history? What do you mean by "better"?
      - 6mian 2y ago
        I had a European peasant in the 1600-1700s in mind when I wrote about the amount of work. During the season, they worked all day; off-season, they had "free time" that went into taking care of the household, inventory, etc., so it's still work. Can't quickly find a reliable source in English I could link, so I can be wrong here. "Better" was referring to what OP wrote in the top comment. I guess 10x faster, 10x longer context, and 100x less prone to hallucinations would make a good "10k x better" than GPT-4.
snewman 2y ago
I think this is unlikely. There has never (in the visible fossil record) been a mutation that suddenly made tigers an order of magnitude stronger and faster, or humans an order of magnitude more intelligent. It's been a long time (if ever?) since chip transistor density made a multiple-order-of-magnitude leap. Any complex optimized system has many limiting factors and it's unlikely that all of them would leap forward at once. The current generation of LLMs are not as complex or optimized as tigers or humans, but they're far enough along that changing one thing is unlikely to result in a giant leap. If and when something radically better comes along, say an alternative to back-propagation that is more like the way our brains learn, it will need a lot of scaling and refinement to catch up with the then-current LLM.
- the8472 2y ago
  Comparing it to evolution and SNPs isn't really a good analogy. Novel network architectures are much larger changes, maybe comparable to new organelles or metabolic pathways? And those have caused catastrophic changes. Evolution also operates on much longer time-scales due to its blind parallel search. https://en.wikipedia.org/wiki/Oxygen_catastrophe https://en.wikipedia.org/wiki/Oxygen_catastrophe

DeathArrow 2y ago

>some algorithmic advance will enhance LLMs by orders of magnitude I would worry if I'd own Nvidia shares.

krasin 2y ago
Actually, that would be fantastic for NVIDIA shares; 1. A new architecture would make all/most of these upcoming Transformer accelerators obsolete => back to GPUs. 2. Higher performance LLMs on GPUs => we can speed up LLMs with 1T+ parameters. So, LLMs become more useful, so more of GPUs would be purchased.
- mindcrime 2y ago
  1. A new architecture would make all/most of these upcoming Transformer accelerators obsolete => back to GPUs. There's no guarantee that that is what would happen. The right (or wrong, depending on your POV) algorithmic breakthrough might make GPU's obsolete for AI, by making CPU's (or analog computing units, or DSP's, or "other") the preferred platform to run AI.
  - edmara 2y ago
    Assuming there is a development that makes GPUs obsolete, I think it's safe to assume that what will replace them at scale will still take the form dedicated AI card/rack 1. Tight integration necessary for fundamental compute constraints like memory latency. 2. Economies of scale 3. Opportunity cost to AI orgs. Meta, OpenAI etc want 50k h100s to arrive in shipping container and plug in so they can focus on their value-add. Everyone will have to readjust to this paradigm. Even if next get AI runs better on CPU, Intel won't suddenly be signing contracts to sell 1,000,000 xeons and 1,000,000 motherboards etc Also, Nvidia have 25bn cash in hand and almost 10 billion yearly r&d spend. They've been an AI-first company for over a decade now, they're more prepared to pivot than anyone else Edit: nearly forgot - Nvidia can issue 5% new stocks and raise 100B like it's nothing.

6mian 2y ago

What to be worried about? Technical progress will happen, sometimes by sudden jumps. Some company will become a leader, competitors will catch up after a while.

cess11 2y ago
"Technical progress" has been destroying our habitat for centuries, causing lots of other species to go extinct. Pretty much the entire planet surface has been 'technically progressed', spreading plastics, climate change and whatnot over the entirety of it. Are you assuming that this particular "progress" would be relatively innocent?
- 6mian 2y ago
  On the other hand, the same "technical progress" (if we're putting machine learning, deforestation, and mining in the same bag) gave you medicine, which turns many otherwise deadly diseases into inconveniences and allows you to work less than 12 hrs/7 days per week to not die from hunger in a large portion of the world. A few hundred years ago, unless you were born into the lucky 0.01% of the ruling population, working from dawn to sunset was the norm for a lot more people than now. I'm not assuming that something 10k x better than GPT-4 will be good or bad; I don't know. I was just curious what exactly to be worried about. I think in the current state, LLMs are already advanced enough for bad uses like article generation for SEO, spam, scams, etc., and I wonder if an order of magnitude better model would allow for something worse.
  - cess11 2y ago
    Where did you learn that history? What do you mean by "better"?
    - 6mian 2y ago
      I had a European peasant in the 1600-1700s in mind when I wrote about the amount of work. During the season, they worked all day; off-season, they had "free time" that went into taking care of the household, inventory, etc., so it's still work. Can't quickly find a reliable source in English I could link, so I can be wrong here. "Better" was referring to what OP wrote in the top comment. I guess 10x faster, 10x longer context, and 100x less prone to hallucinations would make a good "10k x better" than GPT-4.

snewman 2y ago

I think this is unlikely. There has never (in the visible fossil record) been a mutation that suddenly made tigers an order of magnitude stronger and faster, or humans an order of magnitude more intelligent. It's been a long time (if ever?) since chip transistor density made a multiple-order-of-magnitude leap. Any complex optimized system has many limiting factors and it's unlikely that all of them would leap forward at once. The current generation of LLMs are not as complex or optimized as tigers or humans, but they're far enough along that changing one thing is unlikely to result in a giant leap. If and when something radically better comes along, say an alternative to back-propagation that is more like the way our brains learn, it will need a lot of scaling and refinement to catch up with the then-current LLM.

the8472 2y ago
Comparing it to evolution and SNPs isn't really a good analogy. Novel network architectures are much larger changes, maybe comparable to new organelles or metabolic pathways? And those have caused catastrophic changes. Evolution also operates on much longer time-scales due to its blind parallel search. https://en.wikipedia.org/wiki/Oxygen_catastrophe https://en.wikipedia.org/wiki/Oxygen_catastrophe