3 ms·

A 5090 has a peak theoretical limit of GenAI 3356TOPS. So we're "already" an order of magnitude greater than what was considered enough for AGI. One question is

A 5090 has a peak theoretical limit of GenAI 3356TOPS. So we're "already" an order of magnitude greater than what was considered enough for AGI. One question is, "What happened here?" Was the original estimate wrong? Have we notfound the "right" algorithm yet? Something else?

SoftTalker 1y ago
Nature needed 3.5 billion years to work it out, and we're going to solve it in a few decades?
- kelseyfrog 1y ago
  It depends on where we draw the starting line. We're already at parity with 3.5BYA to 541Mya because no neurons existed in that duration. Only more recently, in the Cambrian, do we have evidence that voltage gated potassium signaling evolved[1]. That changes the calculus likely very little, but it feels more accurate. 1. https://www.cell.com/current-biology/pdf/S0960-9822(16)30489-4.pdf https://www.cell.com/current-biology/pdf/S0960-9822(16)30489...
- mathgeek 1y ago
  I know it’s a silly question to begin with, but if you analyze it seriously, you’d want to at most compare human intelligence->superintelligence with the 20 million years between the first homidinae and homo (and even that is probably too large for some folks to compare with). One could even argue you should only compare it back to the discovery of writing or similar.
- Jyaif 1y ago
  That's not an argument. Nature never worked out going into space, yet we solved it in a few decades.
  - SoftTalker 1y ago
    It worked out flying though, millions of years before we did and we still don't do it as well. We can't even do walking as well as nature did.
    - baq 1y ago
      Walking is easy compared to elbows, fingers and thumbs. It’s just falling over in a controlled fashion. I hear at least one company in Boston figured it out. Anyway, humanoid robots should be big in the next 10-20 years. The compute, the batteries, the algorithms are all coming together.
    - derektank 1y ago
      We do flying better. If you adjust for our body weight, a modern airliner uses less energy per traveller mile than your average migratory bird. And the airliner goes much faster.
  - jll29 1y ago
    Yes but that's "in a few decades" ON TOP of millions of years. If I had to give an estimate, I would consider less the time taken to date, but the current state of our knowledge of how the brain works, and how it has grown in the last decades. There is almost nothing that we know so little about as the human brain, how thoughts are represented, modern imaging techniques notwithstanding.
    - exe34 1y ago
      > Yes but that's "in a few decades" ON TOP of millions of years. If that's the bar, then anything else can fit in "a few decades", since that also rests "ON TOP of millions of years".
  - gnz11 1y ago
    One could argue nature solved it by evolving homo sapiens.
lukeschlather 1y ago
"We haven't found the 'right' algorithm yet." seems like the obvious answer, but the numbers in the paper all make sense and I'm interested in some more exotic explanations why it could actually be some orders of magnitude more than a 5090. Although that's not looking at memory, and I am also interested in some explanation of what... a 5090 has 32GB which, a human brain has more like a petabyte of memory assuming 1 byte/synapse. Which is to say 1 million GB in which case even a large cluster of H100s has an absurd amount of TOPS but nowhere near enough high-speed memory.
nvch 1y ago
We are constantly learning (updating the network) in addition to inference. Quite possibly that our brains allocate more resources to learning than to inference. Perhaps AI companies don’t know how to run continuous learning on their models: * it’s unrealistic to do it for one big model because it will instantly start shifting in an unknown direction * they can’t make millions of clones of their model, run them separately and set them free like it happens with humans
- Mars008 1y ago
  > possibly that our brains allocate more resources to learning than to inference It's likely in brain inference is learning. If you want a technical analog it's like a conversation in LLMs. Previous tokens do affect the currently generated. I.e. it's inference time learning, well known and widely used.

SoftTalker 1y ago

Nature needed 3.5 billion years to work it out, and we're going to solve it in a few decades?

kelseyfrog 1y ago
It depends on where we draw the starting line. We're already at parity with 3.5BYA to 541Mya because no neurons existed in that duration. Only more recently, in the Cambrian, do we have evidence that voltage gated potassium signaling evolved[1]. That changes the calculus likely very little, but it feels more accurate. 1. https://www.cell.com/current-biology/pdf/S0960-9822(16)30489-4.pdf https://www.cell.com/current-biology/pdf/S0960-9822(16)30489...
mathgeek 1y ago
I know it’s a silly question to begin with, but if you analyze it seriously, you’d want to at most compare human intelligence->superintelligence with the 20 million years between the first homidinae and homo (and even that is probably too large for some folks to compare with). One could even argue you should only compare it back to the discovery of writing or similar.
Jyaif 1y ago
That's not an argument. Nature never worked out going into space, yet we solved it in a few decades.
- SoftTalker 1y ago
  It worked out flying though, millions of years before we did and we still don't do it as well. We can't even do walking as well as nature did.
  - baq 1y ago
    Walking is easy compared to elbows, fingers and thumbs. It’s just falling over in a controlled fashion. I hear at least one company in Boston figured it out. Anyway, humanoid robots should be big in the next 10-20 years. The compute, the batteries, the algorithms are all coming together.
  - derektank 1y ago
    We do flying better. If you adjust for our body weight, a modern airliner uses less energy per traveller mile than your average migratory bird. And the airliner goes much faster.
- jll29 1y ago
  Yes but that's "in a few decades" ON TOP of millions of years. If I had to give an estimate, I would consider less the time taken to date, but the current state of our knowledge of how the brain works, and how it has grown in the last decades. There is almost nothing that we know so little about as the human brain, how thoughts are represented, modern imaging techniques notwithstanding.
  - exe34 1y ago
    > Yes but that's "in a few decades" ON TOP of millions of years. If that's the bar, then anything else can fit in "a few decades", since that also rests "ON TOP of millions of years".
- gnz11 1y ago
  One could argue nature solved it by evolving homo sapiens.

lukeschlather 1y ago

"We haven't found the 'right' algorithm yet." seems like the obvious answer, but the numbers in the paper all make sense and I'm interested in some more exotic explanations why it could actually be some orders of magnitude more than a 5090. Although that's not looking at memory, and I am also interested in some explanation of what... a 5090 has 32GB which, a human brain has more like a petabyte of memory assuming 1 byte/synapse. Which is to say 1 million GB in which case even a large cluster of H100s has an absurd amount of TOPS but nowhere near enough high-speed memory.

nvch 1y ago

We are constantly learning (updating the network) in addition to inference. Quite possibly that our brains allocate more resources to learning than to inference. Perhaps AI companies don’t know how to run continuous learning on their models: * it’s unrealistic to do it for one big model because it will instantly start shifting in an unknown direction * they can’t make millions of clones of their model, run them separately and set them free like it happens with humans

Mars008 1y ago
> possibly that our brains allocate more resources to learning than to inference It's likely in brain inference is learning. If you want a technical analog it's like a conversation in LLMs. Previous tokens do affect the currently generated. I.e. it's inference time learning, well known and widely used.