7 ms·

"The current round" of AI accelerators you are referring to are things that were designed 2015-2022; There are a number of startups (including my own) that are

"The current round" of AI accelerators you are referring to are things that were designed 2015-2022; There are a number of startups (including my own) that are actually designing for the real bottlenecks that differentiate Transformers (plus SSMs and other emerging architectures) from "old" CNNs, RNNs, etc. Obviously I think my company is doing this in an unique and "correct" way, but I know of half a dozen other companies founded in the past ~18 months that are focused on the memory capacity and bandwidth bottlenecks that exist... the massive failures of the previous decade do not mean that they are going to be repeated.

pk-protect-ai 3y ago
Is there any startup which is ready to compete with this: https://www.redsharknews.com/nvidia-wants-to-increase-computing-power-by-a-factor-of-1-million https://www.redsharknews.com/nvidia-wants-to-increase-comput... ?
- simne 3y ago
  It is known for electronics designers, that specialized circuits outperforms GPUs for few times. Before appear Tensor cores, GPUs was about 4 times worse (speed, power consumption). With Tensor cores, GPUs become better, but they still need to carry video hardware (ramdac, video connectors, 3D processing units, network to connect all this stuff), so they still late. Really GPUs are interest just because current AI applications are not achieve enough revenue to pay for large scale production of special chips. I don't know, if Altman have something Big to get revenue to pay for special chips. Exists speculations that GPT-5 will be enough to replace human at work. If this is real, AI chips will be worth it.
  - pk-protect-ai 3y ago
    We are indeed talking about a 10^6 factor here ... It's not just 10x or 100x, or even 1000x ... If NVIDIA strips away everything not required from their chips, adds more SDRAM and HBM, it won't improve performance by 100x, maybe they'll make it 10x-15x with this. But they claim they are going to achieve a 10^6x improvement in performance. Even if they end up delivering an ARM-compatible CPU with built-in Tensor core, built-in HBM, and vast SDRAM, without DDR RAM at all, how fast can it be? This promise of 10^6x performance improve is a paradigm shift. They know something that we are not. Or they are just bluffing.
    - simne 3y ago
      > But they claim they are going to achieve a 10^6x Classics of management, to ask people more then they could, and they will do most possible, so I don't bother much on such claims. And also this is teambuilding bs, to motivate people claiming impossible targets. Will see, how Jensen Huang will use all his diplomatic skills and rhetoric art, to round corners, when become clear, that claimed things impossible. And this is not first time, such things happen, there are near infinite number of examples. I just few days ago read about IBM 7030 fail, which delivered ~1/10 of claimed, and yesterday people remembered me about Itanium and i960.
    - simne 3y ago
      Exists one important thing, many people don't aware of. When some good smart team (business or not it is not much important), focus on some task and have corresponding resources, it really could make things, impossible for universal team, targeted for some wide outcome. What I see, NVIDIA is good, strong team, they bet very high stakes, when made great acquisitions in 2000s and they won. But NVIDIA made wide targeted product, they cannot made very narrow focus on just neural net. So it is possible to make NN product better then NVIDIA. Real question is to predict, if Altman team could achieve so good economy, to pay expenses for hardware development.
    - simne 3y ago
      For about tech questions you asked. You asked right questions, but you missing context. What really main bottlenecks of NN hardware are neither number crunching, nor memory. Real bottleneck is that GPT-2 is may be last LLM for which was possible train on one machine (even on one card). About GPT-3 usually people said about 32-GPUs installations (possible to install into one machine), for GPT-4 scale said about clouds. And modern clouds are NUMA beasts. I could say, modern clouds networking is slow, but it is not right words, as they are slow as hell. What all these mean, NN are good target for parallel processing in clouds, but not good enough. Real benchmarks said, mentioned 32-cards machine is about 10 times faster than 1 card with such amount of memory, and when on GPT-4 things scaled, benchmarks become much worse. So, just improve network to move bottleneck to something else and will got additional 50-100x improve. And with good team of AI scientists, it is more real to make special hardware network for NN processing, or to tune algorithms, than with team of GPU video processing specialized team.
cma 3y ago
Will your arch work for SSMs?
- trsohmers 3y ago
  Yes; Mamba was a very easy match, with Hyena also being a good match, but could be greatly optimized with some minimal changes to the model architecture or hardware design.
EvgeniyZh 3y ago
What can you actually do hardware wise with memory bottleneck except for use faster memory?

pk-protect-ai 3y ago

Is there any startup which is ready to compete with this: https://www.redsharknews.com/nvidia-wants-to-increase-computing-power-by-a-factor-of-1-million https://www.redsharknews.com/nvidia-wants-to-increase-comput... ?

simne 3y ago
It is known for electronics designers, that specialized circuits outperforms GPUs for few times. Before appear Tensor cores, GPUs was about 4 times worse (speed, power consumption). With Tensor cores, GPUs become better, but they still need to carry video hardware (ramdac, video connectors, 3D processing units, network to connect all this stuff), so they still late. Really GPUs are interest just because current AI applications are not achieve enough revenue to pay for large scale production of special chips. I don't know, if Altman have something Big to get revenue to pay for special chips. Exists speculations that GPT-5 will be enough to replace human at work. If this is real, AI chips will be worth it.
- pk-protect-ai 3y ago
  We are indeed talking about a 10^6 factor here ... It's not just 10x or 100x, or even 1000x ... If NVIDIA strips away everything not required from their chips, adds more SDRAM and HBM, it won't improve performance by 100x, maybe they'll make it 10x-15x with this. But they claim they are going to achieve a 10^6x improvement in performance. Even if they end up delivering an ARM-compatible CPU with built-in Tensor core, built-in HBM, and vast SDRAM, without DDR RAM at all, how fast can it be? This promise of 10^6x performance improve is a paradigm shift. They know something that we are not. Or they are just bluffing.
  - simne 3y ago
    > But they claim they are going to achieve a 10^6x Classics of management, to ask people more then they could, and they will do most possible, so I don't bother much on such claims. And also this is teambuilding bs, to motivate people claiming impossible targets. Will see, how Jensen Huang will use all his diplomatic skills and rhetoric art, to round corners, when become clear, that claimed things impossible. And this is not first time, such things happen, there are near infinite number of examples. I just few days ago read about IBM 7030 fail, which delivered ~1/10 of claimed, and yesterday people remembered me about Itanium and i960.
  - simne 3y ago
    Exists one important thing, many people don't aware of. When some good smart team (business or not it is not much important), focus on some task and have corresponding resources, it really could make things, impossible for universal team, targeted for some wide outcome. What I see, NVIDIA is good, strong team, they bet very high stakes, when made great acquisitions in 2000s and they won. But NVIDIA made wide targeted product, they cannot made very narrow focus on just neural net. So it is possible to make NN product better then NVIDIA. Real question is to predict, if Altman team could achieve so good economy, to pay expenses for hardware development.
  - simne 3y ago
    For about tech questions you asked. You asked right questions, but you missing context. What really main bottlenecks of NN hardware are neither number crunching, nor memory. Real bottleneck is that GPT-2 is may be last LLM for which was possible train on one machine (even on one card). About GPT-3 usually people said about 32-GPUs installations (possible to install into one machine), for GPT-4 scale said about clouds. And modern clouds are NUMA beasts. I could say, modern clouds networking is slow, but it is not right words, as they are slow as hell. What all these mean, NN are good target for parallel processing in clouds, but not good enough. Real benchmarks said, mentioned 32-cards machine is about 10 times faster than 1 card with such amount of memory, and when on GPT-4 things scaled, benchmarks become much worse. So, just improve network to move bottleneck to something else and will got additional 50-100x improve. And with good team of AI scientists, it is more real to make special hardware network for NN processing, or to tune algorithms, than with team of GPU video processing specialized team.

cma 3y ago

Will your arch work for SSMs?

trsohmers 3y ago
Yes; Mamba was a very easy match, with Hyena also being a good match, but could be greatly optimized with some minimal changes to the model architecture or hardware design.

EvgeniyZh 3y ago

What can you actually do hardware wise with memory bottleneck except for use faster memory?