4 ms·

i don't understand what the second paragraph is saying.

otterdude 4mo ago
Basically getting around the branch predictor problem with generalized compute architectures https://en.wikipedia.org/wiki/Branch_predictor https://en.wikipedia.org/wiki/Branch_predictor
nine_k 4mo ago
In very crude terms, AFAICT, if you have a bunch of matrix multiplications, but one of matrices (the one with model weights) doesn't change, you can seriously speed up the computation. One thing is that you don't need to re-fetch the elements of the constant matrix, you can keep it near the ALUs. Then you maybe can detect and ignore sparse / empty blocks by marking them once. IDK how the custom hardware exploits this; would love to hear any ideas!
- guyomes 4mo ago
  > IDK how the custom hardware exploits this; would love to hear any ideas! You might like this article [1], titled "FPGA-based CNN Acceleration using Pattern-Aware Pruning". More context and details can be found in the PhD thesis of Léo Pradels [2]. [1]: https://inria.hal.science/hal-04689673/document https://inria.hal.science/hal-04689673/document [2]: https://theses.hal.science/tel-05021575v1/file/PRADELS_Leo.pdf https://theses.hal.science/tel-05021575v1/file/PRADELS_Leo.p...
- cm2187 4mo ago
  Random thought. Once models stabilise, could you possibly hardcode the model in gates? Or are they too large for a single chip?
  - 8note 4mo ago
    https://www.anuragk.com/blog/posts/Taalas.html https://www.anuragk.com/blog/posts/Taalas.html
  - lsaferite 4mo ago
    https://taalas.com/ https://taalas.com/
    - jwHollister 4mo ago
      wow if they can get something like this working, what happens to all this infrastructure? Hyperscalers have to be assuming the lifespan of that stuff wrong considering the next gen will be 1000x more efficient.
      - otterley 4mo ago
        The question isn’t whether it works (it does); the question is whether there are buyers for hardware that is obsolete the day it ships. Models evolve much more quickly than hardware can keep up.
        simondotau 4mo ago
        Presumably at some point the rapid progress of models will plateau, at least insofar as a model could be frozen in time and remain economically useful for the expected life of hardware. Especially if it comes with compelling benefits e.g. dramatically lower latency and/or dramatically higher performance per watt. If you can build chips that could run one specific LLM 100x faster than anything else, it would have a use case that nothing else could match.
        fragmede 4mo ago
        https://www.cerebras.ai/ https://www.cerebras.ai/ is exactly that! Holy shit it's fast.
        otterley 4mo ago
        Cerebras is not that. Cerebras isn’t tied to a particular model like Taalas is. The latter is even faster than Cerebras.
- fulafel 4mo ago
  Current accelerators (TPUs, various onchip NPUs) are something close to this. Systolic array is the estabilished computer architecture term for flowing data from computation to computation without the overhead of a register file or von Neumann bottleneck.

Basically getting around the branch predictor problem with generalized compute architectures https://en.wikipedia.org/wiki/Branch_predictor https://en.wikipedia.org/wiki/Branch_predictor

nine_k 4mo ago

In very crude terms, AFAICT, if you have a bunch of matrix multiplications, but one of matrices (the one with model weights) doesn't change, you can seriously speed up the computation. One thing is that you don't need to re-fetch the elements of the constant matrix, you can keep it near the ALUs. Then you maybe can detect and ignore sparse / empty blocks by marking them once. IDK how the custom hardware exploits this; would love to hear any ideas!

guyomes 4mo ago
> IDK how the custom hardware exploits this; would love to hear any ideas! You might like this article [1], titled "FPGA-based CNN Acceleration using Pattern-Aware Pruning". More context and details can be found in the PhD thesis of Léo Pradels [2]. [1]: https://inria.hal.science/hal-04689673/document https://inria.hal.science/hal-04689673/document [2]: https://theses.hal.science/tel-05021575v1/file/PRADELS_Leo.pdf https://theses.hal.science/tel-05021575v1/file/PRADELS_Leo.p...
cm2187 4mo ago
Random thought. Once models stabilise, could you possibly hardcode the model in gates? Or are they too large for a single chip?
- 8note 4mo ago
  https://www.anuragk.com/blog/posts/Taalas.html https://www.anuragk.com/blog/posts/Taalas.html
- lsaferite 4mo ago
  https://taalas.com/ https://taalas.com/
  - jwHollister 4mo ago
    wow if they can get something like this working, what happens to all this infrastructure? Hyperscalers have to be assuming the lifespan of that stuff wrong considering the next gen will be 1000x more efficient.
    - otterley 4mo ago
      The question isn’t whether it works (it does); the question is whether there are buyers for hardware that is obsolete the day it ships. Models evolve much more quickly than hardware can keep up.
      - simondotau 4mo ago
        Presumably at some point the rapid progress of models will plateau, at least insofar as a model could be frozen in time and remain economically useful for the expected life of hardware. Especially if it comes with compelling benefits e.g. dramatically lower latency and/or dramatically higher performance per watt. If you can build chips that could run one specific LLM 100x faster than anything else, it would have a use case that nothing else could match.
        fragmede 4mo ago
        https://www.cerebras.ai/ https://www.cerebras.ai/ is exactly that! Holy shit it's fast.
        otterley 4mo ago
        Cerebras is not that. Cerebras isn’t tied to a particular model like Taalas is. The latter is even faster than Cerebras.
fulafel 4mo ago
Current accelerators (TPUs, various onchip NPUs) are something close to this. Systolic array is the estabilished computer architecture term for flowing data from computation to computation without the overhead of a register file or von Neumann bottleneck.