5 ms·

For those hearing about Cerebras for the first time, they make a chipset that's similar to a GPU in matrix multiplication speed but way bigger (a whole wafer) s

by icyfox 3y ago

For those hearing about Cerebras for the first time, they make a chipset that's similar to a GPU in matrix multiplication speed but way bigger (a whole wafer) so it can fit more transistors and memory onto one chip. They achieve this small LOC count because they don't need to shard across multiple devices / backprop consolidate on a central CPU / etc. These tricks are usually what blows up a project from a single architecture proof of concept to a robust training pipeline that can handle the billions of parameters on modern models. This is more akin to training a whole model on a single GPU because... it kind of is. Even with a wafer scale chipset this approach has limits. You eventually will still need to shard to fit more parameters / use different training modalities / etc. I'd look at this more as a proof of concept for the ergonomics of what LLM training can look like when you have access to a much larger compute primitive versus a new state of the art in feature-equivalent clean code. Disclaimer: I'm a small investor in Cerebras.

TeMPOraL 3y ago
This does however feel a bit like that "big data" phenomenon, where most companies deploy ridiculously overcomplicated distributed data clusters, where their actual problems could be handled much more simply, cheaply and efficiently, by a single server with a lots of RAM and a solution somewhere on the spectrum between "bunch of UNIX pipes with standard UNIX text processing tools" and "tuned PostgreSQL" / "tuned in-memory SQLite". That is: a lot of distributed big data processing tasks don't need to be distributed. Perhaps with beefy enough matrix multiplication chips, a lot of "big ML" tasks won't need to be distributed either.
- icyfox 3y ago
  I'm a big believer in the approach you're laying out too. Bugs are much easier to diagnose, crash reporting is more straightforward, and you don't need augmented services to consolidate everything at the output layer. That said - I've been predicting a shift back to simple architectures for awhile now and they haven't really come to pass. Maybe there's too much pressure or financial incentives for increasing complexity to solve increasing complexity? At the end of the day I do believe ergonomics are going to win out. I think that's in large part why pytorch won out over tensorflow and jax; it provided the just-in-time computation that would allow people to more easily find bugs & visualize results without having to `compile()` everything down to a static computation graph. Hardware seems like a natural place for that abstraction layer - but maybe the silver bullet will really be on the software side, since we already have too many "low-RAM" equivalent ML devices in the wild. Cheaper to string things together after the fact vs. shipping net new hardware.
  - cs702 3y ago
    The other thing to keep in mind is that Transformers may well be end up being supplanted by more efficient alternatives with different hardware requirements. For instance, right now there's a new crop of "linear RNNs" (RWKV, Mamba, retnet, etc.) claiming to be as good as Transformers for language modeling but with two advantages: their compute cost is O(n) instead of O(n²), and they don't need to keep past context in memory. I don't know if these linear RNNs will actually supplant Transformers, but I do think hardware requirements are likely to change over time.
  - happytiger 3y ago
    Which architecture more closely resembles the human brain?
    - esafak 3y ago
      Spiking neural networks, I think.
- kristjansson 3y ago
  You can hide a whole lot of essential complexity in a hardware layer. However, the very next question a researcher will ask once a model fits on one device is “can I make it twice as fast/big if I use two?”
  - TeMPOraL 3y ago
    I'd think that such researcher would've already heard of adages like "you can't get nine women to give birth in one month", or "where there's six cooks, there's nothing to eat". Or more directly, perhaps one should ask such researcher, "if your team was to double in head count, would you do this project twice as fast?".
    - andy99 3y ago
      > Or more directly, perhaps one should ask such researcher, "if your team was to double in head count, would you do this project twice as fast?". If everybody's job was just to do dot products all day I'd hope the answer would be yes.
      - mejutoco 3y ago
        Which dot products did you do, I'll do the next one. Oh, that was John's, but he is away on vacation today. Let me take care of John's and tomorrow we have a quick meeting to see which matrix he takes next. Sounds a lot like a bus :)
    - kristjansson 3y ago
      8 GPUs do a pretty bang-up job of doing 8 months of compute in a month :). I think the broader point is that the last x0 years of ML research show that more compute is better, both for iteration speed and for resulting performance. Distribution is just the natural outgrowth of that imperative once it reaches the limit of a single device/node. If Cerebras can address models at today's scale on one device, the immediate next step is "what can N of these devices do together to build models at tomorrow's scale" [0]: https://www.cerebras.net/condor-galaxy-1 https://www.cerebras.net/condor-galaxy-1
- cbsmith 3y ago
  In fairness, it's more a case of companies incorrectly identifying their problem as a "big data" problem, usually due to ego or resume padding. If you genuinely have a "big data" problem, you probably do need distributed data clusters.
- spacecadet 3y ago
  Agreed and thank you for also posting. I haven't incurred any cloud costs and my home server upgrades have been done for cheap, if you really look. I laughed because my "pipe" is as you put it, "Unix pipes with standard text processing". Hey, it works.
- sailingparrot 3y ago
  I don't think that's a totally fair comparison. We don't have overcomplicated distributed training infra. They are pretty much as complicated as needed. Scaling vertically (having a bigger chip) is very hard. There are tons of tradeoff when making the chip, and it's overall an insanely complex problem. That's why Cerebras is a 8 years old company and yet you would be hard pressed to find anyone using them still. And even if you give me a Cerebras chip that works perfectly, it will still be much easier for me to buy two of those chips and link them together in distributed training mode, than it will be for Cerebras to build a chip that is 2x the size. The scale of the current generation of clusters to train models the size of GPT-4 are in the range of 25,000+ GPUs with 80GB of memory each, so no matter your chip size, complicated distributed infra is a necessity. Even assuming everything on Cerebras' marketing page is fully accurate, you would still need to distribute the training over 500+ of those massive chips to replicate a 25k GPU cluster.
jpetrucc 3y ago
Are these chipsets anywhere in the price range that would make them feasible for consumer/pro-sumer? I wasn't able to find anything related to pricing without contacting sales. With the numbers being thrown about, this would appear to be "enterprise-only"
- choppaface 3y ago
  No, three years ago the CS1 was already $1-2m minimum per chip https://www.anandtech.com/show/15838/cerebras-wafer-scale-engine-scores-a-sale-5m-buys-two-for-the-pittsburgh-supercomputing-center https://www.anandtech.com/show/15838/cerebras-wafer-scale-en...
WhitneyLand 3y ago
I didn’t notice performance mentioned anywhere in the article and I think that’s not a good sign at all.
- choppaface 3y ago
  For transformers, especially multi-device training pipelines, yes the codebase can normally be 100ksloc and require a team to do at industry scale. See e.g. Hugginface Transformers, or the Megatron impl they cite. Cerebras is trying to show how easy it is to on-board single ICs and demo their pytorch integration. But yeah, where's the wallclock time comparison?! Surely they did one during development, and surely the Sales team knows (or they do once the article was published), yet not even a hint of what their throughput is like. For Cerebras to be this far and not be plastering benchmarks everywhere is a bad sign. Maybe they're going to just die off like Graphcore.
  - deleted 3y ago
    [deleted]
waynesonfire 3y ago
> I'm a small investor in Cerebras. Hey investor, we want chatgpt 5. The issue is not lines of code.
- icyfox 3y ago
  Well thankfully my day job is actually ML research. :) Lines of code is not strictly speaking a bottleneck for the next generation of models, but it ties with other objectives that are: researcher productivity, hardware efficiency, and model verifiability. GPT-5 might be another case of simply scaling up the existing transformer models, but the next step-function change of model quality are going to involve a lot more R&D about the right architectural primitives to take before scaling it up. And in that case lines of code do matter - because they allow simple concepts to be robustly tested, optimized, and iterated against. Doing that against a 50k monolith is a much harder task.

by icyfox 3y ago

TeMPOraL 3y ago

This does however feel a bit like that "big data" phenomenon, where most companies deploy ridiculously overcomplicated distributed data clusters, where their actual problems could be handled much more simply, cheaply and efficiently, by a single server with a lots of RAM and a solution somewhere on the spectrum between "bunch of UNIX pipes with standard UNIX text processing tools" and "tuned PostgreSQL" / "tuned in-memory SQLite". That is: a lot of distributed big data processing tasks don't need to be distributed. Perhaps with beefy enough matrix multiplication chips, a lot of "big ML" tasks won't need to be distributed either.

icyfox 3y ago
I'm a big believer in the approach you're laying out too. Bugs are much easier to diagnose, crash reporting is more straightforward, and you don't need augmented services to consolidate everything at the output layer. That said - I've been predicting a shift back to simple architectures for awhile now and they haven't really come to pass. Maybe there's too much pressure or financial incentives for increasing complexity to solve increasing complexity? At the end of the day I do believe ergonomics are going to win out. I think that's in large part why pytorch won out over tensorflow and jax; it provided the just-in-time computation that would allow people to more easily find bugs & visualize results without having to `compile()` everything down to a static computation graph. Hardware seems like a natural place for that abstraction layer - but maybe the silver bullet will really be on the software side, since we already have too many "low-RAM" equivalent ML devices in the wild. Cheaper to string things together after the fact vs. shipping net new hardware.
- cs702 3y ago
  The other thing to keep in mind is that Transformers may well be end up being supplanted by more efficient alternatives with different hardware requirements. For instance, right now there's a new crop of "linear RNNs" (RWKV, Mamba, retnet, etc.) claiming to be as good as Transformers for language modeling but with two advantages: their compute cost is O(n) instead of O(n²), and they don't need to keep past context in memory. I don't know if these linear RNNs will actually supplant Transformers, but I do think hardware requirements are likely to change over time.
- happytiger 3y ago
  Which architecture more closely resembles the human brain?
  - esafak 3y ago
    Spiking neural networks, I think.
kristjansson 3y ago
You can hide a whole lot of essential complexity in a hardware layer. However, the very next question a researcher will ask once a model fits on one device is “can I make it twice as fast/big if I use two?”
- TeMPOraL 3y ago
  I'd think that such researcher would've already heard of adages like "you can't get nine women to give birth in one month", or "where there's six cooks, there's nothing to eat". Or more directly, perhaps one should ask such researcher, "if your team was to double in head count, would you do this project twice as fast?".
  - andy99 3y ago
    > Or more directly, perhaps one should ask such researcher, "if your team was to double in head count, would you do this project twice as fast?". If everybody's job was just to do dot products all day I'd hope the answer would be yes.
    - mejutoco 3y ago
      Which dot products did you do, I'll do the next one. Oh, that was John's, but he is away on vacation today. Let me take care of John's and tomorrow we have a quick meeting to see which matrix he takes next. Sounds a lot like a bus :)
  - kristjansson 3y ago
    8 GPUs do a pretty bang-up job of doing 8 months of compute in a month :). I think the broader point is that the last x0 years of ML research show that more compute is better, both for iteration speed and for resulting performance. Distribution is just the natural outgrowth of that imperative once it reaches the limit of a single device/node. If Cerebras can address models at today's scale on one device, the immediate next step is "what can N of these devices do together to build models at tomorrow's scale" [0]: https://www.cerebras.net/condor-galaxy-1 https://www.cerebras.net/condor-galaxy-1
cbsmith 3y ago
In fairness, it's more a case of companies incorrectly identifying their problem as a "big data" problem, usually due to ego or resume padding. If you genuinely have a "big data" problem, you probably do need distributed data clusters.
spacecadet 3y ago
Agreed and thank you for also posting. I haven't incurred any cloud costs and my home server upgrades have been done for cheap, if you really look. I laughed because my "pipe" is as you put it, "Unix pipes with standard text processing". Hey, it works.
sailingparrot 3y ago
I don't think that's a totally fair comparison. We don't have overcomplicated distributed training infra. They are pretty much as complicated as needed. Scaling vertically (having a bigger chip) is very hard. There are tons of tradeoff when making the chip, and it's overall an insanely complex problem. That's why Cerebras is a 8 years old company and yet you would be hard pressed to find anyone using them still. And even if you give me a Cerebras chip that works perfectly, it will still be much easier for me to buy two of those chips and link them together in distributed training mode, than it will be for Cerebras to build a chip that is 2x the size. The scale of the current generation of clusters to train models the size of GPT-4 are in the range of 25,000+ GPUs with 80GB of memory each, so no matter your chip size, complicated distributed infra is a necessity. Even assuming everything on Cerebras' marketing page is fully accurate, you would still need to distribute the training over 500+ of those massive chips to replicate a 25k GPU cluster.

jpetrucc 3y ago

Are these chipsets anywhere in the price range that would make them feasible for consumer/pro-sumer? I wasn't able to find anything related to pricing without contacting sales. With the numbers being thrown about, this would appear to be "enterprise-only"

choppaface 3y ago
No, three years ago the CS1 was already $1-2m minimum per chip https://www.anandtech.com/show/15838/cerebras-wafer-scale-engine-scores-a-sale-5m-buys-two-for-the-pittsburgh-supercomputing-center https://www.anandtech.com/show/15838/cerebras-wafer-scale-en...

WhitneyLand 3y ago

I didn’t notice performance mentioned anywhere in the article and I think that’s not a good sign at all.

choppaface 3y ago
For transformers, especially multi-device training pipelines, yes the codebase can normally be 100ksloc and require a team to do at industry scale. See e.g. Hugginface Transformers, or the Megatron impl they cite. Cerebras is trying to show how easy it is to on-board single ICs and demo their pytorch integration. But yeah, where's the wallclock time comparison?! Surely they did one during development, and surely the Sales team knows (or they do once the article was published), yet not even a hint of what their throughput is like. For Cerebras to be this far and not be plastering benchmarks everywhere is a bad sign. Maybe they're going to just die off like Graphcore.
- deleted 3y ago
  [deleted]

waynesonfire 3y ago

> I'm a small investor in Cerebras. Hey investor, we want chatgpt 5. The issue is not lines of code.

icyfox 3y ago
Well thankfully my day job is actually ML research. :) Lines of code is not strictly speaking a bottleneck for the next generation of models, but it ties with other objectives that are: researcher productivity, hardware efficiency, and model verifiability. GPT-5 might be another case of simply scaling up the existing transformer models, but the next step-function change of model quality are going to involve a lot more R&D about the right architectural primitives to take before scaling it up. And in that case lines of code do matter - because they allow simple concepts to be robustly tested, optimized, and iterated against. Doing that against a 50k monolith is a much harder task.