3 ms·
This does however feel a bit like that "big data" phenomenon, where most companies deploy ridiculously overcomplicated distributed data clusters, where their ac
by TeMPOraL 3y ago
This does however feel a bit like that "big data" phenomenon, where most companies deploy ridiculously overcomplicated distributed data clusters, where their actual problems could be handled much more simply, cheaply and efficiently, by a single server with a lots of RAM and a solution somewhere on the spectrum between "bunch of UNIX pipes with standard UNIX text processing tools" and "tuned PostgreSQL" / "tuned in-memory SQLite".
That is: a lot of distributed big data processing tasks don't need to be distributed. Perhaps with beefy enough matrix multiplication chips, a lot of "big ML" tasks won't need to be distributed either.
- icyfox 3y agoI'm a big believer in the approach you're laying out too. Bugs are much easier to diagnose, crash reporting is more straightforward, and you don't need augmented services to consolidate everything at the output layer. That said - I've been predicting a shift back to simple architectures for awhile now and they haven't really come to pass. Maybe there's too much pressure or financial incentives for increasing complexity to solve increasing complexity? At the end of the day I do believe ergonomics are going to win out. I think that's in large part why pytorch won out over tensorflow and jax; it provided the just-in-time computation that would allow people to more easily find bugs & visualize results without having to `compile()` everything down to a static computation graph. Hardware seems like a natural place for that abstraction layer - but maybe the silver bullet will really be on the software side, since we already have too many "low-RAM" equivalent ML devices in the wild. Cheaper to string things together after the fact vs. shipping net new hardware.
- cs702 3y agoThe other thing to keep in mind is that Transformers may well be end up being supplanted by more efficient alternatives with different hardware requirements. For instance, right now there's a new crop of "linear RNNs" (RWKV, Mamba, retnet, etc.) claiming to be as good as Transformers for language modeling but with two advantages: their compute cost is O(n) instead of O(n²), and they don't need to keep past context in memory. I don't know if these linear RNNs will actually supplant Transformers, but I do think hardware requirements are likely to change over time.
- happytiger 3y agoWhich architecture more closely resembles the human brain?
- esafak 3y agoSpiking neural networks, I think.
- kristjansson 3y agoYou can hide a whole lot of essential complexity in a hardware layer. However, the very next question a researcher will ask once a model fits on one device is “can I make it twice as fast/big if I use two?”
- TeMPOraL 3y agoI'd think that such researcher would've already heard of adages like "you can't get nine women to give birth in one month", or "where there's six cooks, there's nothing to eat". Or more directly, perhaps one should ask such researcher, "if your team was to double in head count, would you do this project twice as fast?".
- andy99 3y ago> Or more directly, perhaps one should ask such researcher, "if your team was to double in head count, would you do this project twice as fast?". If everybody's job was just to do dot products all day I'd hope the answer would be yes.
- mejutoco 3y agoWhich dot products did you do, I'll do the next one. Oh, that was John's, but he is away on vacation today. Let me take care of John's and tomorrow we have a quick meeting to see which matrix he takes next. Sounds a lot like a bus :)
- kristjansson 3y ago8 GPUs do a pretty bang-up job of doing 8 months of compute in a month :). I think the broader point is that the last x0 years of ML research show that more compute is better, both for iteration speed and for resulting performance. Distribution is just the natural outgrowth of that imperative once it reaches the limit of a single device/node. If Cerebras can address models at today's scale on one device, the immediate next step is "what can N of these devices do together to build models at tomorrow's scale" [0]: https://www.cerebras.net/condor-galaxy-1 https://www.cerebras.net/condor-galaxy-1
- cbsmith 3y agoIn fairness, it's more a case of companies incorrectly identifying their problem as a "big data" problem, usually due to ego or resume padding. If you genuinely have a "big data" problem, you probably do need distributed data clusters.
- spacecadet 3y agoAgreed and thank you for also posting. I haven't incurred any cloud costs and my home server upgrades have been done for cheap, if you really look. I laughed because my "pipe" is as you put it, "Unix pipes with standard text processing". Hey, it works.
- sailingparrot 3y agoI don't think that's a totally fair comparison. We don't have overcomplicated distributed training infra. They are pretty much as complicated as needed. Scaling vertically (having a bigger chip) is very hard. There are tons of tradeoff when making the chip, and it's overall an insanely complex problem. That's why Cerebras is a 8 years old company and yet you would be hard pressed to find anyone using them still. And even if you give me a Cerebras chip that works perfectly, it will still be much easier for me to buy two of those chips and link them together in distributed training mode, than it will be for Cerebras to build a chip that is 2x the size. The scale of the current generation of clusters to train models the size of GPT-4 are in the range of 25,000+ GPUs with 80GB of memory each, so no matter your chip size, complicated distributed infra is a necessity. Even assuming everything on Cerebras' marketing page is fully accurate, you would still need to distribute the training over 500+ of those massive chips to replicate a 25k GPU cluster.