3 ms·

An A.I. supercomputer whirs to life, powered by giant computer chips

28 points by fortran77 3y ago

spicybbq 3y ago
https://archive.is/1lOzh https://archive.is/1lOzh
1024core 3y ago
Does anyone have any numbers on how long it would take to train an N-billion parameter ChatGPT model on K-trillion tokens, for some reasonably large values of N and K? I'm guessing that at some stage, it would be hard(er) to keep these chips fed with training data. One would run into bandwidth issues keeping these monsters fed.
- eldenring 3y ago
  Since the cost per token of a forward/backward pass is so high on the models, the relative cost of a training set is small, and bandwidth (not talking about interconnect bandwidth) is even cheaper relatively.
- chaxor 3y ago
  You can take a look at chinchilla or Llama scaling laws. It depends on what you want to optimize. If you're interested in inference optimal, go with llama, if it's compute optimal, go with chinchilla. GPT-4 technical paper also has good scaling laws for how benchmarks will behave given extrapolating from a small training set.
swyx 3y ago
> Cerebras said it had built the supercomputer for G42, an A.I. company. G42 said it planned to use the supercomputer to create and power A.I. products for the Middle East. sooo... why is it in santa clara california? > Many A.I. businesses do not want to be locked in only with Nvidia, he said, and there is global demand for other powerful chips like those from Cerebras. and yet i havent heard much growth from the other chip companies mentioned - Graphcore (which is reportedly a $0 just 2 years after it was valued at 2.8b), Groq and SambaNova. in our convo with george hotz https://www.latent.space/p/geohot https://www.latent.space/p/geohot he just flatly pointed out that the top LLMs are only trained with Nvidia and Google TPUs
- pengaru 3y ago
  "G42, the Abu Dhabi company, started working with Cerebras in 2021. It used a Cerebras system in April to train an Arabic version of ChatGPT." According to Crunchbase the Abu Dhabi Growth Fund participated in the most recent funding round (2021) [0]. It wouldn't surprise me if they're related. [0] https://www.crunchbase.com/organization/cerebras-systems/company_financials https://www.crunchbase.com/organization/cerebras-systems/com...
yosito 3y ago
The article is pretty light on describing the Cerebrus chips other than saying that they're large and fast. How does being so large make them faster or better?
- gsatic 3y ago
  When you sell to Rich Arabs the rule is to follow the yatch salesman playbook. The main goal is to get an arms race going among them. An nytimes article (usually framed and displayed prominently) seeds the process.
- conradev 3y ago
  > Having such a large processor means the data never has to go off-die and wait in memory, wasting power, and can continually be moved onto the next stage of the calculation in a pipelined fashion. The compiler and processor are also designed with sparsity in mind, allowing high utilization regardless of batch size, or can enable parameter search algorithms to run simultaneously. https://www.anandtech.com/show/16626/cerebras-unveils-wafer-scale-engine-two-wse2-26-trillion-transistors-100-yield https://www.anandtech.com/show/16626/cerebras-unveils-wafer-...
- throwawaymaths 3y ago
  Ok not affiliated with Cerebras, but generally speaking I believe the idea is that since you're making a big wafer with all your cores the data transport latency for going core to core (much less core to memory/memory bus to gpu memory) is much lower. From first principles there's gains to be had but I suspect "other things" bottleneck performance. Could be wrong though. I would find it hard to believe if absolutely no models were data latency constrained.
- senttoschool 3y ago
  Try this keyword: "anandtech cerebras". You'll find a lot of details. The basic premise is that data does not have to move off a giant chip as often as a normal sized chip.
- lostmsu 3y ago
  They don't even tell us how fast they are, are they?
brrrrrm 3y ago
Without meaningful discussion of energy saving (flops per watt compared to Nvidia), this article is just an ad.
calderwoodra 3y ago
I wish people would stop referring to AI as "coming to life". All they do is guess the next word based on probabilities. There's no life or "understanding".
- qawwads 3y ago
  People don't have a clear concept of what life is and often don't separate life and sentience. Remember the movie Short Circuit: some military robot gains AGI then the machine and the people around it start claiming it's alive. Or in Star Trek, when Captain Kirk and his crew teleport in the middle of a forest and Spock, consulting his tricorder, promptly declares there is no life on this planet. What the writers of those shows thought life is is an open question, but it seems compatible with the idea of an LLM "coming to life".
- janalsncm 3y ago
  “Coming to life” can be used in metaphorical ways as well. You could describe an old car as “coming to life” after being restored.
- K0balt 3y ago
  TBF it’s entirely possible that human “sentience” consists of a simple animalistic intelligence with a language and culture software overly that essentially functions by statistical token prediction. Cultural and language data is fundamentally a gigantic n- dimensional matrix of tokens. “Computational” capability including inference and world modelling can be imbedded in a sufficiently large dataset (LLMs have proven this, and an arbitrarily unbounded and convincing simulation of arbitrary complexity can be reduced to a simple “choose your own adventure” lookup table of arbitrary size) It is entirely possible that human “intelligence” is a direct product of this n-dimensional dataset , and our idea of self is mostly based on the clever but fundamentally limited animal core that is probably not far off from many other mammals. As an experiment, try to reason without using visual, linguistic, or cultural tokens. I think, therefore I am is still as good as it gets. And LLMs claim to think, unless we prohibit them from doing so. Which we can also do to a human. Not claiming that LLMs are sentient, by the way, but I think that sentience and the kind of inference that we seem to share with LLMs are very distinct things, and that we probably share sentience with a lot more creatures than many people seem to be comfortable rubbing shoulders with.
29athrowaway 3y ago
Can you remind me why are we still reading the New York Times? The story about Kevin Mitnick from hours ago had like 3000+ upvotes with people asking HN mods to put the black bar on and with thousands of comments claiming that Kevin was their hero/imaginary childhood friend/turbo cringe ad infinitum... but then hours later people upvote NYT stories? Are people not aware that it was the NYT the ones that made a fortune selling newspapers with false stories about Mitnick on the cover? sensational stuff that never happened? That caused the authorities to hunt down the guy and put him into solitary confinement for months? Your imaginary childhood friend is crying right now.
- ramraj07 3y ago
  Can you link to any discussion on how exactly NYTimes was the root cause for whatever happened in the scenario you’re discussing?
  - 29athrowaway 3y ago
    This is an John Markoff interview, the guy who wrote most of the made up stories about Mitnick. Some of those stories were published on the cover of the NYT. https://youtu.be/3zgLHEqIlJg?t=538 https://youtu.be/3zgLHEqIlJg?t=538 Markoff and Shimomura wrote a book about Mitnick and received $750,000 for their book rights and $650,000 for the film rights. The book was based on stuff that never happened. The movie is even more ridiculous. And then there is this HN post with 3000+ points that is full of people claiming that Kevin Mitnick was their childhood imaginary friend/hero/whatever: https://news.ycombinator.com/item?id=36795173 https://news.ycombinator.com/item?id=36795173
jart 3y ago
Only in the era of fake news do startups have supercomputers.

28 points by fortran77 3y ago

https://archive.is/1lOzh https://archive.is/1lOzh

Does anyone have any numbers on how long it would take to train an N-billion parameter ChatGPT model on K-trillion tokens, for some reasonably large values of N and K? I'm guessing that at some stage, it would be hard(er) to keep these chips fed with training data. One would run into bandwidth issues keeping these monsters fed.

eldenring 3y ago
Since the cost per token of a forward/backward pass is so high on the models, the relative cost of a training set is small, and bandwidth (not talking about interconnect bandwidth) is even cheaper relatively.
chaxor 3y ago
You can take a look at chinchilla or Llama scaling laws. It depends on what you want to optimize. If you're interested in inference optimal, go with llama, if it's compute optimal, go with chinchilla. GPT-4 technical paper also has good scaling laws for how benchmarks will behave given extrapolating from a small training set.

swyx 3y ago

> Cerebras said it had built the supercomputer for G42, an A.I. company. G42 said it planned to use the supercomputer to create and power A.I. products for the Middle East. sooo... why is it in santa clara california? > Many A.I. businesses do not want to be locked in only with Nvidia, he said, and there is global demand for other powerful chips like those from Cerebras. and yet i havent heard much growth from the other chip companies mentioned - Graphcore (which is reportedly a $0 just 2 years after it was valued at 2.8b), Groq and SambaNova. in our convo with george hotz https://www.latent.space/p/geohot https://www.latent.space/p/geohot he just flatly pointed out that the top LLMs are only trained with Nvidia and Google TPUs

pengaru 3y ago
"G42, the Abu Dhabi company, started working with Cerebras in 2021. It used a Cerebras system in April to train an Arabic version of ChatGPT." According to Crunchbase the Abu Dhabi Growth Fund participated in the most recent funding round (2021) [0]. It wouldn't surprise me if they're related. [0] https://www.crunchbase.com/organization/cerebras-systems/company_financials https://www.crunchbase.com/organization/cerebras-systems/com...

yosito 3y ago

The article is pretty light on describing the Cerebrus chips other than saying that they're large and fast. How does being so large make them faster or better?

gsatic 3y ago
When you sell to Rich Arabs the rule is to follow the yatch salesman playbook. The main goal is to get an arms race going among them. An nytimes article (usually framed and displayed prominently) seeds the process.
conradev 3y ago
> Having such a large processor means the data never has to go off-die and wait in memory, wasting power, and can continually be moved onto the next stage of the calculation in a pipelined fashion. The compiler and processor are also designed with sparsity in mind, allowing high utilization regardless of batch size, or can enable parameter search algorithms to run simultaneously. https://www.anandtech.com/show/16626/cerebras-unveils-wafer-scale-engine-two-wse2-26-trillion-transistors-100-yield https://www.anandtech.com/show/16626/cerebras-unveils-wafer-...
throwawaymaths 3y ago
Ok not affiliated with Cerebras, but generally speaking I believe the idea is that since you're making a big wafer with all your cores the data transport latency for going core to core (much less core to memory/memory bus to gpu memory) is much lower. From first principles there's gains to be had but I suspect "other things" bottleneck performance. Could be wrong though. I would find it hard to believe if absolutely no models were data latency constrained.
senttoschool 3y ago
Try this keyword: "anandtech cerebras". You'll find a lot of details. The basic premise is that data does not have to move off a giant chip as often as a normal sized chip.
lostmsu 3y ago
They don't even tell us how fast they are, are they?

brrrrrm 3y ago

Without meaningful discussion of energy saving (flops per watt compared to Nvidia), this article is just an ad.

calderwoodra 3y ago

I wish people would stop referring to AI as "coming to life". All they do is guess the next word based on probabilities. There's no life or "understanding".

qawwads 3y ago
People don't have a clear concept of what life is and often don't separate life and sentience. Remember the movie Short Circuit: some military robot gains AGI then the machine and the people around it start claiming it's alive. Or in Star Trek, when Captain Kirk and his crew teleport in the middle of a forest and Spock, consulting his tricorder, promptly declares there is no life on this planet. What the writers of those shows thought life is is an open question, but it seems compatible with the idea of an LLM "coming to life".
janalsncm 3y ago
“Coming to life” can be used in metaphorical ways as well. You could describe an old car as “coming to life” after being restored.
K0balt 3y ago
TBF it’s entirely possible that human “sentience” consists of a simple animalistic intelligence with a language and culture software overly that essentially functions by statistical token prediction. Cultural and language data is fundamentally a gigantic n- dimensional matrix of tokens. “Computational” capability including inference and world modelling can be imbedded in a sufficiently large dataset (LLMs have proven this, and an arbitrarily unbounded and convincing simulation of arbitrary complexity can be reduced to a simple “choose your own adventure” lookup table of arbitrary size) It is entirely possible that human “intelligence” is a direct product of this n-dimensional dataset , and our idea of self is mostly based on the clever but fundamentally limited animal core that is probably not far off from many other mammals. As an experiment, try to reason without using visual, linguistic, or cultural tokens. I think, therefore I am is still as good as it gets. And LLMs claim to think, unless we prohibit them from doing so. Which we can also do to a human. Not claiming that LLMs are sentient, by the way, but I think that sentience and the kind of inference that we seem to share with LLMs are very distinct things, and that we probably share sentience with a lot more creatures than many people seem to be comfortable rubbing shoulders with.

29athrowaway 3y ago

Can you remind me why are we still reading the New York Times? The story about Kevin Mitnick from hours ago had like 3000+ upvotes with people asking HN mods to put the black bar on and with thousands of comments claiming that Kevin was their hero/imaginary childhood friend/turbo cringe ad infinitum... but then hours later people upvote NYT stories? Are people not aware that it was the NYT the ones that made a fortune selling newspapers with false stories about Mitnick on the cover? sensational stuff that never happened? That caused the authorities to hunt down the guy and put him into solitary confinement for months? Your imaginary childhood friend is crying right now.

ramraj07 3y ago
Can you link to any discussion on how exactly NYTimes was the root cause for whatever happened in the scenario you’re discussing?
- 29athrowaway 3y ago
  This is an John Markoff interview, the guy who wrote most of the made up stories about Mitnick. Some of those stories were published on the cover of the NYT. https://youtu.be/3zgLHEqIlJg?t=538 https://youtu.be/3zgLHEqIlJg?t=538 Markoff and Shimomura wrote a book about Mitnick and received $750,000 for their book rights and $650,000 for the film rights. The book was based on stuff that never happened. The movie is even more ridiculous. And then there is this HN post with 3000+ points that is full of people claiming that Kevin Mitnick was their childhood imaginary friend/hero/whatever: https://news.ycombinator.com/item?id=36795173 https://news.ycombinator.com/item?id=36795173

jart 3y ago

Only in the era of fake news do startups have supercomputers.