6 ms·

whitepaper: https://www.cerebras.net/wp-content/uploads/2019/08/Cerebras-Wafer-Scale-Engine-Whitepaper.pdf https://www.cerebras.net/wp-content/uploads/2019/08/C

whitepaper: https://www.cerebras.net/wp-content/uploads/2019/08/Cerebras-Wafer-Scale-Engine-Whitepaper.pdf https://www.cerebras.net/wp-content/uploads/2019/08/Cerebras... I don't expect good yields from a chip that takes up the whole wafer. They must disable cores and pieces of SRAM that are damaged. How is this programmed?

groundlogic 7y ago
If you divide that giant piece of silicon into 400k processors, and then only use the ones that actually work... I wonder if they figure that out every time the CPU boots, or at the factory. At this scale, maybe it makes sense to do it all in parallel at boot. Or, even dynamically during runtime. There may be edge case cores that sort of work, and then won't work at different temps, or after aging?
- kingosticks 7y ago
  They'll aim to catch the logic failures and memory failures during wafer test at the factory. This testing is done at room temperature and at hot. There are margins built in to allow for ageing. If they want to ship a decent product they'll also need to repeat the memory testing every boot, and ideally during runtime but maybe the latter isn't a big deal for something like this. EDIT: to add a bit more and possibly address the original question (which I think keveman may have misunderstood), there will usually be some hardware dedicated to controlling the chip's redundancy. Part of that is often a OTP fuse-type thing that can be programmed during wafer test to indicate parts of the chip that don't work. Something (software or hardware) will read that during boot and not use those parts of the chip.
  - groundlogic 7y ago
    Sure, that makes sense. With this many cores it seems like the probability that a core dies during a multi-hour job (or in case it's used for inference, during a very long-lived realtime job) is pretty high, so the software in all layers would need to handle this kind of exception. They probably don't, today, since we haven't seen a 400k core chip before.
keveman 7y ago
> How is this programmed? Full disclosure: I am a Cerebras employee. There is extensive support for TensorFlow. A wide range of models expressed in the TensorFlow will be accelerated transparently.
- paulsutter 7y ago
  He was asking about the implications for yields. Do you route around bad dies/cores, and what are the implication for programming and performance? For everyone else: normally a wafer is divded into dies, each of which (loosely) are a chip. Yield is a percentage of good parts and it's very unlikely that an entire wafer is good. Gene Amdahl estimated that 99.99% yield is needed for successful wafer scale integration: https://en.wikipedia.org/wiki/Wafer-scale_integration https://en.wikipedia.org/wiki/Wafer-scale_integration
  - Veedrac 7y ago
    > For example, the typical 300mm wafer from TSMC may contain “a modest hundred number of flaws,” said Feldman. Cerebras gave its Swarm interconnect redundant links to route around defective tiles and allocated “a little over 1% [of the tiles] as spares.” https://www.eetimes.com/document.asp?doc_id=1335043&page_number=2 https://www.eetimes.com/document.asp?doc_id=1335043&page_num...
- streetcat1 7y ago
  So why are you not publishing benchmarks against nvidia?
  - sanxiyn 7y ago
    Cerebras is an MLPerf member, so they will publish MLPerf numbers some day and then we will talk.
    - streetcat1 7y ago
      They probably run the benchmark (I guess many times, and not only against nvidia). But yet it is not in the white paper. I was an SE at an hardware company and it is the first thing that you do as a product manager.

groundlogic 7y ago

If you divide that giant piece of silicon into 400k processors, and then only use the ones that actually work... I wonder if they figure that out every time the CPU boots, or at the factory. At this scale, maybe it makes sense to do it all in parallel at boot. Or, even dynamically during runtime. There may be edge case cores that sort of work, and then won't work at different temps, or after aging?

kingosticks 7y ago
They'll aim to catch the logic failures and memory failures during wafer test at the factory. This testing is done at room temperature and at hot. There are margins built in to allow for ageing. If they want to ship a decent product they'll also need to repeat the memory testing every boot, and ideally during runtime but maybe the latter isn't a big deal for something like this. EDIT: to add a bit more and possibly address the original question (which I think keveman may have misunderstood), there will usually be some hardware dedicated to controlling the chip's redundancy. Part of that is often a OTP fuse-type thing that can be programmed during wafer test to indicate parts of the chip that don't work. Something (software or hardware) will read that during boot and not use those parts of the chip.
- groundlogic 7y ago
  Sure, that makes sense. With this many cores it seems like the probability that a core dies during a multi-hour job (or in case it's used for inference, during a very long-lived realtime job) is pretty high, so the software in all layers would need to handle this kind of exception. They probably don't, today, since we haven't seen a 400k core chip before.

keveman 7y ago

> How is this programmed? Full disclosure: I am a Cerebras employee. There is extensive support for TensorFlow. A wide range of models expressed in the TensorFlow will be accelerated transparently.

paulsutter 7y ago
He was asking about the implications for yields. Do you route around bad dies/cores, and what are the implication for programming and performance? For everyone else: normally a wafer is divded into dies, each of which (loosely) are a chip. Yield is a percentage of good parts and it's very unlikely that an entire wafer is good. Gene Amdahl estimated that 99.99% yield is needed for successful wafer scale integration: https://en.wikipedia.org/wiki/Wafer-scale_integration https://en.wikipedia.org/wiki/Wafer-scale_integration
- Veedrac 7y ago
  > For example, the typical 300mm wafer from TSMC may contain “a modest hundred number of flaws,” said Feldman. Cerebras gave its Swarm interconnect redundant links to route around defective tiles and allocated “a little over 1% [of the tiles] as spares.” https://www.eetimes.com/document.asp?doc_id=1335043&page_number=2 https://www.eetimes.com/document.asp?doc_id=1335043&page_num...
streetcat1 7y ago
So why are you not publishing benchmarks against nvidia?
- sanxiyn 7y ago
  Cerebras is an MLPerf member, so they will publish MLPerf numbers some day and then we will talk.
  - streetcat1 7y ago
    They probably run the benchmark (I guess many times, and not only against nvidia). But yet it is not in the white paper. I was an SE at an hardware company and it is the first thing that you do as a product manager.