4 ms·

As someone who works for another startup in this area, building the chip is only half the battle. The other half is tooling for compiling benchmark networks ont

As someone who works for another startup in this area, building the chip is only half the battle. The other half is tooling for compiling benchmark networks onto the chip in a performant manner. With 400k cores and their 'duplicate and re-route' defect strategy, this might literally be the most challenging compilation target ever made. It probably stacks up absolutely terribly in every metric right now. That's not to say it will necessarily get better, most of the people I've talked to don't think the megachip will ultimately amount to much more than a clever marketing ploy.

dnautics 7y ago
Is it not the case that the defect identification and rerouting happens at the hw level in a QA phase post production? If not I'm even less bullish on cerebras.
- HereBeBeasties 7y ago
  Yes, that's what their web site says.
joe_the_user 7y ago
With 400k cores and their 'duplicate and re-route' defect strategy, this might literally be the most challenging compilation target ever made. While I'd be generally skeptical, it seems like the compilation for the rerouting could be done on a single low level, below whatever their assembler is, and so the could just look like a regular array of cores - just a single array that translates from i to the ith "real" core and similar structures seems like it could be enough. Edit: I mean, if they're smart, it seems like they'd make the thing look as much as possible like a generic GPU capable of OpenCL. I have no idea if they'll do that but since they have size, they won't have to sell their stuff an otherwise custom approach.
Veedrac 7y ago
A bit baffled by this because on every axis I look this seems like a dream of a compilation target. * No DRAM or caches, everything is in SRAM, and all local SRAM loads are 1 cycle. * Model parallel alone is full performance, no need for data parallel if you size to fit. * Defects are handled in hardware; any latency differences are hidden & not in load path anyway. * Fully asynchronous/dataflow by default, only need minimal synchronization between forward/backward passes. I genuinely don't know how you'd build a simpler system than this.
- IshKebab 7y ago
  I'm afraid recursivecaveat is right. This is an insanely difficult compilation target. I think you're possibly talking about a different kind of "compilation" - i.e. the Clang/GCC bit that converts C++ to machine code. That is indeed trivial. But "compilation" for these chips includes much more than that. The really complicated bit is converting the tensorflow model to some kind of computation plan. Where do you put all the tensor data? How do you move it around the chip. It's insanely complicated. If anything kills Cerebras it will be the software.
  - Veedrac 7y ago
    It's model parallel, so the first thing you do is lay out your floorplan for the model, which looks like this. https://secureservercdn.net/198.12.145.239/a7b.fcb.myftpupload.com/wp-content/uploads/2020/01/f33f23fa38b3f85a893165c25473cf89-2200x950-c-default.png https://secureservercdn.net/198.12.145.239/a7b.fcb.myftpuplo... Then you put your data next to the core that uses it. Simples. (Optimal placement is tricky, but approximate techniques work fine.)
    - deleted 7y ago
      [deleted]
    - IshKebab 7y ago
      When you consider the things that that diagram doesn't show, it doesn't look at all simple. Does that graph even have training? It'll have to be pipelined too. Probably will have to use recomputation due to the shortage of memory. What about within the boxes? You can't nicely separate a matmul into pieces like that. I work on something similar but less ambitious, trust me it is crazy complicated.
      - Veedrac 7y ago
        Could you be more explicit? What about the naïve approach to training (same graph but backwards, computing gradients) is going to fail? Wrt. matmul, if you couldn't split them up, today's AI accelerators wouldn't work full stop. But regardless, even if it was much more complex on CS-1 than on all the other sea-of-multipliers accelerators, it's obviously a problem they've solved and so irrelevant to the compilation issue.

dnautics 7y ago

Is it not the case that the defect identification and rerouting happens at the hw level in a QA phase post production? If not I'm even less bullish on cerebras.

HereBeBeasties 7y ago
Yes, that's what their web site says.

joe_the_user 7y ago

With 400k cores and their 'duplicate and re-route' defect strategy, this might literally be the most challenging compilation target ever made. While I'd be generally skeptical, it seems like the compilation for the rerouting could be done on a single low level, below whatever their assembler is, and so the could just look like a regular array of cores - just a single array that translates from i to the ith "real" core and similar structures seems like it could be enough. Edit: I mean, if they're smart, it seems like they'd make the thing look as much as possible like a generic GPU capable of OpenCL. I have no idea if they'll do that but since they have size, they won't have to sell their stuff an otherwise custom approach.

Veedrac 7y ago

A bit baffled by this because on every axis I look this seems like a dream of a compilation target. * No DRAM or caches, everything is in SRAM, and all local SRAM loads are 1 cycle. * Model parallel alone is full performance, no need for data parallel if you size to fit. * Defects are handled in hardware; any latency differences are hidden & not in load path anyway. * Fully asynchronous/dataflow by default, only need minimal synchronization between forward/backward passes. I genuinely don't know how you'd build a simpler system than this.

IshKebab 7y ago
I'm afraid recursivecaveat is right. This is an insanely difficult compilation target. I think you're possibly talking about a different kind of "compilation" - i.e. the Clang/GCC bit that converts C++ to machine code. That is indeed trivial. But "compilation" for these chips includes much more than that. The really complicated bit is converting the tensorflow model to some kind of computation plan. Where do you put all the tensor data? How do you move it around the chip. It's insanely complicated. If anything kills Cerebras it will be the software.
- Veedrac 7y ago
  It's model parallel, so the first thing you do is lay out your floorplan for the model, which looks like this. https://secureservercdn.net/198.12.145.239/a7b.fcb.myftpupload.com/wp-content/uploads/2020/01/f33f23fa38b3f85a893165c25473cf89-2200x950-c-default.png https://secureservercdn.net/198.12.145.239/a7b.fcb.myftpuplo... Then you put your data next to the core that uses it. Simples. (Optimal placement is tricky, but approximate techniques work fine.)
  - deleted 7y ago
    [deleted]
  - IshKebab 7y ago
    When you consider the things that that diagram doesn't show, it doesn't look at all simple. Does that graph even have training? It'll have to be pipelined too. Probably will have to use recomputation due to the shortage of memory. What about within the boxes? You can't nicely separate a matmul into pieces like that. I work on something similar but less ambitious, trust me it is crazy complicated.
    - Veedrac 7y ago
      Could you be more explicit? What about the naïve approach to training (same graph but backwards, computing gradients) is going to fail? Wrt. matmul, if you couldn't split them up, today's AI accelerators wouldn't work full stop. But regardless, even if it was much more complex on CS-1 than on all the other sea-of-multipliers accelerators, it's obviously a problem they've solved and so irrelevant to the compilation issue.