9 ms·

From the perspective of an outsider, I can't see how a company like this could survive. They claim on the one hand to have done something really amazing and are

by wbhart 7y ago

From the perspective of an outsider, I can't see how a company like this could survive. They claim on the one hand to have done something really amazing and are at the stage where they are looking for customers. Normally, you'd expect them to be touting performance figures to secure such investment. Instead, they've decided to keep the performance secret. And they've managed to find some "expert" who says this is normal. Does anyone here have expertise in this area? Is this the model for a successful company in this area?

tedivm 7y ago
I got a demo of this two years ago, and honestly I don't think it matters that they aren't sharing these numbers. Any company that is going to consider this is going to want to benchmark it on their own models and systems, and as long as Cerebras allows that they aren't going to have trouble finding customers (assuming their claims line up with reality). Even if that doesn't work out most of the people on these time have built companies that were acquired by either AMD or another chip maker.
- justicezyx 7y ago
  Mass market customers are just going to skip without benchmarks. Although, at this stage, Crebras does not care about mass market yet.
deleted 7y ago
[deleted]
phonon 7y ago
There are probably only a few hundred prospective customers. (Some may buy several units). Each unit will cost millions. They can discuss the expected workloads/performance with each prospective customer individually.
jandrese 7y ago
Keeping the performance figures a secret is a red flag on the level of "run, don't walk, away from this company". At best their solution is on par with GPUs in a performance per watt/dollar sense. At worst they're scammers looking for a sucker.
- privateSFacct 7y ago
  I'm also very curious about their performance per watt / dollar for the standard ML datasets out there (facial recognition etc). We have a reasonable sense of both training time and runtime for these in the cloud (and it is falling FAST).
recursivecaveat 7y ago
As someone who works for another startup in this area, building the chip is only half the battle. The other half is tooling for compiling benchmark networks onto the chip in a performant manner. With 400k cores and their 'duplicate and re-route' defect strategy, this might literally be the most challenging compilation target ever made. It probably stacks up absolutely terribly in every metric right now. That's not to say it will necessarily get better, most of the people I've talked to don't think the megachip will ultimately amount to much more than a clever marketing ploy.
- dnautics 7y ago
  Is it not the case that the defect identification and rerouting happens at the hw level in a QA phase post production? If not I'm even less bullish on cerebras.
  - HereBeBeasties 7y ago
    Yes, that's what their web site says.
- joe_the_user 7y ago
  With 400k cores and their 'duplicate and re-route' defect strategy, this might literally be the most challenging compilation target ever made. While I'd be generally skeptical, it seems like the compilation for the rerouting could be done on a single low level, below whatever their assembler is, and so the could just look like a regular array of cores - just a single array that translates from i to the ith "real" core and similar structures seems like it could be enough. Edit: I mean, if they're smart, it seems like they'd make the thing look as much as possible like a generic GPU capable of OpenCL. I have no idea if they'll do that but since they have size, they won't have to sell their stuff an otherwise custom approach.
- Veedrac 7y ago
  A bit baffled by this because on every axis I look this seems like a dream of a compilation target. * No DRAM or caches, everything is in SRAM, and all local SRAM loads are 1 cycle. * Model parallel alone is full performance, no need for data parallel if you size to fit. * Defects are handled in hardware; any latency differences are hidden & not in load path anyway. * Fully asynchronous/dataflow by default, only need minimal synchronization between forward/backward passes. I genuinely don't know how you'd build a simpler system than this.
Veedrac 7y ago
They have customers already, one (Argonne National Labs) is given explicitly. The issue with using ‘industry standard’ benchmarks is that it's like measuring a bus' efficiency by shuttling around a single person at a time. The CS-1 is just bigger than that; the workloads that it provides the most value on are ones that are sized to fit, and specifically built for the device. This does make it hard to evaluate as outsiders (certainly for similar reasons I never liked Graphcore), but I don't think it means anything as grim as you say. The recipe fits.
- m0zg 7y ago
  That' sounds like horseshit to me. Very large public datasets and models are available to test training on a chip or system of any size. ImageNet is large enough for this. But if that's not sufficient, OpenImages is also available. To me as a practitioner a meaningful metric would be "it trains an ImageNet classifier to e.g. 80% top1 in a minute". If it's not suitable for CNNs, do BERT or something else non-convolutional. Even better if I can replicate this result in a public cloud somewhere. They know this, and yet all we have is a single mention of a customer under an NDA and no public benchmarks of any kind, let alone any verifiable ones. If it did excel at those, we'd already know.
  - tynpeddler 7y ago
    > Cerebras hasn’t released MLPerf results or any other independently verifiable apples-to-apples comparisons. Instead the company prefers to let customers try out the CS-1 using their own neural networks and data. > This approach is not unusual, according to analysts. “Everybody runs their own models that they developed for their own business,” says Karl Freund, an AI analyst at Moor Insights. “That’s the only thing that matters to buyers.” Sounds like instead of benchmarks, prospective customers get a chance to run a workload of their choice on the core before purchase. Assuming support is good, that's way better than looking at benchmarks, because you're guaranteed that the performances you're comparing are for workloads you care about.
  - Veedrac 7y ago
    The appropriately large models with public recognition I know of use attention, which is too memory-hungry to work effectively on the CS-1. The datasets aren't the issue. I'm fine with skepticism. It's certainly plausible that they don't actually do all that well.

by wbhart 7y ago

tedivm 7y ago

I got a demo of this two years ago, and honestly I don't think it matters that they aren't sharing these numbers. Any company that is going to consider this is going to want to benchmark it on their own models and systems, and as long as Cerebras allows that they aren't going to have trouble finding customers (assuming their claims line up with reality). Even if that doesn't work out most of the people on these time have built companies that were acquired by either AMD or another chip maker.

justicezyx 7y ago
Mass market customers are just going to skip without benchmarks. Although, at this stage, Crebras does not care about mass market yet.

deleted 7y ago

[deleted]

phonon 7y ago

There are probably only a few hundred prospective customers. (Some may buy several units). Each unit will cost millions. They can discuss the expected workloads/performance with each prospective customer individually.

jandrese 7y ago

Keeping the performance figures a secret is a red flag on the level of "run, don't walk, away from this company". At best their solution is on par with GPUs in a performance per watt/dollar sense. At worst they're scammers looking for a sucker.

privateSFacct 7y ago
I'm also very curious about their performance per watt / dollar for the standard ML datasets out there (facial recognition etc). We have a reasonable sense of both training time and runtime for these in the cloud (and it is falling FAST).

recursivecaveat 7y ago

As someone who works for another startup in this area, building the chip is only half the battle. The other half is tooling for compiling benchmark networks onto the chip in a performant manner. With 400k cores and their 'duplicate and re-route' defect strategy, this might literally be the most challenging compilation target ever made. It probably stacks up absolutely terribly in every metric right now. That's not to say it will necessarily get better, most of the people I've talked to don't think the megachip will ultimately amount to much more than a clever marketing ploy.

dnautics 7y ago
Is it not the case that the defect identification and rerouting happens at the hw level in a QA phase post production? If not I'm even less bullish on cerebras.
- HereBeBeasties 7y ago
  Yes, that's what their web site says.
joe_the_user 7y ago
With 400k cores and their 'duplicate and re-route' defect strategy, this might literally be the most challenging compilation target ever made. While I'd be generally skeptical, it seems like the compilation for the rerouting could be done on a single low level, below whatever their assembler is, and so the could just look like a regular array of cores - just a single array that translates from i to the ith "real" core and similar structures seems like it could be enough. Edit: I mean, if they're smart, it seems like they'd make the thing look as much as possible like a generic GPU capable of OpenCL. I have no idea if they'll do that but since they have size, they won't have to sell their stuff an otherwise custom approach.
Veedrac 7y ago
A bit baffled by this because on every axis I look this seems like a dream of a compilation target. * No DRAM or caches, everything is in SRAM, and all local SRAM loads are 1 cycle. * Model parallel alone is full performance, no need for data parallel if you size to fit. * Defects are handled in hardware; any latency differences are hidden & not in load path anyway. * Fully asynchronous/dataflow by default, only need minimal synchronization between forward/backward passes. I genuinely don't know how you'd build a simpler system than this.

Veedrac 7y ago

They have customers already, one (Argonne National Labs) is given explicitly. The issue with using ‘industry standard’ benchmarks is that it's like measuring a bus' efficiency by shuttling around a single person at a time. The CS-1 is just bigger than that; the workloads that it provides the most value on are ones that are sized to fit, and specifically built for the device. This does make it hard to evaluate as outsiders (certainly for similar reasons I never liked Graphcore), but I don't think it means anything as grim as you say. The recipe fits.

m0zg 7y ago
That' sounds like horseshit to me. Very large public datasets and models are available to test training on a chip or system of any size. ImageNet is large enough for this. But if that's not sufficient, OpenImages is also available. To me as a practitioner a meaningful metric would be "it trains an ImageNet classifier to e.g. 80% top1 in a minute". If it's not suitable for CNNs, do BERT or something else non-convolutional. Even better if I can replicate this result in a public cloud somewhere. They know this, and yet all we have is a single mention of a customer under an NDA and no public benchmarks of any kind, let alone any verifiable ones. If it did excel at those, we'd already know.
- tynpeddler 7y ago
  > Cerebras hasn’t released MLPerf results or any other independently verifiable apples-to-apples comparisons. Instead the company prefers to let customers try out the CS-1 using their own neural networks and data. > This approach is not unusual, according to analysts. “Everybody runs their own models that they developed for their own business,” says Karl Freund, an AI analyst at Moor Insights. “That’s the only thing that matters to buyers.” Sounds like instead of benchmarks, prospective customers get a chance to run a workload of their choice on the core before purchase. Assuming support is good, that's way better than looking at benchmarks, because you're guaranteed that the performances you're comparing are for workloads you care about.
- Veedrac 7y ago
  The appropriately large models with public recognition I know of use attention, which is too memory-hungry to work effectively on the CS-1. The datasets aren't the issue. I'm fine with skepticism. It's certainly plausible that they don't actually do all that well.