29 ms·
Cerebras Systems unveils a 1.2T transistor chip for AI
- navaati 7y agoWow ! 56 (!) times more core than the biggest Nvidia Volta core, a single 46 square millimeters chip (no chiplets like recent AMD chips), and an incredible wooping 18 GB of SRAM (that’s like 18GB of CPU cache basically) ! I don’t know if you guys are used to that scale but I find it monstruous !
- 14 7y agoI was a little confused when I read your comment and 46 square millimeters. I am guessing you meant 46k square millimeters as the article stated 46,225 square millimeters which yes you are right that is monstrous. Very cool! As a care giver I often discuss with my clients the cool things that have come into existence in their life time and wonder when I get old what things will my children reflect back on and say grandpa you were alive when “x” was invented how neat. Personally I am hoping it is fusion power.
- thfuran 7y agoI'll put my vote in for commodity room-temperature superconductors.
- duderific 7y agoThe young people I work with are always shocked when I tell them there was no internet (or at least, nothing terribly useful at the consumer level) when I went to college.
- navaati 7y agoAh, stupid me :). I had no idea of the scale of things and in my native locale the comma is the decimal point...
- fartismartass 7y agoIt is monstrous. But the way I read it, it is actually similar to chiplets. Only chiplets are cut from the wafer, individually tested and then combined for the final chip. Here all parts stay on the wafer and faulty ones are routed around.
- rbanffy 7y agoSince they are wiring the chips on the wafer and routing around defects, I imagine a chiplet design on a wafer-sized interposer would help them deal with yield issues. I wonder what their competition will do. It's certainly possible for Nvidia or AMD to bundle GPU dies on top of large interposers and TSMC has already shown large ones, though nothing on this scale.
- Tuna-Fish 7y agoIt's wafer scale integration. All the major AI learning chips are probably heading this way. That is, CPUs and GPUs are made on silicon wafers, typically 30cm in diameter, with many chips built on a single wafer and then cut from the wafer and packaged into products. The idea of wafer-scale integration is that instead of cutting the wafer, you just build all the communication between the computation elements in the wafer into the wafer and therefore get a "network on wafer", a single very massive "chip". The reason to do this is that the lowest-energy way to communicate data is on-chip, and in major AI learning setups, by far the most of the power is spent on data movement. By making the largest possible chip, you minimize the data movement power requirements and spare more power for the computation.
- sanxiyn 7y agoWafer scale integration does seem to be a logical conclusion of integrated circuit. When Jack Kilby invented IC it indeed was a stroke of genius.
- groundlogic 7y ago> The 46,225 square millimeters of silicon in the Cerebras WSE house 400,000 AI-optimized, no-cache, no-overhead, compute cores > But Cerebras has designed its chip to be redundant, so one impurity won’t disable the whole chip. This sounds kinda clever to a semi-layperson. Has this been attempted before? Edit: at this scale. Not single-digit cores CPUs being binned, but 100k+ CPU core chips with some kind of automated core-deactivation on failure.
- rrss 7y agoYes, every chip of moderate size does this. Selling floorswept chips as lower performance SKUs is very common.
- groundlogic 7y agoSure, that practice is pretty wellknown. But at this scale?
- deleted 7y ago[deleted]
- vardump 7y ago>> But Cerebras has designed its chip to be redundant, so one impurity won’t disable the whole chip. > This sounds kinda clever to a semi-layperson. Has this been attempted before? I think every large chip is like that nowadays. It's just a matter of degree.
- robin_reala 7y agoYep, it’s standard practice. The old AMD tri-core chips for example were all physically four cores with a (usually) broken one disabled. Modern AMD chips use multiple chiplets on a single board to extend yields.
- happycube 7y ago... and some of the new Epyc chips use only two cores per die out of eight. Customers still come out ahead since those chips have far more cache than a chip with 2 fully enabled 8 core dies.
- CaliforniaKarl 7y agoThis is one of the first of what will probably be a number of announcements out of the Hot Chips conference, happening now through Tuesday on the Stanford campus. https://www.hotchips.org https://www.hotchips.org
- riku_iki 7y agoInteresting what will be the price and power consumption. Does it need specialized server with huge power supply module?
- morphle 7y agoA 300mm wafer at 16nm node would cost more than $6500 a piece. The power consumption would be more than 20kW if all transistors are in use simultaneously. We am designing a special reconfigurable AC-DC and DC-DC power router (inverter) to supply this huge amount of power. Our WSI design will be liquid immersively cooled.
- wolf550e 7y agowhitepaper: https://www.cerebras.net/wp-content/uploads/2019/08/Cerebras-Wafer-Scale-Engine-Whitepaper.pdf https://www.cerebras.net/wp-content/uploads/2019/08/Cerebras... I don't expect good yields from a chip that takes up the whole wafer. They must disable cores and pieces of SRAM that are damaged. How is this programmed?
- groundlogic 7y agoIf you divide that giant piece of silicon into 400k processors, and then only use the ones that actually work... I wonder if they figure that out every time the CPU boots, or at the factory. At this scale, maybe it makes sense to do it all in parallel at boot. Or, even dynamically during runtime. There may be edge case cores that sort of work, and then won't work at different temps, or after aging?
- kingosticks 7y agoThey'll aim to catch the logic failures and memory failures during wafer test at the factory. This testing is done at room temperature and at hot. There are margins built in to allow for ageing. If they want to ship a decent product they'll also need to repeat the memory testing every boot, and ideally during runtime but maybe the latter isn't a big deal for something like this. EDIT: to add a bit more and possibly address the original question (which I think keveman may have misunderstood), there will usually be some hardware dedicated to controlling the chip's redundancy. Part of that is often a OTP fuse-type thing that can be programmed during wafer test to indicate parts of the chip that don't work. Something (software or hardware) will read that during boot and not use those parts of the chip.
- groundlogic 7y agoSure, that makes sense. With this many cores it seems like the probability that a core dies during a multi-hour job (or in case it's used for inference, during a very long-lived realtime job) is pretty high, so the software in all layers would need to handle this kind of exception. They probably don't, today, since we haven't seen a 400k core chip before.
- program_whiz 7y agoAn amazing stride in computing power. I'm not convinced that the issue is really about more hardware. While more hardware will definitely be useful once we understand AI, we still don't have a fundamental understanding of how AGI works. There seem to be more open questions than solved ones. I think what we have now is best described as: "Given a well stated problem with a known set of hypotheses, a metric that indicates progress towards the correct answer, and enough data to statistically model this hypothesis space, we can efficiently search for the local optimum hypothesis." I'm not sure doing that faster is going to really "create" AGI out of thin air, but it may be necessary once we understand how it can be done (it may be an exponential time algo that requires massive hardware to execute).
- narrator 7y agoExponential time algos aren't really practical on any hardware except quantum computers, and only for algorithms that benefit from quantum speedup such as the quantum fourier transform[1]. The quantum fourier transform goes from O(n2^n) in a classical computer to O(n^2) in a quantum computer. [1] https://en.wikipedia.org/wiki/Quantum_Fourier_transform https://en.wikipedia.org/wiki/Quantum_Fourier_transform
- yters 7y agoIf AGI is a halting oracle, then hardware performance is irrelevant.
- gridlockd 7y agoWhat's there to understand? We know how intelligence "happened". We just need to build a few million human-sized brains, attach the universe, and simulate a few billion years of evolution. Whatever comes out of that could just be called "AGI" by definition.
- yters 7y agoWe don't know that's how intelligence happened. That's just speculation based on materialist assumptions. Intelligence is most likely immaterial, i.e. abstract concepts, free will, mathematics, consciousness, etc. In which case, it's beyond anything in this physical universe.
- rbanffy 7y agoThere are more details here: https://fortune.com/2019/08/19/ai-artificial-intelligence-cerebras-wafer-scale-chip https://fortune.com/2019/08/19/ai-artificial-intelligence-ce...
- bhassel 7y ago> In another break with industry practice, the chip won’t be sold on its own, but will be packaged into a computer “appliance” that Cerebras has designed. One reason is the need for a complex system of water-cooling, a kind of irrigation network to counteract the extreme heat generated by a chip running at 15 kilowatts of power. 15 kW, yikes.
- dmitrygr 7y agoSo, Azul Systems 2.0? clever(ish) hardware with good(ish) results, for too much $$$ for anyone to actually buy?
- rbanffy 7y agoThe main difference is that what Azul built had more limits - once you run your workload well enough, there is little incentive to have more compute power. When it comes to ML, the more compute power you throw at it, the better.
- voldacar 7y agoThis is very neat, but I really wish the press would say "neural net" instead of "AI". "AI" just means a computer program that has some ability to reason about data similarly to a human, neural nets are a subset of that I guess "AI" gets you the clicks though
- curiousgal 7y ago> neural net Which is basically matrix multiplication.
- voldacar 7y agoEverything is matrix multiplication at some level!
- make3 7y agoCall it "just function approximation" if you want, but calling it just matrix multiplication removes the only important part, that the matrix's value are found through tiny adjustments made to generalize the empirical distribution of a dataset, and end up approximating it's function
- ericd 7y agoWell, plus nonlinearities and backprop/gradient descent, to turn those matrix multiplications into a universal function approximator.
- samcheng 7y agoI remember watching the (excellent!) Tesla Autonomy Day presentation, and an analyst was asking about 'chiplets' but was met with a bit of dismissal from the Tesla team. Maybe THIS is what the analyst had in mind! Pretty cool stuff, although I question how interconnect / inter-processing-unit communication would work. Notably, no benchmarks in the press release...
- jaboutboul 7y agoThere goes Intel. Behind the curve again...
- temac 7y agoImpressive but is this a good idea? Either you need a crazy thermal solution, or it must be way less thermally dense than smaller chips can be. And is it really that much of an advantage to stay on chip compared to going through a PCB, if distances are crazy?
- sanxiyn 7y agoIt does seem to enable interconnect with impressive latency and bandwidth. It probably is worth it considering both Google and Nvidia systems are interconnect-limited.
- deepnotderp 7y agoThey are using vertical water pipes for cooling to solve the thermal density problem. At 15 kw they're doing OK there.
- gnode 7y agoI'm surprised this hasn't been done before; a monolithic wafer can achieve denser interconnects, and is arguably simpler than a multi-chip module. On the other hand, multi-chip modules can combine the working chips in low-yield wafers, whereas a monolithic wafer would likely contain many failed blocks, uselessly taking up space / distancing the working chips. Cooling isn't really a problem, as the TDP scales with area as in other chips. Water cooling or heat pipes can transport the heat away to a large heat sink. 3D / die-stacked chips have a harder cooling problem, potentially requiring something like an intra-chip circulatory system.
- bryanlarsen 7y ago> I'm surprised this hasn't been done before It has, a couple of people have linked to https://en.wikipedia.org/wiki/Wafer-scale_integration https://en.wikipedia.org/wiki/Wafer-scale_integration
- ivalm 7y agoIt's 15kW, you need a custom liquid cooling solution that they package the chip with.
- mschuster91 7y ago18 GByte of memory with 1 clock cycle latency? That's impressive.
- dkersten 7y agoDepends on the how fast a clock cycle is, I suppose. Its easy to make something one clock cycle if the clock ticks verrrry slowly ;)
- hwillis 7y agoI didn't see any indication, but I'd give ~0% chance that the full address space can be accessed in one clock.
- bryanlarsen 7y agoI assume that each byte of that memory has a 1 clock cycle latency to only one of the 400,000 cores on the wafer. That's about 45KByte of memory per core; 1 clock cycle latency to a block that small is quite reasonable.
- kingosticks 7y agoHow do they go and package something this big? Is this supposed to be used or is it just a headline?
- kingosticks 7y agoThere's some information on this towards the end of the slides at https://www.anandtech.com/show/14758/hot-chips-31-live-blogs-cerebras-wafer-scale-deep-learning https://www.anandtech.com/show/14758/hot-chips-31-live-blogs.... Sadly there's very little concrete info on the packaging methodology (hardly a surprise) which to me is the only truely novel thing about this chip. But it must cost an absolute fortune.
- p1esk 7y agoBut... can I play Crysis - oops sorry - can I train ResNet on this thing?
- ineedasername 7y agoWould this really be more efficient in term of cost/performance? It seems the specialized nature of the chip push the price high enough that you could build equivalent systems with traditional hardware for the same or less, and it would all be known-quantities rather that working with something brand new and not as well understood.
- mv4 7y agoSpecialized AI chips don't seem like a very good business idea to me. The way we do things (in AI) today - we may be doing things completely different tomorrow. It's not like there's a standard everyone has agreed on. There is a very real risk these specialized, expensive devices will go the way of the Bitcoin ASIC miner (which saturated secondary markets at a fraction of its original cost). Source: I do ML consulting and build AI hardware.
- p1esk 7y agoThe way we do things in AI today is multiplication of two large matrices. Just like we did it 30 years ago: http://yann.lecun.com/exdb/publis/pdf/lecun-89e.pdf http://yann.lecun.com/exdb/publis/pdf/lecun-89e.pdf
- mv4 7y agoWhile the theoretical innovations have mostly been incremental, there has been a lot of progress in the development of "light" deep learning frameworks - so the tasks that previously required massive GPUs can now run on your phone. And this trend will continue.
- mv4 7y agoWhile this is certainly a very impressive achievement, I am personally interested in small and light AI. World's tiniest AI chip? That would get me excited!
- modeless 7y agoThere are far more transistors in this chip than neurons in the human brain. In 100 of these chips, there are more transistors than there are synapses in the human brain. I don't mean to suggest that transistors are equivalent to neurons or synapses; clearly they are very different things (though it is not clear that neurons are much more computationally powerful than transistors, despite assertions from many people that this is the case). But I think it is still useful to compare the complexity of structure of chips vs. the human brain. We are finally approaching the same order of magnitude of complexity in structure. Also note that this is not manufactured on TSMC's highest density process. Assuming TSMC's 3nm process development is successful, that will probably be 6x denser than the 16nm process used here.
- jdsully 7y agoI haven’t seen a perceptron circuit done in less than 4 transistors plus a few diodes. However these chips are almost certainly digital and will use a lot more to perform the floating point math. Single transistors can’t even compute our simplified model of a neuron let alone the complexities of the real thing.
- mlyle 7y agoIf we're going to talk about net computational power per unit (neuron/transistor), we also need to note that single transistors do stuff a whole lot faster than biological neurons, too. If we had neurons that were 1,000x faster, could we use fewer of them for the same task?
- deleted 7y ago[deleted]
- goldemerald 7y agoIs there any comparison of training speed of a neural net for one of these chips and a typical one? I'd be interested to see how long it takes to train an imagenet classifier on one of these compared to other hardware.
- sp332 7y agoThere's a photo on the homepage: https://www.cerebras.net/ https://www.cerebras.net/ It's just about 8.5" across. You could just barely print a 1:1 photo of it on a letter-sized sheet of paper, with less than a half-milimeter margin on each side.
- ohazi 7y agoI wonder if they bond it to a copper slab. At that scale, the tiniest amount of PCB flex would probably shatter the die/wafer...
- oITAZt 7y agoThey do not. They have "developed [a] custom connector to connect wafer to PCB": https://imgur.com/a/sXxGbiD https://imgur.com/a/sXxGbiD
- ohazi 7y agoI think "cold plate" is the slab.
- oITAZt 7y agoYes, I it is probably a copper slab (they never mentioned), but there is no electrical connection, as the rest of the slides make clear: https://imgur.com/a/Rbd7e4D https://imgur.com/a/Rbd7e4D It just provides a thermal connection for water cooling. The electrical connection is made through the PCB (and probably through a thick copper plate on the opposite side of the PCB)
- natpalmer1776 7y agoI know very little about this sub-field, but from a layman's perspective, and given competition between Intel and NVidia in the field of deep learning, I would not be surprised if Intel tried to acquire or takeover this company for their single-chip designs.
- bmh 7y agoIn their whitepaper they claim "with all model parameters in on-chip memory, all of the time", yet that entire 15 kW monster has only 18 GB of memory. Given the memory vs compute numbers that you see in Nvidia cards, this seems strangely low.
- Veedrac 7y ago18GB is huge! An NVIDIA V100 has 6MB of L2 memory. HBM is off-chip, and vastly (~100x) slower.
- Traster 7y agoI'd love to see some metrics on whether this idea has any merit. Because obviously any problem that can be done on a massive wafer can be done on multiple smaller wafers. The question is, are the compromises to get something working on a massive wafer killing performance to the point where just splitting the problem up efficiently would have been better. It's also important to think: Stuff isn't just going to happen to be the size to fit on this wafer, it's either huge in scale, or not. If it's not, you don't need this wafer, if it is, you probably need to partition your problem onto multiple copies of this wafer anyway. Take their comparison to a Nvidia GPU. It might be 57 times bigger, but I can rent 1000 Nvidia GPUs from AWS at the drop of a hat (more or less). So yes, maybe they've done some interesting stuff, but they need some decent benchmarks to show off before we can really distinguish between whether this is the Boeing 747 Dreamliner or a Spruce Goose.
- streetcat1 7y agoThe problem is not if it bigger than nvidia (which is kind of a strange metric). It is when Amdhal law kicks in. https://en.wikipedia.org/wiki/Amdahl%27s_law https://en.wikipedia.org/wiki/Amdahl%27s_law
- dgacmu 7y agoYou get much higher bandwidth (at lower power cost) between chiplets than if you used a multi-chip design or even an interposer. The drawback is that you suffer a really interesting thermal problem and have to figure out what to do with the wafer space that doesn't fit in your square -- probably creating lower-scale designs that you sell. The second drawback is that you can't really match your available computation to memory in the same way you can with a more conventional GPU. So you have to be able to split your model across chips and train model-parallel. The advantage is that model-parallel lets you throw a heck of a lot more computation at the problem and can help you scale better than using only data parallelism. Model-parallel training is typically harder than data-parallel because you need high bandwidth between the computation units. But that's exactly what Cerebras's design is intended to provide. You also have a yield management issue, where you have to build in the capability to route around dead chips, but that's not too nasty a technical detail. But if your "chip"-level yield (note that their chip is still a replicated array of subunits) is too low, it kills your overall yield. So they're going to be conservative with their manufacturing to keep yield high. It's not obviously broken, but it's certainly true we need benchmarks -- but not just benchmarks, time for people to come up with models that are optimized for training/inference on the cerebras platform, which will take even longer.
- deleted 7y ago[deleted]
- rightbyte 7y ago"The Cerebras software stack is designed to meet users where they are, integrating with open source ML frameworks like TensorFlow and PyTorch" What's the instruction-set? They don't say. I assume you need to program in some DSL VERILOG-ish macro-assembler for that monster contraption. Python is probably not what works well ...
- sanxiyn 7y agoYour CoreML model can run on Apple Neural Engine, but Apple doesn't expose that hardware's instruction set. This probably works similarly.
- ChuckMcM 7y agoWafer scale integration, the stupid idea that just won't die :-) Okay. I'm not quite that cynical but I was avidly following Trillogy Systems (Gene Amdahl started it to make super computers using a single wafer). Conceptually awesome, in practice not so much. The thing that broke down in the '80s was that different parts of the system evolve at different rates. As a result your wafer computer was always going to be sub-optimal at something, whether it was memory access or an I/O channel standard that was new, changing that part meant all new wafers and fab companies new that every time you change the masks, you have to re-qualify everything. Very time consuming. I thought the AMD "chiplet" solution to making processors that could evolve outside the interconnect was a good engineering solution to this problem. Dave Ditzel, of Transmeta fame, was pushing at one point a 'stackable' chip. Sort of 'chip on chip' like some cell phone SoCs have for memory, but generalized to allow more stacking than just 2 chips. The problem becomes getting the heat out of such a system as only the bottom chip is in contact with the substrate and the top chip with the case. Conceptually though, another angle where you could replace parts of the design without new masks for the other parts. I really liked the SeaMicro clusters (tried to buy one at the previous company but it was too far off axis to pass the toy vs tool test). Perhaps they will solve the problems of WSI and turn it into something amazing.
- samstave 7y agoHavent hear Tranmeta in a LONG time. I recall when I was at intel in 1996 and I used to work a few feet from Andy Grove... and I would ask naive questions like "how come we cant stack mutiple CPUs on top of eachother" and make naive statements like: "When google figures out how to sell their services to customers (GCP) we are fucked" (this was made in the 2000's when I was on a hike with Intels then head of tech marketing, not 1996) ((During that hike he was telling me abt a secret project where they were able to make a proc that 48 cores)) (((I didnt believe it and I was like "what the fuck are they going to do with it)))?? -- welp this is how the future happens. and here we are. and ask financially stupid questions like: "what are you working on?" response "trying to figure out how to make our ERP financial system handel numbers in the billions of dollars" I made a bunch of other stupid comments... like "apple is going to start using Intel procs" and was yelled at by my apple fan boi " THATS NEVER GOING TO FUCKING HAPPEN" But im just a moron. --- But transmeta... there was a palpable fear of them at intel at that time....
- fnord77 7y agosince the article didn't post a picture of the bare chip: https://i.imgur.com/cMo4w0C.jpg https://i.imgur.com/cMo4w0C.jpg
- panabee 7y agocan anyone comment on the differences between cerebras and other chip startups trying to rethink the semiconductor architecture for AI? what are the main technical forks?
- lnsru 7y agoI am just wondering, why there is no similar product as FPGA array. As far as I know, it’s the cheapest way to see if there is product/market fit for a semiconductor product. High speed transceiver as well as memory controllers are included in FPGA. This single wafer approach looks very interesting to me. I was intern at Infineon some time ago and was working on device distribution characterization across 200 mm wafer. The chips in the middle were 2-3 better performing than these in the border. So how Cerberas’s chip manages this issue? Middle parts are throttled or low performing areas near wafer’s boarder are disabled? How much does it cost?.. I can imagine it being shipped on thermal pad with liquid nitrogen cooling bellow. There must be some wires bonded for interface to the host. Very interesting technical project. I am very curious what are the clients for such huge specialized chip.
- morphle 7y agoAn FPGA would make it much more generic a product so you can sell to more markets. But you would loose a factor 200 (a factor 1000 with traditional FPGA design) to make transistors reconfigurable. If you leave the wafer intact, you get 70,000 mm2. Cerebras cut off almost half of the wafer. At 7nm you would get 2.7 trillon transistors with 300mm wafers, more with 450 mm wafers. You disable those reticle-sized areas with impurities or damage at runtime. You can cool it with immersive liquid cooling. Instead of wire bonding you can stack chips on top or use free space optics [1]. [1] https://www.youtube.com/watch?v=7hWWyuesmhs https://www.youtube.com/watch?v=7hWWyuesmhs
- The_rationalist 7y agoASICS do not support CUDA. There is a forked tensorflow with opencl support from AMD but I doubt people will use it for this ASIC. So how can tensorflow/cntk/pytorch use such hardware?
- sanxiyn 7y agoThe exact same way TensorFlow supports TPU: by writing another backend. TPU doesn't support CUDA either.
- deleted 7y ago[deleted]
- The_rationalist 7y agoSo instead of having common optimized kernels for AMD, Intel, all ARM firms, all FPGAs and all ASICS. Each member is reinventing the wheel in it's own backend? Not so surprising ^^