7 ms·
Tenstorrent unveils Grayskull, its RISC-V answer to GPUs
- neilv 3y ago[flagged]
- curious_cat_163 3y ago> "...including BERT for natural language processing tasks, ResNet for image recognition, Whisper for speech recognition and translation, YOLOv5 for real-time object detection, and U-Net for image segmentation." I wonder why they are starting with these models. One could speculate that they are going for power efficiencies but that does not quite add up entirely.
- camel-cdr 3y agoFrom what I've gathered from the Tenstorrent discord: The grayskull cards only have 8 GB of RAM and don't have a fast enough memory to NoC bandwidth to make working with multiple cards practical. The next generations, that is wormhole and newer don't have this limitations and are specifically designed to work in server racks, see the galaxy system. [0] The Grayskull really only is a devboard. It has some other quirks that will be improved by wormhole, like native 19 bit floats in their SIMT engines, instead of 32 bit, in wormhole. [0] https://tenstorrent.com/systems/galaxy/ https://tenstorrent.com/systems/galaxy/
- deleted 3y ago[deleted]
- hhh 3y agoYOLOv5 is pretty widely used, and it can be useful to get efficient compute at the edge. Disclosure: I work for a Tenstorrent customer.
- VHRanger 3y ago> I wonder why they are starting with these models. They were building Greyskull in 2020/2021 initially. BERT_Large and similar were the SOTA at that time.
- transitionnel 3y agoLike Intel SSE for media optimization probably? Those models do seem to be the best currently available (IMO), so yeah power efficiencies for guaranteed common use cases should be a 0-day integration.
- transitionnel 3y agoWhich reminds me... Hope those libraries are open and vetted...
- binarymax 3y agoNext-gen BERT models are still very popular for embeddings.
- usrusr 3y agoHighly speculative, and I suspect that I don't do these models justice: The purpose of these boards seems to get people acquainted with their programming model. Not as in using board and model as a turnkey solution (if that happens, fine, but this is not the goal), but as in getting potential customers for future boards to learn how to make their own models or third party models run on the board. The more models they supported out of the box, the less well the goal of building buyer-side expertise in the programming model would be served.
- PhilippGille 3y agoDev kits: - Grayskull e75 | drawing 75W | 96 Tensix cores | 1 GHz clock | 96 MB SRAM | 8GB LPDDR4 @ 102.4 GB/s | $599 - Grayskull e150 | drawing 200W | 120 Tensix cores | 1.2 GHz clock | 120 MB SRAM | 8GB LPDDR4 @ 118.4 GB/s | $799 It will be interesting to see their inference performance, compared to graphics cards. Will they be interesting for home labs? I found one interview with unboxing a preview version (if I understood correctly), with some background info, but no performance numbers: https://morethanmoore.substack.com/p/unboxing-the-tenstorrent-grayskull https://morethanmoore.substack.com/p/unboxing-the-tenstorren...
- christkv 3y agoMemory seems very low for inference on bigger models?. Memory bandwidth is also massively lower than something like a 4090 or 7900xtx?. Am I missing some fairy dust magic here?
- ChainOfFools 3y agoWhat's the memory interface like? Is an extreme NUMA thing with several orders greater potential for parallelization? I'm probably dreaming; new CPU core designs are one thing, a completely new memory design is likely high fantasy..
- 3y ago
- deleted 3y ago[deleted]
- camel-cdr 3y agoSince people seem to be wondering how the architecture works, and the software stack is open (although I'm not sure to what extend), I'll share my understanding of it. The basic system consists of the cards consists of a bunch of Tensix cores and shared memory: > Each Tensix core contains a high-density tensor math unit (FPU) which performs most of the heavy lifting, a SIMD engine (SFPU), five Risc-V CPU cores, and a large local memory storage. [0] > The cores are connected with two torus-shaped, going in opposite directions. [0] The RISC-V cores in the Tensix cores, are tiny rv32i cores, that can control the FPU, SFPU, and are also used to prepare/move data. The FPU, does "dense tensor math", so I think it's probably a matmul engine, but I don't know any more specifics. [1] The SFPU is a more general purpose SIMT engine, that can be driven from the RISC-V cores. There is a SFPU simulator you can play around with on their github. [2] See the low level Kernels example for how the programming model works. [3] The grayskull SFPU has 4 general purpose LRegs, which each hold 64 19-bit values. Wormhole has 8 general purpose LRegs, which each hold 32 32-bit values. I've been told that wormhole SFPU has a ~3x IPC increase over grayskull, and a few new SFPU instructions. You can probably find out more by browsing the docs and rummaging through the github repos. [4] [0] https://docs.tenstorrent.com/tenstorrent/v/tt-buda/hardware https://docs.tenstorrent.com/tenstorrent/v/tt-buda/hardware [1] https://docs.tenstorrent.com/tenstorrent/v/tt-buda/terminology#fpu https://docs.tenstorrent.com/tenstorrent/v/tt-buda/terminolo... [2] https://github.com/tenstorrent-metal/sfpi/blob/master/tests/grayskull/sfpi/sfpu.cc https://github.com/tenstorrent-metal/sfpi/blob/master/tests/... [3] https://tenstorrent-metal.github.io/tt-metal/latest/tt_metal/apis/kernel_apis/sfpu/llk.html https://tenstorrent-metal.github.io/tt-metal/latest/tt_metal... [4] https://github.com/tenstorrent-metal/ https://github.com/tenstorrent-metal/
- BirAdam 3y agoSo, modern transputer but RISC-V. Seems neat.
- cmrdporcupine 3y agoOh, so am I to read this right that the actual vector work being done here is not done via the RiscV [V]ector extension, but by a purely custom core?
- margorczynski 3y agoI wonder if with the recent research coming to light about the effectiveness of using 3 state weights do they plan to release something along those lines? Can they really compete in FLOP/$ with the likes on Nvidia even if it is a more bespoke architecture than a modified GPU?
- mastax 3y agoNvidia has insane margins so it isn't hard* for a well-funded player to beat them in FLOPS/$. The hard part is the compiler toolchains, the libraries, the documentation, getting models to run fast. * Everything is relative. It's hard to make even a simple microcontroller - but also it isn't hard, you know?
- transitionnel 3y ago"Life in 1 comment"
- VHRanger 3y ago> Nvidia has insane margins so it isn't hard* for a well-funded player to beat them in FLOPS/$ It's hard when you can't buy TSMC capacity
- dannyw 3y agoI’d wait till the dust settles. Maybe 2bit encoding (-1, 0, 0.5, 1) would be easier to design hardware for; as 0.5 can be a bitwise shift.
- imtringued 3y agoNobody cares about FLOPs anymore when it comes to transformer based architectures and considering the transformer architecture is applicable to almost any usecase beyond LLMs, memory bandwidth and memory capacity are the most important metrics. Grayskull has enough 16bit float performance to outrun its memory bandwidth even with 1 ternary bit quantization.
- ribit 3y agoI was unable to find any reference to performance or architectural details. Memory bandwidth is very low for an ML-focused device. Price is extremely high. What am I missing?
- nabla9 3y agoThose are DevKits. I assume they are sold to hardware manufacturers and maybe software developers so that they check out and test their systems against real processor. Processors are probably manufactured in some relatively old process technology. The real thing will be manufactured using different 2nm process according to the article and performance will be different.
- deleted 3y ago[deleted]
- physPop 3y agoOutsider to the field and just curious: Does anyone know how these kinds of processors compare to the custom silicon by aws/google/tesla?
- VHRanger 3y agoMuch more efficient per watt for some specialized types of operations. Obviously immature ecosystem, expect a couple of years at minimum before adoption if it's the real deal.
- bschne 3y agoBeen following this for a while b/c Jim Keller, but every time I look at the arch [1; as linked by other commenter] as someone who doesn't know the first thing about CPU/ASIC design it just looks sort of... "wacky"? Does anyone who understands this have a good explainer for the rationale behind having a grid of cores with memory and IFs interspersed between and then something akin to a network interconnecting them with that topology? What is it about the target workloads here that makes this a good approach? 1. https://docs.tenstorrent.com/tenstorrent/v/tt-buda/hardware https://docs.tenstorrent.com/tenstorrent/v/tt-buda/hardware
- tormeh 3y agoYou want memory to be close to where it's used because at the speeds of high-performance ICs, the latency caused by distance is actually significant.
- bschne 3y agobut isn't that aspect common among this, CPUs, GPUs, ...? And it feels like the whole NOC thing would add quite some overhead to moving things around. Are you saying proximity here more than offsets this vs. e.g. each core having its own cache as I think they do in a "normal" CPU? And if so, is this more true of ML inference workloads than other workloads, for some reason?
- loeg 3y agoI think the NOC approach is fundamentally similar to Intel's rings that they used for core interconnect back in the mid-2010s. It works. https://www.realworldtech.com/includes/images/articles/snbep-1.jpg https://www.realworldtech.com/includes/images/articles/snbep... https://en.wikichip.org/wiki/intel/microarchitectures/sandy_bridge_(client)#Ring_Interconnect https://en.wikichip.org/wiki/intel/microarchitectures/sandy_...
- paulmd 3y agoThey still use this today, and the ring interconnect is also the topology inside zen3/zen4 CCXs (with 8 cores). Ring is one of the simplest and best systems for >4 cores, until you get to about 8-10 cores (at which point you generally split it into multiple “tiers” like multiple CCXs etc).
- binarymax 3y agoThe System Requirements on this page say 64GB RAM is a prerequisite on the host system. Why? Wouldn’t an inference server be barebones aside from the Inference hardware? https://tenstorrent.com/cards/ https://tenstorrent.com/cards/
- camel-cdr 3y agoFrom the tenstorrent discord: > It has more to do with the memory resources required during model compilation. The requirement will vary from model to model, so 64GB is a safe limit for all.
- VHRanger 3y agoAnd why do they specify PCIe 4.0? I imagine it's just to leverage bandwidth correctly? Or is there a necessary feature in there?
- deleted 3y ago[deleted]
- loeg 3y agoThis article is extremely low detail. Does anyone have a source with more information?
- VHRanger 3y agodocs.tenstorrent.com All the details you need!
- pella 3y agoGrayskull™ e150 = Tensix Cores: 120 * 5 RISC-V => 600 RISC-V CPU core ( 1 Tensix core = 5 RISC-V core : https://docs.tenstorrent.com/tenstorrent/v/tt-buda/hardware https://docs.tenstorrent.com/tenstorrent/v/tt-buda/hardware )
- hollerith 3y agoI don't think that is as clear as you think it is. If you had to communicate it orally, what words would you add? ADDED. I think I figured out what it means: a Grayskull e150 contains 120 'Tensix cores', each of which contains 5 RISC-V cores.
- loeg 3y agoDo you think this thing will run Linux (re: now removed comment about Linux default core limit)?
- pella 3y agoAs I see this is : TT-Metalium [1] = a low level API [1] "TT-Metalium is a platform for programming heterogeneous collection of CPUs (host processors) and Tenstorrent acceleration devices, composed of many RISC-V processors. Target users of TT-Metal are expert parallel programmers wanting to write parallel and efficient code with full access to the Tensix hardware via low-level kernel APIs." https://tenstorrent-metal.github.io/tt-metal/latest/tt_metal/programming_model/index.html https://tenstorrent-metal.github.io/tt-metal/latest/tt_metal... [1] "The software stacks come in two varieties – a high level and a low level. The high-level is called TT-Buda, using higher-level APIs to get things up and running, along with interfaces into modern machine learning frameworks. The lower level is TT-Metalium, which provides fine-grained control over the hardware for custom operators, custom control, and even non-machine learning code. Tenstorrent states that there are no black boxes, no encrypted APIs, and no hidden functions." https://morethanmoore.substack.com/p/unboxing-the-tenstorrent-grayskull https://morethanmoore.substack.com/p/unboxing-the-tenstorren...
- deleted 3y ago[deleted]
- audiofish 3y agoThis looks very similar to the Picochip designs used in a lot of small cellular base stations for SDR. I hope it is similar, because those were fantastic chips to program for. That influence could have come via Intel's acquisition of Picochip.
- bragr 3y agoThe stated architecture reminds me of how the Intel Project Larrabee GPU was supposed to work, except with RISC-V instead stripped x86 cores.
- usrusr 3y agoDid Larrabee contain anything that wasn't an x86 core? According to other comments, this seems to consist of massive highly specialized processing units with just a few tiny CPU sprinkled in to keep the former fed. Quite the opposite of how I remember Larabee.
- VHRanger 3y agoNot really, Xeon Phis were basically the a x86 CPU abusing AVX extensions to their limit
- shrubble 3y agoDo you see any similarities with the Xeon Phi boards?
- multiphonics 3y agoHow is the networked multi CPU core Grayskull a different approach compared to, say, Ampere's 100+ ARM core chip?
- spintin 3y agoI bought a 3050 low profile / half length with DDR6 at 14GHz for half this price. And it can play games too.
- wtallis 3y agoGDDR6, 14 GT/s if we want to be accurate. So about double the DRAM bandwidth, but less than 1/20th the SRAM capacity, and fairly different architectures on-chip. I'm not sure what ML workloads would benefit from the extra SRAM enough to overcome the DRAM bandwidth deficit, but they probably exist or Tenstorrent wouldn't be making such a SRAM-heavy chip.
- _0ffh 3y agoI'm getting sick of this: Groq, Tenstorrent, all the promising startups are offering inference-only solutions. I have it from Groq official channels that they do not plan to invest development time into enabling training, because inference is where the big money is at. Which I understand insofar as inference demand probably outstrips training demand by ~millions, but I still can't help but find all of this so egreriously disappointing!
- mugivarra69 3y agohow does 8gb cut it?