9 ms·
Dev kits: - Grayskull e75 | drawing 75W | 96 Tensix cores | 1 GHz clock | 96 MB SRAM | 8GB LPDDR4 @ 102.4 GB/s | $599 - Grayskull e150 | drawing 200W | 120 Te
by PhilippGille 3y ago
Dev kits:
- Grayskull e75 | drawing 75W | 96 Tensix cores | 1 GHz clock | 96 MB SRAM | 8GB LPDDR4 @ 102.4 GB/s | $599
- Grayskull e150 | drawing 200W | 120 Tensix cores | 1.2 GHz clock | 120 MB SRAM | 8GB LPDDR4 @ 118.4 GB/s | $799
It will be interesting to see their inference performance, compared to graphics cards. Will they be interesting for home labs?
I found one interview with unboxing a preview version (if I understood correctly), with some background info, but no performance numbers: https://morethanmoore.substack.com/p/unboxing-the-tenstorrent-grayskull https://morethanmoore.substack.com/p/unboxing-the-tenstorren...
- christkv 3y agoMemory seems very low for inference on bigger models?. Memory bandwidth is also massively lower than something like a 4090 or 7900xtx?. Am I missing some fairy dust magic here?
- ChainOfFools 3y agoWhat's the memory interface like? Is an extreme NUMA thing with several orders greater potential for parallelization? I'm probably dreaming; new CPU core designs are one thing, a completely new memory design is likely high fantasy..
- onlyrealcuzzo 3y agoWill graphics cards eventually get renamed? Are the optimizations for inference basically identical to graphics? I imagine the workloads are different, and that inference has been the main market for years now...
- yencabulator 3y agoThe new things are already getting renamed, but the dust hasn't settled yet. NPU: https://en.wikipedia.org/wiki/AI_accelerator https://en.wikipedia.org/wiki/AI_accelerator TPU: https://en.wikipedia.org/wiki/Tensor_Processing_Unit https://en.wikipedia.org/wiki/Tensor_Processing_Unit
- adgjlsfhk1 3y agowhy bother renaming? just re-acronym to "greatly parallel unit"
- algo_trader 3y agoThe website indicates each Tensix core ~1TOP theoretical. So not amazing. It will have to compete on price/memory in the crowded inference space.
- andy99 3y agoThis is purely a developer kit, letting people get used to the hardware configuration and the software stacks before the big hardware comes later in future generations. I'd like to know what different precisions/ quantizations if any are supported. For LLMs 8GB is fine for playing around if weights are quantized. And as the article mentions it's more than enough for lots of computer vision models.
- camel-cdr 3y agoGrayskull supports FP8, FP16, BFP16, VFP2, BFP4, BFP8 [0], and some sort of 19-bit floating point in the SFPU. [1] I don't know how and with what performance those are supported. FP4 has 221 TFLOPS on the Grayskull e75, and 332 TFLOPS on the Grayskull e150. [0] While wormhole has 82 INT8 TOPS per card. [2] I haven't found any other numbers. Edit: Some more info about data formats: https://docs.tenstorrent.com/tenstorrent/v/tt-buda/dataformats https://docs.tenstorrent.com/tenstorrent/v/tt-buda/dataforma... [0] https://docs.tenstorrent.com/tenstorrent/add-in-boards-and-cooling-kits/grayskull-tm-e75-e150/specifications https://docs.tenstorrent.com/tenstorrent/add-in-boards-and-c... [1] https://tenstorrent-metal.github.io/tt-metal/latest/tt_metal/apis/kernel_apis/sfpu/llk.html https://tenstorrent-metal.github.io/tt-metal/latest/tt_metal... [2] https://tenstorrent.com/systems/galaxy/ https://tenstorrent.com/systems/galaxy/
- dannyw 3y agoWhy are they so skimpy on memory capacity? I guess this is a dev kit for anything but LLMs?
- marty1885 3y agoGrayskull is make before LLMs being a thing. And their plan is like Groq, to distribute the compute graph across multiple processors to get higher effective memory and throughput by pipelining. But better-ish by having RAM so you can fit models on much less cards. Grayskull doesn't have this ability. The next generation Wormhole does by having 100GbE interfaces on the cards. Also the CPUs on Grayskull is 32bit. Memory is addressed through the bank address so it works for now. But they'll have to upgrade to 64bit soon.
- aleph_minus_one 3y agoThese specifications rather look like some AI acceleration ICs, and not like GPUs (recall that "GPU" stands for Graphics Processing Unit).
- KeplerBoy 3y agoSure, no one actually cares about the graphics part anymore in these contexts. Many Nvidia Datacenter GPUs are also no longer GPUs in that sense.
- mise_en_place 3y agoYep just need to load the model. It can and probably should be headless.
- bee_rider 3y agoI know it isn’t the what you mean, but as an aside (rather than a contradiction), the gaming marked is still much bigger than the ML model training one, right? So presumably people care about the graphics part in the sense that development there funds the development of the cards, haha. I always assumed that was what killed Xeon Phi. Hard to justify a GEMM card in and of itself. However clever you get, NVIDIA will be back next year with twice as much bandwidth.
- KeplerBoy 3y agoNo, datacenter GPU revenue has surpassed gaming revenue some time ago. At least for Nvidia and I fully expect this trend to continue.
- krapht 3y agoI'll bet against that. NVIDIA's margin is so large that Google/Microsoft and others can fund / are funding large teams to make TPUs and other dedicated hardware competitive. I also believe LLMs are part of the hype cycle just like crypto. In ten years the gaming market will return to being NVIDIA's primary market.