5 ms·
Memory seems very low for inference on bigger models?. Memory bandwidth is also massively lower than something like a 4090 or 7900xtx?. Am I missing some fairy
by christkv 3y ago
Memory seems very low for inference on bigger models?. Memory bandwidth is also massively lower than something like a 4090 or 7900xtx?. Am I missing some fairy dust magic here?
- ChainOfFools 3y agoWhat's the memory interface like? Is an extreme NUMA thing with several orders greater potential for parallelization? I'm probably dreaming; new CPU core designs are one thing, a completely new memory design is likely high fantasy..
- onlyrealcuzzo 3y agoWill graphics cards eventually get renamed? Are the optimizations for inference basically identical to graphics? I imagine the workloads are different, and that inference has been the main market for years now...
- yencabulator 3y agoThe new things are already getting renamed, but the dust hasn't settled yet. NPU: https://en.wikipedia.org/wiki/AI_accelerator https://en.wikipedia.org/wiki/AI_accelerator TPU: https://en.wikipedia.org/wiki/Tensor_Processing_Unit https://en.wikipedia.org/wiki/Tensor_Processing_Unit
- adgjlsfhk1 3y agowhy bother renaming? just re-acronym to "greatly parallel unit"
- algo_trader 3y agoThe website indicates each Tensix core ~1TOP theoretical. So not amazing. It will have to compete on price/memory in the crowded inference space.
- andy99 3y agoThis is purely a developer kit, letting people get used to the hardware configuration and the software stacks before the big hardware comes later in future generations. I'd like to know what different precisions/ quantizations if any are supported. For LLMs 8GB is fine for playing around if weights are quantized. And as the article mentions it's more than enough for lots of computer vision models.
- camel-cdr 3y agoGrayskull supports FP8, FP16, BFP16, VFP2, BFP4, BFP8 [0], and some sort of 19-bit floating point in the SFPU. [1] I don't know how and with what performance those are supported. FP4 has 221 TFLOPS on the Grayskull e75, and 332 TFLOPS on the Grayskull e150. [0] While wormhole has 82 INT8 TOPS per card. [2] I haven't found any other numbers. Edit: Some more info about data formats: https://docs.tenstorrent.com/tenstorrent/v/tt-buda/dataformats https://docs.tenstorrent.com/tenstorrent/v/tt-buda/dataforma... [0] https://docs.tenstorrent.com/tenstorrent/add-in-boards-and-cooling-kits/grayskull-tm-e75-e150/specifications https://docs.tenstorrent.com/tenstorrent/add-in-boards-and-c... [1] https://tenstorrent-metal.github.io/tt-metal/latest/tt_metal/apis/kernel_apis/sfpu/llk.html https://tenstorrent-metal.github.io/tt-metal/latest/tt_metal... [2] https://tenstorrent.com/systems/galaxy/ https://tenstorrent.com/systems/galaxy/
- dannyw 3y agoWhy are they so skimpy on memory capacity? I guess this is a dev kit for anything but LLMs?
- marty1885 3y agoGrayskull is make before LLMs being a thing. And their plan is like Groq, to distribute the compute graph across multiple processors to get higher effective memory and throughput by pipelining. But better-ish by having RAM so you can fit models on much less cards. Grayskull doesn't have this ability. The next generation Wormhole does by having 100GbE interfaces on the cards. Also the CPUs on Grayskull is 32bit. Memory is addressed through the bank address so it works for now. But they'll have to upgrade to 64bit soon.