3 ms·

Hard to know who this is aimed at until we see the price. I guess they are going for the “slow memory, but lots of it” market that is less sensitive to how fast

by jl6 2y ago

Hard to know who this is aimed at until we see the price. I guess they are going for the “slow memory, but lots of it” market that is less sensitive to how fast their very large LLMs run as long as they run at all. Hobbyists will rejoice if they can afford it, but is there a commercial use case that can tolerate low tokens per second?

gessha 2y ago
Large scale unstructured text extraction? Anything you can set up a job for and go to bed?
zamadatix 2y ago
Ignoring the DDR5 completely the 4c-26 is still 256 GB of 1.092 TB/s memory in a single card. Looking to the "starter" 1c26-032 it still has a 400G connection on top of 32 lanes of PCIe Gen 5, so I'm not sure if the card overall will really be cost competitive for "local but slow" LLM use. Prices will say all though. The actual RT performance will be really interesting for rendering use cases too.
Palomides 2y ago
a 400gbe network card runs like $1500, these aren't going to be priced for hobbyists
gymbeaux 2y ago
1TB/s memory bandwidth is competitive with NVIDIA GPUs. But yes, I’m skeptical that there’s really a market for “runs the LLM, but it’s not usable in real-time”. I guess if you had a list of prompts and you ran them throughout the night while you were sleeping? NVIDIA’s Digits or whatever it’s called now has mediocre memory bandwidth, but it only pulls like 200W at full load. So on the one hand, it can’t be used like most people use ChatGPT or Gemini or DeepSeek, but if the alternative is “not running the LLM at all”, I can see some people wanting Digits. It may run LLMs very slowly (technically we don’t know for sure, but it’s likely based on hardware specs), but at least it’s very small and doesn’t use a lot of electricity or put out a lot of heat. Although I guess 200W 24/7 is worse than 1kW for an hour or two a day.

by jl6 2y ago

gessha 2y ago

Large scale unstructured text extraction? Anything you can set up a job for and go to bed?

zamadatix 2y ago

Ignoring the DDR5 completely the 4c-26 is still 256 GB of 1.092 TB/s memory in a single card. Looking to the "starter" 1c26-032 it still has a 400G connection on top of 32 lanes of PCIe Gen 5, so I'm not sure if the card overall will really be cost competitive for "local but slow" LLM use. Prices will say all though. The actual RT performance will be really interesting for rendering use cases too.

Palomides 2y ago

a 400gbe network card runs like $1500, these aren't going to be priced for hobbyists

gymbeaux 2y ago

1TB/s memory bandwidth is competitive with NVIDIA GPUs. But yes, I’m skeptical that there’s really a market for “runs the LLM, but it’s not usable in real-time”. I guess if you had a list of prompts and you ran them throughout the night while you were sleeping? NVIDIA’s Digits or whatever it’s called now has mediocre memory bandwidth, but it only pulls like 200W at full load. So on the one hand, it can’t be used like most people use ChatGPT or Gemini or DeepSeek, but if the alternative is “not running the LLM at all”, I can see some people wanting Digits. It may run LLMs very slowly (technically we don’t know for sure, but it’s likely based on hardware specs), but at least it’s very small and doesn’t use a lot of electricity or put out a lot of heat. Although I guess 200W 24/7 is worse than 1kW for an hour or two a day.