3 ms·
1TB/s memory bandwidth is competitive with NVIDIA GPUs. But yes, I’m skeptical that there’s really a market for “runs the LLM, but it’s not usable in real-time
by gymbeaux 2y ago
1TB/s memory bandwidth is competitive with NVIDIA GPUs.
But yes, I’m skeptical that there’s really a market for “runs the LLM, but it’s not usable in real-time”. I guess if you had a list of prompts and you ran them throughout the night while you were sleeping?
NVIDIA’s Digits or whatever it’s called now has mediocre memory bandwidth, but it only pulls like 200W at full load. So on the one hand, it can’t be used like most people use ChatGPT or Gemini or DeepSeek, but if the alternative is “not running the LLM at all”, I can see some people wanting Digits. It may run LLMs very slowly (technically we don’t know for sure, but it’s likely based on hardware specs), but at least it’s very small and doesn’t use a lot of electricity or put out a lot of heat.
Although I guess 200W 24/7 is worse than 1kW for an hour or two a day.