3 ms·
Dedicated PCIe or memory-attached inference units will continue to exist, and will continue to do the heavy lifting, but an ISA extension provides latencies any
by rbanffy 4mo ago
Dedicated PCIe or memory-attached inference units will continue to exist, and will continue to do the heavy lifting, but an ISA extension provides latencies any external unit would have trouble matching. You can, with some work, extract some decent throughput with CPUs alone, with a large enough CPU you can use for non-AI jobs the rest of the time. There was a nice writeup not that long ago here on HN describing the flags and the reasoning behind them to use on that specific machine.
Funny thing, I can't find the article.
- usrusr 4mo agoYeah, I guess in these days of hardware not getting completely outdated even after half a decade, it might well make sense to spend chip area on specialist units that are shut down almost the entire time. In the old days, when it was all about squeezing maximum performance from a given chip area budget you'd prefer using those transistors for some improvement of the general purpose parts, no matter how marginal. But now it's the thermal envelope, or power consumption of you prefer looking at it from that angle, and even slightly esoteric special purpose units can start making sense.
- rbanffy 4mo agoDark silicon is also a thermal management strategy. Areal usage can go down, spreading heat generation over a larger space and making cooling the chip easier.