3 ms·
Is it cheaper in $/GB than used Vega 56(HBM2 8GB) besides? There are mining boards with bunch of x1 slots that probably can run half a dozen of them for same 48
by numpad0 2y ago
Is it cheaper in $/GB than used Vega 56(HBM2 8GB) besides? There are mining boards with bunch of x1 slots that probably can run half a dozen of them for same 48GB.
- BoredPositron 2y agoWould take a bunch of time just to load the model...
- magicalhippo 2y agoAFAIK this doesn't really work for interactive use, as LLMs process data serially. So your request needs to pass through all of the cards for each token, one at a time. Thus a lot of PCIe traffic and hence latency. Better than nothing, but only really useful if you can batch requests so you can keep each GPU working all the time, rather than just one at a time.
- numpad0 2y agoClearly I wasn't aware enough that DNN is by default like a mesh. Makes sense that it's going to be bottlenecked by the tightest link. Thanks...