4 ms·
What doe they mean by "single-node inference"? Do they mean inference done on a single machine?
by tonetegeatinst 2y ago
What doe they mean by "single-node inference"?
Do they mean inference done on a single machine?
- simonw 2y agoYes, albeit a really expensive one. Large models like GPT-4 are rumored to run inference on multiple machines because they don't fit in VRAM for even the most expensive GPUs. (I wouldn't be surprised if GPT-4o mini is small enough to fit on a single large instance though, would explain how they could drop the price so much.)
- bjornsing 2y agoYeah that’s how I read it. Probably means 8 x 80 GB GPUs.