3 ms·
i guess you'd need a good relationship with a cloud provider, because I'm not even sure a regular customer could rent that many GPUs through normal means, or ju
by FrenchDevRemote 3y ago
i guess you'd need a good relationship with a cloud provider, because I'm not even sure a regular customer could rent that many GPUs through normal means, or just make your own servers/datacenters
then I guess some basic decent load balancing/queuing on your inference endpoints
- dbish 3y agoExactly. All the big ones have partnerships/allegiances or own their cloud already, look at OpenAI and Microsoft or Google Bard on top of GCP, and similarly Stability seems to have a strategic partnership with AWS. You won't run at their scale without something like that, especially with GPU shortages we're seeing.