5 ms·
I have not found this to be true at all in my field (natural language generation). We have a 7 figure GPU setup that is running 24/7 at 100% utilization just t
by cardine 4y ago
I have not found this to be true at all in my field (natural language generation).
We have a 7 figure GPU setup that is running 24/7 at 100% utilization just to handle inference.
- fartcannon 4y agoHow do you train new models if your GPUs are being used for inference? I guess the training happens significantly less frequently? Forgive my ignorance.
- jacquesm 4y agoTypically a different set of hardware for model training.
- cardine 4y agoWe have different servers for each. But the split is usually 80%/20% for inference/training. As our product grows in usage the 80% number is steadily increasing. That isn't because we aren't training that often - we are almost always training many new models. It is just that inference is so computationally expensive!
- godelski 4y agoAre you training new models from scratch or just fine tuning LLMs? I'm from the CV side and we tend to train stuff from scratch because we're still highly focused on finding new architectures and how to scale. The NLP people I know tend to use LLMs and existing checkpoints so their experiments tend to be a lot cheaper. Not that anyone should think any aspect (training nor inference) is cheap.
- dheera 4y agoAlso true of self-driving. You train a perception model for a week and then log millions of vehicle-hours on inference.