I do much more inference than training, but a bit of both. Speed is generally not a serious issue: - I'm not trying to scale to millions of users on inference

I do much more inference than training, but a bit of both. Speed is generally not a serious issue: - I'm not trying to scale to millions of users on inference; it's a dev machine - Training is a bit of fine-tuning; I'm not building big models from scratch. - Worst-case speed has gone into "annoying" territory, but never into prohibitive. Worst case, I can grab a coffee while things run. - It's not a lot of coffee, since I'm trying to visualize some interesting things about some data. The majority of that is data cleaning, visualization, data engineering, UI/UX, meetings, and a million other things. My GPU is sitting idle >>95% of the time. I am constrained by RAM. In the era of GPT2 -> GPT3, Stable Diffusion, etc., RAM becomes a major bottleneck. There are cloud workflows, but they aren't a great fit, both due to data security / regulatory issues, to how we plan to do dev ops / deployment, as well as just to time for integration. Dropping $8k on 4x 24GB GPUs would be a no-brainer if it let me work with bigger models. I'm sure people can describe Better Ways to Do Things, but changing workflows, system architecture, etc. to those workflows would be obviously out-of-scope for what I'm trying to do. At that point, I'm better off working with smaller models. They're good enough.