3 ms·
just curious how you're hosting this? The cheapest GCP GPU instance is $0.5/hr, which is pretty expensive long-term.
by Jack000 7y ago
just curious how you're hosting this? The cheapest GCP GPU instance is $0.5/hr, which is pretty expensive long-term.
- jeffshek 7y agoGCP Instances that are scaling up and down based on usage. Lot of image freezing, for faster boots, etc. One hack is for the medium-level models, you can actually run them on Cascade Lake (which are sort-of more optimized for ML) than traditional processors. There's a 30% performance there. Mathematically, GPU vs CPU in performance at inference time, you're paying an annoying premium! Right now, the default writing style "medium" is running in Cascade Lake (no gpu). I also (over)optimized the microservices running the endpoints too.
- Jack000 7y agodidn't know about Cascade Lake, thanks! Every time I did the numbers it didn't add up for cloud GPU. I could buy a new gaming desktop, serve the model from that, and make my money back in 2-3 months