4 ms·
The real question here is what makes you think you need a product that automatically scales? If you have to think about it, chances are you don't need it. It's
by julianlam 4y ago
The real question here is what makes you think you need a product that automatically scales?
If you have to think about it, chances are you don't need it. It's often the case about a lot of optimizations. I've had clients ask for help sharding their database, when that's almost never the correct course of action.
- paulgb 4y agoGP mentioned GPUs. GPUs + bursty traffic = either you autoscale, or you burn through a bunch of credits and VC cash on idle GPUs.
- mattnewton 4y agoThis, we burn a lot of cash for every idle machine. Manually scaling up and down resulted in both worse performance and a lot more cost.
- thundergolfer 4y agoFor GPUs startups can now leverage serverless GPU cloud providers: https://ramsrigoutham.medium.com/the-landscape-of-serverless-gpu-providers-in-2023-a21b0ff18901 https://ramsrigoutham.medium.com/the-landscape-of-serverless.... Much simpler than setting up K8s with scale-to-zero autoscaling nodegroups.
- mattnewton 4y agoWill have to revisit this, when we originally evaluated banana.dev and similar platforms they lacked the ability to mount a network drive to quickly load a bunch of model weights after spinup, which is a weird requirement we had with a previous pipeline that we don’t need with other products.
- thundergolfer 4y agoModal.com has a network filesystem feature called 'shared volumes'. Disclaimer: I work for modal.com, but honestly Modal is the most comprehensive of the serverless GPU platforms because it didn't start as just a 'serverlessly run the latest open-source model' platform, it's aiming to be an end-to-end cloud platform.
- chinaman425 4y ago[dead]