> However, that GPU could probably handle only half dozen users It may only be able to host a handful of users making requests within the same second, but ther

> However, that GPU could probably handle only half dozen users It may only be able to host a handful of users making requests within the same second, but there's a lot of downtime when users are not triggering inference requests – time reading code, navigating around, etc. Add to that the fact that most engineers are having meetings, writing docs, testing things, they do a lot that isn't writing code. Then you add the fact that users don't work 24/7. Between all of this, I'd be surprised if a GPU wasn't able to handle ~hundreds to ~thousands of monthly active users.