3 ms·
If I set a limit, and you cut off my service because I reached the limit, I would definitely not "complain just as much" as if I set a limit and you allowed me
by radlad 6mo ago
If I set a limit, and you cut off my service because I reached the limit, I would definitely not "complain just as much" as if I set a limit and you allowed me to spend past it.
We're not talking about an EC2 or EBS volume here, this is access to an API.
- deleted 6mo ago[deleted]
- walthamstow 6mo agoMeh, you probably would complain. Maybe you forgot you set it. Now your project is taking off, making money, and it got nuked. Why aren't we talking about an EC2 - is that not a cloud compute service? People have been complaining about cloud billing since long before LLMs. Anything to say about the technical problem of constantly monitoring many services against a project or account-level limit?
- ChromaticPanic 6mo agoYou mean we can implement rate limiting on API for security purposes no problem but suddenly having it track costs as well is technically impossible?
- radlad 6mo agoWhy aren't we talking about an EC2? Because this is a thread about the Gemini inference API. Loss of service would be restored on payment, not permanent. But that's besides the point: I as the customer set a limit, and you as as the service provider did not adhere to it. I've worked on a number of systems and while it is sometimes impossible to stop at an exact limit, I am confident that it is feasible to stop with less slippage than occurred in this scenario. And at the companies I've worked at, within a margin of error that we're able to absorb any slippage ourselves, as these losses are made up elsewhere, and are worth the customer goodwill. If we can do it, I'm sure Google can.