36 ms·

Suggests to me that they have a LOT of compute that they can't, or have decided not to, scale down. Interesting way for them to keep the infra up and cover cost

Suggests to me that they have a LOT of compute that they can't, or have decided not to, scale down. Interesting way for them to keep the infra up and cover costs.

jstummbillig 3y ago
It mostly suggests large spikes in usage, no? Which does not seem very surprising.
- tony_cannistra 3y ago
  Totally. There are other explanations (like wanting to offer a discount), but my mind initally went to "they have a lot of provisioned compute that they pre-paid for, and they want higher-utilization when API request rates are low." Just hypothesizing.
  - amluto 3y ago
    Or they don't have enough provisioned compute to cover peak usage, and they therefore want to reduce peak usage.
blackoil 3y ago
Or simply that traffic is uneven. Like the Spot instances of EC2.
danielmarkbruce 3y ago
I don't think that's right. There is an optimal batch size, and this allows them to just keep everything optimal or close to it for as much time as possible. I'd bet every single minute they miss chances to run things optimally because they need to hit some latency number, and this enables them to not have to make that trade off.
- tony_cannistra 3y ago
  Yeah, that's another good explanation.
yumraj 3y ago
Not really. It could be they have massive traffic during US daytime hours, which is not evenly matched by traffic from the rest of the world during US night times, so they'll just run this batch during US night time and hence the 24 hour period.
- tony_cannistra 3y ago
  Yah, that's what I'm saying. In those low periods they could just scale down the infra, but perhaps there's a constraint that doesn't allow for that. This lets them not waste it.
  - yumraj 3y ago
    Depends on what you mean by scale down infra. If they're running their own DC (or their own hardware in a colo) then I believe the only scaling down can by shutting down the systems at night, which, while non-trivial amount in electricity savings, is not really much when compared to the total hardware CapEx or even the total OpEx. And, even then, you have to consider the cost of shutting down and bringing them up and the complexity it brings. Now, if they're using a cloud provider they can theoretically terminate the instances. But, even in that case, to do that, they'll have to use on-demand instances rather than use reserved instances which will probably wipe out all the savings.
pornel 3y ago
Maybe this gives them more control and improves ability to handle latency-sensitive traffic. Or maybe they’ve found a way to batch process many requests together in a way that’s more efficient.
8organicbits 3y ago
For anyone trying to figure out these sorts of pricing questions when running a service in the cloud: I wrote up a blog post about how (d one of) the math works out. https://alexsci.com/blog/modeling-on-demand-pricing/ https://alexsci.com/blog/modeling-on-demand-pricing/

Suggests to me that they have a LOT of compute that they can't, or have decided not to, scale down. Interesting way for them to keep the infra up and cover costs.

jstummbillig 3y ago

It mostly suggests large spikes in usage, no? Which does not seem very surprising.

tony_cannistra 3y ago
Totally. There are other explanations (like wanting to offer a discount), but my mind initally went to "they have a lot of provisioned compute that they pre-paid for, and they want higher-utilization when API request rates are low." Just hypothesizing.
- amluto 3y ago
  Or they don't have enough provisioned compute to cover peak usage, and they therefore want to reduce peak usage.

blackoil 3y ago

Or simply that traffic is uneven. Like the Spot instances of EC2.

danielmarkbruce 3y ago

I don't think that's right. There is an optimal batch size, and this allows them to just keep everything optimal or close to it for as much time as possible. I'd bet every single minute they miss chances to run things optimally because they need to hit some latency number, and this enables them to not have to make that trade off.

tony_cannistra 3y ago
Yeah, that's another good explanation.

yumraj 3y ago

Not really. It could be they have massive traffic during US daytime hours, which is not evenly matched by traffic from the rest of the world during US night times, so they'll just run this batch during US night time and hence the 24 hour period.

tony_cannistra 3y ago
Yah, that's what I'm saying. In those low periods they could just scale down the infra, but perhaps there's a constraint that doesn't allow for that. This lets them not waste it.
- yumraj 3y ago
  Depends on what you mean by scale down infra. If they're running their own DC (or their own hardware in a colo) then I believe the only scaling down can by shutting down the systems at night, which, while non-trivial amount in electricity savings, is not really much when compared to the total hardware CapEx or even the total OpEx. And, even then, you have to consider the cost of shutting down and bringing them up and the complexity it brings. Now, if they're using a cloud provider they can theoretically terminate the instances. But, even in that case, to do that, they'll have to use on-demand instances rather than use reserved instances which will probably wipe out all the savings.

pornel 3y ago

Maybe this gives them more control and improves ability to handle latency-sensitive traffic. Or maybe they’ve found a way to batch process many requests together in a way that’s more efficient.

8organicbits 3y ago

For anyone trying to figure out these sorts of pricing questions when running a service in the cloud: I wrote up a blog post about how (d one of) the math works out. https://alexsci.com/blog/modeling-on-demand-pricing/ https://alexsci.com/blog/modeling-on-demand-pricing/