4 ms·

> they avoid the need for busywork polling The fact that you know that you can only expect an answer after 24h also avoids the need for busywork polling. > In

> they avoid the need for busywork polling The fact that you know that you can only expect an answer after 24h also avoids the need for busywork polling. > Instead of a single "one batch with maybe up to 24hr latency" the offer could be a series of tiered queues with different SLAs and costs, and you can pick which you submit your work to depending on how important it is. But that is what they have. They have one "ASAP" API and one "within 24h" one. These are two big distinct use cases. You use the fast one when there is a user waiting for an answer realtime, and you use the slow one when nobody is waiting for it realtime. These are two distinct use cases from the user's perspective. You are correct in identifying that there could be an API which offers 2h returns, or one which offer 14day returns, but why would you complicate the offering with that? It adds a lot of complication to the documentation, a lot of complication on the scheduling side and a lot of complication on the pricing side, and for what upside?

mike_hearn 3y ago
There's almost no complexity on the scheduling side done properly. Just pop work off the queues in priority order. There are use cases for multiple latency tiers 1. At the lower end you have realtime-ish responses for when the chat is via SMS/WhatsApp/tickets/email, i.e. the social convention is that responses may be delayed by a few minutes or even 10-20 minutes and that's reasonable, but not by 24 hours. 2. At the high end you have a database with 200 million documents, a prompt that works, and you just want to submit all of them in one go as quickly as your client can work, then get the results as they get processed. There is no latency requirement whatsoever, but you'd prefer not to have to chunk things up and track batch IDs and such yourself. What you DO really want though is to have exactly-once submission, so you don't accidentally lose your connection half way through a batch upload and then end up paying for the same batch twice. These are all basic problems of the type that occur all the time in the IT industry. The solutions are well known and mature. Your bank runs on message queues of various kinds. The tech really isn't that fancy.

mike_hearn 3y ago

There's almost no complexity on the scheduling side done properly. Just pop work off the queues in priority order. There are use cases for multiple latency tiers 1. At the lower end you have realtime-ish responses for when the chat is via SMS/WhatsApp/tickets/email, i.e. the social convention is that responses may be delayed by a few minutes or even 10-20 minutes and that's reasonable, but not by 24 hours. 2. At the high end you have a database with 200 million documents, a prompt that works, and you just want to submit all of them in one go as quickly as your client can work, then get the results as they get processed. There is no latency requirement whatsoever, but you'd prefer not to have to chunk things up and track batch IDs and such yourself. What you DO really want though is to have exactly-once submission, so you don't accidentally lose your connection half way through a batch upload and then end up paying for the same batch twice. These are all basic problems of the type that occur all the time in the IT industry. The solutions are well known and mature. Your bank runs on message queues of various kinds. The tech really isn't that fancy.