6 ms·

It's a pity that this sort of feature is done with a custom ad-hoc HTTP protocol instead of using a message queuing protocol like AMQP or STOMP. Those are desig

It's a pity that this sort of feature is done with a custom ad-hoc HTTP protocol instead of using a message queuing protocol like AMQP or STOMP. Those are designed for this kind of thing and already have libraries available, they avoid the need for busywork polling and avoid the need to break work up into batches client side. You should just be able to submit messages containing work to a queue and then let the broker inform you when new responses arrive on the response queue if you're connected. If you're not connected they buffer up for a while until you reconnect and drain the queue. The advantage of that approach would be that latency could be minimized or tiered. Instead of a single "one batch with maybe up to 24hr latency" the offer could be a series of tiered queues with different SLAs and costs, and you can pick which you submit your work to depending on how important it is. I wonder why they went for this custom JSON/HTTP approach instead. Maybe they feel that they can't expect developers to understand anything other than POSTing JSON.

dmw_ng 3y ago
There's a reason pretty much everything that does not require low latency replies avoids stateful networking - everything from RSS to video streaming prefers stateless polling designs because it is vastly easier for both parties to implement and scale. Meanwhile, I couldn't name a single system in widespread use built around a MQ paradigm in its public interface, except for actual MQ APIs, and many of those (e.g. from AWS) are still built on polling for the reasons just described
- jerf 3y ago
  Behind each request made to OpenAI is a staggering amount of GPU computation. If the price of the queue request is even a hundred thousandth of the overall price of a single request I'd be stunned. There is no message queue scaling issue here. Message queue scaling issues arise when you are blasting around a lot of messages, but each of them take minimal resources on an individual basis to service, so it's feasible for the queue itself to be the bottleneck. I wouldn't be surprised a single Raspberry Pi could handle the entire queuing load here, and if it couldn't it's not off by a very large factor, because the GPU resources behind what it would take to service a full RPi's queuing capacity would be staggeringly enormous, I think well beyond what OpenAI actually has.
  - tsimionescu 3y ago
    Isn't the same true then of an HTTP server? Handling the polling requests is a minute amount of work compared to running the real jobs. And you've addressed the scalability problem, but not the connectivity issues that generally plague long-lived connections on the Internet at large.
    - mike_hearn 3y ago
      Inevitably some users will decide to poll every 60 seconds or whatever, because they have no idea when the work will be completed and because what they really want is "results ASAP but willing to tolerate latency to pay less". And then your servers are doing a ton of TLS negotiation, user authentication, request serving and database lookups, just to answer "not yet". I think people are getting distracted by the idea of connections being somehow expensive. They aren't really compared to polling (unless the poll is genuinely very rare). A stateless request is expensive because you have to go back to your source of truth on every request (probably an expensive and hard to scale RDBMS), and you don't control how often the user makes such requests. CPU load is potentially unbounded and users don't pay unless you introduce pay-per-poll micropayments. Compare that to an MQ design: the overhead is a single TCP connection and a bit of memory to map that connection to an internal queue. Whilst the work sits in the queue or is being processed, nothing is happening and there's no DB load. Overhead is a matter of bytes and in the event that you run out of RAM you can always kick users off at random and let them exponentially back off and retry (automatically - because the libraries handle this and make it transparent). Or just use swap, after all, latency is not that important.
      - tsimionescu 3y ago
        Nothing prevents, in principle, a long lived HTTP connection where the server only replies once the response is available (long polling). However, on the real internet, such long lived connections just don't work, for a large minority of users. There are numerous devices, typically close to the client, which kill "idle" connections. NAT gateways and stateful firewalls are some of the most common culprits. So, you just can't rely on your customers being able to keep around a long connection. Not to mention the numerous corporate environments in which it is hard to even open an outgoing connection which is not HTTPS or a handful of other known protocols.
- btown 3y ago
  Dropbox's approach here in the early 2010s (not sure if still used) was actually quite clever. Both official clients and third parties could open up a long-timeout HTTP connection that wouldn't be responded to until the web server was delivered a message in their internal message queue. Avoided the overhead of polling and allowed for extremely low latency, while letting clients still use their favorite HTTP library. https://dropbox.tech/developers/low-latency-notification-of-dropbox-file-changes https://dropbox.tech/developers/low-latency-notification-of-...
  - tsimionescu 3y ago
    That's typically called HTTP long polling, and it's a commonly used alternative to things like WebSockets.
  - hamandcheese 3y ago
    This is just long-polling, right?
  - smarx007 3y ago
    > long-timeout HTTP connection that wouldn't be responded to This open connection IS the overhead. Some approaches have more overhead, some less. Long-polling these days is not needed and can be replaced by SSE [1] (if you are OK with unidirectional communication), websockets (bidirectional), or callback URIs in the request. The latter is a per-request webhook, essentially, and would have the lowest overhead at the cost of ceremony (the client now needs to have a running web server). By the way, long polling was popularized by Comet [2] around 2006. [1]: https://en.wikipedia.org/wiki/Server-sent_events https://en.wikipedia.org/wiki/Server-sent_events [2]: https://en.wikipedia.org/wiki/Comet_(programming) https://en.wikipedia.org/wiki/Comet_(programming)
    - TeMPOraL 3y ago
      Wait, how is having the client run its own http(s) server and accept regular HTTP(S) requests the lowest overhead option? Sounds like the highest overhead one, with all the protocol statelessness.
- dec0dedab0de 3y ago
  webhooks are a little closer in that they remove the need for constant polling, atleast for longer running processes
- mike_hearn 3y ago
  Think about every messenger out there, any Slack or Slack-like app that uses WebSockets, email clients that use IMAP etc. They aren't polling once a minute like it's 1995. It's not really easier to implement or scale, in my view. It may seem that way if you've never worked on large scale stateful connection serving. If you want users to get the answer as soon as it's ready, and you do or at least your users do, then you need users to hold open connections and at that point it's really a question of how much your protocols and libraries do for you. If you use HTTP the answer is "not much" because it was designed to download hypertext. The actual task of managing lots of connections server side isn't especially hard. There are MQ engines that support sharding and can have a regular TCP LB like a NetScaler stuck in front of them, or of course, you can just implement client-side LB as well.
  - dmw_ng 3y ago
    I'm not promoting one way or the other, just pointing out why things are the way they are. Restarting a service with stateful networking is reason enough to avoid it where possible, watching entire buildings melt for 15 minutes because a single binary SEGV'd is a real outcome. For an extra helping of pain, add a herd of third party clients of random versions to a system that never needed the comms capabilities on offer and you have a problem to solve that never needed to exist in the first place
    - mike_hearn 3y ago
      Good MQ libraries know how to do backoff and retry transparently, so if you aren't provisioned for peak load (which is what melting implies) you can just reject connections for a while until everyone is back. The alternative with polling rapidly turns into being melted all the time - it's literally handling constant reconnection load - especially as you don't control the polling code and as the user is expected to write it themselves you can safely bet it will be MVP. Retries if they occur at all will be in a tight loop. I worked on Gmail for a few years and we had a ton of clients permanently connected for IMAP, web long polling and Talk. Those clients tended to be well written and it wasn't the big problem people seem to think it is, especially as some could do client side LB which reduced resource requirements server side. I also experienced (luckily from a distance) fully HTTP polling based systems where backoff/retry wasn't implemented properly and caused services to go hard down because they couldn't handle being hit with all the polling simultaneously, and the clients just broke or did something stupid like immediately retry if they got server errors. Fundamentally, state is fundamental and can't be removed, so it's going to be sitting somewhere. If state isn't in the protocol stack then it's in your database or session cache instead. Sometimes that's better, sometimes it's worse.
simonw 3y ago
That sounds really difficult to use to me. I'd need to figure out how to connect to a custom messaging protocol, and how to keep that connection online (maybe routing through firewalls etc), and handle retries and connection failures and suchlike. An HTTP polling API may be less efficient but I know how to use that from all of the programming languages I work with already.
- alexchamberlain 3y ago
  FWIW, AMQP is the same protocol as RabbitMQ- there's a library for pretty much every language/ecosystem.
  - simonw 3y ago
    Yeah, I've used that in the past and found it significantly harder than HTTP.
- mike_hearn 3y ago
  AMQP isn't a custom messaging protocol and MQ libraries do all of that for you already (handling reconnects and retries etc). Connection interruptions aren't an issue. You're actually worrying about low level details of the sort MQ systems are specifically designed to solve for you, but HTTP stacks do not. For example, someone below asks "how do I see my submitted jobs". MQ protocols solve that for you out of the box. You can browse the queues that you own. It's all already standardized and implemented. Because they rolled their own protocol, OpenAI lacks this function and will have to develop it themselves. Then because it's all oriented around batches (when the underlying problem does not actually need batching), the users will have to run a local DB to keep track of what prompts map to what batches or they'll find themselves redownloading the whole batch to find out what's in it. As for firewalls, you open the port. The only issue there is if you're behind a firewall you don't have any control over and where the people who do don't want you using AI, but that's not an issue in the datacenter deployments that are presumably generating inferencing batch workloads, and such firewalls or proxies can break HTTP APIs too anyway.
tuckerconnelly 3y ago
Message queues are difficult to maintain. Webhooks is the best solution IMO.
krisoft 3y ago
> they avoid the need for busywork polling The fact that you know that you can only expect an answer after 24h also avoids the need for busywork polling. > Instead of a single "one batch with maybe up to 24hr latency" the offer could be a series of tiered queues with different SLAs and costs, and you can pick which you submit your work to depending on how important it is. But that is what they have. They have one "ASAP" API and one "within 24h" one. These are two big distinct use cases. You use the fast one when there is a user waiting for an answer realtime, and you use the slow one when nobody is waiting for it realtime. These are two distinct use cases from the user's perspective. You are correct in identifying that there could be an API which offers 2h returns, or one which offer 14day returns, but why would you complicate the offering with that? It adds a lot of complication to the documentation, a lot of complication on the scheduling side and a lot of complication on the pricing side, and for what upside?
- mike_hearn 3y ago
  There's almost no complexity on the scheduling side done properly. Just pop work off the queues in priority order. There are use cases for multiple latency tiers 1. At the lower end you have realtime-ish responses for when the chat is via SMS/WhatsApp/tickets/email, i.e. the social convention is that responses may be delayed by a few minutes or even 10-20 minutes and that's reasonable, but not by 24 hours. 2. At the high end you have a database with 200 million documents, a prompt that works, and you just want to submit all of them in one go as quickly as your client can work, then get the results as they get processed. There is no latency requirement whatsoever, but you'd prefer not to have to chunk things up and track batch IDs and such yourself. What you DO really want though is to have exactly-once submission, so you don't accidentally lose your connection half way through a batch upload and then end up paying for the same batch twice. These are all basic problems of the type that occur all the time in the IT industry. The solutions are well known and mature. Your bank runs on message queues of various kinds. The tech really isn't that fancy.
deleted 3y ago
[deleted]
willcipriano 3y ago
You'd have to hold a ton of connections open for 24 hours+
- mike_hearn 3y ago
  That isn't hard. They aren't going to serve a billion connections are they, and WhatsApp scaled a full NxN low latency messaging system up with a handful of people. At any rate, that's only the case if every request actually sits in the queue for 24 hours. The 24 hour period is clearly chosen to let them absorb free capacity overnight (all such services have diurnal cycles), but unless their implementation is really shoddy at any given moment their workers will be popping work off an internal queue anyway to take advantage of moment-sized lulls in traffic. Done properly you can saturate your hardware 100% all the time... there's certainly enough inferencing demand out there. So at that point the question is just one of external API.
dragonwriter 3y ago
> I wonder why they went for this custom JSON/HTTP approach instead. Maybe they feel that they can't expect developers to understand anything other than POSTing JSON. Its simpler for them to build and operate since it fits their existing API (and user validation, quota tracking, etc.) infrastructure, its simpler for them to update their SDK to handle another HTTP endpoint than to build an AMQP or STOMP client into the SDK for this feature, its easier for most consumers who aren't using the SDK, and a message queue interface for this one bit of functionality would offer minimal benefits. Easy choice. (If you are consuming from an environment where you are alreayd using AMQP or STOMP, sure, build a quick adapter for the HTTP interface. But I don't see any good reason OpenAI should have done one of those instead of HTTP.)
vibl 3y ago
> Instead of a single "one batch with maybe up to 24hr latency" the offer could be a series of tiered queues with different SLAs and costs The API includes a `completion_window` parameter ("The time frame within which the batch should be processed. Currently only 24h is supported."). https://platform.openai.com/docs/api-reference/batch/create https://platform.openai.com/docs/api-reference/batch/create

dmw_ng 3y ago

There's a reason pretty much everything that does not require low latency replies avoids stateful networking - everything from RSS to video streaming prefers stateless polling designs because it is vastly easier for both parties to implement and scale. Meanwhile, I couldn't name a single system in widespread use built around a MQ paradigm in its public interface, except for actual MQ APIs, and many of those (e.g. from AWS) are still built on polling for the reasons just described

jerf 3y ago
Behind each request made to OpenAI is a staggering amount of GPU computation. If the price of the queue request is even a hundred thousandth of the overall price of a single request I'd be stunned. There is no message queue scaling issue here. Message queue scaling issues arise when you are blasting around a lot of messages, but each of them take minimal resources on an individual basis to service, so it's feasible for the queue itself to be the bottleneck. I wouldn't be surprised a single Raspberry Pi could handle the entire queuing load here, and if it couldn't it's not off by a very large factor, because the GPU resources behind what it would take to service a full RPi's queuing capacity would be staggeringly enormous, I think well beyond what OpenAI actually has.
- tsimionescu 3y ago
  Isn't the same true then of an HTTP server? Handling the polling requests is a minute amount of work compared to running the real jobs. And you've addressed the scalability problem, but not the connectivity issues that generally plague long-lived connections on the Internet at large.
  - mike_hearn 3y ago
    Inevitably some users will decide to poll every 60 seconds or whatever, because they have no idea when the work will be completed and because what they really want is "results ASAP but willing to tolerate latency to pay less". And then your servers are doing a ton of TLS negotiation, user authentication, request serving and database lookups, just to answer "not yet". I think people are getting distracted by the idea of connections being somehow expensive. They aren't really compared to polling (unless the poll is genuinely very rare). A stateless request is expensive because you have to go back to your source of truth on every request (probably an expensive and hard to scale RDBMS), and you don't control how often the user makes such requests. CPU load is potentially unbounded and users don't pay unless you introduce pay-per-poll micropayments. Compare that to an MQ design: the overhead is a single TCP connection and a bit of memory to map that connection to an internal queue. Whilst the work sits in the queue or is being processed, nothing is happening and there's no DB load. Overhead is a matter of bytes and in the event that you run out of RAM you can always kick users off at random and let them exponentially back off and retry (automatically - because the libraries handle this and make it transparent). Or just use swap, after all, latency is not that important.
    - tsimionescu 3y ago
      Nothing prevents, in principle, a long lived HTTP connection where the server only replies once the response is available (long polling). However, on the real internet, such long lived connections just don't work, for a large minority of users. There are numerous devices, typically close to the client, which kill "idle" connections. NAT gateways and stateful firewalls are some of the most common culprits. So, you just can't rely on your customers being able to keep around a long connection. Not to mention the numerous corporate environments in which it is hard to even open an outgoing connection which is not HTTPS or a handful of other known protocols.
btown 3y ago
Dropbox's approach here in the early 2010s (not sure if still used) was actually quite clever. Both official clients and third parties could open up a long-timeout HTTP connection that wouldn't be responded to until the web server was delivered a message in their internal message queue. Avoided the overhead of polling and allowed for extremely low latency, while letting clients still use their favorite HTTP library. https://dropbox.tech/developers/low-latency-notification-of-dropbox-file-changes https://dropbox.tech/developers/low-latency-notification-of-...
- tsimionescu 3y ago
  That's typically called HTTP long polling, and it's a commonly used alternative to things like WebSockets.
- hamandcheese 3y ago
  This is just long-polling, right?
- smarx007 3y ago
  > long-timeout HTTP connection that wouldn't be responded to This open connection IS the overhead. Some approaches have more overhead, some less. Long-polling these days is not needed and can be replaced by SSE [1] (if you are OK with unidirectional communication), websockets (bidirectional), or callback URIs in the request. The latter is a per-request webhook, essentially, and would have the lowest overhead at the cost of ceremony (the client now needs to have a running web server). By the way, long polling was popularized by Comet [2] around 2006. [1]: https://en.wikipedia.org/wiki/Server-sent_events https://en.wikipedia.org/wiki/Server-sent_events [2]: https://en.wikipedia.org/wiki/Comet_(programming) https://en.wikipedia.org/wiki/Comet_(programming)
  - TeMPOraL 3y ago
    Wait, how is having the client run its own http(s) server and accept regular HTTP(S) requests the lowest overhead option? Sounds like the highest overhead one, with all the protocol statelessness.
dec0dedab0de 3y ago
webhooks are a little closer in that they remove the need for constant polling, atleast for longer running processes
mike_hearn 3y ago
Think about every messenger out there, any Slack or Slack-like app that uses WebSockets, email clients that use IMAP etc. They aren't polling once a minute like it's 1995. It's not really easier to implement or scale, in my view. It may seem that way if you've never worked on large scale stateful connection serving. If you want users to get the answer as soon as it's ready, and you do or at least your users do, then you need users to hold open connections and at that point it's really a question of how much your protocols and libraries do for you. If you use HTTP the answer is "not much" because it was designed to download hypertext. The actual task of managing lots of connections server side isn't especially hard. There are MQ engines that support sharding and can have a regular TCP LB like a NetScaler stuck in front of them, or of course, you can just implement client-side LB as well.
- dmw_ng 3y ago
  I'm not promoting one way or the other, just pointing out why things are the way they are. Restarting a service with stateful networking is reason enough to avoid it where possible, watching entire buildings melt for 15 minutes because a single binary SEGV'd is a real outcome. For an extra helping of pain, add a herd of third party clients of random versions to a system that never needed the comms capabilities on offer and you have a problem to solve that never needed to exist in the first place
  - mike_hearn 3y ago
    Good MQ libraries know how to do backoff and retry transparently, so if you aren't provisioned for peak load (which is what melting implies) you can just reject connections for a while until everyone is back. The alternative with polling rapidly turns into being melted all the time - it's literally handling constant reconnection load - especially as you don't control the polling code and as the user is expected to write it themselves you can safely bet it will be MVP. Retries if they occur at all will be in a tight loop. I worked on Gmail for a few years and we had a ton of clients permanently connected for IMAP, web long polling and Talk. Those clients tended to be well written and it wasn't the big problem people seem to think it is, especially as some could do client side LB which reduced resource requirements server side. I also experienced (luckily from a distance) fully HTTP polling based systems where backoff/retry wasn't implemented properly and caused services to go hard down because they couldn't handle being hit with all the polling simultaneously, and the clients just broke or did something stupid like immediately retry if they got server errors. Fundamentally, state is fundamental and can't be removed, so it's going to be sitting somewhere. If state isn't in the protocol stack then it's in your database or session cache instead. Sometimes that's better, sometimes it's worse.

simonw 3y ago

That sounds really difficult to use to me. I'd need to figure out how to connect to a custom messaging protocol, and how to keep that connection online (maybe routing through firewalls etc), and handle retries and connection failures and suchlike. An HTTP polling API may be less efficient but I know how to use that from all of the programming languages I work with already.

alexchamberlain 3y ago
FWIW, AMQP is the same protocol as RabbitMQ- there's a library for pretty much every language/ecosystem.
- simonw 3y ago
  Yeah, I've used that in the past and found it significantly harder than HTTP.
mike_hearn 3y ago
AMQP isn't a custom messaging protocol and MQ libraries do all of that for you already (handling reconnects and retries etc). Connection interruptions aren't an issue. You're actually worrying about low level details of the sort MQ systems are specifically designed to solve for you, but HTTP stacks do not. For example, someone below asks "how do I see my submitted jobs". MQ protocols solve that for you out of the box. You can browse the queues that you own. It's all already standardized and implemented. Because they rolled their own protocol, OpenAI lacks this function and will have to develop it themselves. Then because it's all oriented around batches (when the underlying problem does not actually need batching), the users will have to run a local DB to keep track of what prompts map to what batches or they'll find themselves redownloading the whole batch to find out what's in it. As for firewalls, you open the port. The only issue there is if you're behind a firewall you don't have any control over and where the people who do don't want you using AI, but that's not an issue in the datacenter deployments that are presumably generating inferencing batch workloads, and such firewalls or proxies can break HTTP APIs too anyway.

tuckerconnelly 3y ago

Message queues are difficult to maintain. Webhooks is the best solution IMO.

krisoft 3y ago

> they avoid the need for busywork polling The fact that you know that you can only expect an answer after 24h also avoids the need for busywork polling. > Instead of a single "one batch with maybe up to 24hr latency" the offer could be a series of tiered queues with different SLAs and costs, and you can pick which you submit your work to depending on how important it is. But that is what they have. They have one "ASAP" API and one "within 24h" one. These are two big distinct use cases. You use the fast one when there is a user waiting for an answer realtime, and you use the slow one when nobody is waiting for it realtime. These are two distinct use cases from the user's perspective. You are correct in identifying that there could be an API which offers 2h returns, or one which offer 14day returns, but why would you complicate the offering with that? It adds a lot of complication to the documentation, a lot of complication on the scheduling side and a lot of complication on the pricing side, and for what upside?

mike_hearn 3y ago
There's almost no complexity on the scheduling side done properly. Just pop work off the queues in priority order. There are use cases for multiple latency tiers 1. At the lower end you have realtime-ish responses for when the chat is via SMS/WhatsApp/tickets/email, i.e. the social convention is that responses may be delayed by a few minutes or even 10-20 minutes and that's reasonable, but not by 24 hours. 2. At the high end you have a database with 200 million documents, a prompt that works, and you just want to submit all of them in one go as quickly as your client can work, then get the results as they get processed. There is no latency requirement whatsoever, but you'd prefer not to have to chunk things up and track batch IDs and such yourself. What you DO really want though is to have exactly-once submission, so you don't accidentally lose your connection half way through a batch upload and then end up paying for the same batch twice. These are all basic problems of the type that occur all the time in the IT industry. The solutions are well known and mature. Your bank runs on message queues of various kinds. The tech really isn't that fancy.

deleted 3y ago

[deleted]

willcipriano 3y ago

You'd have to hold a ton of connections open for 24 hours+

mike_hearn 3y ago
That isn't hard. They aren't going to serve a billion connections are they, and WhatsApp scaled a full NxN low latency messaging system up with a handful of people. At any rate, that's only the case if every request actually sits in the queue for 24 hours. The 24 hour period is clearly chosen to let them absorb free capacity overnight (all such services have diurnal cycles), but unless their implementation is really shoddy at any given moment their workers will be popping work off an internal queue anyway to take advantage of moment-sized lulls in traffic. Done properly you can saturate your hardware 100% all the time... there's certainly enough inferencing demand out there. So at that point the question is just one of external API.

dragonwriter 3y ago

> I wonder why they went for this custom JSON/HTTP approach instead. Maybe they feel that they can't expect developers to understand anything other than POSTing JSON. Its simpler for them to build and operate since it fits their existing API (and user validation, quota tracking, etc.) infrastructure, its simpler for them to update their SDK to handle another HTTP endpoint than to build an AMQP or STOMP client into the SDK for this feature, its easier for most consumers who aren't using the SDK, and a message queue interface for this one bit of functionality would offer minimal benefits. Easy choice. (If you are consuming from an environment where you are alreayd using AMQP or STOMP, sure, build a quick adapter for the HTTP interface. But I don't see any good reason OpenAI should have done one of those instead of HTTP.)

vibl 3y ago

> Instead of a single "one batch with maybe up to 24hr latency" the offer could be a series of tiered queues with different SLAs and costs The API includes a `completion_window` parameter ("The time frame within which the batch should be processed. Currently only 24h is supported."). https://platform.openai.com/docs/api-reference/batch/create https://platform.openai.com/docs/api-reference/batch/create