3 ms·

Then you design your system to always respond as quickly as possible. HTTP even has a status code for that: 202 Accepted. You put the job in a queue and return

Then you design your system to always respond as quickly as possible. HTTP even has a status code for that: 202 Accepted. You put the job in a queue and return some ID or cookie to the client, possibly also an ETA. Client can then poll for results (with bounded exponential back-off), again polling is just a quick key look-up; alternatively, if you have some means to push results to clients (like via WebSockets), use that. If Python is your thing, Celery makes this pattern trivial to implement. This also simplifies so many other things: for example, deployments (and rollbacks!) can proceed much faster, since you don't need to wait obnoxiously long for all connections to drain. Of course you're pushing more complexity up the stack, but in my experience it's a good trade-off.

hluska 9y ago
I've spent enough time building systems and doing ops that this complexity genuinely scares me. Have you used this pattern in highly scalable production apps? What's it like to debug when everything goes to hell?
- rollcat 9y ago
  So first, this is not my idea, but something that people much smarter than me came up with. This is the pattern that AppEngine enforces. Apps that regularly take more than 1s to respond are penalized, and there's a 60s hard deadline on all requests. Things like a task queue or push channels are built-in to the platform, with a high-level API exposed, so you just focus on writing the application / business logic. Celery tries to basically do the same for you, w/o the proprietary Google SDK's. > What's it like to debug when everything goes to hell? From my experience so far... You will find bugs in your application code way more often than in a battle-tested task queuing system, much like you're quite unlikely to find a bug in nginx or Python. Fixing things is MUCH easier than in a request / response model. I know it's not a "web scale" example, but imagine an installation involving camera rigs, some networked hardware, a transcoding server, etc. Dude walks into a photo booth, types in his email/phone number on a tablet, etc. When I see a failure in a component of the pipeline, I can hack together and deploy a patch in seconds / minutes (from alert to running the fix in production) and just tell the system to retry a failed job. Dude gets his silly photo via email/sms within two minutes instead of one. Picture a similar situation in a web application that processes user-uploaded media. You write your batch processing logic to bail early on any sign of error, and when you get the chance to fix some bug that e.g. affected 3% of your users, again - you resubmit the failed jobs and users just see the result with a delay, instead of having to re-upload.
  - hluska 9y ago
    Thanks for the response - I understand it much better now.
jakobegger 9y ago
I'm not sure how that would solve the problem. Not every server is a front end web server. Think of something like video transcoding, or complex database queries. At some point you are going to have a number of servers that do the work, and a load balancer needs to balance the work between them. If one server is busy with a job that will take an hour, there's no point in randomly queueing up lots of work while others are idle.
- rollcat 9y ago
  Read up on how task queues work - they do fill a role of a load balancer, your video transcoding case is a perfect example application. Basically workers can leave/join the pool at any time, poll for more work whenever idle/done, and signal failures so that jobs can be retried later. There's no such thing as an idle worker, unless the queue is empty. No "overloaded" workers either, running two CPU-bound tasks concurrently won't get either of them done sooner. Your monitoring / auto-scaling system can watch the length of the task queue and add/remove workers as needed. (You can also try to combine this with AWS spot instances.)
  - jakobegger 9y ago
    Okay, so that would require totally changing the application architecture, and in the end it would suffer from exactly the same issue: A broken worker that takes tasks from the queue and immediately returns an incorrect result will quickly take over a significant portion of the tasks. That’s exactly the issue the article was talking about.
    - rollcat 9y ago
      > A broken worker that takes tasks from the queue and immediately returns an incorrect result will quickly take over a significant portion of the tasks. Except now you can use a round-robin/"dumb" LB on the frontend, fixing that exact issue (and a whole class of other problems). Yes, a broken worker will quickly drain the queue, except once you find out and remove it, you can (hopefully) resubmit the failed jobs. It gives you a new primitive to work with - HTTP requests are 1:1 to HTTP responses, but a job's result can be updated, and the new result propagated up the chain. (Think: media transcoding / post-processing, rendering pipelines, data analysis, etc.) > that would require totally changing the application architecture True, usual cost/benefit assessment when dealing with technical debt, etc.

hluska 9y ago

I've spent enough time building systems and doing ops that this complexity genuinely scares me. Have you used this pattern in highly scalable production apps? What's it like to debug when everything goes to hell?

rollcat 9y ago
So first, this is not my idea, but something that people much smarter than me came up with. This is the pattern that AppEngine enforces. Apps that regularly take more than 1s to respond are penalized, and there's a 60s hard deadline on all requests. Things like a task queue or push channels are built-in to the platform, with a high-level API exposed, so you just focus on writing the application / business logic. Celery tries to basically do the same for you, w/o the proprietary Google SDK's. > What's it like to debug when everything goes to hell? From my experience so far... You will find bugs in your application code way more often than in a battle-tested task queuing system, much like you're quite unlikely to find a bug in nginx or Python. Fixing things is MUCH easier than in a request / response model. I know it's not a "web scale" example, but imagine an installation involving camera rigs, some networked hardware, a transcoding server, etc. Dude walks into a photo booth, types in his email/phone number on a tablet, etc. When I see a failure in a component of the pipeline, I can hack together and deploy a patch in seconds / minutes (from alert to running the fix in production) and just tell the system to retry a failed job. Dude gets his silly photo via email/sms within two minutes instead of one. Picture a similar situation in a web application that processes user-uploaded media. You write your batch processing logic to bail early on any sign of error, and when you get the chance to fix some bug that e.g. affected 3% of your users, again - you resubmit the failed jobs and users just see the result with a delay, instead of having to re-upload.
- hluska 9y ago
  Thanks for the response - I understand it much better now.

jakobegger 9y ago

I'm not sure how that would solve the problem. Not every server is a front end web server. Think of something like video transcoding, or complex database queries. At some point you are going to have a number of servers that do the work, and a load balancer needs to balance the work between them. If one server is busy with a job that will take an hour, there's no point in randomly queueing up lots of work while others are idle.

rollcat 9y ago
Read up on how task queues work - they do fill a role of a load balancer, your video transcoding case is a perfect example application. Basically workers can leave/join the pool at any time, poll for more work whenever idle/done, and signal failures so that jobs can be retried later. There's no such thing as an idle worker, unless the queue is empty. No "overloaded" workers either, running two CPU-bound tasks concurrently won't get either of them done sooner. Your monitoring / auto-scaling system can watch the length of the task queue and add/remove workers as needed. (You can also try to combine this with AWS spot instances.)
- jakobegger 9y ago
  Okay, so that would require totally changing the application architecture, and in the end it would suffer from exactly the same issue: A broken worker that takes tasks from the queue and immediately returns an incorrect result will quickly take over a significant portion of the tasks. That’s exactly the issue the article was talking about.
  - rollcat 9y ago
    > A broken worker that takes tasks from the queue and immediately returns an incorrect result will quickly take over a significant portion of the tasks. Except now you can use a round-robin/"dumb" LB on the frontend, fixing that exact issue (and a whole class of other problems). Yes, a broken worker will quickly drain the queue, except once you find out and remove it, you can (hopefully) resubmit the failed jobs. It gives you a new primitive to work with - HTTP requests are 1:1 to HTTP responses, but a job's result can be updated, and the new result propagated up the chain. (Think: media transcoding / post-processing, rendering pipelines, data analysis, etc.) > that would require totally changing the application architecture True, usual cost/benefit assessment when dealing with technical debt, etc.