6 ms·

That model works if you have a large number of short lived requests. On the other hand, if you have a small number of long running requests that you want to di

That model works if you have a large number of short lived requests. On the other hand, if you have a small number of long running requests that you want to distribute over a number of servers, then your load balancer needs to track which servers are busy, or you'll spend a lot of time waiting for the randomly assigned server to finish, while others are idle.

rollcat 9y ago
Then you design your system to always respond as quickly as possible. HTTP even has a status code for that: 202 Accepted. You put the job in a queue and return some ID or cookie to the client, possibly also an ETA. Client can then poll for results (with bounded exponential back-off), again polling is just a quick key look-up; alternatively, if you have some means to push results to clients (like via WebSockets), use that. If Python is your thing, Celery makes this pattern trivial to implement. This also simplifies so many other things: for example, deployments (and rollbacks!) can proceed much faster, since you don't need to wait obnoxiously long for all connections to drain. Of course you're pushing more complexity up the stack, but in my experience it's a good trade-off.
- hluska 9y ago
  I've spent enough time building systems and doing ops that this complexity genuinely scares me. Have you used this pattern in highly scalable production apps? What's it like to debug when everything goes to hell?
  - rollcat 9y ago
    So first, this is not my idea, but something that people much smarter than me came up with. This is the pattern that AppEngine enforces. Apps that regularly take more than 1s to respond are penalized, and there's a 60s hard deadline on all requests. Things like a task queue or push channels are built-in to the platform, with a high-level API exposed, so you just focus on writing the application / business logic. Celery tries to basically do the same for you, w/o the proprietary Google SDK's. > What's it like to debug when everything goes to hell? From my experience so far... You will find bugs in your application code way more often than in a battle-tested task queuing system, much like you're quite unlikely to find a bug in nginx or Python. Fixing things is MUCH easier than in a request / response model. I know it's not a "web scale" example, but imagine an installation involving camera rigs, some networked hardware, a transcoding server, etc. Dude walks into a photo booth, types in his email/phone number on a tablet, etc. When I see a failure in a component of the pipeline, I can hack together and deploy a patch in seconds / minutes (from alert to running the fix in production) and just tell the system to retry a failed job. Dude gets his silly photo via email/sms within two minutes instead of one. Picture a similar situation in a web application that processes user-uploaded media. You write your batch processing logic to bail early on any sign of error, and when you get the chance to fix some bug that e.g. affected 3% of your users, again - you resubmit the failed jobs and users just see the result with a delay, instead of having to re-upload.
    - hluska 9y ago
      Thanks for the response - I understand it much better now.
- jakobegger 9y ago
  I'm not sure how that would solve the problem. Not every server is a front end web server. Think of something like video transcoding, or complex database queries. At some point you are going to have a number of servers that do the work, and a load balancer needs to balance the work between them. If one server is busy with a job that will take an hour, there's no point in randomly queueing up lots of work while others are idle.
  - rollcat 9y ago
    Read up on how task queues work - they do fill a role of a load balancer, your video transcoding case is a perfect example application. Basically workers can leave/join the pool at any time, poll for more work whenever idle/done, and signal failures so that jobs can be retried later. There's no such thing as an idle worker, unless the queue is empty. No "overloaded" workers either, running two CPU-bound tasks concurrently won't get either of them done sooner. Your monitoring / auto-scaling system can watch the length of the task queue and add/remove workers as needed. (You can also try to combine this with AWS spot instances.)
    - jakobegger 9y ago
      Okay, so that would require totally changing the application architecture, and in the end it would suffer from exactly the same issue: A broken worker that takes tasks from the queue and immediately returns an incorrect result will quickly take over a significant portion of the tasks. That’s exactly the issue the article was talking about.
      - rollcat 9y ago
        > A broken worker that takes tasks from the queue and immediately returns an incorrect result will quickly take over a significant portion of the tasks. Except now you can use a round-robin/"dumb" LB on the frontend, fixing that exact issue (and a whole class of other problems). Yes, a broken worker will quickly drain the queue, except once you find out and remove it, you can (hopefully) resubmit the failed jobs. It gives you a new primitive to work with - HTTP requests are 1:1 to HTTP responses, but a job's result can be updated, and the new result propagated up the chain. (Think: media transcoding / post-processing, rendering pipelines, data analysis, etc.) > that would require totally changing the application architecture True, usual cost/benefit assessment when dealing with technical debt, etc.
Johnny555 9y ago
Also if you have moderately long lived requests, if one server starts to get overloaded and it starts to slow down, new requests will still be sent to that server, which will make it more overloaded and slower, and jobs start to build up on that server, making it slower and slower.
grosjona 9y ago
If each server can handle thousands of operations at a time, the duration of each operation shouldn't matter because long operations should also be evenly spread across servers... Also due to probability. When you're dealing with big numbers across relatively few servers, randomness averages out everything.
- kd5bjo 9y ago
  It depends on how long your long operations are. Back when I was working on Justin.tv, we had to write a custom load balancer because we had to maintain a sizable number of TCP connections that lasted for hours or days without disconnecting.
  - grosjona 9y ago
    I've worked on a cryptocurrency trading platform that had WebSocket connections which also sometimes lasted for days (including trading bots). Each user-facing server could handle 10k concurrent sockets and the long connections were spread out evenly. When you have a sample of 10k users, they tend to behave consistently with any other 10k sample... Assuming that all machines are the same size and the code is the same on all machines.

rollcat 9y ago

Then you design your system to always respond as quickly as possible. HTTP even has a status code for that: 202 Accepted. You put the job in a queue and return some ID or cookie to the client, possibly also an ETA. Client can then poll for results (with bounded exponential back-off), again polling is just a quick key look-up; alternatively, if you have some means to push results to clients (like via WebSockets), use that. If Python is your thing, Celery makes this pattern trivial to implement. This also simplifies so many other things: for example, deployments (and rollbacks!) can proceed much faster, since you don't need to wait obnoxiously long for all connections to drain. Of course you're pushing more complexity up the stack, but in my experience it's a good trade-off.

hluska 9y ago
I've spent enough time building systems and doing ops that this complexity genuinely scares me. Have you used this pattern in highly scalable production apps? What's it like to debug when everything goes to hell?
- rollcat 9y ago
  So first, this is not my idea, but something that people much smarter than me came up with. This is the pattern that AppEngine enforces. Apps that regularly take more than 1s to respond are penalized, and there's a 60s hard deadline on all requests. Things like a task queue or push channels are built-in to the platform, with a high-level API exposed, so you just focus on writing the application / business logic. Celery tries to basically do the same for you, w/o the proprietary Google SDK's. > What's it like to debug when everything goes to hell? From my experience so far... You will find bugs in your application code way more often than in a battle-tested task queuing system, much like you're quite unlikely to find a bug in nginx or Python. Fixing things is MUCH easier than in a request / response model. I know it's not a "web scale" example, but imagine an installation involving camera rigs, some networked hardware, a transcoding server, etc. Dude walks into a photo booth, types in his email/phone number on a tablet, etc. When I see a failure in a component of the pipeline, I can hack together and deploy a patch in seconds / minutes (from alert to running the fix in production) and just tell the system to retry a failed job. Dude gets his silly photo via email/sms within two minutes instead of one. Picture a similar situation in a web application that processes user-uploaded media. You write your batch processing logic to bail early on any sign of error, and when you get the chance to fix some bug that e.g. affected 3% of your users, again - you resubmit the failed jobs and users just see the result with a delay, instead of having to re-upload.
  - hluska 9y ago
    Thanks for the response - I understand it much better now.
jakobegger 9y ago
I'm not sure how that would solve the problem. Not every server is a front end web server. Think of something like video transcoding, or complex database queries. At some point you are going to have a number of servers that do the work, and a load balancer needs to balance the work between them. If one server is busy with a job that will take an hour, there's no point in randomly queueing up lots of work while others are idle.
- rollcat 9y ago
  Read up on how task queues work - they do fill a role of a load balancer, your video transcoding case is a perfect example application. Basically workers can leave/join the pool at any time, poll for more work whenever idle/done, and signal failures so that jobs can be retried later. There's no such thing as an idle worker, unless the queue is empty. No "overloaded" workers either, running two CPU-bound tasks concurrently won't get either of them done sooner. Your monitoring / auto-scaling system can watch the length of the task queue and add/remove workers as needed. (You can also try to combine this with AWS spot instances.)
  - jakobegger 9y ago
    Okay, so that would require totally changing the application architecture, and in the end it would suffer from exactly the same issue: A broken worker that takes tasks from the queue and immediately returns an incorrect result will quickly take over a significant portion of the tasks. That’s exactly the issue the article was talking about.
    - rollcat 9y ago
      > A broken worker that takes tasks from the queue and immediately returns an incorrect result will quickly take over a significant portion of the tasks. Except now you can use a round-robin/"dumb" LB on the frontend, fixing that exact issue (and a whole class of other problems). Yes, a broken worker will quickly drain the queue, except once you find out and remove it, you can (hopefully) resubmit the failed jobs. It gives you a new primitive to work with - HTTP requests are 1:1 to HTTP responses, but a job's result can be updated, and the new result propagated up the chain. (Think: media transcoding / post-processing, rendering pipelines, data analysis, etc.) > that would require totally changing the application architecture True, usual cost/benefit assessment when dealing with technical debt, etc.

Johnny555 9y ago

Also if you have moderately long lived requests, if one server starts to get overloaded and it starts to slow down, new requests will still be sent to that server, which will make it more overloaded and slower, and jobs start to build up on that server, making it slower and slower.

grosjona 9y ago

If each server can handle thousands of operations at a time, the duration of each operation shouldn't matter because long operations should also be evenly spread across servers... Also due to probability. When you're dealing with big numbers across relatively few servers, randomness averages out everything.

kd5bjo 9y ago
It depends on how long your long operations are. Back when I was working on Justin.tv, we had to write a custom load balancer because we had to maintain a sizable number of TCP connections that lasted for hours or days without disconnecting.
- grosjona 9y ago
  I've worked on a cryptocurrency trading platform that had WebSocket connections which also sometimes lasted for days (including trading bots). Each user-facing server could handle 10k concurrent sockets and the long connections were spread out evenly. When you have a sample of 10k users, they tend to behave consistently with any other 10k sample... Assuming that all machines are the same size and the code is the same on all machines.