5 ms·

That's why I like random load balancing. If each machine is powerful enough and can handle a few thousand users then the distribution averages out. Smart load

That's why I like random load balancing. If each machine is powerful enough and can handle a few thousand users then the distribution averages out. Smart load balancers are only really necessary if you have inefficient servers that can't handle more than 100 connections per second and they're difficult to get right. If you toss a coin 10 times, you're much more likely to get >=80% heads than if you were to toss that coin 1000 times.

foota 9y ago
This is true, but if you have a tighter tolerance on how much load each machine needs to be able to handle then you may be able to waste less resources.
- deleted 9y ago
  [deleted]
matt4077 9y ago
Random balancing only works with homogenous servers.
- ec109685 9y ago
  You can weight them.
- randerson 9y ago
  Isn't it almost always advisable to have homogenous servers in a web farm? What are some example cases there that doesn't make sense?
  - sethammons 9y ago
    I'm on mobile, so won't look for the source. I heard Netflix benchmarks each new instance bootstrapped in AWS because they have found a dramatically high level of variability in performance from what should be home get out hardware. They toss out low performers and rebootstrap the instance.
  - zepolen 9y ago
    Containerization and spot instances.
jakobegger 9y ago
That model works if you have a large number of short lived requests. On the other hand, if you have a small number of long running requests that you want to distribute over a number of servers, then your load balancer needs to track which servers are busy, or you'll spend a lot of time waiting for the randomly assigned server to finish, while others are idle.
- rollcat 9y ago
  Then you design your system to always respond as quickly as possible. HTTP even has a status code for that: 202 Accepted. You put the job in a queue and return some ID or cookie to the client, possibly also an ETA. Client can then poll for results (with bounded exponential back-off), again polling is just a quick key look-up; alternatively, if you have some means to push results to clients (like via WebSockets), use that. If Python is your thing, Celery makes this pattern trivial to implement. This also simplifies so many other things: for example, deployments (and rollbacks!) can proceed much faster, since you don't need to wait obnoxiously long for all connections to drain. Of course you're pushing more complexity up the stack, but in my experience it's a good trade-off.
  - hluska 9y ago
    I've spent enough time building systems and doing ops that this complexity genuinely scares me. Have you used this pattern in highly scalable production apps? What's it like to debug when everything goes to hell?
    - rollcat 9y ago
      So first, this is not my idea, but something that people much smarter than me came up with. This is the pattern that AppEngine enforces. Apps that regularly take more than 1s to respond are penalized, and there's a 60s hard deadline on all requests. Things like a task queue or push channels are built-in to the platform, with a high-level API exposed, so you just focus on writing the application / business logic. Celery tries to basically do the same for you, w/o the proprietary Google SDK's. > What's it like to debug when everything goes to hell? From my experience so far... You will find bugs in your application code way more often than in a battle-tested task queuing system, much like you're quite unlikely to find a bug in nginx or Python. Fixing things is MUCH easier than in a request / response model. I know it's not a "web scale" example, but imagine an installation involving camera rigs, some networked hardware, a transcoding server, etc. Dude walks into a photo booth, types in his email/phone number on a tablet, etc. When I see a failure in a component of the pipeline, I can hack together and deploy a patch in seconds / minutes (from alert to running the fix in production) and just tell the system to retry a failed job. Dude gets his silly photo via email/sms within two minutes instead of one. Picture a similar situation in a web application that processes user-uploaded media. You write your batch processing logic to bail early on any sign of error, and when you get the chance to fix some bug that e.g. affected 3% of your users, again - you resubmit the failed jobs and users just see the result with a delay, instead of having to re-upload.
Groxx 9y ago
Is this better than round-robin somehow?

foota 9y ago

This is true, but if you have a tighter tolerance on how much load each machine needs to be able to handle then you may be able to waste less resources.

deleted 9y ago
[deleted]

matt4077 9y ago

Random balancing only works with homogenous servers.

ec109685 9y ago
You can weight them.
randerson 9y ago
Isn't it almost always advisable to have homogenous servers in a web farm? What are some example cases there that doesn't make sense?
- sethammons 9y ago
  I'm on mobile, so won't look for the source. I heard Netflix benchmarks each new instance bootstrapped in AWS because they have found a dramatically high level of variability in performance from what should be home get out hardware. They toss out low performers and rebootstrap the instance.
- zepolen 9y ago
  Containerization and spot instances.

jakobegger 9y ago

That model works if you have a large number of short lived requests. On the other hand, if you have a small number of long running requests that you want to distribute over a number of servers, then your load balancer needs to track which servers are busy, or you'll spend a lot of time waiting for the randomly assigned server to finish, while others are idle.

rollcat 9y ago
Then you design your system to always respond as quickly as possible. HTTP even has a status code for that: 202 Accepted. You put the job in a queue and return some ID or cookie to the client, possibly also an ETA. Client can then poll for results (with bounded exponential back-off), again polling is just a quick key look-up; alternatively, if you have some means to push results to clients (like via WebSockets), use that. If Python is your thing, Celery makes this pattern trivial to implement. This also simplifies so many other things: for example, deployments (and rollbacks!) can proceed much faster, since you don't need to wait obnoxiously long for all connections to drain. Of course you're pushing more complexity up the stack, but in my experience it's a good trade-off.
- hluska 9y ago
  I've spent enough time building systems and doing ops that this complexity genuinely scares me. Have you used this pattern in highly scalable production apps? What's it like to debug when everything goes to hell?
  - rollcat 9y ago
    So first, this is not my idea, but something that people much smarter than me came up with. This is the pattern that AppEngine enforces. Apps that regularly take more than 1s to respond are penalized, and there's a 60s hard deadline on all requests. Things like a task queue or push channels are built-in to the platform, with a high-level API exposed, so you just focus on writing the application / business logic. Celery tries to basically do the same for you, w/o the proprietary Google SDK's. > What's it like to debug when everything goes to hell? From my experience so far... You will find bugs in your application code way more often than in a battle-tested task queuing system, much like you're quite unlikely to find a bug in nginx or Python. Fixing things is MUCH easier than in a request / response model. I know it's not a "web scale" example, but imagine an installation involving camera rigs, some networked hardware, a transcoding server, etc. Dude walks into a photo booth, types in his email/phone number on a tablet, etc. When I see a failure in a component of the pipeline, I can hack together and deploy a patch in seconds / minutes (from alert to running the fix in production) and just tell the system to retry a failed job. Dude gets his silly photo via email/sms within two minutes instead of one. Picture a similar situation in a web application that processes user-uploaded media. You write your batch processing logic to bail early on any sign of error, and when you get the chance to fix some bug that e.g. affected 3% of your users, again - you resubmit the failed jobs and users just see the result with a delay, instead of having to re-upload.

Groxx 9y ago

Is this better than round-robin somehow?