3 ms·

I honestly had never thought hard about reversing the relationship and having workers pull. This point about no longer requiring health checks is a real "woah"

• throwaway894345 4y ago
  Agreed on the “whoa moment”. Not needing health checks seems pretty compelling. I’m curious if there are any off-the-shelf pull-based load balancer that don’t require the full HTTP request copied onto the queue before handing off to a worker?
  • adql 4y ago
    You need health checks. If not for LB then for your own monitoring.
    • preseinger 4y ago
      you don't a server can return 200 OK to every health check query and 5xx to every production request, is that server up or down? hopefully clear it is down similarly, if it returns 5xx to every health check query but 200 to every prod request, is it up or down? hopefully clear it is up if you want to monitor the status of a server (or application) for internal purposes, that's fine, but that's (at best) supplementary signal for the decisions made by a load balancer, not something you can make load balancing decisions on in isolation
• ramses0 4y ago
  https://beanstalkd.github.io https://beanstalkd.github.io
• 0x457 4y ago
  You still need health checks, though? Otherwise, how do you tell the difference between: "no traffic + server alive" vs "some traffic + server is dead". Yeah, you can monitor throughput on a load balancer, but if I ever again wake up from an alert about not traffic being served - I will throw hands.
  • samwho 4y ago
    Yeah, they’d have to exist but it would be for quite a different purpose. I wonder if that would mean we could implement them in different ways, e.g have a health check service that everything pings and if a ping isn’t received for N minutes, assume it’s dead and trigger some replacement routine or alert.
    • throwway120385 4y ago
      This begins to look a lot more like a software watchdog at that point, and you can even have each service provide a count of outstanding/processed requests per tick and if a server never gets outstanding or processed requests you could have the watchdog kill it off.
  • paulddraper 4y ago
    Your server scaling system will need healthchecks. But: 1. It can be more tolerant of failures, since an unresponsive server doesn't mean failed requests. 2. Hooking up your server scaling system to HAProxy, etc. isn't the easiest thing in the world anyway.
  • preseinger 4y ago
    server health is necessarily a function of the actual production traffic it receives, determined at the application layer, as observed by a specific observer it can't be known by the server itself, as (among many other reasons) the server can't know about network issues between itself and any upstream caller it can't be determined by out-of-band health check queries, because those queries don't represent actual traffic, the simplifying assumption that they _do_ introduces many common failure modes that any seasoned engineer can speak at length about health checks can be a nice additional signal on top of monitoring actual prod traffic, but they can't be used by themselves, they just don't capture enough relevant information