9 ms·
I’ve done something like this in large service ecosystems. It has two very unfortunate properties: - if the actual performance deviates from the predicted (sc
by shermantanktop 2y ago
I’ve done something like this in large service ecosystems. It has two very unfortunate properties:
- if the actual performance deviates from the predicted (scored) performance, the system easily enters a degenerate bottlenecked state.
- and if that happens, the many internal queues make diagnosis, root causing, and confidence in a fix all exponentially worse.
Now you might assert that this will be applied in situations where scores are accurate and brown failures do not occur. Those aren’t the situations I deal with.
- Joker_vD 2y agoThe algorithm in TFA, as I understand it, requires the scheduler to constantly monitor the queue lengths and reschedule the workers appropriately, as opposed to manually estimating the required number of workers for each stage and then never changing it.
- shermantanktop 2y agoThis is what happens with PID controllers. It is by definition a trailing view, and a single catastrophically bad request can cause catastrophic deviation of the whole system before the queue monitoring can "catch up."
- hinkley 2y agoIt’s possible to write a scheduler that looks like work stealing and can route unordered tasks around a poison pill. But we’ve generally found it works better for the worker to pull when its queue is empty, and of course if you have partially ordered events nobody can process (start or reject) those ordered tasks until the borked one finishes or times out. So the situation you describe can absolutely happen, but is not a given. It’ll depend on other decisions.