4 ms·

I see that on my continuous integration system. We use Teamcity with ~50 agents for build tasks that take 20-30 minutes. Each agent can only be running a single

I see that on my continuous integration system. We use Teamcity with ~50 agents for build tasks that take 20-30 minutes. Each agent can only be running a single task. During the day, all agents are busy and the queue fills up with 30 or so pending tasks. If one agent gets into a bad state where, say, it fails to checkout from source control and fails within the first 20 seconds of a build, then it will very quickly chew through the entire queue of pending tasks, failing them all. You'd think the more agents you have the better insulated you are from the failure of a single one, but this particular failure mode actually becomes more common the more agents you add!!!

ithkuil 9y ago
> this particular failure mode actually becomes more common the more agents you add I'm surprised. I would expect the length of the pending job queue to also have an impact on this failure _mode_ (as opposed to the failure _cause_), and the queue length is inversely proportional to the number of agents, flooring to 0 when you have to many agents than necessary to deal with peak demand. If the bad agents (I'm assuming that the fault to check out from SCM lies in the agents themselves, not in the overloaded SCM server due to too many agents polling, otherwise this is not an example relevant to the thread) are e.g. 1 every 50 agents and the number of agents is much larger than the peak load (e.g. a million agents), the probability of a job of failing approaches 1/50, because the good agents blocked on doing work are just a small fraction and thus they don't significantly skew the probability of a new job to get scheduled to a bad agent (which are more likely to be ready). If on the other hand you have less agents than peak load, the queue will contain some jobs, and bad agents will chew through the whole queue when they get the chance, as you described. And they always get the chance, since they are almost always ready (if they fail very quickly), so that the pool of ready agents will almost always contain only bad agents. If you add more agents you have to deal with these bad agents "floating on the top of the pool", and the new ones you add (at the rate of e.g. 1 every 50). So, if you have 5 bad agents and you add another 5 you still have a pretty high chance that new jobs will land on the bad ones. But you can add more and the probability of success just gets better.
- cecilpl2 9y ago
  That is a good point and I like the visual metaphor of bad agents "floating on top of the pool". I was considering the fairly narrow problem domain between say 20 and 40 agents, and the queue fills up to perhaps 50 pending tasks mid-day. You are correct in the case that once you add enough agents to keep the queue empty, then additional agents reduce the fail chance. Sadly, though, the CI system I use (Teamcity) prioritizes agents that have a history of completing that particular task the fastest. That's great if your agents are all different specs, but fails spectacularly in this case, where a bad agent will be selected above a good one.

ithkuil 9y ago

> this particular failure mode actually becomes more common the more agents you add I'm surprised. I would expect the length of the pending job queue to also have an impact on this failure _mode_ (as opposed to the failure _cause_), and the queue length is inversely proportional to the number of agents, flooring to 0 when you have to many agents than necessary to deal with peak demand. If the bad agents (I'm assuming that the fault to check out from SCM lies in the agents themselves, not in the overloaded SCM server due to too many agents polling, otherwise this is not an example relevant to the thread) are e.g. 1 every 50 agents and the number of agents is much larger than the peak load (e.g. a million agents), the probability of a job of failing approaches 1/50, because the good agents blocked on doing work are just a small fraction and thus they don't significantly skew the probability of a new job to get scheduled to a bad agent (which are more likely to be ready). If on the other hand you have less agents than peak load, the queue will contain some jobs, and bad agents will chew through the whole queue when they get the chance, as you described. And they always get the chance, since they are almost always ready (if they fail very quickly), so that the pool of ready agents will almost always contain only bad agents. If you add more agents you have to deal with these bad agents "floating on the top of the pool", and the new ones you add (at the rate of e.g. 1 every 50). So, if you have 5 bad agents and you add another 5 you still have a pretty high chance that new jobs will land on the bad ones. But you can add more and the probability of success just gets better.

cecilpl2 9y ago
That is a good point and I like the visual metaphor of bad agents "floating on top of the pool". I was considering the fairly narrow problem domain between say 20 and 40 agents, and the queue fills up to perhaps 50 pending tasks mid-day. You are correct in the case that once you add enough agents to keep the queue empty, then additional agents reduce the fail chance. Sadly, though, the CI system I use (Teamcity) prioritizes agents that have a history of completing that particular task the fastest. That's great if your agents are all different specs, but fails spectacularly in this case, where a bad agent will be selected above a good one.