10 ms·

Lessons we learned when debugging a scaling problem

167 points by sahin-boydas 7y ago

afraca 7y ago
Bravo to the people that can navigate such a complex (not an insult) architecture and pull useful information from it! I expect it to be quite tricky to get in between the production environment(s) and capture packets or change logging etc. I did wonder though: the authors seemed a bit unsure about the relation between MaxStartups and resource usage, wouldn't it be wise to just send an email to the openSSH mailing list or something to not be surprised by possible future problems?
- EricE 7y ago
  Might be valuable if it reacted the same way in all environments (not always the case). And if someone answered faster (correctly - without sending them down an unrelated rabbit hole too!) then they could have found the answer out by just experimenting. Don't get me wrong - I'm a huge fan of community support, but sometimes it's faster to just try things and see how the systems reacts. Not to say you can't do both in parallel; and they might have - you aren't going to detail everything you did in a writeup like this. As others said, I really enjoy reading write-ups like these. Who needs fiction when you have real world mysteries to solve?
adamc 7y ago
I thought this was a pretty good piece, and points to the problems caused by running services at regular intervals.
amenod 7y ago
> It would make everyone more comfortable if OpenSSH let us see the how close we were to hitting MaxStartups at any point, rather than having to go in blind and only find out we were close when the limit is breached and connections are dropped. If they are not afraid to dive into source, it should be fairly easy to add some logging when the value reaches 75% for example. Of course one needs to be careful not to break anything else (issuing thousands of warning logs at the same time that the system is at peak load is probably not a good idea), but it shouldn't be that difficult. Also, I assume this would be something that OpenSSH would be happy to get a PR (or MR, coming from GitLab? ;) ) for. Nice writeup. What I miss is how they added monitoring for these kinds of errors, so it doesn't escalate to 1.5% of all connections again. But other than that, good job!
- certifiable 7y ago
  Oh, that's a fun story. We use https://github.com/google/mtail https://github.com/google/mtail to turn log data into metrics exported for prometheus. We were already doing this for our haproxy logs; we just added a stanza to capture a secondary indicator that the problem is occurring (termination_state 'SD', and 0 bytes_read), and export that as a metric. You can see the relevant MR at https://gitlab.com/gitlab-cookbooks/gitlab-mtail/merge_requests/47 https://gitlab.com/gitlab-cookbooks/gitlab-mtail/merge_reque... (and https://gitlab.com/gitlab-cookbooks/gitlab-mtail/merge_requests/48 https://gitlab.com/gitlab-cookbooks/gitlab-mtail/merge_reque... to fix some bugs) Then we just hooked it up through the usual sort of prometheus alerting rules. We made it really twitchy, because this combination of logs Should Not Happen if everything is working right, and we want to alert as soon as it starts occurring, so we can bump the limits again (or re-evaluate in general)
funkjunky 7y ago
It's blog posts like this that make me realize I have no idea what I'm doing. Great work on troubleshooting a complex and interesting problem.
wyldfire 7y ago
> Lesson #3: It is polite to log interesting information at default levels and deliberately dropping a connection for any reason is definitely interesting to system administrators. Agreed. But I suppose part of the reason this feature is present is because so many ssh servers are the targets of attacks, often brute force. So logging these might be problematic. Speaking of "Rate limiting" -- openssh logging something like "dropped N connections exceeding MaxStartups" once per minute (if N > 0) [at WARN level] seems like a sane compromise.
tilolebo 7y ago
Isn't it possible to just set the haproxy maxconn to a slightly lower value than what the backend can deal with, and then let the reverse proxy retry once with another backend? Or even queue it for some hundreds of milliseconds before that? This way you avoid overloading backends. Also, haproxy provides tons of real-time metrics, included the highwatermark for concurrent and queued connections.
- tilolebo 7y ago
  Aaand this is what happens when one comments before reading all the article...
- ignoramous 7y ago
  What you suggest is what was followed by a large-scale distributed system I worked on. Your suggestion is also in-line with Google' SRE book section on Handling Overload: https://landing.google.com/sre/sre-book/chapters/handling-overload/ https://landing.google.com/sre/sre-book/chapters/handling-ov... > Mandate that batch client jobs use a separate set of batch proxy backend tasks that do nothing but forward requests to the underlying backends and hand their responses back to the clients in a controlled way. Therefore, instead of "batch client → backend," you have "batch client → batch proxy → backend." In this case, when the very large job starts, only the batch proxy job suffers, shielding the actual backends (and higher-priority clients). Effectively, the batch proxy acts like a fuse. Another advantage of using the proxy is that it typically reduces the number of connections against the backend, which can improve the load balancing against the backend. Other chapters on Load Balancing, Addressing Cascading Failures are related too.
cfors 7y ago
I love these sorts of write ups. I wish there was a way to encourage internal transparency like this, as opposed to my least favorite way to be greeted when I let someone know about an issue with their product: "Can you try again?" Nothing as infuriating as that, it could be a valuable learning lesson for many people just like this!
- donmcronald 7y ago
  I would love to see a blog post explaining the non-technical interaction for something like this. Is it a single person that understands all the networking plus the whole software stack, from HA-Proxy to the SSH hosts, or is it a team of people working together on the problem? I feel like everyone involved would have to be more eager to solve the problem than worrying about assigning blame. Does that work because it's all within the same company or does GitLab make extra effort to dis-incentivize blame shifting and other unproductive behavior? In my life (all small scale - local vendors) I call people and the only answer I ever get is roughly "our stuff if perfect. It's you."
  - certifiable 7y ago
    We definitely take deliberate effort to dis-incentivize blame; it's specifically called out in our values at https://about.gitlab.com/handbook/values/#blameless-problem-solving https://about.gitlab.com/handbook/values/#blameless-problem-... but speaking as an employee, I can confirm that this is my daily experience. Of course for this specific issue there really wasn't anything or anyone to blame: no-one really did anything wrong at any point. Systems simply get complicated at certain scales and circumstances, and we have to engineer our way around the issues.
- vmurthy 7y ago
  Back when I was studying for a cybersecurity course, I had the pleasure of reading a few posts on the Cloudflare blog. There are about 88 pages(!) of blogs at blog.cloudflare.com A few recent ones might pique your curiosity https://blog.cloudflare.com/on-the-recent-http-2-dos-attacks/ https://blog.cloudflare.com/on-the-recent-http-2-dos-attacks... https://blog.cloudflare.com/a-gentle-introduction-to-linux-kernel-fuzzing/ https://blog.cloudflare.com/a-gentle-introduction-to-linux-k... https://blog.cloudflare.com/the-deep-dive-into-how-verizon-and-a-bgp-optimizer-knocked-large-parts-of-the-internet-offline-monday/ https://blog.cloudflare.com/the-deep-dive-into-how-verizon-a...
mattip 7y ago
They should have led with the bottom line - add a `sleep 7` to any cron job that is going out to the network and set it to start at the top of the hour plus 3 minutes
ignoramous 7y ago
Scheduled tasks are a great way to brown-out your downstream dependencies. In one instance, MDAM RAID checks caused P99 latency spikes first Sunday of every month [0] (default setting). It caused a lot of pain to our customers until the check was IO throttled, which meant spikes weren't as high, but lasted for a longer time. Scheduled tasks are a great way to brown-out yourself. In another case, the client process hadn't set a socket timeout on a blocking tcp connection [1] (default setting), and so it'd run out of worker threads blocked on recv routinely when the server (fronted by reverse-proxy) started rejecting incoming due to overload. Only a restart of the process would recover the client. Scheduled tasks are a great way to prove HAProxy will scale way better than your backend. Thanks u/wtarreau Speaking of HAProxy: It fronted a thread-based server processing all sorts of heavy and light queries with segregated thread-pools for various workloads. During peak, proxy would accept connections and queue work faster than the worker threads could handle, and on occasion, the work-queue would grow so big that it not only contained retries of same work scheduled by the desperate clients but a laundry list of work that wasn't valid anymore (processed by some other back-end in the retry path, or simply expired as time-to-service exceeded SLA). Yet, there the back-end was, in a quagmire, chugging through never-ending work, in constant overload when ironically the client wasn't even waiting on the other end and had long closed the connection. The health-checks were passing because, well, that was on a separate thread-pool, with a different work-queue. Smiles all around. Least conns and the event horizon. Tread carefully. Least conns load balancing bit us hard on multiple occasions, and is now banned for similar reasons outlined here: https://rachelbythebay.com/w/2018/04/21/lb/ https://rachelbythebay.com/w/2018/04/21/lb/ [0] https://serverfault.com/questions/199096/linux-software-raid-runs-checkarray-on-the-first-sunday-of-the-month-why https://serverfault.com/questions/199096/linux-software-raid... [1] https://stackoverflow.com/questions/667640/how-to-tell-if-a-connection-is-dead-in-python https://stackoverflow.com/questions/667640/how-to-tell-if-a-...
- qw 7y ago
  I guess it would be nice to have a "fuzzy" cron job setting, so you could specify "run it at a random time during the day/hour/minute"
  - 7y ago
darkhorn 7y ago
I remember that few years ago same issue occured to either Github or Bitbucket and they made a blog post about it.
jacques_chester 7y ago
> Lesson #4: When you choose specific non-default settings, leave a comment or link to documentation/issues as to why, future people will thank you. Put it in the commit message. There is no better place to put it. That is the exact point in time at which you have the most information about that exact change. Commit messages are cheap, be liberal with your words and links. It drives me nuts to read commit messages of the form "fix", "fixed 2", "add test", "new var" and on and on. You took an hour to make the change, take a damn minute to save some future colleague a day of frustration.
- RussianCow 7y ago
  I agree with you about the importance of commit messages, but the problem with using them in place of comments is that commit messages become much harder to find as the history becomes longer, as opposed to comments which will always be immediately visible for as long as they are relevant.
  - jacques_chester 7y ago
    This is what git blame is for. Comments are notorious for drifting out of sync with reality.
    - umvi 7y ago
      Git blame only shows you the last person that touched the line, not the person who introduced a breaking change to that line (but it wasn't caught for weeks/months). Comments are just as cheap as commit messages. Why not both?
      - jacques_chester 7y ago
        It shows you the commit that introduced the current version of the line. That's what you want if you want to understand the current version of the line. I feel like I'm missing something here. I hate this feeling of impending dumb.
auslander 7y ago
> we have 27 SSH backends, and 18 HAproxy No autoscaling? This load pattern is a prime candidate for it.
- btilly 7y ago
  I disagree. Autoscaling sounds great, but in practice it notices that there is a problem after you already have once, and by the time you've scaled up, there is no problem any more. Which is the worst of all worlds. For example in this case you'd realize there is a problem at the top of the hour. And 2 minutes later you'd have a bunch of autoscaled instances up, all wondering what the fuss was about. So they scale down again, but then at the top of the next hour the same thing happens again with the same result. The same thing happens with ad servers. An ad buy goes in for 30 seconds. When it hits, you have a firehose. That then shuts off. In fact this problem is so common that I recommend against autoscaling unless you have specific reason to believe that it will work for you.
  - auslander 7y ago
    > For example .. In fact this problem is so common .. So whats the problem here? Assuming there are no misconfigurations anymore, and just spiking loads.
    - btilly 7y ago
      The problem is that people try to use autoscaling to handle a spiking load, and it fails abysmally. In my opinion for those with a spiking load profile, autoscaling should be considered harmful.
  - hinkley 7y ago
    And god help you if that starts your autoscaler to flapping. Too much CPU load, start some servers. Too little, kill some servers. Oops now CPU is too high, bring them back. No, it's too low... It reminds me of the old UI bug where you get an off-by-one error for scrollbar logic and so it appears and disappears in a cycle until you resize. Edit: I think we want something akin to a learning thermostat, but for servers. Figure out that I get a CPU spike at 5:20 on week days and spin some servers up at 5:10. Then spin em down at 8:00 when EST starts to go to bed.
navinsylvester 7y ago
Interesting read. But i bumped into the same problem recently while working on distributed simulation. I was writing an agent for an application which talks to clients over ssh. I started getting the same error when i cranked up machine count to 60. I googled like every layman and zeroed in on MaxStartups. Looked at the manual to cross verify and fixed the issue in 5 minutes. I agree with what ever he did after he found out the issue but it was a clear case of SSH throwing an error. Why not just look up manual or google instead of wasting time on laborious packet inspection.
- abrookewood 7y ago
  Because at their scale, people actually need to understand the issue rather than just applying fixes blindly plucked from the Internet. Don't get me wrong, Google is my go-to as well, but I suspect that they need to dig a little deeper.
  - navinsylvester 7y ago
    Fair enough. But wouldn't doing investigation on the error provide data point to help debug the issue. They are trying to fix a customer issue and time is also of importance here. > just applying fixes blindly plucked from the Internet. I wasn't recommending that.
- toast0 7y ago
  > Why not just look up manual or google instead of wasting time on laborious packet inspection. When you've got a load balancing layer, and virtual machines, and interacting with the real world, there's a reasonable chance that the problem is not a (relatively) simple application configuration issue. If you've looked through enough of these, getting a pcap (especially with an identified customer) and looking through it with Wireshark is relatively quick.
  - navinsylvester 7y ago
    I could have worded it better like "why packet inspection to start with".
vmurthy 7y ago
I can't help but notice that these kind of blogs should be recommended reading for CS/engineering students as well and not just to people who are already working. The way an issue is to be analyzed (this post is a great example of Systems thinking IMO), what happens at scale and why not to blindly Google your way out of problems are valuable skills any engineer should have and which are unfortunately not taught in many schools.

167 points by sahin-boydas 7y ago

Bravo to the people that can navigate such a complex (not an insult) architecture and pull useful information from it! I expect it to be quite tricky to get in between the production environment(s) and capture packets or change logging etc. I did wonder though: the authors seemed a bit unsure about the relation between MaxStartups and resource usage, wouldn't it be wise to just send an email to the openSSH mailing list or something to not be surprised by possible future problems?

EricE 7y ago
Might be valuable if it reacted the same way in all environments (not always the case). And if someone answered faster (correctly - without sending them down an unrelated rabbit hole too!) then they could have found the answer out by just experimenting. Don't get me wrong - I'm a huge fan of community support, but sometimes it's faster to just try things and see how the systems reacts. Not to say you can't do both in parallel; and they might have - you aren't going to detail everything you did in a writeup like this. As others said, I really enjoy reading write-ups like these. Who needs fiction when you have real world mysteries to solve?

adamc 7y ago

I thought this was a pretty good piece, and points to the problems caused by running services at regular intervals.

amenod 7y ago

> It would make everyone more comfortable if OpenSSH let us see the how close we were to hitting MaxStartups at any point, rather than having to go in blind and only find out we were close when the limit is breached and connections are dropped. If they are not afraid to dive into source, it should be fairly easy to add some logging when the value reaches 75% for example. Of course one needs to be careful not to break anything else (issuing thousands of warning logs at the same time that the system is at peak load is probably not a good idea), but it shouldn't be that difficult. Also, I assume this would be something that OpenSSH would be happy to get a PR (or MR, coming from GitLab? ;) ) for. Nice writeup. What I miss is how they added monitoring for these kinds of errors, so it doesn't escalate to 1.5% of all connections again. But other than that, good job!

certifiable 7y ago
Oh, that's a fun story. We use https://github.com/google/mtail https://github.com/google/mtail to turn log data into metrics exported for prometheus. We were already doing this for our haproxy logs; we just added a stanza to capture a secondary indicator that the problem is occurring (termination_state 'SD', and 0 bytes_read), and export that as a metric. You can see the relevant MR at https://gitlab.com/gitlab-cookbooks/gitlab-mtail/merge_requests/47 https://gitlab.com/gitlab-cookbooks/gitlab-mtail/merge_reque... (and https://gitlab.com/gitlab-cookbooks/gitlab-mtail/merge_requests/48 https://gitlab.com/gitlab-cookbooks/gitlab-mtail/merge_reque... to fix some bugs) Then we just hooked it up through the usual sort of prometheus alerting rules. We made it really twitchy, because this combination of logs Should Not Happen if everything is working right, and we want to alert as soon as it starts occurring, so we can bump the limits again (or re-evaluate in general)

funkjunky 7y ago

It's blog posts like this that make me realize I have no idea what I'm doing. Great work on troubleshooting a complex and interesting problem.

wyldfire 7y ago

> Lesson #3: It is polite to log interesting information at default levels and deliberately dropping a connection for any reason is definitely interesting to system administrators. Agreed. But I suppose part of the reason this feature is present is because so many ssh servers are the targets of attacks, often brute force. So logging these might be problematic. Speaking of "Rate limiting" -- openssh logging something like "dropped N connections exceeding MaxStartups" once per minute (if N > 0) [at WARN level] seems like a sane compromise.

tilolebo 7y ago

Isn't it possible to just set the haproxy maxconn to a slightly lower value than what the backend can deal with, and then let the reverse proxy retry once with another backend? Or even queue it for some hundreds of milliseconds before that? This way you avoid overloading backends. Also, haproxy provides tons of real-time metrics, included the highwatermark for concurrent and queued connections.

tilolebo 7y ago
Aaand this is what happens when one comments before reading all the article...
ignoramous 7y ago
What you suggest is what was followed by a large-scale distributed system I worked on. Your suggestion is also in-line with Google' SRE book section on Handling Overload: https://landing.google.com/sre/sre-book/chapters/handling-overload/ https://landing.google.com/sre/sre-book/chapters/handling-ov... > Mandate that batch client jobs use a separate set of batch proxy backend tasks that do nothing but forward requests to the underlying backends and hand their responses back to the clients in a controlled way. Therefore, instead of "batch client → backend," you have "batch client → batch proxy → backend." In this case, when the very large job starts, only the batch proxy job suffers, shielding the actual backends (and higher-priority clients). Effectively, the batch proxy acts like a fuse. Another advantage of using the proxy is that it typically reduces the number of connections against the backend, which can improve the load balancing against the backend. Other chapters on Load Balancing, Addressing Cascading Failures are related too.

cfors 7y ago

I love these sorts of write ups. I wish there was a way to encourage internal transparency like this, as opposed to my least favorite way to be greeted when I let someone know about an issue with their product: "Can you try again?" Nothing as infuriating as that, it could be a valuable learning lesson for many people just like this!

donmcronald 7y ago
I would love to see a blog post explaining the non-technical interaction for something like this. Is it a single person that understands all the networking plus the whole software stack, from HA-Proxy to the SSH hosts, or is it a team of people working together on the problem? I feel like everyone involved would have to be more eager to solve the problem than worrying about assigning blame. Does that work because it's all within the same company or does GitLab make extra effort to dis-incentivize blame shifting and other unproductive behavior? In my life (all small scale - local vendors) I call people and the only answer I ever get is roughly "our stuff if perfect. It's you."
- certifiable 7y ago
  We definitely take deliberate effort to dis-incentivize blame; it's specifically called out in our values at https://about.gitlab.com/handbook/values/#blameless-problem-solving https://about.gitlab.com/handbook/values/#blameless-problem-... but speaking as an employee, I can confirm that this is my daily experience. Of course for this specific issue there really wasn't anything or anyone to blame: no-one really did anything wrong at any point. Systems simply get complicated at certain scales and circumstances, and we have to engineer our way around the issues.
vmurthy 7y ago
Back when I was studying for a cybersecurity course, I had the pleasure of reading a few posts on the Cloudflare blog. There are about 88 pages(!) of blogs at blog.cloudflare.com A few recent ones might pique your curiosity https://blog.cloudflare.com/on-the-recent-http-2-dos-attacks/ https://blog.cloudflare.com/on-the-recent-http-2-dos-attacks... https://blog.cloudflare.com/a-gentle-introduction-to-linux-kernel-fuzzing/ https://blog.cloudflare.com/a-gentle-introduction-to-linux-k... https://blog.cloudflare.com/the-deep-dive-into-how-verizon-and-a-bgp-optimizer-knocked-large-parts-of-the-internet-offline-monday/ https://blog.cloudflare.com/the-deep-dive-into-how-verizon-a...

mattip 7y ago

They should have led with the bottom line - add a `sleep 7` to any cron job that is going out to the network and set it to start at the top of the hour plus 3 minutes

ignoramous 7y ago

Scheduled tasks are a great way to brown-out your downstream dependencies. In one instance, MDAM RAID checks caused P99 latency spikes first Sunday of every month [0] (default setting). It caused a lot of pain to our customers until the check was IO throttled, which meant spikes weren't as high, but lasted for a longer time. Scheduled tasks are a great way to brown-out yourself. In another case, the client process hadn't set a socket timeout on a blocking tcp connection [1] (default setting), and so it'd run out of worker threads blocked on recv routinely when the server (fronted by reverse-proxy) started rejecting incoming due to overload. Only a restart of the process would recover the client. Scheduled tasks are a great way to prove HAProxy will scale way better than your backend. Thanks u/wtarreau Speaking of HAProxy: It fronted a thread-based server processing all sorts of heavy and light queries with segregated thread-pools for various workloads. During peak, proxy would accept connections and queue work faster than the worker threads could handle, and on occasion, the work-queue would grow so big that it not only contained retries of same work scheduled by the desperate clients but a laundry list of work that wasn't valid anymore (processed by some other back-end in the retry path, or simply expired as time-to-service exceeded SLA). Yet, there the back-end was, in a quagmire, chugging through never-ending work, in constant overload when ironically the client wasn't even waiting on the other end and had long closed the connection. The health-checks were passing because, well, that was on a separate thread-pool, with a different work-queue. Smiles all around. Least conns and the event horizon. Tread carefully. Least conns load balancing bit us hard on multiple occasions, and is now banned for similar reasons outlined here: https://rachelbythebay.com/w/2018/04/21/lb/ https://rachelbythebay.com/w/2018/04/21/lb/ [0] https://serverfault.com/questions/199096/linux-software-raid-runs-checkarray-on-the-first-sunday-of-the-month-why https://serverfault.com/questions/199096/linux-software-raid... [1] https://stackoverflow.com/questions/667640/how-to-tell-if-a-connection-is-dead-in-python https://stackoverflow.com/questions/667640/how-to-tell-if-a-...

qw 7y ago
I guess it would be nice to have a "fuzzy" cron job setting, so you could specify "run it at a random time during the day/hour/minute"
- 7y ago

darkhorn 7y ago

I remember that few years ago same issue occured to either Github or Bitbucket and they made a blog post about it.

jacques_chester 7y ago

> Lesson #4: When you choose specific non-default settings, leave a comment or link to documentation/issues as to why, future people will thank you. Put it in the commit message. There is no better place to put it. That is the exact point in time at which you have the most information about that exact change. Commit messages are cheap, be liberal with your words and links. It drives me nuts to read commit messages of the form "fix", "fixed 2", "add test", "new var" and on and on. You took an hour to make the change, take a damn minute to save some future colleague a day of frustration.

RussianCow 7y ago
I agree with you about the importance of commit messages, but the problem with using them in place of comments is that commit messages become much harder to find as the history becomes longer, as opposed to comments which will always be immediately visible for as long as they are relevant.
- jacques_chester 7y ago
  This is what git blame is for. Comments are notorious for drifting out of sync with reality.
  - umvi 7y ago
    Git blame only shows you the last person that touched the line, not the person who introduced a breaking change to that line (but it wasn't caught for weeks/months). Comments are just as cheap as commit messages. Why not both?
    - jacques_chester 7y ago
      It shows you the commit that introduced the current version of the line. That's what you want if you want to understand the current version of the line. I feel like I'm missing something here. I hate this feeling of impending dumb.

auslander 7y ago

> we have 27 SSH backends, and 18 HAproxy No autoscaling? This load pattern is a prime candidate for it.

btilly 7y ago
I disagree. Autoscaling sounds great, but in practice it notices that there is a problem after you already have once, and by the time you've scaled up, there is no problem any more. Which is the worst of all worlds. For example in this case you'd realize there is a problem at the top of the hour. And 2 minutes later you'd have a bunch of autoscaled instances up, all wondering what the fuss was about. So they scale down again, but then at the top of the next hour the same thing happens again with the same result. The same thing happens with ad servers. An ad buy goes in for 30 seconds. When it hits, you have a firehose. That then shuts off. In fact this problem is so common that I recommend against autoscaling unless you have specific reason to believe that it will work for you.
- auslander 7y ago
  > For example .. In fact this problem is so common .. So whats the problem here? Assuming there are no misconfigurations anymore, and just spiking loads.
  - btilly 7y ago
    The problem is that people try to use autoscaling to handle a spiking load, and it fails abysmally. In my opinion for those with a spiking load profile, autoscaling should be considered harmful.
- hinkley 7y ago
  And god help you if that starts your autoscaler to flapping. Too much CPU load, start some servers. Too little, kill some servers. Oops now CPU is too high, bring them back. No, it's too low... It reminds me of the old UI bug where you get an off-by-one error for scrollbar logic and so it appears and disappears in a cycle until you resize. Edit: I think we want something akin to a learning thermostat, but for servers. Figure out that I get a CPU spike at 5:20 on week days and spin some servers up at 5:10. Then spin em down at 8:00 when EST starts to go to bed.

navinsylvester 7y ago

Interesting read. But i bumped into the same problem recently while working on distributed simulation. I was writing an agent for an application which talks to clients over ssh. I started getting the same error when i cranked up machine count to 60. I googled like every layman and zeroed in on MaxStartups. Looked at the manual to cross verify and fixed the issue in 5 minutes. I agree with what ever he did after he found out the issue but it was a clear case of SSH throwing an error. Why not just look up manual or google instead of wasting time on laborious packet inspection.

abrookewood 7y ago
Because at their scale, people actually need to understand the issue rather than just applying fixes blindly plucked from the Internet. Don't get me wrong, Google is my go-to as well, but I suspect that they need to dig a little deeper.
- navinsylvester 7y ago
  Fair enough. But wouldn't doing investigation on the error provide data point to help debug the issue. They are trying to fix a customer issue and time is also of importance here. > just applying fixes blindly plucked from the Internet. I wasn't recommending that.
toast0 7y ago
> Why not just look up manual or google instead of wasting time on laborious packet inspection. When you've got a load balancing layer, and virtual machines, and interacting with the real world, there's a reasonable chance that the problem is not a (relatively) simple application configuration issue. If you've looked through enough of these, getting a pcap (especially with an identified customer) and looking through it with Wireshark is relatively quick.
- navinsylvester 7y ago
  I could have worded it better like "why packet inspection to start with".

vmurthy 7y ago

I can't help but notice that these kind of blogs should be recommended reading for CS/engineering students as well and not just to people who are already working. The way an issue is to be analyzed (this post is a great example of Systems thinking IMO), what happens at scale and why not to blindly Google your way out of problems are valuable skills any engineer should have and which are unfortunately not taught in many schools.