3 ms·

> It would make everyone more comfortable if OpenSSH let us see the how close we were to hitting MaxStartups at any point, rather than having to go in blind and

by amenod 7y ago

> It would make everyone more comfortable if OpenSSH let us see the how close we were to hitting MaxStartups at any point, rather than having to go in blind and only find out we were close when the limit is breached and connections are dropped. If they are not afraid to dive into source, it should be fairly easy to add some logging when the value reaches 75% for example. Of course one needs to be careful not to break anything else (issuing thousands of warning logs at the same time that the system is at peak load is probably not a good idea), but it shouldn't be that difficult. Also, I assume this would be something that OpenSSH would be happy to get a PR (or MR, coming from GitLab? ;) ) for. Nice writeup. What I miss is how they added monitoring for these kinds of errors, so it doesn't escalate to 1.5% of all connections again. But other than that, good job!

certifiable 7y ago
Oh, that's a fun story. We use https://github.com/google/mtail https://github.com/google/mtail to turn log data into metrics exported for prometheus. We were already doing this for our haproxy logs; we just added a stanza to capture a secondary indicator that the problem is occurring (termination_state 'SD', and 0 bytes_read), and export that as a metric. You can see the relevant MR at https://gitlab.com/gitlab-cookbooks/gitlab-mtail/merge_requests/47 https://gitlab.com/gitlab-cookbooks/gitlab-mtail/merge_reque... (and https://gitlab.com/gitlab-cookbooks/gitlab-mtail/merge_requests/48 https://gitlab.com/gitlab-cookbooks/gitlab-mtail/merge_reque... to fix some bugs) Then we just hooked it up through the usual sort of prometheus alerting rules. We made it really twitchy, because this combination of logs Should Not Happen if everything is working right, and we want to alert as soon as it starts occurring, so we can bump the limits again (or re-evaluate in general)

by amenod 7y ago

certifiable 7y ago

Oh, that's a fun story. We use https://github.com/google/mtail https://github.com/google/mtail to turn log data into metrics exported for prometheus. We were already doing this for our haproxy logs; we just added a stanza to capture a secondary indicator that the problem is occurring (termination_state 'SD', and 0 bytes_read), and export that as a metric. You can see the relevant MR at https://gitlab.com/gitlab-cookbooks/gitlab-mtail/merge_requests/47 https://gitlab.com/gitlab-cookbooks/gitlab-mtail/merge_reque... (and https://gitlab.com/gitlab-cookbooks/gitlab-mtail/merge_requests/48 https://gitlab.com/gitlab-cookbooks/gitlab-mtail/merge_reque... to fix some bugs) Then we just hooked it up through the usual sort of prometheus alerting rules. We made it really twitchy, because this combination of logs Should Not Happen if everything is working right, and we want to alert as soon as it starts occurring, so we can bump the limits again (or re-evaluate in general)