4 ms·

>It is where I attach a debugger, it is where I install iotop and use it for the first time. It is where I cat out mysterious /proc and /sys values to discover

>It is where I attach a debugger, it is where I install iotop and use it for the first time. It is where I cat out mysterious /proc and /sys values to discover exotic things about cgroups I only learned about 5 minutes prior in obscure system documentation. It is, SSH is indeed the tool for that, but that's because until recently we did not have better tools and interfaces. Once you try newer tools, you don't want to go back. Here's the example of my fairly recent debug session: - Network is really slow on the home server, no idea why - Try to just reboot it, no changes - Run kernel perf, check the flame graph - Kernel spends A LOT of time in nf_* (netfilter functions, iptables) - Check iptables rules - sshguard has banned 13000 IP addresses in its table - Each network packet travels through all the rules - Fix: clean the rules/skip the table for established connections/add timeouts You don't need debugging facilities for many issues. You need observability and tracing. Instead of debugging the issue for tens of minutes at least, I just used observability tool which showed me the path in 2 minutes.

crawshaw 9mo ago
How did you use tracing to check the current state of a machine’s iptables rules?
- ValdikSS 9mo ago
  In this case I used `perf` utility, but only because the server does not have a proper observability tool. Take a look at this Netflix presentation, especially on the screenshots of their web interface tool: https://archives.kernel-recipes.org/wp-content/uploads/2025/01/kernelrecipesperfevents-170929090404-171007142608.pdf https://archives.kernel-recipes.org/wp-content/uploads/2025/...
  - crawshaw 9mo ago
    That is a command line tool run over ssh. If you have invented a new way to run command line tools, that’s great (and very possible, writing a service that can fork+exec and map stdio), but it is the equivalent to using ssh. You cannot run commands using traces.
    - charcircuit 9mo ago
      With that mindset anything is equivalent to ssh. The command line is not the pinnacle of user interfaces and giving admins full control of the machine isn't the pinnacle of security either. We need to accept that UNIX did not get things right decades ago and be willing to evolve UX and security to a better place.
      - crawshaw 9mo ago
        Happy to try an alternative. Traces I have tried, and it is not an alternative.
IgorPartola 9mo ago
See I would not reboot the server first before figuring out what is happening. You lose a lot of info by doing that and the worst thing that can happen is that the problem goes away for a little bit.
- butvacuum 9mo ago
  most failstates arent worth preserving in a SMB environment. In larger environments or ones equipped for it a snapshot can be taken before rebooting- should the issue repeat. Once is chance, twice is coincidence, three times makes a pattern.
  - Ferret7446 9mo ago
    Alternatively, if it doesn't happen again it's not worth fixing, if it does happen again then you can investigate it when it happens again.
- gerdesj 9mo ago
  To be fair, turning it off and on again is unreasonably effective. I recently diagnosed and fixed an issue with Veeam backups that suddenly stopped working part way through the usual window and stopped working from that point on. This particular setup has three sites (prod, my home and DR), and five backup proxies. Anyway, I read logs and Googled somewhat. I rebooted the backup server - no joy, even though it looked like the issue was there. I restarted the proxies and things started working again. The error was basically: there are no available proxies, even though they were all available (but not working but not giving off "not working" vibes). I could bother with trying to look for what went wrong but life is too short. This is the first time that pattern has happened to me (I'll note it down mentally and it was logged in our incident log). So, OK, I'll agree that a reboot should not generally be the first option. Whilst sciencing it or nerding harder is the purist approach, often a cheeky reboot gets the job done. However, do be aware that a Windows box will often decide to install updates if you are not careful 8)
  - akerl_ 9mo ago
gerdesj 9mo ago
I fail to understand how your approach is different to your parent. perf is a shell tool. iptables is a shell tool. sshguard is a log reader and ultimately you will use the CLI to take action. If you are advocating newer tools, look into nft - iptables is sooo last decade 8) I've used the lot: ipfw, ipchains, iptables and nftables. You might also try fail2ban - it is still worthwhile even in the age of the massively distributed botnet, and covers more than just ssh. I also recommend a VPN and not exposing ssh to the wild. Finally, 13,000 address in an ipset is nothing particularly special these days. I hope sshguard is making a properly optimised ipset table and that you running appropriate hardware. My home router is a pfSense jobbie running on a rather elderly APU4 based box and it has over 200,000 IPs in its pfBlocker-NG IP block tables and about 150,000 records in its DNS tables.
- ValdikSS 9mo ago
  >perf is a shell tool. iptables is a shell tool. sshguard is a log reader and ultimately you will use the CLI to take action. Well yes, and to be honest in this case I did that all over SSH: run `perf`, generate flame graph, copy the .svg to the PC over SFTP, open it in the file viewer. What I really wanted is a web interface which will just show me EVERYTHING it knows about the system in a form of charts, graphs, so I can just skim through it and check if everything allright visually, without using the shell and each individual command. Take a look at Netflix presentation, especially on their web interface screenshots: https://archives.kernel-recipes.org/wp-content/uploads/2025/01/kernelrecipesperfevents-170929090404-171007142608.pdf https://archives.kernel-recipes.org/wp-content/uploads/2025/... >look into nft - iptables is sooo last decade It doesn't matter in this context: iptables is using new netfilter (I'm not using iptables-legacy), and this exact scenario is 100% possible with native netfilter nft. >Finally, 13,000 address in an ipset is nothing particularly special these days Oh, the other day I had just 70 `iptables -m set --match-set` rules, and did you know how apparently inefficient source/destination address hashing algorithm for the set match is?! It was debugged with perf as well, but I wish I just had it as a dashboard picture from the start. I'm talking about ~4Gbit/s sudden limitation on a 10Gbit link.
  - gerdesj 9mo ago
kelnos 9mo ago
That only works if the people who built the observability tool have thought of everything. They haven't, of course; no one can. It's great that you were able to solve this problem with your observability tools. But nothing will ever be as comprehensive as what you can do with shell access. I don't get what the big deal is here. Just... use shell access when you need it. If you have other things in place that let you easily debug and fix some classes of issues, great. But some things might be easier to fix with shell access, and you could very easily run into something you can't figure out without ssh. Completely disabling shell access is just making things harder for you. You don't get brownie points or magical benefits from denying yourself that.
- DANmode 9mo ago
  > That only works if the people who built the observability tool have thought of everything. They haven't, of course; no one can. but the tool draws data from forums of people who have had the problem I’m having before.
johnisgood 9mo ago
Your example is a shell debugging session. You ran perf, checked iptables, inspected sshguard - all via SSH (or locally). The "observability tool" here is shell access to system utilities. This proves the parent's point: when the unknown happens, you need a shell.

crawshaw 9mo ago

How did you use tracing to check the current state of a machine’s iptables rules?

ValdikSS 9mo ago
In this case I used `perf` utility, but only because the server does not have a proper observability tool. Take a look at this Netflix presentation, especially on the screenshots of their web interface tool: https://archives.kernel-recipes.org/wp-content/uploads/2025/01/kernelrecipesperfevents-170929090404-171007142608.pdf https://archives.kernel-recipes.org/wp-content/uploads/2025/...
- crawshaw 9mo ago
  That is a command line tool run over ssh. If you have invented a new way to run command line tools, that’s great (and very possible, writing a service that can fork+exec and map stdio), but it is the equivalent to using ssh. You cannot run commands using traces.
  - charcircuit 9mo ago
    With that mindset anything is equivalent to ssh. The command line is not the pinnacle of user interfaces and giving admins full control of the machine isn't the pinnacle of security either. We need to accept that UNIX did not get things right decades ago and be willing to evolve UX and security to a better place.
    - crawshaw 9mo ago
      Happy to try an alternative. Traces I have tried, and it is not an alternative.

IgorPartola 9mo ago

See I would not reboot the server first before figuring out what is happening. You lose a lot of info by doing that and the worst thing that can happen is that the problem goes away for a little bit.

butvacuum 9mo ago
most failstates arent worth preserving in a SMB environment. In larger environments or ones equipped for it a snapshot can be taken before rebooting- should the issue repeat. Once is chance, twice is coincidence, three times makes a pattern.
- Ferret7446 9mo ago
  Alternatively, if it doesn't happen again it's not worth fixing, if it does happen again then you can investigate it when it happens again.
gerdesj 9mo ago
To be fair, turning it off and on again is unreasonably effective. I recently diagnosed and fixed an issue with Veeam backups that suddenly stopped working part way through the usual window and stopped working from that point on. This particular setup has three sites (prod, my home and DR), and five backup proxies. Anyway, I read logs and Googled somewhat. I rebooted the backup server - no joy, even though it looked like the issue was there. I restarted the proxies and things started working again. The error was basically: there are no available proxies, even though they were all available (but not working but not giving off "not working" vibes). I could bother with trying to look for what went wrong but life is too short. This is the first time that pattern has happened to me (I'll note it down mentally and it was logged in our incident log). So, OK, I'll agree that a reboot should not generally be the first option. Whilst sciencing it or nerding harder is the purist approach, often a cheeky reboot gets the job done. However, do be aware that a Windows box will often decide to install updates if you are not careful 8)
- akerl_ 9mo ago

gerdesj 9mo ago

I fail to understand how your approach is different to your parent. perf is a shell tool. iptables is a shell tool. sshguard is a log reader and ultimately you will use the CLI to take action. If you are advocating newer tools, look into nft - iptables is sooo last decade 8) I've used the lot: ipfw, ipchains, iptables and nftables. You might also try fail2ban - it is still worthwhile even in the age of the massively distributed botnet, and covers more than just ssh. I also recommend a VPN and not exposing ssh to the wild. Finally, 13,000 address in an ipset is nothing particularly special these days. I hope sshguard is making a properly optimised ipset table and that you running appropriate hardware. My home router is a pfSense jobbie running on a rather elderly APU4 based box and it has over 200,000 IPs in its pfBlocker-NG IP block tables and about 150,000 records in its DNS tables.

ValdikSS 9mo ago
>perf is a shell tool. iptables is a shell tool. sshguard is a log reader and ultimately you will use the CLI to take action. Well yes, and to be honest in this case I did that all over SSH: run `perf`, generate flame graph, copy the .svg to the PC over SFTP, open it in the file viewer. What I really wanted is a web interface which will just show me EVERYTHING it knows about the system in a form of charts, graphs, so I can just skim through it and check if everything allright visually, without using the shell and each individual command. Take a look at Netflix presentation, especially on their web interface screenshots: https://archives.kernel-recipes.org/wp-content/uploads/2025/01/kernelrecipesperfevents-170929090404-171007142608.pdf https://archives.kernel-recipes.org/wp-content/uploads/2025/... >look into nft - iptables is sooo last decade It doesn't matter in this context: iptables is using new netfilter (I'm not using iptables-legacy), and this exact scenario is 100% possible with native netfilter nft. >Finally, 13,000 address in an ipset is nothing particularly special these days Oh, the other day I had just 70 `iptables -m set --match-set` rules, and did you know how apparently inefficient source/destination address hashing algorithm for the set match is?! It was debugged with perf as well, but I wish I just had it as a dashboard picture from the start. I'm talking about ~4Gbit/s sudden limitation on a 10Gbit link.
- gerdesj 9mo ago

kelnos 9mo ago

That only works if the people who built the observability tool have thought of everything. They haven't, of course; no one can. It's great that you were able to solve this problem with your observability tools. But nothing will ever be as comprehensive as what you can do with shell access. I don't get what the big deal is here. Just... use shell access when you need it. If you have other things in place that let you easily debug and fix some classes of issues, great. But some things might be easier to fix with shell access, and you could very easily run into something you can't figure out without ssh. Completely disabling shell access is just making things harder for you. You don't get brownie points or magical benefits from denying yourself that.

DANmode 9mo ago
> That only works if the people who built the observability tool have thought of everything. They haven't, of course; no one can. but the tool draws data from forums of people who have had the problem I’m having before.

johnisgood 9mo ago

Your example is a shell debugging session. You ran perf, checked iptables, inspected sshguard - all via SSH (or locally). The "observability tool" here is shell access to system utilities. This proves the parent's point: when the unknown happens, you need a shell.