6 ms·

from personal experience, simply moving ssh off of port 22 cuts your drive-by log noise by 99.9%, which makes auditing clean logs much much easier. failed logi

from personal experience, simply moving ssh off of port 22 cuts your drive-by log noise by 99.9%, which makes auditing clean logs much much easier. failed login attempts with ssh on 22: hundreds per week, after: 0. i imagine with knockd it would go negative.

peterwwillis 6y ago
> moving ssh off of port 22 cuts your drive-by log noise I didn't know logs made me unsafe! I'm turning off all logging immediately.
- femiagbabiaka 6y ago
  The point is that 99% of all access attempts are done by simple skiddies or people simply crawling the internet with masscan or similar.
- shoeffner 6y ago
  The quote is not about logs making you unsafe. The quote is about ssh login and connection attempts on port 22 which clutter your log files. Assume you move ssh to another port, say 23. In that case, anyone trying to connect to port 22 would find a closed port and the connection would fail, on a layer before login attempts get logged.
- kelnos 6y ago
  The point is that most login attempts are automated, and the automated scripts just try port 22 and move on if they fail. If you run ssh on port 22, you see in your logs all the failed attempts from the giant sea of automated scanners, and will probably miss the one or two humans who might know something about you specifically (like your username) and are trying a targeted attack. If you put ssh on another port, all of those automated login attempts disappear from your logs. The determined human attacker probably has port-scanned you and found ssh on another port, so you'll see just that attacker and can look into it more deeply if you feel the need. On port 22, you probably never bother to look at your failed-attempt log, because there's so much noise.
  - mirages 6y ago
    Actually I'm receiving about 10 attempts per minute on my non-22 ssh port that I've set up
    - kelnos 6y ago
      Damn, that's a shame. From what other people say when they advocate non-22, they make me believe it would be a lot less. Oh well.
  - belorn 6y ago
    I would be interested to read about peoples process and procedures from those that regularly read their ssh failed attempts, identify human attackers, and react to that information. My question is basically: Do they exist? How is the work flow and how much time per year do they spend on it? What incident occurred and what value did they derive from the process. Was it cost-effective?
    - peterwwillis 6y ago
      About a decade and a half ago it was in vogue to constantly monitor your logs for attacks. There was sophisticated software to detect the latest attacks, and they would notify you constantly, and you'd run around chasing your tail for every unusual port scan. We spent tons of money on reporting software to try to stay on top of it all. But we realized that as attacks increased, it was pointless to look at who or how often we were being attacked, because it wasn't making our systems more secure. What did make them more secure was actually securing them: patching systems, implementing firewall rules, blocking or slowing down multiple login attempts. Non-sophisticated attacks are trivial to prevent, and sophisticated attacks won't show up in logs. Everyone needs to stop logging ssh attempts and just secure their boxes and move on with life. And that's why port knocking is stupid.
    - kelnos 6y ago
      I don't read mine at all (talking about a personal server here[0]), because there's a huge amount of noise. The email alerts for it go to a mail folder that I literally never look at, because it gets hundreds of emails per day. If I only got an email per week or so, I'd probably not filter them to a folder, and actually look at them. [0] At work, other people deal with intrusion detection there, and ssh access is gated behind a VPN anyway.
- EvilTerran 6y ago
  Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
  - peterwwillis 6y ago
    It was a response to their main point, but the subtlety was lost on people who don't think critically. Just because you see or don't see attacks doesn't mean you are more or less secure. It just means you've finally noticed there is risk. It doesn't mean there is more or less risk. It's the same risk, it's just more visible.
tptacek 6y ago
Wouldn't a reasonable rebuttal be that the log filtering rule that discards these failed login attempts would accomplish the same security goal, with less mechanism?
- leeoniya 6y ago
  i prefer knowing that nothing is happening rather than filtering things i dont like. also, lots of people sing the praises of fail2ban, which needs to watch log files to operate, but you can get most of its utility by simply rate-limiting using iptables' session tracking without any additional mechanisms. https://selivan.github.io/2018/07/07/limit-new-connections-frequency-with-iptables-recent.html https://selivan.github.io/2018/07/07/limit-new-connections-f...
  - tptacek 6y ago
    I can tell you right now what's happening and compress all those log records into a single record: Every exposed 22/tcp on the Internet is being continuously exposed to automated SSH scanners running from thousands of points on the Internet. There is nothing you can reasonably do to prevent it (the sources are so diverse you couldn't even realistically block a determined scanner), and, if you've turned off password authentication --- which you must do anyways --- the probes aren't a meaningful threat.
    - leeoniya 6y ago
      > the sources are so diverse you couldn't even realistically block a determined scanner thankfully, i dont have properties which attract determined scanners. and i imagine that's true for 99% of public ip addresses, too.
      - tptacek 6y ago
        So then, of course, there isn't even a theoretical reason why you'd want to pay attention to SSH probes.
WrtCdEvrydy 6y ago
I wonder if it would be interesting to load a ssh honeypot that would feed from the logs so it would add the logins and then just dump the ip + input to a file. Honeypot could basically run in a some sort of isolation layer (like Sandboxie or jails) and then self-destruct after the automated script is gone... and then you slam the door on that user/ip combo for good. I can't help but think this would be interesting...
- EvanAnderson 6y ago
  I think you're talking about something like Kippo: https://www.honeynet.org/projects/old/kippo/ https://www.honeynet.org/projects/old/kippo/
- acidictadpole 6y ago
  One of my university courses offered an opportunity for a project like this and I did it with some classmates. We started by altering the ssh daemon to disallow all logins over this ssh daemon and to log all the usernames and passwords attempted. After a week we gathered thousands of attempts to brute force into the honeypot. Interestingly enough, the passwords used were a combination of the very commonly used ones but also ones that were clearly from other popped boxes. After a week or so of this we altered the ssh daemon again. This time it would log all attempts but also grant access on the 3rd attempt no matter what the credentials were. The few bots that managed to get in all tried to install various rootkits on the machine, all of which where targeted at a different distro of Linux than we were using so it mostly just busted up our shells output. It was a long time ago but a great experience.
- cycomanic 6y ago
  You don't even need a honeypot, simply ban ip addresses for some time after X failed login attempts.
  - lloeki 6y ago
    The point of the honeypot is that there's no heuristic causing any delay or elusive attacker being missed (e.g botnet trying once per IP). You don't even need any processing time, nor even complete syn/synack/ack: any TCP connection attempt to that port triggers an instaban.
Evidlo 6y ago
> i imagine with knockd it would go negative. So it's the people running knockd that are actually the ones generating all these ssh login attempts...
- deleted 6y ago
  [deleted]
MaxBarraclough 6y ago
> simply moving ssh off of port 22 cuts your drive-by log noise by 99.9%, which makes auditing clean logs much much easier. Is there a reason you can't use some form of IP whitelisting? I'm not a proper sysadmin, but when I play about on EC2 I always configure the security-group to block incoming connections to TCP port 22 except from whitelisted IPs. The only time I've had to broaden the whitelist beyond just a few static IPs, is with mobile Internet tethering. They seem to frequently change my IP, so I guessed at my provider's IP range and whitelisted it with a pretty broad wildcard mask. Still much better than accepting connections from any old address. I can see that my rather simple approach might not scale very nicely.
- iso1631 6y ago
  > I'm not a proper sysadmin, but when I play about on EC2 I always configure the security-group to block incoming connections to TCP port 22 except from whitelisted IPs. Because I have multiple IPs I connect to, most of which are unknown. Maybe I'm connecting from a coffee shop, maybe from a tethered phone, maybe from work, maybe from an airplane, or a train, or a friend's wifi, etc.

peterwwillis 6y ago

> moving ssh off of port 22 cuts your drive-by log noise I didn't know logs made me unsafe! I'm turning off all logging immediately.

femiagbabiaka 6y ago
The point is that 99% of all access attempts are done by simple skiddies or people simply crawling the internet with masscan or similar.
shoeffner 6y ago
The quote is not about logs making you unsafe. The quote is about ssh login and connection attempts on port 22 which clutter your log files. Assume you move ssh to another port, say 23. In that case, anyone trying to connect to port 22 would find a closed port and the connection would fail, on a layer before login attempts get logged.
kelnos 6y ago
The point is that most login attempts are automated, and the automated scripts just try port 22 and move on if they fail. If you run ssh on port 22, you see in your logs all the failed attempts from the giant sea of automated scanners, and will probably miss the one or two humans who might know something about you specifically (like your username) and are trying a targeted attack. If you put ssh on another port, all of those automated login attempts disappear from your logs. The determined human attacker probably has port-scanned you and found ssh on another port, so you'll see just that attacker and can look into it more deeply if you feel the need. On port 22, you probably never bother to look at your failed-attempt log, because there's so much noise.
- mirages 6y ago
  Actually I'm receiving about 10 attempts per minute on my non-22 ssh port that I've set up
  - kelnos 6y ago
    Damn, that's a shame. From what other people say when they advocate non-22, they make me believe it would be a lot less. Oh well.
- belorn 6y ago
  I would be interested to read about peoples process and procedures from those that regularly read their ssh failed attempts, identify human attackers, and react to that information. My question is basically: Do they exist? How is the work flow and how much time per year do they spend on it? What incident occurred and what value did they derive from the process. Was it cost-effective?
  - peterwwillis 6y ago
    About a decade and a half ago it was in vogue to constantly monitor your logs for attacks. There was sophisticated software to detect the latest attacks, and they would notify you constantly, and you'd run around chasing your tail for every unusual port scan. We spent tons of money on reporting software to try to stay on top of it all. But we realized that as attacks increased, it was pointless to look at who or how often we were being attacked, because it wasn't making our systems more secure. What did make them more secure was actually securing them: patching systems, implementing firewall rules, blocking or slowing down multiple login attempts. Non-sophisticated attacks are trivial to prevent, and sophisticated attacks won't show up in logs. Everyone needs to stop logging ssh attempts and just secure their boxes and move on with life. And that's why port knocking is stupid.
  - kelnos 6y ago
    I don't read mine at all (talking about a personal server here[0]), because there's a huge amount of noise. The email alerts for it go to a mail folder that I literally never look at, because it gets hundreds of emails per day. If I only got an email per week or so, I'd probably not filter them to a folder, and actually look at them. [0] At work, other people deal with intrusion detection there, and ssh access is gated behind a VPN anyway.
EvilTerran 6y ago
Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- peterwwillis 6y ago
  It was a response to their main point, but the subtlety was lost on people who don't think critically. Just because you see or don't see attacks doesn't mean you are more or less secure. It just means you've finally noticed there is risk. It doesn't mean there is more or less risk. It's the same risk, it's just more visible.

tptacek 6y ago

Wouldn't a reasonable rebuttal be that the log filtering rule that discards these failed login attempts would accomplish the same security goal, with less mechanism?

leeoniya 6y ago
i prefer knowing that nothing is happening rather than filtering things i dont like. also, lots of people sing the praises of fail2ban, which needs to watch log files to operate, but you can get most of its utility by simply rate-limiting using iptables' session tracking without any additional mechanisms. https://selivan.github.io/2018/07/07/limit-new-connections-frequency-with-iptables-recent.html https://selivan.github.io/2018/07/07/limit-new-connections-f...
- tptacek 6y ago
  I can tell you right now what's happening and compress all those log records into a single record: Every exposed 22/tcp on the Internet is being continuously exposed to automated SSH scanners running from thousands of points on the Internet. There is nothing you can reasonably do to prevent it (the sources are so diverse you couldn't even realistically block a determined scanner), and, if you've turned off password authentication --- which you must do anyways --- the probes aren't a meaningful threat.
  - leeoniya 6y ago
    > the sources are so diverse you couldn't even realistically block a determined scanner thankfully, i dont have properties which attract determined scanners. and i imagine that's true for 99% of public ip addresses, too.
    - tptacek 6y ago
      So then, of course, there isn't even a theoretical reason why you'd want to pay attention to SSH probes.

WrtCdEvrydy 6y ago

I wonder if it would be interesting to load a ssh honeypot that would feed from the logs so it would add the logins and then just dump the ip + input to a file. Honeypot could basically run in a some sort of isolation layer (like Sandboxie or jails) and then self-destruct after the automated script is gone... and then you slam the door on that user/ip combo for good. I can't help but think this would be interesting...

EvanAnderson 6y ago
I think you're talking about something like Kippo: https://www.honeynet.org/projects/old/kippo/ https://www.honeynet.org/projects/old/kippo/
acidictadpole 6y ago
One of my university courses offered an opportunity for a project like this and I did it with some classmates. We started by altering the ssh daemon to disallow all logins over this ssh daemon and to log all the usernames and passwords attempted. After a week we gathered thousands of attempts to brute force into the honeypot. Interestingly enough, the passwords used were a combination of the very commonly used ones but also ones that were clearly from other popped boxes. After a week or so of this we altered the ssh daemon again. This time it would log all attempts but also grant access on the 3rd attempt no matter what the credentials were. The few bots that managed to get in all tried to install various rootkits on the machine, all of which where targeted at a different distro of Linux than we were using so it mostly just busted up our shells output. It was a long time ago but a great experience.
cycomanic 6y ago
You don't even need a honeypot, simply ban ip addresses for some time after X failed login attempts.
- lloeki 6y ago
  The point of the honeypot is that there's no heuristic causing any delay or elusive attacker being missed (e.g botnet trying once per IP). You don't even need any processing time, nor even complete syn/synack/ack: any TCP connection attempt to that port triggers an instaban.

Evidlo 6y ago

> i imagine with knockd it would go negative. So it's the people running knockd that are actually the ones generating all these ssh login attempts...

deleted 6y ago
[deleted]

MaxBarraclough 6y ago

> simply moving ssh off of port 22 cuts your drive-by log noise by 99.9%, which makes auditing clean logs much much easier. Is there a reason you can't use some form of IP whitelisting? I'm not a proper sysadmin, but when I play about on EC2 I always configure the security-group to block incoming connections to TCP port 22 except from whitelisted IPs. The only time I've had to broaden the whitelist beyond just a few static IPs, is with mobile Internet tethering. They seem to frequently change my IP, so I guessed at my provider's IP range and whitelisted it with a pretty broad wildcard mask. Still much better than accepting connections from any old address. I can see that my rather simple approach might not scale very nicely.

iso1631 6y ago
> I'm not a proper sysadmin, but when I play about on EC2 I always configure the security-group to block incoming connections to TCP port 22 except from whitelisted IPs. Because I have multiple IPs I connect to, most of which are unknown. Maybe I'm connecting from a coffee shop, maybe from a tethered phone, maybe from work, maybe from an airplane, or a train, or a friend's wifi, etc.