4 ms·

i prefer knowing that nothing is happening rather than filtering things i dont like. also, lots of people sing the praises of fail2ban, which needs to watch lo

i prefer knowing that nothing is happening rather than filtering things i dont like. also, lots of people sing the praises of fail2ban, which needs to watch log files to operate, but you can get most of its utility by simply rate-limiting using iptables' session tracking without any additional mechanisms. https://selivan.github.io/2018/07/07/limit-new-connections-frequency-with-iptables-recent.html https://selivan.github.io/2018/07/07/limit-new-connections-f...

tptacek 6y ago
I can tell you right now what's happening and compress all those log records into a single record: Every exposed 22/tcp on the Internet is being continuously exposed to automated SSH scanners running from thousands of points on the Internet. There is nothing you can reasonably do to prevent it (the sources are so diverse you couldn't even realistically block a determined scanner), and, if you've turned off password authentication --- which you must do anyways --- the probes aren't a meaningful threat.
- leeoniya 6y ago
  > the sources are so diverse you couldn't even realistically block a determined scanner thankfully, i dont have properties which attract determined scanners. and i imagine that's true for 99% of public ip addresses, too.
  - tptacek 6y ago
    So then, of course, there isn't even a theoretical reason why you'd want to pay attention to SSH probes.
    - leeoniya 6y ago
      i mean, you're in the security industry, where targeted attacks are the default assumption. the overwhelming majority of web properties are not valuable enough to get this kind of attention. even if security through obscurity is not real protection from targeted attacks, it is at minimum significant noise reduction, and quite possibly a reasonable barrier against drive-bys which assume default configurations. i'm sure you know how many scanners try to access some default php/wordpress admin route. how much of that would disappear if wordpress was simply installed behind a random 32-char route instead of /wp-admin? all it is is obscurity, sure. but is it not security?
      - hunter2_ 6y ago
        The whole "obscurity isn't security" thing is a super interesting topic. Suppose you have a static file on a public web server and you want to control access to the file... confidentiality. Many people would agree that if you come up with a long string, don't disclose it to untrusted parties, and use it as a basic auth password, then you have secured the file to some extent. And many people would claim that if you take that same undisclosed string, and make it a path segment (parent directory) for the file, then you have merely obscured the file, or perhaps secured it to a far lesser extent. The former makes it "not available to the public" while the latter makes it "available to the public, they just don't know where to look." I call BS. These are pretty much the same thing* which is to say: an adversary trying to get the file needs to know something that they don't know, and can't reasonably guess/discover any time soon. Did you secure it or obscure it? Whatever you call it, you require a "something you know" factor. *Obviously, there's the fact that other parts of the stack -- browser history, access logs, rate limiting, etc. -- actually treat a password payload with the respect it deserves and assume there's nothing sensitive about a URI path... kindly pretend my example doesn't have this flaw.
        heavenlyblue 6y ago
        Semantics of this is wildly different. You may accidentally enable the directory listing. While you may say that certain methods of withholding the information are isomorphic at certain conditions it doesn’t mean that they are semantically equivalent.
        hunter2_ 6y ago
        I admit that my example is flawed for a few reasons and autoindex is another in that list. While obscurity and security are not identical by any means, and therefore not semantically equivalent in general, I'm just pointing out how the "something you know" factor inherently utilizes what common ground they do share, whether you call it a secret, a key, a password, a URL with a long random portion.
        sokoloff 6y ago
        Topically relevant (and amusing) username! http://bash.org/?244321 http://bash.org/?244321
- thaumaturgy 6y ago
  From a systems-administration standpoint, it's nice to be able to easily sift out the random bots and drive-bys from the more personal probes that maybe have a human at the other end. There are myriad bits and bobs of software and services that promise to do anomaly detection in logs, but an easier approach for now is to just move ssh off 22. It affords basically zero extra real security but makes any subsequent login attempts more worthwhile to look at.
  - tptacek 6y ago
    Let me gently challenge you: you've moved SSH to 7163/tcp. You see a pattern of login attempts. Now what, and why?
    - thaumaturgy 6y ago
      Sure. Direct answer: search logs across all services and nodes on the network for activity from that source or netrange. See if it's targeting particular usernames, especially anything that shouldn't be guessable. Treat it first as a source of information: did I screw something up somewhere? Have we had an incident I don't know about? If it looks like nothingburger, ignore it and get on with the day. Broader answer after reading some of your other comments in this thread: in your ideal, highly secured network environment, none of this is necessary because everything's wired up tighter than a gnat's ass hole. Unfortunately I've never had the pleasure of working for one of those places.
      - tptacek 6y ago
        I guess that's fair. But: if you buy into this idea, there's a much, much better thing to do: look into Canarytokens. Canarytokens, unlike port knocking, are criminally underused. Really, you should do something similar for your off-port SSH service; don't actually _run_ SSH there, just run a stateless unprivileged service that spoofs a bit of SSH protocol and generates loud alerts. The networks I'm describing aren't "ideal" or "highly secured". I am describing table stakes. While I was at Latacora, most of the clients we engaged with were already at this level of maturity when we joined up.
belorn 6y ago
Turn on logging for failed connection to port 22 and you have just recreated the same log file, assuming you trust the underlying security in your sshd configuration. You just moved the "nothing happened" event from the sshd to the firewall, and the firewall by default do not log such events. The best argument in favor of knockd that I can think of is the same of fail2ban, which is that ssh connection is actually a finite resource. You can only have so many simultaneous TCP connections. Rejecting packages in the firewall has no limits, and I have had servers which has received service disruption in the past because swarms of ssh bots used up the available tcp connections that the web server could handle. fail2ban in my case was a less intrusive change to our work flow so we use that rather than knockd to solve the problem. I will add as a small note that the default of not logging firewall events has come to bite us once in a while. Just like keeping the default in ssh to log, keeping the default to not log is a trade off between resource usage and keeping logs of events.

tptacek 6y ago

I can tell you right now what's happening and compress all those log records into a single record: Every exposed 22/tcp on the Internet is being continuously exposed to automated SSH scanners running from thousands of points on the Internet. There is nothing you can reasonably do to prevent it (the sources are so diverse you couldn't even realistically block a determined scanner), and, if you've turned off password authentication --- which you must do anyways --- the probes aren't a meaningful threat.

leeoniya 6y ago
> the sources are so diverse you couldn't even realistically block a determined scanner thankfully, i dont have properties which attract determined scanners. and i imagine that's true for 99% of public ip addresses, too.
- tptacek 6y ago
  So then, of course, there isn't even a theoretical reason why you'd want to pay attention to SSH probes.
  - leeoniya 6y ago
    i mean, you're in the security industry, where targeted attacks are the default assumption. the overwhelming majority of web properties are not valuable enough to get this kind of attention. even if security through obscurity is not real protection from targeted attacks, it is at minimum significant noise reduction, and quite possibly a reasonable barrier against drive-bys which assume default configurations. i'm sure you know how many scanners try to access some default php/wordpress admin route. how much of that would disappear if wordpress was simply installed behind a random 32-char route instead of /wp-admin? all it is is obscurity, sure. but is it not security?
    - hunter2_ 6y ago
      The whole "obscurity isn't security" thing is a super interesting topic. Suppose you have a static file on a public web server and you want to control access to the file... confidentiality. Many people would agree that if you come up with a long string, don't disclose it to untrusted parties, and use it as a basic auth password, then you have secured the file to some extent. And many people would claim that if you take that same undisclosed string, and make it a path segment (parent directory) for the file, then you have merely obscured the file, or perhaps secured it to a far lesser extent. The former makes it "not available to the public" while the latter makes it "available to the public, they just don't know where to look." I call BS. These are pretty much the same thing* which is to say: an adversary trying to get the file needs to know something that they don't know, and can't reasonably guess/discover any time soon. Did you secure it or obscure it? Whatever you call it, you require a "something you know" factor. *Obviously, there's the fact that other parts of the stack -- browser history, access logs, rate limiting, etc. -- actually treat a password payload with the respect it deserves and assume there's nothing sensitive about a URI path... kindly pretend my example doesn't have this flaw.
      - heavenlyblue 6y ago
        Semantics of this is wildly different. You may accidentally enable the directory listing. While you may say that certain methods of withholding the information are isomorphic at certain conditions it doesn’t mean that they are semantically equivalent.
        hunter2_ 6y ago
        I admit that my example is flawed for a few reasons and autoindex is another in that list. While obscurity and security are not identical by any means, and therefore not semantically equivalent in general, I'm just pointing out how the "something you know" factor inherently utilizes what common ground they do share, whether you call it a secret, a key, a password, a URL with a long random portion.
      - sokoloff 6y ago
        Topically relevant (and amusing) username! http://bash.org/?244321 http://bash.org/?244321
thaumaturgy 6y ago
From a systems-administration standpoint, it's nice to be able to easily sift out the random bots and drive-bys from the more personal probes that maybe have a human at the other end. There are myriad bits and bobs of software and services that promise to do anomaly detection in logs, but an easier approach for now is to just move ssh off 22. It affords basically zero extra real security but makes any subsequent login attempts more worthwhile to look at.
- tptacek 6y ago
  Let me gently challenge you: you've moved SSH to 7163/tcp. You see a pattern of login attempts. Now what, and why?
  - thaumaturgy 6y ago
    Sure. Direct answer: search logs across all services and nodes on the network for activity from that source or netrange. See if it's targeting particular usernames, especially anything that shouldn't be guessable. Treat it first as a source of information: did I screw something up somewhere? Have we had an incident I don't know about? If it looks like nothingburger, ignore it and get on with the day. Broader answer after reading some of your other comments in this thread: in your ideal, highly secured network environment, none of this is necessary because everything's wired up tighter than a gnat's ass hole. Unfortunately I've never had the pleasure of working for one of those places.
    - tptacek 6y ago
      I guess that's fair. But: if you buy into this idea, there's a much, much better thing to do: look into Canarytokens. Canarytokens, unlike port knocking, are criminally underused. Really, you should do something similar for your off-port SSH service; don't actually _run_ SSH there, just run a stateless unprivileged service that spoofs a bit of SSH protocol and generates loud alerts. The networks I'm describing aren't "ideal" or "highly secured". I am describing table stakes. While I was at Latacora, most of the clients we engaged with were already at this level of maturity when we joined up.

belorn 6y ago

Turn on logging for failed connection to port 22 and you have just recreated the same log file, assuming you trust the underlying security in your sshd configuration. You just moved the "nothing happened" event from the sshd to the firewall, and the firewall by default do not log such events. The best argument in favor of knockd that I can think of is the same of fail2ban, which is that ssh connection is actually a finite resource. You can only have so many simultaneous TCP connections. Rejecting packages in the firewall has no limits, and I have had servers which has received service disruption in the past because swarms of ssh bots used up the available tcp connections that the web server could handle. fail2ban in my case was a less intrusive change to our work flow so we use that rather than knockd to solve the problem. I will add as a small note that the default of not logging firewall events has come to bite us once in a while. Just like keeping the default in ssh to log, keeping the default to not log is a trade off between resource usage and keeping logs of events.