6 ms·

config

72 points by loumf 2y ago

RGamma 2y ago
Put a personal VPS (no domain) online today. Took like 10 seconds for the first ssh login attempts to start. It's toxic out there. I put everything behind wireguard these days.
- siamese_puff 2y ago
  Same. Wireguard port and block the rest. Ez-Pz step 1 to take with minimal effort.
  - kstrauser 2y ago
    I have a default-deny firewall in place, but this was on port 443 that I needed to have exposed already.
- kijin 2y ago
  I've had a dedicated server pwned before I even logged in for the first time. Somehow the attackers must have guessed the (purportedly random) initial root password, or exploited a vulnerability in the somewhat outdated OS image.
  - getcrunk 2y ago
    If the host wasn’t some random cheap spot you HAVE to tell us who! (please :))
    - kijin 2y ago
      It was in South Korea. I don't want to blame the host because they were very quick to acknowledge the problem, which is something even well-known companies with worldwide presence tend not to be good at. They wiped the server and reinstalled a clean OS, free of charge, within hours.
- nextaccountic 2y ago
  > I put everything behind wireguard these days. What do you mean? You don't connect to ssh in the VPS directly, but through a VPN? How does this protects the VPS from attacks?
  - alanfranz 2y ago
    You don’t bind the ssh port to the public internet.
    - nextaccountic 2y ago
      What about doing things like port knocking on non-standard ports? Exposing wireguard to the world isn't much better than exposing ssh
  - paranoidrobot 2y ago
    If there isn't already, there needs to be something like the "Survivability Onion" from the Military[1], which has the first layer of the onion being: Don't Be There. For hosting, one of those layers is going to be: Don't expose ports publicly, if you don't want people connecting to them. Unless you're hosting a public SSH service, you shouldn't expose SSH on a public network. It protects the VPS from a few avenues of attack: - SSH vulnerabilities (either current, or because the server has not been updated regularly) - SSH server misconfiguration - Credentials leaked/stolen - Denial of Service (hammering SSH in an effort to either consume resources on the host, and make it harder to fix/patch an issue, or to force your hosting provider to block SSH) [1] https://en.wikipedia.org/wiki/Survivability#Military https://en.wikipedia.org/wiki/Survivability#Military (at the bottom of the section)
    - SahAssar 2y ago
      > SSH vulnerabilities OpenSSH is probably one of the most audited pieces of software running on your system. > SSH server misconfiguration Just like with TLS use tools to validate your config (like https://www.ssh-audit.com/ https://www.ssh-audit.com/) and you'll be mostly safe. > Denial of Service Usually it's easier to DoS whatever other service your server is running, like a web backend och mail server or similar. --- On the other hand you now need to manage two keys, the wireguard one and the SSH one, and you still need to expose your wireguard server (which is not necessarily more secure than OpenSSH). Just disable SSH password auth, run one of the ssh audit tools like the one I linked above and use hardware keys like the openssh yubikey feature (optional but nice to know that the key can't be cloned). If you really don't want people trying to ssh to you move it to another port or use port-knocking.
- userbinator 2y ago
  Known VPS IP ranges are going to be very heavily scanned.
- mjrpes 2y ago
  Yeah I've been happy with setting DHCP to internally resolve subdomains to local ip addresses, combined with a lets encrypt wildcard cert created with a DNS challenge. Everything secured behind WireGuard and you never for a second need to make your web services public.
kstrauser 2y ago
Hey, it's me! I'd added a new hostname to a long-existing domain. Then I added that hostname as a new virtual host to a Caddy server I've been running for a long time. The requests were to that vhost, i.e. using the `Host: my-new-host.example.com` header, not just running `curl http://1.2.3.4 http://1.2.3.4`. They were asking for the brand-new host by name. After hashing it out with some friends on Mastodon, I think it's most likely because Caddy acquired a Let's Encrypt cert, those certs are logged[0], and attackers pounce on new hosts as soon as they're in the logs. [0]https://letsencrypt.org/docs/ct-logs/ https://letsencrypt.org/docs/ct-logs/
- wrs 2y ago
  Yes, this is a downside of certificate transparency. Also, using Let’s Encrypt certs for internal hosts, as some do for convenience, means you’re publishing your internal hostnames. Use an internal CA if that worries you.
  - singron 2y ago
    If you care, you can use a wildcard cert. E.g. instead of getting a cert for foo.example.com, get one for *.foo.example.com, and then use an obscure subdomain. AFAIK, nobody is bruteforcing subdomains on wildcard certs.
    - bradtheappguy 2y ago
      They are
      - thwarted 2y ago
        Are they brute forcing them, or are they relying on the fact that most users of wildcard certs also have wildcard DNS entries, so it's just that everything is actually available. Wildcard DNS pointing to a web server with a wildcard cert will pass through to the web server, and at that point the web server responds with a 404 if the host header isn't one configured for virtual hosts.
    - 2y ago
randall 2y ago
exact same experience. i happened to look at my logs and was terrified for a minute.
xyst 2y ago
One of these days I’ll setup a honeypot and use a reverse shell technique to backdoor the attacker.
- kstrauser 2y ago
  There's a story behind why this blog is at honeypot.net, and it surprisingly has nothing to do with that at all.
- iJohnDoe 2y ago
  Nothing is impossible, but just having an SSH connection active to a system does not allow the possibility to tunnel backwards through that SSH connection. Unless the person who initiated the connection already configured it for reverse connections. Would like to learn more though.
  - PhilipRoman 2y ago
    For non-interactive scripts there probably isn't much you can do. I've always wanted to setup an interactive honeypot ssh server which prints the standard message for a rejected login and then pretends to close the connection, while running a keylogger to hopefully intercept some command which asks for password. The major obstacle is that people customize their prompts, although many probably use whatever is set by default on Ubuntu.
doubloon 2y ago
sometimes i wonder why the internet wound up this way. was there some alternative development path or is this inherent in the physical network design?
- SoftTalker 2y ago
  Micropayments. If it cost some amount of money to connect to a server, all of this would stop almost immediately.
  - kstrauser 2y ago
    Somewhere, John Stankey feels a shiver up his spine.
  - cdchn 2y ago
    Sure helped the PSTN.
    - SoftTalker 2y ago
      Back when long-distance calls cost real money per minute, yes. You got the occasional telemarketing call, but it was local and in most areas there wasn't much of it happening.
      - cdchn 2y ago
        But accessing the PSTN network still has a cost. Simply having a cost is not a blocker for bad actors.
        SoftTalker 2y ago
        Yes -- the key is a marginal cost for each call that is higher than what they earn on average per call. And that would not be a lot.
        cdchn 2y ago
userbinator 2y ago
I created a new hostname in DNS If you’re relying on obscurity Presumably the former caused your new hostname to be published for all of the Internet to see? That doesn't sound like obscurity to me. I've had a service running on a high port for many years at the same IP. I've seen it get the occasional "knock" from some scanner or bot, but it has been generally quiet. It probably also depends on your IP, as some parts of the Internet are likely scanned far more frequently and aggressively than others.
- kstrauser 2y ago
  It's more likely that it was the Let's Encrypt cert I requested; see https://news.ycombinator.com/item?id=40385643 https://news.ycombinator.com/item?id=40385643 for more.
TZubiri 2y ago
Imagine falling for that lol > having an all purpose 1000kloc http server. > serving your source code root > not using any permissions system
ta988 2y ago
Yes that's my experience too just a few seconds anytime I create a certificate with letsencrypt
gmuslera 2y ago
Having a webserver listening on ipv4 may get something like that sooner or later. All the tine you get vulnerability scanning by different actors and goals, be aware of that or not. With hostnames is just another layer, that may have more requirements, but the motivations are similar.
jmward01 2y ago
What is missing in our society to actually deal with this? I see stories like this and get the impression that it is a forgone conclusion that it is OK for this to be normal. I realize the author isn't saying that, and I appreciate their warning and think it is great they posted this, but where is the call to action to deal with this? And I don't mean we should all just get stronger safeguards. The people doing this type of thing are causing harm on a large scale so how do we get society to recognize and start really caring about this to the point that it wouldn't be a joke to call some law enforcement agency to tell them you are being hacked?
- cdchn 2y ago
  Corruption and the cultural acceptance of being a bad actor against groups different than yourself in certain societies.
- kstrauser 2y ago
  It costs time and money to run those scans, and people wouldn’t be doing it if it wasn’t profitable. I don’t want to blame the victims, but the root is that people put insecure setups online frequently enough to keep the criminals interested. We could require a license or certification to launch a server but that’d be even worse. I don’t have an answer. This normal isn’t OK. I don’t know what to do about it though.
- ryandrake 2y ago
  There's nothing missing from society. This is the natural state of humanity, and one always must be vigilant. The only thing keeping thousands of people from trying every home's physical front door every 5 seconds to see if it's unlocked is that it's not scalable. If it were possible, it would be happening. AND, if it did start happening, it would not be a law enforcement problem unless law enforcement became equally scalable. This is not a popular opinion and it often gets dunked on reflexively, but I truly believe the default is low-trust. We are one evolutionary step away from chimpanzees, with barely enough "society" grafted on to our world, which mostly doesn't function. You need to assume you are under potential attack at all times and be prepared to defend against it. Whether it be a physical attack, social engineering, scams, advertisements... in cyber-space and in meat-space. Any other security posture, long term will eventually open you up to a breach by an adversary.
robertclaus 2y ago
I've also had my hosting provider run those checks to proactively warn me about vulnerabilities.
dilyevsky 2y ago
These days certificate transparency logs seems to be a trigger to immediately get slammed with a ton of scanners. Some “legit” commercial (eg paloaltonetworks) some russian/chinese
- andrewstuart2 2y ago
  Yep. CTL is an important feature but it's definitely also a liability.
darepublic 2y ago
But wouldn't obscurity beat these attacks. I mean aren't they crafted only to explore common weaknesses and not esoteric weird unique set ups?
- dilyevsky 2y ago
  Yes it would but we dont like to talk about it =) Although if enough people start doing it my guess is the scanners will begin knocking on more than just :22,80,443
MBCook 2y ago
Some of these I get and are obvious. Some look like some kind of exploit maybe? And then there’s .DS_Store. What’s the point of that? In case you find a Mac to launch more targeted attacks against known bugs? To know if the developer is in a Mac and just copied files without filtering out dot files?
- dilyevsky 2y ago
  This one is probably to find if there’s a publicly exposed smb partition or something
billy99k 2y ago
I was working on bug bounty work a few years ago for a large company with lots of subdomains in different parts of the world. I found .git/config on a server and I was able to partially reconstruct the entire git repository. This led me to paths that I wouldn't otherwise be able to find and a complete server takeover through remote code execution. One of their developers left test code for a website template and an unrestricted file upload form.
mchenier 2y ago
Port knocking may be a first line defense here with a port scan attack detector to ban IPs that try to find such ports. See Linux knockd and psad for references. This obscurity doesn’t protect again man-in-the-middle but at least protects from unwanted and opportunistic guests. It also gives more time to indirectly protect from 0-day on sshd (aka the fiasco that could have been the xz incident).
pronoiac 2y ago
I've used https://crt.sh https://crt.sh to check my own logged SSL certs before, though it looks like they're seeing issues right now.
eqvinox 2y ago
How many of these accesses were from source IPs owned by Tencent? (this is not a prejudice; >80% of auto-bans my servers are issuing are for Tencent IPs. I should grab some exact numbers at some point.)
abimaelmartell 2y ago
I always wondered who is behind these attacks, they don't seem targeted since they do them on random ips. I did a bunch of Devops a few years ago on a Startup, and whenever i started a new AWS EC2 instance, i started getting request for Wordpress files, and other common CMS files.

72 points by loumf 2y ago

Put a personal VPS (no domain) online today. Took like 10 seconds for the first ssh login attempts to start. It's toxic out there. I put everything behind wireguard these days.

siamese_puff 2y ago
Same. Wireguard port and block the rest. Ez-Pz step 1 to take with minimal effort.
- kstrauser 2y ago
  I have a default-deny firewall in place, but this was on port 443 that I needed to have exposed already.
kijin 2y ago
I've had a dedicated server pwned before I even logged in for the first time. Somehow the attackers must have guessed the (purportedly random) initial root password, or exploited a vulnerability in the somewhat outdated OS image.
- getcrunk 2y ago
  If the host wasn’t some random cheap spot you HAVE to tell us who! (please :))
  - kijin 2y ago
    It was in South Korea. I don't want to blame the host because they were very quick to acknowledge the problem, which is something even well-known companies with worldwide presence tend not to be good at. They wiped the server and reinstalled a clean OS, free of charge, within hours.
nextaccountic 2y ago
> I put everything behind wireguard these days. What do you mean? You don't connect to ssh in the VPS directly, but through a VPN? How does this protects the VPS from attacks?
- alanfranz 2y ago
  You don’t bind the ssh port to the public internet.
  - nextaccountic 2y ago
    What about doing things like port knocking on non-standard ports? Exposing wireguard to the world isn't much better than exposing ssh
- paranoidrobot 2y ago
  If there isn't already, there needs to be something like the "Survivability Onion" from the Military[1], which has the first layer of the onion being: Don't Be There. For hosting, one of those layers is going to be: Don't expose ports publicly, if you don't want people connecting to them. Unless you're hosting a public SSH service, you shouldn't expose SSH on a public network. It protects the VPS from a few avenues of attack: - SSH vulnerabilities (either current, or because the server has not been updated regularly) - SSH server misconfiguration - Credentials leaked/stolen - Denial of Service (hammering SSH in an effort to either consume resources on the host, and make it harder to fix/patch an issue, or to force your hosting provider to block SSH) [1] https://en.wikipedia.org/wiki/Survivability#Military https://en.wikipedia.org/wiki/Survivability#Military (at the bottom of the section)
  - SahAssar 2y ago
    > SSH vulnerabilities OpenSSH is probably one of the most audited pieces of software running on your system. > SSH server misconfiguration Just like with TLS use tools to validate your config (like https://www.ssh-audit.com/ https://www.ssh-audit.com/) and you'll be mostly safe. > Denial of Service Usually it's easier to DoS whatever other service your server is running, like a web backend och mail server or similar. --- On the other hand you now need to manage two keys, the wireguard one and the SSH one, and you still need to expose your wireguard server (which is not necessarily more secure than OpenSSH). Just disable SSH password auth, run one of the ssh audit tools like the one I linked above and use hardware keys like the openssh yubikey feature (optional but nice to know that the key can't be cloned). If you really don't want people trying to ssh to you move it to another port or use port-knocking.
userbinator 2y ago
Known VPS IP ranges are going to be very heavily scanned.
mjrpes 2y ago
Yeah I've been happy with setting DHCP to internally resolve subdomains to local ip addresses, combined with a lets encrypt wildcard cert created with a DNS challenge. Everything secured behind WireGuard and you never for a second need to make your web services public.

kstrauser 2y ago

Hey, it's me! I'd added a new hostname to a long-existing domain. Then I added that hostname as a new virtual host to a Caddy server I've been running for a long time. The requests were to that vhost, i.e. using the `Host: my-new-host.example.com` header, not just running `curl http://1.2.3.4 http://1.2.3.4`. They were asking for the brand-new host by name. After hashing it out with some friends on Mastodon, I think it's most likely because Caddy acquired a Let's Encrypt cert, those certs are logged[0], and attackers pounce on new hosts as soon as they're in the logs. [0]https://letsencrypt.org/docs/ct-logs/ https://letsencrypt.org/docs/ct-logs/

wrs 2y ago
Yes, this is a downside of certificate transparency. Also, using Let’s Encrypt certs for internal hosts, as some do for convenience, means you’re publishing your internal hostnames. Use an internal CA if that worries you.
- singron 2y ago
  If you care, you can use a wildcard cert. E.g. instead of getting a cert for foo.example.com, get one for *.foo.example.com, and then use an obscure subdomain. AFAIK, nobody is bruteforcing subdomains on wildcard certs.
  - bradtheappguy 2y ago
    They are
    - thwarted 2y ago
      Are they brute forcing them, or are they relying on the fact that most users of wildcard certs also have wildcard DNS entries, so it's just that everything is actually available. Wildcard DNS pointing to a web server with a wildcard cert will pass through to the web server, and at that point the web server responds with a 404 if the host header isn't one configured for virtual hosts.
  - 2y ago

randall 2y ago

exact same experience. i happened to look at my logs and was terrified for a minute.

xyst 2y ago

One of these days I’ll setup a honeypot and use a reverse shell technique to backdoor the attacker.

kstrauser 2y ago
There's a story behind why this blog is at honeypot.net, and it surprisingly has nothing to do with that at all.
iJohnDoe 2y ago
Nothing is impossible, but just having an SSH connection active to a system does not allow the possibility to tunnel backwards through that SSH connection. Unless the person who initiated the connection already configured it for reverse connections. Would like to learn more though.
- PhilipRoman 2y ago
  For non-interactive scripts there probably isn't much you can do. I've always wanted to setup an interactive honeypot ssh server which prints the standard message for a rejected login and then pretends to close the connection, while running a keylogger to hopefully intercept some command which asks for password. The major obstacle is that people customize their prompts, although many probably use whatever is set by default on Ubuntu.

doubloon 2y ago

sometimes i wonder why the internet wound up this way. was there some alternative development path or is this inherent in the physical network design?

SoftTalker 2y ago
Micropayments. If it cost some amount of money to connect to a server, all of this would stop almost immediately.
- kstrauser 2y ago
  Somewhere, John Stankey feels a shiver up his spine.
- cdchn 2y ago
  Sure helped the PSTN.
  - SoftTalker 2y ago
    Back when long-distance calls cost real money per minute, yes. You got the occasional telemarketing call, but it was local and in most areas there wasn't much of it happening.
    - cdchn 2y ago
      But accessing the PSTN network still has a cost. Simply having a cost is not a blocker for bad actors.
      - SoftTalker 2y ago
        Yes -- the key is a marginal cost for each call that is higher than what they earn on average per call. And that would not be a lot.
        cdchn 2y ago

userbinator 2y ago

I created a new hostname in DNS If you’re relying on obscurity Presumably the former caused your new hostname to be published for all of the Internet to see? That doesn't sound like obscurity to me. I've had a service running on a high port for many years at the same IP. I've seen it get the occasional "knock" from some scanner or bot, but it has been generally quiet. It probably also depends on your IP, as some parts of the Internet are likely scanned far more frequently and aggressively than others.

kstrauser 2y ago
It's more likely that it was the Let's Encrypt cert I requested; see https://news.ycombinator.com/item?id=40385643 https://news.ycombinator.com/item?id=40385643 for more.

TZubiri 2y ago

Imagine falling for that lol > having an all purpose 1000kloc http server. > serving your source code root > not using any permissions system

ta988 2y ago

Yes that's my experience too just a few seconds anytime I create a certificate with letsencrypt

gmuslera 2y ago

Having a webserver listening on ipv4 may get something like that sooner or later. All the tine you get vulnerability scanning by different actors and goals, be aware of that or not. With hostnames is just another layer, that may have more requirements, but the motivations are similar.

jmward01 2y ago

What is missing in our society to actually deal with this? I see stories like this and get the impression that it is a forgone conclusion that it is OK for this to be normal. I realize the author isn't saying that, and I appreciate their warning and think it is great they posted this, but where is the call to action to deal with this? And I don't mean we should all just get stronger safeguards. The people doing this type of thing are causing harm on a large scale so how do we get society to recognize and start really caring about this to the point that it wouldn't be a joke to call some law enforcement agency to tell them you are being hacked?

cdchn 2y ago
Corruption and the cultural acceptance of being a bad actor against groups different than yourself in certain societies.
kstrauser 2y ago
It costs time and money to run those scans, and people wouldn’t be doing it if it wasn’t profitable. I don’t want to blame the victims, but the root is that people put insecure setups online frequently enough to keep the criminals interested. We could require a license or certification to launch a server but that’d be even worse. I don’t have an answer. This normal isn’t OK. I don’t know what to do about it though.
ryandrake 2y ago
There's nothing missing from society. This is the natural state of humanity, and one always must be vigilant. The only thing keeping thousands of people from trying every home's physical front door every 5 seconds to see if it's unlocked is that it's not scalable. If it were possible, it would be happening. AND, if it did start happening, it would not be a law enforcement problem unless law enforcement became equally scalable. This is not a popular opinion and it often gets dunked on reflexively, but I truly believe the default is low-trust. We are one evolutionary step away from chimpanzees, with barely enough "society" grafted on to our world, which mostly doesn't function. You need to assume you are under potential attack at all times and be prepared to defend against it. Whether it be a physical attack, social engineering, scams, advertisements... in cyber-space and in meat-space. Any other security posture, long term will eventually open you up to a breach by an adversary.

robertclaus 2y ago

I've also had my hosting provider run those checks to proactively warn me about vulnerabilities.

dilyevsky 2y ago

These days certificate transparency logs seems to be a trigger to immediately get slammed with a ton of scanners. Some “legit” commercial (eg paloaltonetworks) some russian/chinese

andrewstuart2 2y ago
Yep. CTL is an important feature but it's definitely also a liability.

darepublic 2y ago

But wouldn't obscurity beat these attacks. I mean aren't they crafted only to explore common weaknesses and not esoteric weird unique set ups?

dilyevsky 2y ago
Yes it would but we dont like to talk about it =) Although if enough people start doing it my guess is the scanners will begin knocking on more than just :22,80,443

MBCook 2y ago

Some of these I get and are obvious. Some look like some kind of exploit maybe? And then there’s .DS_Store. What’s the point of that? In case you find a Mac to launch more targeted attacks against known bugs? To know if the developer is in a Mac and just copied files without filtering out dot files?

dilyevsky 2y ago
This one is probably to find if there’s a publicly exposed smb partition or something

billy99k 2y ago

I was working on bug bounty work a few years ago for a large company with lots of subdomains in different parts of the world. I found .git/config on a server and I was able to partially reconstruct the entire git repository. This led me to paths that I wouldn't otherwise be able to find and a complete server takeover through remote code execution. One of their developers left test code for a website template and an unrestricted file upload form.

mchenier 2y ago

Port knocking may be a first line defense here with a port scan attack detector to ban IPs that try to find such ports. See Linux knockd and psad for references. This obscurity doesn’t protect again man-in-the-middle but at least protects from unwanted and opportunistic guests. It also gives more time to indirectly protect from 0-day on sshd (aka the fiasco that could have been the xz incident).

pronoiac 2y ago

I've used https://crt.sh https://crt.sh to check my own logged SSL certs before, though it looks like they're seeing issues right now.

eqvinox 2y ago

How many of these accesses were from source IPs owned by Tencent? (this is not a prejudice; >80% of auto-bans my servers are issuing are for Tencent IPs. I should grab some exact numbers at some point.)

abimaelmartell 2y ago

I always wondered who is behind these attacks, they don't seem targeted since they do them on random ips. I did a bunch of Devops a few years ago on a Startup, and whenever i started a new AWS EC2 instance, i started getting request for Wordpress files, and other common CMS files.