7 ms·
An interesting thing that I've noticed is that some of the attackers watch the Certificate Transparency logs for newly issued certificates to get their targets.
by pferde 3y ago
An interesting thing that I've noticed is that some of the attackers watch the Certificate Transparency logs for newly issued certificates to get their targets.
I've had several instances of a new server being up on a new IP address for over a week, with only a few random probing hits in access logs, but then, maybe an hour after I got a certificate from Let's Encrypt, it suddenly started getting hundreds of hits just like those listed in the article. After a few hours, it always dies down somewhat.
The take-away is, secure your new stuff as early as possible, ideally even before the service is exposed to the Internet.
- elashri 3y agoIt is also useful to rely more on wildcard certs, as it makes it difficult to determine from CT logs the specific subdomains to attack.
- bombcar 3y agoThere's really no reason to avoid wildcard certs for your domains, unless you have so many subdomains that are managed by various business interests. I use LE wildcard certs and they're great, you can use them internally.
- couchand 3y agoIt seems like the principle of least power would apply here. There's value in restricting capability to no more than strictly necessary. Consider the risk of a compromised some-small-obscure-system.corporate.com in the presence of a mission-critical-system.corporate.com when both are issued wildcard certs. Wildcard certs are indeed a valuable tool, but there is no free lunch.
- baobun 3y agoYou'd usually put a reverse proxy exposing the services and terminating TLS with the wildcard cert. The individual services can still have individual non-wildcard internal-only certs signed by an internal CA. These don't need to touch an external CA or appear in CT logs - only the reverse proxy/proxies should ever hit these, and can be configured to trust the internal CA (only) explicitly.
- nullindividual 3y agoA compromised wildcard certificate has a much higher potential for abuse. The strong preference in IT security is a single-host or UCC (SAN) certificate. Renewing a wildcard is also unfun when you have services which require a manual import.
- dfc 3y agoRenewing any certificate that requires a manual import is not fun. Why are wildcard certs less fun to manually import than individual certificates?
- nullindividual 3y agoPresumably one purchases a wildcard for multiple distinct systems.
- dfc 3y agoUsing them like that never occurred to me. I was thinking multiple sites on one host or vanity hostnames: dfc.example.com / nullindividual.example.com. etc.
- nullindividual 3y agoUnless you're running some sort of automated system to churn out vanity host names (like an Azure or AWS would to provide you an OOTB URI), a UCC/SAN cert is a better choice. More restrictive is better than less restrictive when it comes to certificates.
- stefandesu 3y agoYeah, I switched to wildcard certs at some point for this reason.
- maccam912 3y agoSame! As soon as a new cert is registered for a new subdomain, I get a small burst of traffic. It threw me off at first assuming I had some tool running that was scanning it.
- supriyo-biswas 3y agoThese aren't attackers - they're usually services like urlscan.io and others who crawl the web for malware by monitoring CT logs.
- joshspankit 3y agoThe thread is specifically talking about logs of attacks
- fulafel 3y agoThe message they responded to was not, it was: "I've had several instances of a new server being up on a new IP address for over a week, with only a few random probing hits in access logs, but then, maybe an hour after I got a certificate from Let's Encrypt, it suddenly started getting hundreds of hits"
- pferde 3y agoWhat I wrote was ".. hundreds of hits just like those listed in the article ...", and the article listed attacks.
- heywoodlh 3y agoWas looking into Certificate Transparency logs recently. Are there any convenient tools/methods for querying CT logs? i.e. search for domains within a timeframe Cloudflare’s Merkle Town[0] is useful for getting overviews, but I haven’t found an easy way to query CT logs. ct-woodpecker[1] seems promising, too [0] https://ct.cloudflare.com/ https://ct.cloudflare.com/ [1] https://github.com/letsencrypt/ct-woodpecker https://github.com/letsencrypt/ct-woodpecker
- j0hnyl 3y agohttps://certstream.calidog.io/ https://certstream.calidog.io/
- H8crilA 3y agohttps://crt.sh/ https://crt.sh/
- simonw 3y agoSteampipe have a fun SQLite extension that lets you query them via SQL: https://til.simonwillison.net/sqlite/steampipe#user-content-running-extensions-in-datasette https://til.simonwillison.net/sqlite/steampipe#user-content-... It uses an API provided by https://crt.sh/ https://crt.sh/
- high_priest 3y agoQuerying crt.sh helped me identify a dev service I was supposed to take down, but forgot about it. Nice alternative use case :D
- KronisLV 3y ago> The take-away is, secure your new stuff as early as possible, ideally even before the service is exposed to the Internet. Honestly it feels like you'll need at least something like basicauth in front of your stuff from the first minutes it's publicly exposed. Well, either that, or run on your own CA and use self-signed certs (with mTLS) before switching over. For example, when some software still has initial install/setup screens where you create the admin user, connect to the DB and so on, as opposed to specifying everything initially in the environment variables, config files, or other more specialized secret management solutions.
- p_l 3y agoGenerally I'd recommend not exposing anything unless you deployed the security for it. Just SSH scanning can be a big issue.
- psanford 3y agoA big issue how? If you block password auth, ssh scanning is a nonissue.
- aunderscored 3y agoUnless an attack on the sshd is employed. Which is also possible
- psanford 3y agoPre-auth sshd vulnerabilities are extremely rare and are not what ssh scanners are looking for.
- bradknowles 3y agoDDoS attack on your sshd?
- indigodaddy 3y ago
- pkulak 3y agoI host so many services, but I gave up totally on exposing them to the internet. Modern VPNs are just too good. It lets me sleep at night. Some of my stuff is, for example, photo hosting and backup. Just nope all the way.
- justaj 3y agoIf you're the only one accessing those services, then why use a VPN instead of port mapping those services to localhost of the server, and then forwarding that localhost port to your client machine's localhost port via SSH?
- pkulak 3y agoI didn’t understand any of that, sorry. Haha A VPN lets me access my stuff from my phone while out of the house, for example.
- justaj 3y agoThis might clear up a few things then: https://web.archive.org/web/20220522192804/https://www.dbsysnet.com/how-to-access-a-linux-server-behind-nat-via-reverse-ssh-tunnel/ https://web.archive.org/web/20220522192804/https://www.dbsys... Original article which doesn't contain the first graphic: https://www.xmodulo.com/access-linux-server-behind-nat-reverse-ssh-tunnel.html https://www.xmodulo.com/access-linux-server-behind-nat-rever...
- redleader55 3y agoI am in the same situation with the grandparent. I don't even expose the SSH port to the outside. The only port open is the UDP port of Wireguard which allows only the packets signed by the correct key. Everything works perfectly, no issues with NAT, I even give my mobile devices an IPv6 that my ISP allocates. Tunneling through SSH is significantly worse because you encapsulate a TCP connection inside a TCP connection and it's userspace.
- nemothekid 3y agoFun anecdote - I wrote a new load balancer for our services to direct traffic to an ECS cluster. The services are exposed by domain name (e.g. api-tools.mycompany.com), and the load balancer was designed to produce certificates via letsencrypt for any host that came in. I had planned to make the move over the next day, but I moved a single service over to make sure everything was working. Next day as I'm testing moving traffic over, I find that I've been rate limited by Lets Encrypt for a week. I check the database and I had provisioned dozens of certificates for vpn.api-tools.mycompany.com, phpmyadmin.api-tools.mycompany.com, down the list of anything you can think of. There was no security issue, but it was very annoying that I had to delay the rollout by a week and add a whitelist feature.
- feitingen 3y agoI'm still getting crawlers looking for an old printer i got a letsencrypt certificate for.
- throwbadubadu 3y ago> The take-away is, secure your new stuff as early as possible, ideally even before the service is exposed to the Internet. What? Ideally..before? Seriously? It is 2024.. and this was true even decades ago, absolutely mandatory. (Still remembering that dev that discovered file sharing in his exposed mongo instance (yes, that!! :D) without password only hours after putting it up.. "but how could they know the host it is secret!!" :D ).
- kafrofrite 3y agoI work as a security engineer and, yes, the CT logs are extremely useful not only for identifying new targets the moment you get a certificate but also for identifying patterns in naming your infra (e.g., dev-* etc.). A good starting point for hardening your servers is CIS Hardening Guides and the relevant scripts.
- illiac786 3y agoOn censys.io you can search by domain for example. Some internet facing appliances generate certificate automatically with letsencrypt but use a central DNS server, meaning every one of these appliances is on the same domain, using random Subdomains. Once you figured out what the domain is, you can easily build a list of IPs out of the cert transparency log and if there is ever an exploit for this specific type of appliances, attackers now have a bespoke list of IPs to hack, a dream come true. I don't see a solution for this particular use case, I would argue self signed certs would be more secure in this case.