46 ms·

Encrypted traffic interception on Hetzner and Linode targeting Jabber service

731 points by f311a 3y ago

kristjank 3y ago
This really wants me to re-evaluate all the times I have gracefuly TOFU'd while SSHing into a machine or even logging onto a website.
- lukevp 3y ago
  What does TOFU mean in this context? I haven’t heard that word related to tech before OpenTofu just recently!
  - holmesworcester 3y ago
    Trust On First Use
  - deleted 3y ago
    [deleted]
  - sirl1on 3y ago
    Trust On First Use. No relation to OpenTofu where Tofu is a word play in TerraForm.
  - mmh0000 3y ago
    https://en.wikipedia.org/wiki/Trust_on_first_use https://en.wikipedia.org/wiki/Trust_on_first_use
- marcosdumay 3y ago
  Given that the attack subverted the TLS certificates, I'd say key-based TOFU is still a better alternative. Of course, checking the server key is better than not doing so, but any knee-jerk response to TOFU will probably make your security worse.
- rodlette 3y ago
  Consider including SSH known hosts in your cfgmgmt. I have all my machines' public keys, and all major git forge's SSH keys in my git-managed cfgmgmt repo. For NixOS users: https://search.nixos.org/options?channel=23.05&show=programs.ssh.knownHosts https://search.nixos.org/options?channel=23.05&show=programs...
  - ehPReth 3y ago
    one issue is that most hosting providers don't display ssh host keys anywhere in the control panels so unless you have a... lot of fun (and spend time) with the virtual console... (or maybe a custom cloud-init script?) you're SOL for the very very first connection
    - rudasn 3y ago
      Isn't more important that the known key does not change unless you made the change, than having the "right" key to begin with?
      - ehPReth 3y ago
        hmmmm... if someone MITMs you the first thing could they 'in theory' just change the key on the host to match the MITM one?
- 0xbadcafebee 3y ago
  At DEFCON, fifteen years ago, MITM'ing TOFU ssh connections was what you did when you were bored in between talks. If you wanted to capture a new first connection, just keep dropping connections until they try a different host or device. I don't really get why people use ssh at all if they aren't using certs. Might as well use telnet.
- thriftwy 3y ago
  An unknow key when connecting to a known server will raise all kinds of alarms in my head.
LinuxBender 3y ago
Archive [1] Their theories are interesting and would explain how they obtained the certs if true. Perhaps the take-away here is to have multiple probes on multiple providers checking all of ones TLS fingerprints and alerting on unknown fingerprints and then checking the certificate transparency log or lack thereof. openssl s_client -servername news.ycombinator.com -connect news.ycombinator.com:443 < /dev/null 2>/dev/null | openssl x509 -fingerprint -noout -in /dev/stdin SHA1 Fingerprint=7E:49:BA:40:86:87:B3:39:66:93:94:9E:9C:45:71:85:3C:8D:95:16 [1] - https://archive.ph/C0jYJ https://archive.ph/C0jYJ [updated]
- ValdikSS 3y ago
  I've added "Could you prevent or monitor this kind of attack?" section. There are several indications which could be used to discover the attack from day 1: * All issued SSL/TLS certificates are subject to certificate transparency. It is worth configuring certificate transparency monitoring, such as Cert Spotter (source on github), which will notify you by email of new certificates issued for your domain names * Limit validation methods and set exact account identifier which could issue new certificates with Certification Authority Authorization (CAA) Record Extensions for Account URI and Automatic Certificate Management Environment (ACME) Method Binding (RFC 8657) to prevent certificate issue for your domain using other certificate authorities, ACME accounts or validation methods * Monitor SSL/TLS certificate changes on all your services using external service * Monitor MAC address of default gateway for changes
  - LinuxBender 3y ago
    Nice addition. Me personally being the paranoid type, I don't trust the transparency log for monitoring but rather for writing up the root cause analysis. Reason being is that if someone can legally compel Hetzner and/or Linode and/or LetsEncrypt to do or not do something then the same entity can compel the certificate transparency site to ignore something. But you covered what I would do and that is to have multiple nodes doing active monitoring of TLS changes using an external service. That service being openssl s_client in my case. In the case of Jabber it might be interesting to add some monitoring in the application that uses a cryptographic signed payload to list all the valid fingerprints and send an alert message to the server if something odd is happening like public key pinning but without a hard-fail. That list could be pre-loaded with new certificate fingerprints prior to being deployed. If the oddness is confirmed then perhaps add some way to tell the clients that certificate is likely forged. That way both server operator and client are aware that something evil this way comes. [Edit] - Looks like based on the comment from MattJ100 that some Jabber servers and some clients can already do something like this.
throwaway77384 3y ago
This is baseless speculation, but I'm assuming Jabber is being targeted as it's famously used on darknet markets for drug trades (or other illicit activity). Goes to show that you should never just trust "it's encrypted, bro". You need to PGP your messages at the very least. Is PGP crackable by quantum computers? Will there be hardening against those kinds of attacks in the future? Since, if the messages have been hoovered up in encrypted form, it's just a matter of time until they get decrypted. And this appears to be done for just about all web traffic they can get their hands on... see https://en.wikipedia.org/wiki/Utah_Data_Center https://en.wikipedia.org/wiki/Utah_Data_Center
- ndriscoll 3y ago
  Jabber/XMPP has had e2e encryption for at least like 10-15 years. I used to use it with even my normie friends back when Facebook/Google Talk supported XMPP and you could use pidgin, kopete, etc. Obviously securely exchanging keys with an anonymous drug dealer over the Internet is error-prone though...
- ValdikSS 3y ago
  This mitm may be as a part of Genesis Market takedown, but it's just out of the blue. https://therecord.media/genesis-market-takedown-cybercrime https://therecord.media/genesis-market-takedown-cybercrime
- theamk 3y ago
  A second layer of encryption would help, but I don't recommend PGP in particular. If you haven't heard, it has lots of problems and a lot of people recommend avoiding it (for example https://www.latacora.com/blog/2019/07/16/the-pgp-problem/ https://www.latacora.com/blog/2019/07/16/the-pgp-problem/ / https://news.ycombinator.com/item?id=20455780 https://news.ycombinator.com/item?id=20455780)
  - upofadown 3y ago
    "The PGP Problem" is generally misleading and is straight out wrong in some places. I ended up writing an article to save time: * https://articles.59.ca/doku.php?id=pgpfan:tpp https://articles.59.ca/doku.php?id=pgpfan:tpp PGP certainly has its problems, but isn't really special compared to other similar things. The big advantage that PGP has is that it is a stable and well known standard. There is a tendency to imply that it is insecure in some way, but no real evidence seems to exist to that effect.
zhovner 3y ago
>The attacker managed to issue multiple SSL/TLS certificates via Let’s Encrypt for jabber.ru and xmpp.ru domains since 18 Apr 2023 Why is it even possible to issue more than 1 certificate on the same domain via Let’s Encrypt? Shouldn't the previous certificate be revoked when a new one is issued?
- MattJ100 3y ago
  It's fairly common for people to obtain multiple certificates for different machines or services, so they can be selectively revoked and they don't have to share keys across machines. More use-cases: - You might obtain a new certificate, but deploy it gradually, so you want the old one to remain valid while you do that. - One certificate may cover different sets of domain names. If you have a certificate for "example.com, foo.example.com" and then request a certificate for only "foo.example.com", should the earlier one be revoked? (leaving "example.com" without a certificate).
- aaomidi 3y ago
  You should always have more than a single certificate for your domain honestly. Cloudflare for example, tries to optimize certificate delivery (and have backup certificates available for you just in case a CA needs to revoke theirs). Also, on distributed systems its less safe to share private keys between the various frontends.
- blibble 3y ago
  provisioning a 2nd machine into your webserver cluster before activating it?
  - KMag 3y ago
    You could sync certificates across hosts for this purpose, though. The advantage of multiple certificates is being able to revoke a subset of certificates if you can determine only a subset of your hosts have been compromised.
    - blibble 3y ago
beeboobaa 3y ago
Are there any services that monitor the certificate transparency logs and send an email when a new certificate is issued for your domain? That would alert you to this kind of MITM
- detourdog 3y ago
  Kerberos right?
- jauntywundrkind 3y ago
  > Could you prevent or monitor this kind of attack? There are several indications which could be used to discover the attack from day 1: > All issued SSL/TLS certificates are subject to certificate transparency. It is worth configuring certificate transparency monitoring, such as Cert Spotter (source on github), which will notify you by email of new certificates issued for your domain names
  - radlad 3y ago
    > All issued SSL/TLS certificates are subject to certificate transparency. It is worth configuring certificate transparency monitoring, such as Cert Spotter (source on github), which will notify you by email of new certificates issued for your domain names This seems like the kind of thing domain registrars should do for you.
    - Operyl 3y ago
      Who could in theory also be told “don’t notify your client about this specific one”. You’re best off self hosting if you think you’d be a target for these kind of attacks.
      - ratsmack 3y ago
        It seems that trust is a highly variable quantity, maybe similar to intuition.
- tptacek 3y ago
vb-8448 3y ago
> The attacker managed to issue multiple SSL/TLS certificates via Let’s Encrypt for jabber.ru and xmpp.ru domains since 18 Apr 2023 > We tend to assume this is lawful interception Hetzner and Linode were forced to setup based on German police request. > Another possible, although much more unlikely scenario is an intrusion on the internal networks of both Hetzner and Linode targeting specifically jabber.ru — much harder to believe but not entirely impossible. And what if the attacker tricked somehow the letsencrypt challenges? Or this is supposed to be impossible?
- janosdebugs 3y ago
  If that were the case, they wouldn't have seen weird connectivity issues/behaviors.
  - vb-8448 3y ago
    why not? They discover the issue only because the attacker failed to renew the certificates.
    - janosdebugs 3y ago
      Yeah, but just getting a hold of a valid certificate doesn't automatically mean a MITM. For that the network connections need to be intercepted too.
- nmjohn 3y ago
  The attacker effectively controlled the IP the domain was pointed to. If you have this, getting a cert issued from any CA is trivial - you've proved to them you control the domain in question.
  - 3np 3y ago
    As mentioned elsewhere in the thread, RFC 8657 can prevent this. https://news.ycombinator.com/item?id=37958831 https://news.ycombinator.com/item?id=37958831
- 3y ago
upofadown 3y ago
>End-to-end encrypted communications, such as OMEMO, OTR or PGP, are protected from the interception only if both parties have validated the encryption keys. The users are asked to check their accounts for new unauthorized OMEMO and PGP keys in their PEP storage, and change passwords. The attacker would need to set up a separate MiTM for the particular E2EE scheme used. Some of the XMPP clients I have encountered will not let you use a particular cryptographic identity unless you have explicitly claimed to have verified it. Still a good reminder. If you have not seen and dealt with a ridiculously long number (or equivalent) you have not achieved end to end.
Andrew_nenakhov 3y ago
I wonder if there is a legal way to demand Hetzner/Linode comments on this situation. Likely, the entity behind the interception is some government agency or police.
- elliewithcolor 3y ago
  For Hetzner as German Company: If it is legal interception no. You can’t demand an answer, without going to a lawyer first. On the other hand, if it is not lawful interception, I doubt Hetzner would allow it. Because that’s also against law.
- baybal2 3y ago
  [dead]
- dathinab 3y ago
  there isn't even any guarantee that the wiretapping was done through them instead of e.g. the carriers which pretty much in any country have since decades been forced to help with lawful wiretapping...
  - chatmasta 3y ago
    Carriers meaning the interconnect providers eg Level3, Cogent etc? How would this intercept be implemented in practice? Surely it'd be much easier to add a node as close as possible to the origin host, i.e. within the Hetzner network, rather than redirecting traffic from the outside with some sort of BGP hijack?
    - dathinab 3y ago
      any carrier on any level but like other have pointed out this seems to have been in the hetzner network through wire taping laws also extend to datacenter internal interconnects I mean servers of different people can communicated with each other without the traffic leaving the server so it kinda makes sense
rodlette 3y ago
I agree with most of the mitigation suggestions. For high risk targets, consider layering an additional auth mechanism that doesn't rely on trusted CAs: Tor onion services, SSH, or Wireguard. > All issued SSL/TLS certificates are subject to certificate transparency. +1. crt.sh has RSS feeds for this. > Limit validation methods and set exact account identifiers Using CAA is a good idea in general, but would it help in this case? The attacker would just request the exact cert configuration that is permitted by CAA. Maybe this helps if you can strengthen one validation method? > Monitor SSL/TLS certificate changes on all your services using external service +1. High-risk targets should be aware of what certs are valid at any time, and be checking for those. > Monitor MAC address of default gateway for changes A more sophisticated attack could preserve the MAC address. > "Channel binding" is a feature in XMPP which can detect a MiTM even if the interceptor present a valid certificate. TIL.
- hlandau 3y ago
  >Using CAA is a good idea in general, but would it help in this case? The attacker would just request the exact cert configuration that is permitted by CAA. Maybe this helps if you can strengthen one validation method? Author of ACME-CAA (RFC 8657) here. ACME-CAA can mitigate this because you can put a unique identifier for your ACME account in the CAA record, so it is not possible for an attacker to do this unless they can get your ACME account private key (or coerce the ACME service). This assumes you have DNSSEC-secured nameservers, of course, otherwise DNS requests can potentially also be intercepted when queried by the CA. See RFC 8657 for a full list of security caveats. The RFC is designed to be more readable than most. Blogpost by me with more background: https://www.devever.net/~hl/acme-caa-live https://www.devever.net/~hl/acme-caa-live
  - aaomidi 3y ago
    Yep! This is one way of preventing this type of attack.
  - crosser 3y ago
    Could you please elaborate the reasons to implement this outside of DANE (RFC-7671) framework?
zxwrt 3y ago
What a nefarious move by Hetzner and Linode. How to trust them after this?
- Nextgrid 3y ago
  If it's a "lawful intercept" then they most likely have no choice.
- dathinab 3y ago
  first the article jumped the gun when saying it's by Hetzner/Linode, as it as much could have been done by the carrier the data-center connects to and from what I know about German espionage/wiretapping by police law that would be far more likely second pretty much _every_ country has laws which require carriers to help them wiretap in case of an investigation with appropriate court orders thirdly even if it went through Hetzner/Linode instead of carriers it wasn't done "a move from them" but something they where legally binding ordered to silently tolerate lastly if as unlikely as it seems it was not lawful interception of police or similar then they (or a carrier) would have been hacked, i.e. there is absolutely no chance that server providers will do such an attack on their own violation, especially Hetzner (they also don't have the legal means to get the necessary certificates)
  - cassonmars 3y ago
    That "lawful" interception allows certificate issuance to be a means of wiretap completely undermines any trust one should have in CAs. It seems that an alternative is greatly needed.
  - scraptor 3y ago
    The fact that they were legally required to MITM their customers does not make them more trustworthy (in the sense of unlikely to do it in the future), just the opposite! Of course that applies equally to any other cloud host (modulo jurisdiction games) but that does little to restore my interest in running my software on other peoples computers.
    - 3y ago
wutwutwat 3y ago
As an end user, how can I monitor if I’m being MITM’d? If everything I do is proxied through an attacker’s network, nothing I do online could be trusted to properly test if I were being proxied, right? I know apps can do cert pinning, but as an end user how can I validate that when connect to anything that it’s the real one and not a request from being middled?
- LinuxBender 3y ago
  One can get fingerprints of TLS endpoints ahead of time and/or from a known clean network then compare and contrast to what you see from the location you suspect might be MITM'd. I believe there are browser addons that display certificate fingerprints and alert if a certificate changes however one can click on the lock symbol and drill down to the same info. openssl s_client -servername news.ycombinator.com -connect news.ycombinator.com:443 < /dev/null 2>/dev/null | openssl x509 -fingerprint -noout -in /dev/stdin SHA1 Fingerprint=7E:49:BA:40:86:87:B3:39:66:93:94:9E:9C:45:71:85:3C:8D:95:16 A higher friction method would be to use a browser addon that would pin a certificate to a site. Not useful in the case of the Jabber client but useful if one was visiting Jabbers website to validate something. Some Jabber clients and servers have their own method described in another part of this thread [1] to protect against interception. The openssl s_client method is likely the most versatile for testing from different locations, ports and applications. [1] - https://news.ycombinator.com/item?id=37956911 https://news.ycombinator.com/item?id=37956911
- polski-g 3y ago
  https://github.com/matteocorti/check_ssl_cert https://github.com/matteocorti/check_ssl_cert
costco 3y ago
This is almost certainly related to some sort of Russian cybercrime investigation. If you ever read Krebs or peruse some of the seedier Russian forums, xmpp.ru will sound familiar to you, because it is. Not imputing anything to the operator, that's just the nature of operating an anonymous service. https://ddanchev.blogspot.com/2021/03/exposing-currently-active-portfolio-of_82.html https://ddanchev.blogspot.com/2021/03/exposing-currently-act... https://ddanchev.blogspot.com/2019/07/profiling-currently-active-portfolio-of.html https://ddanchev.blogspot.com/2019/07/profiling-currently-ac... https://blog.talosintelligence.com/picking-apart-remcos/ https://blog.talosintelligence.com/picking-apart-remcos/ https://flashpoint.io/wp-content/uploads/Plea-Agreement-USA-v.-Vitalii-Chychasov.pdf https://flashpoint.io/wp-content/uploads/Plea-Agreement-USA-... Really interesting writeup though. I guess it's a practical example of why everyone should get a CT monitoring service!
- ValdikSS 3y ago
  Yes, I tend to believe that as well. My guess it's either related to Genesis Market or Qakbot takedowns.
- orbital-decay 3y ago
  Drug gangs widely use XMPP as a secure communication channel in combination with Tor. Hydra (the darknet market) has been taken down by German police in 2022 after trying to extend their "service" to the EU; something similar might or might not have happened here.
  - codedokode 3y ago
    [flagged]
    - _lvbh 3y ago
      Telegram? Not Signal or Matrix?
      - toastal 3y ago
lima 3y ago
Perfectly pulling off an actual MitM attack and then forgetting to renew the certificate is certainly a very German thing :-)
- jakub_g 3y ago
  I wonder if someone didn't "forget" on purpose, so that people learn about it. Or, someone very diligently followed the orders - there was an order to set up a cert, but there was no requirement that it has to auto-renew :)
  - luma 3y ago
    It was claimed to have been running for 6 months so it must have renewed certs at least once - LE certs are good for 90 days.
    - a1369209993 3y ago
      > 6 months [...] 90 days. So they were told to renew the certificate, but not how many times to renew it?
      - JPHPJ 3y ago
        Shouldn't the certificate transparency logs have it?
        tgsovlerkhgsel 3y ago
        I suspect it's not trivial to distinguish between the legit and fake ones just based on CT logs. Unless Let's Encrypt publicly logs the account used to issue the certificate (I think they don't), only the logs held at Let's Encrypt will reveal this information. I expect their security team to be looking at those logs right now.
        pgporada 3y ago
deleted 3y ago
[deleted]
mak234 3y ago
so is jabber.ru still secure now? what needs to be done on the user side to make it so? can one reissue/request new server certificate from the server and not from let's encrypt? thanks
- zaik 3y ago
  If you have used OMEMO or OpenPGP to encrypt your messages, the content of your messages is secure, provided you have verified the public keys of your contacts. If you didn't verify the keys, you should do it now and check that no suspicious keys have been injected.
dathinab 3y ago
> We tend to assume this is lawful interception Hetzner and Linode were forced to setup based on German police request. This seems to be somewhat jumping the gun I think. Given the certs and the target assuming it's an lawful interception seems reasonable. But there is nothing there which requires Hetzner or Lindoe complying or knowing about this. Given the nature of the attacks you can do the interception on the carrier level, and carriers being forced to comply with lawful wiretapping is pretty much anywhere in the world pretty much standard and many laws are based around that approach. Much less so around approaches involving data centers.
- Hello71 3y ago
  according to the article, the added hop visible in traceroute to Linode machine is after two 10/8 hops, so it's likely an internal machine. more importantly, the affected VM has a strange gateway MAC address, which can only be controlled by the local network administrator, not an intermediate carrier.
Libre___ 3y ago
I was a bit curious about the MAC address since it isn't locally administered which indicates to me that someone likely deliberately chose this range of addresses. https://www.google.com/search?q=%2290%253Ade%253A01%22+linode https://www.google.com/search?q=%2290%253Ade%253A01%22+linod... Googling for '"90:de:01" linode' and '"90:de:00" linode' indicates that addresses from these blocks have been assigned to other linode VMs recently. I sadly don't have a linode VM of my own to compare with right now but it would seem like the traffic has been routed to another VM on linode infrastructure
0xbadcafebee 3y ago
Getting a valid cert for a domain you don't own is getting close to trivial. All of the mitigation strategies are defeated a number of ways, have been for years. I've commented on incidents like these for years on HN. I'm not some internet big wig, and any blog I write isn't going to trend, so this never gets traction. But anyone who understands the tech can find a half dozen ways to get a cert. If you really need to trust your connection, don't rely on internet PKI / public certs. It was secure enough for general purpose in the aughts, it's not anymore.
- dewey 3y ago
  What are these trival steps to get a valid certificate for news.ycombinator.com?
  - escalt 3y ago
    this is left as an exercise for the reader
  - LinuxBender 3y ago
    This would not be trivial and would require successful phishing. I've sent an email requesting to make this even harder to do. leaving out those details on purpose
- gouggoug 3y ago
  Bold claim lacking substantiation. Care to share? Maybe it will get traction this time.
  - thriftwy 3y ago
    If you are your hoster and can reroute requests, you can get all kinds of valid SSL certs for any domains that you host. Just grab these IP packets when CA comes to validate that you own that domain. Perhaps EV could solve that to some extent but it is never mandated. Even if you tried to put any stuff into WHOIS to mitigate this, your hoster can serve any bullshit on this channel too. It does look very bad and SSH approach to certificates is just infinitely better. If Jabber used SSL keys instead, they will be alerted immediately. Come to think of it, your hoster can also find ways to steal keys directly from hardware, though.
AtNightWeCode 3y ago
The way that certbot works with let's encrypt the only surprise is that this does not occur more often. We have monitoring on several certs used for TLS. We should probably add an alert if an A-record changes as well. I assume the root cause is DNS tricks.
- ilyt 3y ago
  > I assume the root cause is DNS tricks. Maybe read the article
hnarn 3y ago
http://archive.today/0R7Or http://archive.today/0R7Or
SergeAx 3y ago
And this is why, kids, you need to enforce certificate pinning on your critical infrastructure!
- LinuxBender 3y ago
  Public key pinning has been mostly abandoned on popular websites due to high risk of prolonged outages should someone make a mistake and mistakes do happen. There are browser addons that replicate this behavior however and some Jabber clients/servers apparently have their own mechanism for this [1] but one can only hope everyone is using the combination of servers and clients that support this feature. [1] - https://news.ycombinator.com/item?id=37956911 https://news.ycombinator.com/item?id=37956911
Traubenfuchs 3y ago
We are using self developed application layer security. It requires specialized client code and is terminated at a gateway that proxies requests to our normal applications. I really, really hate the idea of this kind of eavesdropping.
gigatexal 3y ago
This is a deal breaker for me. Which is a shame because Hetzner has such great pricing. Oh well.
- ComputerGuru 3y ago
  It's likely legally enforced, and any other provider would be subject to the same.
  - gigatexal 3y ago
    Fair enough
- LinuxBender 3y ago
  In fairness to them if it's a lawful intercept then they have no choice. Non compliance can shut down a business. Your only alternatives in that case would be the underground style hosters sometimes marketed as bullet-proof hosting though they are just shady resellers with bold claims. They do not typically last long. There are a couple I know of in Amsterdam that are right down the street from The Hague that have been around for a while so I suspect they are just honeypots.
codedokode 3y ago
Wouldn't be surprised if this happened in Russia; but not in Europe. Also, Let's Encrypt security is a joke - they issued fake certificate without serious checking. Using unencrypted HTTP for confirmation is a vulnerability - doesn't this mean that large national ISPs can easily issue ceritificates for any site hosted within a country?
- deleted 3y ago
  [deleted]
- ItsABytecode 3y ago
  > Also, Let's Encrypt security is a joke - they issued fake certificate without serious checking. Using unencrypted HTTP for confirmation is a vulnerability They can’t use HTTPS if you don’t have a certificate yet
  - codedokode 3y ago
    This doesn't mean you should use unsecure methods instead and issue certificates to anyone capable to do MitM.
    - blueflow 3y ago
      How could Letsencrypt even verify a server setup if not via DNS/HTTP? Also, verify against what? The servers are basically random strangers without identity when they first talk to LE.
      - codedokode 3y ago
        In this case there has been a valid certificate for the site; this alone should raise suspicion. Also, if they cannot do secure validation then maybe they should stop issuing certificates for sites that already have a proper certificate.
        blueflow 3y ago
wizzard0 3y ago
well i must admit monitoring certificate issuance with LetsEncrypt is quite boring even if you HAVE alerts set up (I do) so… not surprised. Still cool. What a time to be alive
whytevuhuni 3y ago
Could this be a blue pill attack? A vulnerability in the xmpp server exploited to inject a rootkit, which then hides itself inside the kernel? Or creates network/pid namespaces and puts you in them, while leaving the mitm server in the original one? If so, the mitm could be on the same host, and wouldn't need the cooperation of the hosting provider. I'm not sure how to check for either of these without restarting (which the admin does not seem to want to do, as it is a live service). https://en.wikipedia.org/wiki/Blue_Pill_(software) https://en.wikipedia.org/wiki/Blue_Pill_(software)
- codedokode 3y ago
  No because traffic redirection occured in provider's infrastructure, not on a server.
  - whytevuhuni 3y ago
    Let's say you are blue-pilled and what you see as your eth0 interface is actually a virtual interface controlled by the rootkit. In that case, where the redirection happened is no longer something you would be able to tell, right?
- londons_explore 3y ago
  If this had happened, the attacker would have likely stolen the servers TLS certificate and keys Whereas this attack generated new keys (and was detected!), suggesting the attacker didn't compromise the server itself.
riku_iki 3y ago
So, if I rent server in those services, it all potentially can be compromised and there is no way to make sure things are secure?..
- bananapub 3y ago
  ? did you not read the article?
  - riku_iki 3y ago
    I briefly read the article, maybe you can explain what is your point?
    - _flux 3y ago
      The point is probably the heading of the final chapter "Could you prevent or monitor this kind of attack?" which enumerates the ways how you could detect this attack.
      - riku_iki 3y ago
        lets say I opened account on hentzner, rented server, they sent me email with password, I SSH-ed to it and connection got wire tapped. What way you think could help me to detect this?
        _flux 3y ago
        Well you could check the SSH server fingerprint that it matches what the SSH client sees. In principle if the SSH connection is MitM'd, they could change the output from the shell so that it would look like it wouldn't be changed. But ultimately they created the keys to begin with, so they can just keep another copy for the purpose of MitM. I doubt they do this nowadays, but they _could_. edit: I actually thought this story was about virtual machines, but I noticed it was about physical hosts, so those have a bit more hope; but ultimately you're still trusting devices that are not physically under your control, so they cannot be trusted. But let's say that in the case of virtual hosts: basically there is no way to ensure you have non-wiretapped connection to a virtual computer provided by hosting service. They don't even need to MitM it, they can just look at the memory of the virtual machine. The only attempt at trying to do that properly has been AMD's "Secure Encrypted Virtualization" https://www.amd.com/en/developer/sev.html https://www.amd.com/en/developer/sev.html, but I think it was broken by some researchers and I don't know if it could currently provide some means to do this safely. I suspect even in that case it might be challenging to install the initial operating systems so that it won't contain the MitM functionality in itself. And even then you would be trusting AMD's implementation of the security layer. Frankly that the wiretap was discovered this time is just a learning experience for the provider to do better next time. I doubt there are good reasons why it should be possible to detect it, if implemented competently.
mrbonner 3y ago
I’m wondering if mTLS (aka zero-trust) is used, would take prevent this kind of MITM attack? My understanding is that it should since the cert is self-signed to be used from both ends.
snvzz 3y ago
A reminder e2ee is a must.
chgo1 3y ago
@dang: .org.ru should probably be treated as TLD
mermaidsalad 3y ago
Next time someone in Matrix discussion will ask "why we need Matrix when there is jabber/xmpp" show them this: > All jabber.ru and xmpp.ru communications between these dates should be assumed compromised. Given the nature of the interception, the attacker have been able to execute any action as if it is executed from the authorized account, without knowing the account password. This means that the attacker could download account's roster, lifetime unencrypted server-side message history, send new messages or alter them in real time.
- Arathorn 3y ago
  The same attack could be mounted against a Matrix server tbh. But the fact Matrix is E2EE by default would mitigate it a bit, and if folks verified who they were talking to then the attack would be defeated.
  - huggingmouth 3y ago
    Matrix clients typically complain very loudly when keys change or unverified sessions get added to your chats. Id's say it would mitigate it A LOT. Though I have to agree that there is still a lot of room for improvement. I personally prefer the approach taken by briar: your public key is your address. Pretty cut and dry, not much room for shenanigans.
mkup 3y ago
There's one thing I can't understand in this story: if that's lawful interception, why Hetzner and Linode bothered to set up MitM interception with different LE certificate and key, rather than extract the TLS private key directly from the RAM and/or storage device of the VPS? Even if this is a physically dedicated server, they can extract the private key from the RAM by dumping the RAM contents after unscheduled reboot. Extraction of the private key isn't visible in CT logs, much more stealthier, practically undetectable.
- tgsovlerkhgsel 3y ago
  Because it was easier, most likely. There's also a possibility that one would be a "search" and the other would be an "interception" with different levels of approvals requested, but I don't know what the current legal situation in Germany is right now.
- ValdikSS 3y ago
  Likely because it's 'more illegal'. I'd bet they are not allowed to hack into the server if it's not directly involved in the cybercriminal activity.
- xorcist 3y ago
  On a physical server, couldn't you just hotplug a PCIe card in there and DMA out any data you are interested in? Something like a network card with firmware specifically for the purpose should do it. It sounds so much a standard thing for law enforcement that I imagine such equipment should be available off the shelf?
hhsectech 3y ago
I've always thought a solution like LetsEncrypt would be an ideal backdoor for LEAs wanting to circumvent SSL/TLS. The same thing applies to VPN services. Why bother trying to develop tools etc that can decrypt traffic when you can just have people send their traffic to you?
- MaxBarraclough 3y ago
  Why are your concerns specific to LetsEncrypt? They're just another CA, no?
psvz 3y ago
let's encrypt is a public non-profit, who would definitely assist in finding out how domain control verification (DCV) happened - somehow that obvious line of pursuit was missing from the investigation. The mystery of the MAC 90:de:01:49:76:ae is interesting though, but CIA-custom-Made NIC might be a bit superfluous as an idea. Linode (acquired by Akamai) might use something in-house designed and OEM-manufactured, so I wouldn't treat it as strong evidence. Talk to Let's Encrypt - that will likely unveil what happened. Interestingly, search "90:de:01" in this page: https://www.cyberciti.biz/faq/howto-linux-configuring-default-route-with-ipcommand/ https://www.cyberciti.biz/faq/howto-linux-configuring-defaul... ^^ looks like another victim VM of the alleged mysterious interceptor :-)
olegarch 3y ago
Using DANE or so (RRs CAA/TLSA) can help in this case (when provider route traffic to specific port), but does not solve the problem completely. If DNS record will be compromised, then hacker can setup "correct" TLSA/CAA record to trust his fake certificate. As result, problem can be solved, if we will have reliable DNS subsystem, where is impossible to compromise DNS provider or infrastructure. I think, emerDNS can be used in such critical application: https://emercoin.com/en/documentation/blockchain-services/emerdns/emerdns-introduction/ https://emercoin.com/en/documentation/blockchain-services/em...
harrymit907 3y ago
A user reported this on Reddit 3 days ago: https://www.reddit.com/r/hetzner/comments/17ankoh/does_hetzner_run_a_proxy_in_front_of_my_server/ https://www.reddit.com/r/hetzner/comments/17ankoh/does_hetzn...
withinboredom 3y ago
Interesting reddit thread ending here: https://www.reddit.com/r/hetzner/comments/17ccp3i/hetzner_does_run_a_mitm_proxy_in_front_of_my/ https://www.reddit.com/r/hetzner/comments/17ccp3i/hetzner_do...

731 points by f311a 3y ago

This really wants me to re-evaluate all the times I have gracefuly TOFU'd while SSHing into a machine or even logging onto a website.

lukevp 3y ago
What does TOFU mean in this context? I haven’t heard that word related to tech before OpenTofu just recently!
- holmesworcester 3y ago
  Trust On First Use
- deleted 3y ago
  [deleted]
- sirl1on 3y ago
  Trust On First Use. No relation to OpenTofu where Tofu is a word play in TerraForm.
- mmh0000 3y ago
  https://en.wikipedia.org/wiki/Trust_on_first_use https://en.wikipedia.org/wiki/Trust_on_first_use
marcosdumay 3y ago
Given that the attack subverted the TLS certificates, I'd say key-based TOFU is still a better alternative. Of course, checking the server key is better than not doing so, but any knee-jerk response to TOFU will probably make your security worse.
rodlette 3y ago
Consider including SSH known hosts in your cfgmgmt. I have all my machines' public keys, and all major git forge's SSH keys in my git-managed cfgmgmt repo. For NixOS users: https://search.nixos.org/options?channel=23.05&show=programs.ssh.knownHosts https://search.nixos.org/options?channel=23.05&show=programs...
- ehPReth 3y ago
  one issue is that most hosting providers don't display ssh host keys anywhere in the control panels so unless you have a... lot of fun (and spend time) with the virtual console... (or maybe a custom cloud-init script?) you're SOL for the very very first connection
  - rudasn 3y ago
    Isn't more important that the known key does not change unless you made the change, than having the "right" key to begin with?
    - ehPReth 3y ago
      hmmmm... if someone MITMs you the first thing could they 'in theory' just change the key on the host to match the MITM one?
0xbadcafebee 3y ago
At DEFCON, fifteen years ago, MITM'ing TOFU ssh connections was what you did when you were bored in between talks. If you wanted to capture a new first connection, just keep dropping connections until they try a different host or device. I don't really get why people use ssh at all if they aren't using certs. Might as well use telnet.
thriftwy 3y ago
An unknow key when connecting to a known server will raise all kinds of alarms in my head.

LinuxBender 3y ago

Archive [1] Their theories are interesting and would explain how they obtained the certs if true. Perhaps the take-away here is to have multiple probes on multiple providers checking all of ones TLS fingerprints and alerting on unknown fingerprints and then checking the certificate transparency log or lack thereof. openssl s_client -servername news.ycombinator.com -connect news.ycombinator.com:443 < /dev/null 2>/dev/null | openssl x509 -fingerprint -noout -in /dev/stdin SHA1 Fingerprint=7E:49:BA:40:86:87:B3:39:66:93:94:9E:9C:45:71:85:3C:8D:95:16 [1] - https://archive.ph/C0jYJ https://archive.ph/C0jYJ [updated]

ValdikSS 3y ago
I've added "Could you prevent or monitor this kind of attack?" section. There are several indications which could be used to discover the attack from day 1: * All issued SSL/TLS certificates are subject to certificate transparency. It is worth configuring certificate transparency monitoring, such as Cert Spotter (source on github), which will notify you by email of new certificates issued for your domain names * Limit validation methods and set exact account identifier which could issue new certificates with Certification Authority Authorization (CAA) Record Extensions for Account URI and Automatic Certificate Management Environment (ACME) Method Binding (RFC 8657) to prevent certificate issue for your domain using other certificate authorities, ACME accounts or validation methods * Monitor SSL/TLS certificate changes on all your services using external service * Monitor MAC address of default gateway for changes
- LinuxBender 3y ago
  Nice addition. Me personally being the paranoid type, I don't trust the transparency log for monitoring but rather for writing up the root cause analysis. Reason being is that if someone can legally compel Hetzner and/or Linode and/or LetsEncrypt to do or not do something then the same entity can compel the certificate transparency site to ignore something. But you covered what I would do and that is to have multiple nodes doing active monitoring of TLS changes using an external service. That service being openssl s_client in my case. In the case of Jabber it might be interesting to add some monitoring in the application that uses a cryptographic signed payload to list all the valid fingerprints and send an alert message to the server if something odd is happening like public key pinning but without a hard-fail. That list could be pre-loaded with new certificate fingerprints prior to being deployed. If the oddness is confirmed then perhaps add some way to tell the clients that certificate is likely forged. That way both server operator and client are aware that something evil this way comes. [Edit] - Looks like based on the comment from MattJ100 that some Jabber servers and some clients can already do something like this.

throwaway77384 3y ago

This is baseless speculation, but I'm assuming Jabber is being targeted as it's famously used on darknet markets for drug trades (or other illicit activity). Goes to show that you should never just trust "it's encrypted, bro". You need to PGP your messages at the very least. Is PGP crackable by quantum computers? Will there be hardening against those kinds of attacks in the future? Since, if the messages have been hoovered up in encrypted form, it's just a matter of time until they get decrypted. And this appears to be done for just about all web traffic they can get their hands on... see https://en.wikipedia.org/wiki/Utah_Data_Center https://en.wikipedia.org/wiki/Utah_Data_Center

ndriscoll 3y ago
Jabber/XMPP has had e2e encryption for at least like 10-15 years. I used to use it with even my normie friends back when Facebook/Google Talk supported XMPP and you could use pidgin, kopete, etc. Obviously securely exchanging keys with an anonymous drug dealer over the Internet is error-prone though...
ValdikSS 3y ago
This mitm may be as a part of Genesis Market takedown, but it's just out of the blue. https://therecord.media/genesis-market-takedown-cybercrime https://therecord.media/genesis-market-takedown-cybercrime
theamk 3y ago
A second layer of encryption would help, but I don't recommend PGP in particular. If you haven't heard, it has lots of problems and a lot of people recommend avoiding it (for example https://www.latacora.com/blog/2019/07/16/the-pgp-problem/ https://www.latacora.com/blog/2019/07/16/the-pgp-problem/ / https://news.ycombinator.com/item?id=20455780 https://news.ycombinator.com/item?id=20455780)
- upofadown 3y ago
  "The PGP Problem" is generally misleading and is straight out wrong in some places. I ended up writing an article to save time: * https://articles.59.ca/doku.php?id=pgpfan:tpp https://articles.59.ca/doku.php?id=pgpfan:tpp PGP certainly has its problems, but isn't really special compared to other similar things. The big advantage that PGP has is that it is a stable and well known standard. There is a tendency to imply that it is insecure in some way, but no real evidence seems to exist to that effect.

zhovner 3y ago

>The attacker managed to issue multiple SSL/TLS certificates via Let’s Encrypt for jabber.ru and xmpp.ru domains since 18 Apr 2023 Why is it even possible to issue more than 1 certificate on the same domain via Let’s Encrypt? Shouldn't the previous certificate be revoked when a new one is issued?

MattJ100 3y ago
It's fairly common for people to obtain multiple certificates for different machines or services, so they can be selectively revoked and they don't have to share keys across machines. More use-cases: - You might obtain a new certificate, but deploy it gradually, so you want the old one to remain valid while you do that. - One certificate may cover different sets of domain names. If you have a certificate for "example.com, foo.example.com" and then request a certificate for only "foo.example.com", should the earlier one be revoked? (leaving "example.com" without a certificate).
aaomidi 3y ago
You should always have more than a single certificate for your domain honestly. Cloudflare for example, tries to optimize certificate delivery (and have backup certificates available for you just in case a CA needs to revoke theirs). Also, on distributed systems its less safe to share private keys between the various frontends.
blibble 3y ago
provisioning a 2nd machine into your webserver cluster before activating it?
- KMag 3y ago
  You could sync certificates across hosts for this purpose, though. The advantage of multiple certificates is being able to revoke a subset of certificates if you can determine only a subset of your hosts have been compromised.
  - blibble 3y ago

beeboobaa 3y ago

Are there any services that monitor the certificate transparency logs and send an email when a new certificate is issued for your domain? That would alert you to this kind of MITM

detourdog 3y ago
Kerberos right?
jauntywundrkind 3y ago
> Could you prevent or monitor this kind of attack? There are several indications which could be used to discover the attack from day 1: > All issued SSL/TLS certificates are subject to certificate transparency. It is worth configuring certificate transparency monitoring, such as Cert Spotter (source on github), which will notify you by email of new certificates issued for your domain names
- radlad 3y ago
  > All issued SSL/TLS certificates are subject to certificate transparency. It is worth configuring certificate transparency monitoring, such as Cert Spotter (source on github), which will notify you by email of new certificates issued for your domain names This seems like the kind of thing domain registrars should do for you.
  - Operyl 3y ago
    Who could in theory also be told “don’t notify your client about this specific one”. You’re best off self hosting if you think you’d be a target for these kind of attacks.
    - ratsmack 3y ago
      It seems that trust is a highly variable quantity, maybe similar to intuition.
tptacek 3y ago

vb-8448 3y ago

> The attacker managed to issue multiple SSL/TLS certificates via Let’s Encrypt for jabber.ru and xmpp.ru domains since 18 Apr 2023 > We tend to assume this is lawful interception Hetzner and Linode were forced to setup based on German police request. > Another possible, although much more unlikely scenario is an intrusion on the internal networks of both Hetzner and Linode targeting specifically jabber.ru — much harder to believe but not entirely impossible. And what if the attacker tricked somehow the letsencrypt challenges? Or this is supposed to be impossible?

janosdebugs 3y ago
If that were the case, they wouldn't have seen weird connectivity issues/behaviors.
- vb-8448 3y ago
  why not? They discover the issue only because the attacker failed to renew the certificates.
  - janosdebugs 3y ago
    Yeah, but just getting a hold of a valid certificate doesn't automatically mean a MITM. For that the network connections need to be intercepted too.
nmjohn 3y ago
The attacker effectively controlled the IP the domain was pointed to. If you have this, getting a cert issued from any CA is trivial - you've proved to them you control the domain in question.
- 3np 3y ago
  As mentioned elsewhere in the thread, RFC 8657 can prevent this. https://news.ycombinator.com/item?id=37958831 https://news.ycombinator.com/item?id=37958831
3y ago

upofadown 3y ago

>End-to-end encrypted communications, such as OMEMO, OTR or PGP, are protected from the interception only if both parties have validated the encryption keys. The users are asked to check their accounts for new unauthorized OMEMO and PGP keys in their PEP storage, and change passwords. The attacker would need to set up a separate MiTM for the particular E2EE scheme used. Some of the XMPP clients I have encountered will not let you use a particular cryptographic identity unless you have explicitly claimed to have verified it. Still a good reminder. If you have not seen and dealt with a ridiculously long number (or equivalent) you have not achieved end to end.

Andrew_nenakhov 3y ago

I wonder if there is a legal way to demand Hetzner/Linode comments on this situation. Likely, the entity behind the interception is some government agency or police.

elliewithcolor 3y ago
For Hetzner as German Company: If it is legal interception no. You can’t demand an answer, without going to a lawyer first. On the other hand, if it is not lawful interception, I doubt Hetzner would allow it. Because that’s also against law.
baybal2 3y ago
[dead]
dathinab 3y ago
there isn't even any guarantee that the wiretapping was done through them instead of e.g. the carriers which pretty much in any country have since decades been forced to help with lawful wiretapping...
- chatmasta 3y ago
  Carriers meaning the interconnect providers eg Level3, Cogent etc? How would this intercept be implemented in practice? Surely it'd be much easier to add a node as close as possible to the origin host, i.e. within the Hetzner network, rather than redirecting traffic from the outside with some sort of BGP hijack?
  - dathinab 3y ago
    any carrier on any level but like other have pointed out this seems to have been in the hetzner network through wire taping laws also extend to datacenter internal interconnects I mean servers of different people can communicated with each other without the traffic leaving the server so it kinda makes sense

rodlette 3y ago

I agree with most of the mitigation suggestions. For high risk targets, consider layering an additional auth mechanism that doesn't rely on trusted CAs: Tor onion services, SSH, or Wireguard. > All issued SSL/TLS certificates are subject to certificate transparency. +1. crt.sh has RSS feeds for this. > Limit validation methods and set exact account identifiers Using CAA is a good idea in general, but would it help in this case? The attacker would just request the exact cert configuration that is permitted by CAA. Maybe this helps if you can strengthen one validation method? > Monitor SSL/TLS certificate changes on all your services using external service +1. High-risk targets should be aware of what certs are valid at any time, and be checking for those. > Monitor MAC address of default gateway for changes A more sophisticated attack could preserve the MAC address. > "Channel binding" is a feature in XMPP which can detect a MiTM even if the interceptor present a valid certificate. TIL.

hlandau 3y ago
>Using CAA is a good idea in general, but would it help in this case? The attacker would just request the exact cert configuration that is permitted by CAA. Maybe this helps if you can strengthen one validation method? Author of ACME-CAA (RFC 8657) here. ACME-CAA can mitigate this because you can put a unique identifier for your ACME account in the CAA record, so it is not possible for an attacker to do this unless they can get your ACME account private key (or coerce the ACME service). This assumes you have DNSSEC-secured nameservers, of course, otherwise DNS requests can potentially also be intercepted when queried by the CA. See RFC 8657 for a full list of security caveats. The RFC is designed to be more readable than most. Blogpost by me with more background: https://www.devever.net/~hl/acme-caa-live https://www.devever.net/~hl/acme-caa-live
- aaomidi 3y ago
  Yep! This is one way of preventing this type of attack.
- crosser 3y ago
  Could you please elaborate the reasons to implement this outside of DANE (RFC-7671) framework?

zxwrt 3y ago

What a nefarious move by Hetzner and Linode. How to trust them after this?

Nextgrid 3y ago
If it's a "lawful intercept" then they most likely have no choice.
dathinab 3y ago
first the article jumped the gun when saying it's by Hetzner/Linode, as it as much could have been done by the carrier the data-center connects to and from what I know about German espionage/wiretapping by police law that would be far more likely second pretty much _every_ country has laws which require carriers to help them wiretap in case of an investigation with appropriate court orders thirdly even if it went through Hetzner/Linode instead of carriers it wasn't done "a move from them" but something they where legally binding ordered to silently tolerate lastly if as unlikely as it seems it was not lawful interception of police or similar then they (or a carrier) would have been hacked, i.e. there is absolutely no chance that server providers will do such an attack on their own violation, especially Hetzner (they also don't have the legal means to get the necessary certificates)
- cassonmars 3y ago
  That "lawful" interception allows certificate issuance to be a means of wiretap completely undermines any trust one should have in CAs. It seems that an alternative is greatly needed.
- scraptor 3y ago
  The fact that they were legally required to MITM their customers does not make them more trustworthy (in the sense of unlikely to do it in the future), just the opposite! Of course that applies equally to any other cloud host (modulo jurisdiction games) but that does little to restore my interest in running my software on other peoples computers.
  - 3y ago

wutwutwat 3y ago

As an end user, how can I monitor if I’m being MITM’d? If everything I do is proxied through an attacker’s network, nothing I do online could be trusted to properly test if I were being proxied, right? I know apps can do cert pinning, but as an end user how can I validate that when connect to anything that it’s the real one and not a request from being middled?

LinuxBender 3y ago
One can get fingerprints of TLS endpoints ahead of time and/or from a known clean network then compare and contrast to what you see from the location you suspect might be MITM'd. I believe there are browser addons that display certificate fingerprints and alert if a certificate changes however one can click on the lock symbol and drill down to the same info. openssl s_client -servername news.ycombinator.com -connect news.ycombinator.com:443 < /dev/null 2>/dev/null | openssl x509 -fingerprint -noout -in /dev/stdin SHA1 Fingerprint=7E:49:BA:40:86:87:B3:39:66:93:94:9E:9C:45:71:85:3C:8D:95:16 A higher friction method would be to use a browser addon that would pin a certificate to a site. Not useful in the case of the Jabber client but useful if one was visiting Jabbers website to validate something. Some Jabber clients and servers have their own method described in another part of this thread [1] to protect against interception. The openssl s_client method is likely the most versatile for testing from different locations, ports and applications. [1] - https://news.ycombinator.com/item?id=37956911 https://news.ycombinator.com/item?id=37956911
polski-g 3y ago
https://github.com/matteocorti/check_ssl_cert https://github.com/matteocorti/check_ssl_cert

costco 3y ago

This is almost certainly related to some sort of Russian cybercrime investigation. If you ever read Krebs or peruse some of the seedier Russian forums, xmpp.ru will sound familiar to you, because it is. Not imputing anything to the operator, that's just the nature of operating an anonymous service. https://ddanchev.blogspot.com/2021/03/exposing-currently-active-portfolio-of_82.html https://ddanchev.blogspot.com/2021/03/exposing-currently-act... https://ddanchev.blogspot.com/2019/07/profiling-currently-active-portfolio-of.html https://ddanchev.blogspot.com/2019/07/profiling-currently-ac... https://blog.talosintelligence.com/picking-apart-remcos/ https://blog.talosintelligence.com/picking-apart-remcos/ https://flashpoint.io/wp-content/uploads/Plea-Agreement-USA-v.-Vitalii-Chychasov.pdf https://flashpoint.io/wp-content/uploads/Plea-Agreement-USA-... Really interesting writeup though. I guess it's a practical example of why everyone should get a CT monitoring service!

ValdikSS 3y ago
Yes, I tend to believe that as well. My guess it's either related to Genesis Market or Qakbot takedowns.
orbital-decay 3y ago
Drug gangs widely use XMPP as a secure communication channel in combination with Tor. Hydra (the darknet market) has been taken down by German police in 2022 after trying to extend their "service" to the EU; something similar might or might not have happened here.
- codedokode 3y ago
  [flagged]
  - _lvbh 3y ago
    Telegram? Not Signal or Matrix?
    - toastal 3y ago

lima 3y ago

Perfectly pulling off an actual MitM attack and then forgetting to renew the certificate is certainly a very German thing :-)

jakub_g 3y ago
I wonder if someone didn't "forget" on purpose, so that people learn about it. Or, someone very diligently followed the orders - there was an order to set up a cert, but there was no requirement that it has to auto-renew :)
- luma 3y ago
  It was claimed to have been running for 6 months so it must have renewed certs at least once - LE certs are good for 90 days.
  - a1369209993 3y ago
    > 6 months [...] 90 days. So they were told to renew the certificate, but not how many times to renew it?
    - JPHPJ 3y ago
      Shouldn't the certificate transparency logs have it?
      - tgsovlerkhgsel 3y ago
        I suspect it's not trivial to distinguish between the legit and fake ones just based on CT logs. Unless Let's Encrypt publicly logs the account used to issue the certificate (I think they don't), only the logs held at Let's Encrypt will reveal this information. I expect their security team to be looking at those logs right now.
        pgporada 3y ago

deleted 3y ago

[deleted]

mak234 3y ago

so is jabber.ru still secure now? what needs to be done on the user side to make it so? can one reissue/request new server certificate from the server and not from let's encrypt? thanks

zaik 3y ago
If you have used OMEMO or OpenPGP to encrypt your messages, the content of your messages is secure, provided you have verified the public keys of your contacts. If you didn't verify the keys, you should do it now and check that no suspicious keys have been injected.

dathinab 3y ago

> We tend to assume this is lawful interception Hetzner and Linode were forced to setup based on German police request. This seems to be somewhat jumping the gun I think. Given the certs and the target assuming it's an lawful interception seems reasonable. But there is nothing there which requires Hetzner or Lindoe complying or knowing about this. Given the nature of the attacks you can do the interception on the carrier level, and carriers being forced to comply with lawful wiretapping is pretty much anywhere in the world pretty much standard and many laws are based around that approach. Much less so around approaches involving data centers.

Hello71 3y ago
according to the article, the added hop visible in traceroute to Linode machine is after two 10/8 hops, so it's likely an internal machine. more importantly, the affected VM has a strange gateway MAC address, which can only be controlled by the local network administrator, not an intermediate carrier.

Libre___ 3y ago

I was a bit curious about the MAC address since it isn't locally administered which indicates to me that someone likely deliberately chose this range of addresses. https://www.google.com/search?q=%2290%253Ade%253A01%22+linode https://www.google.com/search?q=%2290%253Ade%253A01%22+linod... Googling for '"90:de:01" linode' and '"90:de:00" linode' indicates that addresses from these blocks have been assigned to other linode VMs recently. I sadly don't have a linode VM of my own to compare with right now but it would seem like the traffic has been routed to another VM on linode infrastructure

0xbadcafebee 3y ago

Getting a valid cert for a domain you don't own is getting close to trivial. All of the mitigation strategies are defeated a number of ways, have been for years. I've commented on incidents like these for years on HN. I'm not some internet big wig, and any blog I write isn't going to trend, so this never gets traction. But anyone who understands the tech can find a half dozen ways to get a cert. If you really need to trust your connection, don't rely on internet PKI / public certs. It was secure enough for general purpose in the aughts, it's not anymore.

dewey 3y ago
What are these trival steps to get a valid certificate for news.ycombinator.com?
- escalt 3y ago
  this is left as an exercise for the reader
- LinuxBender 3y ago
  This would not be trivial and would require successful phishing. I've sent an email requesting to make this even harder to do. leaving out those details on purpose
gouggoug 3y ago
Bold claim lacking substantiation. Care to share? Maybe it will get traction this time.
- thriftwy 3y ago
  If you are your hoster and can reroute requests, you can get all kinds of valid SSL certs for any domains that you host. Just grab these IP packets when CA comes to validate that you own that domain. Perhaps EV could solve that to some extent but it is never mandated. Even if you tried to put any stuff into WHOIS to mitigate this, your hoster can serve any bullshit on this channel too. It does look very bad and SSH approach to certificates is just infinitely better. If Jabber used SSL keys instead, they will be alerted immediately. Come to think of it, your hoster can also find ways to steal keys directly from hardware, though.

AtNightWeCode 3y ago

The way that certbot works with let's encrypt the only surprise is that this does not occur more often. We have monitoring on several certs used for TLS. We should probably add an alert if an A-record changes as well. I assume the root cause is DNS tricks.

ilyt 3y ago
> I assume the root cause is DNS tricks. Maybe read the article

hnarn 3y ago

http://archive.today/0R7Or http://archive.today/0R7Or

SergeAx 3y ago

And this is why, kids, you need to enforce certificate pinning on your critical infrastructure!

LinuxBender 3y ago
Public key pinning has been mostly abandoned on popular websites due to high risk of prolonged outages should someone make a mistake and mistakes do happen. There are browser addons that replicate this behavior however and some Jabber clients/servers apparently have their own mechanism for this [1] but one can only hope everyone is using the combination of servers and clients that support this feature. [1] - https://news.ycombinator.com/item?id=37956911 https://news.ycombinator.com/item?id=37956911

Traubenfuchs 3y ago

We are using self developed application layer security. It requires specialized client code and is terminated at a gateway that proxies requests to our normal applications. I really, really hate the idea of this kind of eavesdropping.

gigatexal 3y ago

This is a deal breaker for me. Which is a shame because Hetzner has such great pricing. Oh well.

ComputerGuru 3y ago
It's likely legally enforced, and any other provider would be subject to the same.
- gigatexal 3y ago
  Fair enough
LinuxBender 3y ago
In fairness to them if it's a lawful intercept then they have no choice. Non compliance can shut down a business. Your only alternatives in that case would be the underground style hosters sometimes marketed as bullet-proof hosting though they are just shady resellers with bold claims. They do not typically last long. There are a couple I know of in Amsterdam that are right down the street from The Hague that have been around for a while so I suspect they are just honeypots.

codedokode 3y ago

Wouldn't be surprised if this happened in Russia; but not in Europe. Also, Let's Encrypt security is a joke - they issued fake certificate without serious checking. Using unencrypted HTTP for confirmation is a vulnerability - doesn't this mean that large national ISPs can easily issue ceritificates for any site hosted within a country?

deleted 3y ago
[deleted]
ItsABytecode 3y ago
> Also, Let's Encrypt security is a joke - they issued fake certificate without serious checking. Using unencrypted HTTP for confirmation is a vulnerability They can’t use HTTPS if you don’t have a certificate yet
- codedokode 3y ago
  This doesn't mean you should use unsecure methods instead and issue certificates to anyone capable to do MitM.
  - blueflow 3y ago
    How could Letsencrypt even verify a server setup if not via DNS/HTTP? Also, verify against what? The servers are basically random strangers without identity when they first talk to LE.
    - codedokode 3y ago
      In this case there has been a valid certificate for the site; this alone should raise suspicion. Also, if they cannot do secure validation then maybe they should stop issuing certificates for sites that already have a proper certificate.
      - blueflow 3y ago

wizzard0 3y ago

well i must admit monitoring certificate issuance with LetsEncrypt is quite boring even if you HAVE alerts set up (I do) so… not surprised. Still cool. What a time to be alive

whytevuhuni 3y ago

Could this be a blue pill attack? A vulnerability in the xmpp server exploited to inject a rootkit, which then hides itself inside the kernel? Or creates network/pid namespaces and puts you in them, while leaving the mitm server in the original one? If so, the mitm could be on the same host, and wouldn't need the cooperation of the hosting provider. I'm not sure how to check for either of these without restarting (which the admin does not seem to want to do, as it is a live service). https://en.wikipedia.org/wiki/Blue_Pill_(software) https://en.wikipedia.org/wiki/Blue_Pill_(software)

codedokode 3y ago
No because traffic redirection occured in provider's infrastructure, not on a server.
- whytevuhuni 3y ago
  Let's say you are blue-pilled and what you see as your eth0 interface is actually a virtual interface controlled by the rootkit. In that case, where the redirection happened is no longer something you would be able to tell, right?
londons_explore 3y ago
If this had happened, the attacker would have likely stolen the servers TLS certificate and keys Whereas this attack generated new keys (and was detected!), suggesting the attacker didn't compromise the server itself.

riku_iki 3y ago

So, if I rent server in those services, it all potentially can be compromised and there is no way to make sure things are secure?..

bananapub 3y ago
? did you not read the article?
- riku_iki 3y ago
  I briefly read the article, maybe you can explain what is your point?
  - _flux 3y ago
    The point is probably the heading of the final chapter "Could you prevent or monitor this kind of attack?" which enumerates the ways how you could detect this attack.
    - riku_iki 3y ago
      lets say I opened account on hentzner, rented server, they sent me email with password, I SSH-ed to it and connection got wire tapped. What way you think could help me to detect this?
      - _flux 3y ago
        Well you could check the SSH server fingerprint that it matches what the SSH client sees. In principle if the SSH connection is MitM'd, they could change the output from the shell so that it would look like it wouldn't be changed. But ultimately they created the keys to begin with, so they can just keep another copy for the purpose of MitM. I doubt they do this nowadays, but they _could_. edit: I actually thought this story was about virtual machines, but I noticed it was about physical hosts, so those have a bit more hope; but ultimately you're still trusting devices that are not physically under your control, so they cannot be trusted. But let's say that in the case of virtual hosts: basically there is no way to ensure you have non-wiretapped connection to a virtual computer provided by hosting service. They don't even need to MitM it, they can just look at the memory of the virtual machine. The only attempt at trying to do that properly has been AMD's "Secure Encrypted Virtualization" https://www.amd.com/en/developer/sev.html https://www.amd.com/en/developer/sev.html, but I think it was broken by some researchers and I don't know if it could currently provide some means to do this safely. I suspect even in that case it might be challenging to install the initial operating systems so that it won't contain the MitM functionality in itself. And even then you would be trusting AMD's implementation of the security layer. Frankly that the wiretap was discovered this time is just a learning experience for the provider to do better next time. I doubt there are good reasons why it should be possible to detect it, if implemented competently.

mrbonner 3y ago

I’m wondering if mTLS (aka zero-trust) is used, would take prevent this kind of MITM attack? My understanding is that it should since the cert is self-signed to be used from both ends.

snvzz 3y ago

A reminder e2ee is a must.

chgo1 3y ago

@dang: .org.ru should probably be treated as TLD

mermaidsalad 3y ago

Next time someone in Matrix discussion will ask "why we need Matrix when there is jabber/xmpp" show them this: > All jabber.ru and xmpp.ru communications between these dates should be assumed compromised. Given the nature of the interception, the attacker have been able to execute any action as if it is executed from the authorized account, without knowing the account password. This means that the attacker could download account's roster, lifetime unencrypted server-side message history, send new messages or alter them in real time.

Arathorn 3y ago
The same attack could be mounted against a Matrix server tbh. But the fact Matrix is E2EE by default would mitigate it a bit, and if folks verified who they were talking to then the attack would be defeated.
- huggingmouth 3y ago
  Matrix clients typically complain very loudly when keys change or unverified sessions get added to your chats. Id's say it would mitigate it A LOT. Though I have to agree that there is still a lot of room for improvement. I personally prefer the approach taken by briar: your public key is your address. Pretty cut and dry, not much room for shenanigans.

mkup 3y ago

There's one thing I can't understand in this story: if that's lawful interception, why Hetzner and Linode bothered to set up MitM interception with different LE certificate and key, rather than extract the TLS private key directly from the RAM and/or storage device of the VPS? Even if this is a physically dedicated server, they can extract the private key from the RAM by dumping the RAM contents after unscheduled reboot. Extraction of the private key isn't visible in CT logs, much more stealthier, practically undetectable.

tgsovlerkhgsel 3y ago
Because it was easier, most likely. There's also a possibility that one would be a "search" and the other would be an "interception" with different levels of approvals requested, but I don't know what the current legal situation in Germany is right now.
ValdikSS 3y ago
Likely because it's 'more illegal'. I'd bet they are not allowed to hack into the server if it's not directly involved in the cybercriminal activity.
xorcist 3y ago
On a physical server, couldn't you just hotplug a PCIe card in there and DMA out any data you are interested in? Something like a network card with firmware specifically for the purpose should do it. It sounds so much a standard thing for law enforcement that I imagine such equipment should be available off the shelf?

hhsectech 3y ago

I've always thought a solution like LetsEncrypt would be an ideal backdoor for LEAs wanting to circumvent SSL/TLS. The same thing applies to VPN services. Why bother trying to develop tools etc that can decrypt traffic when you can just have people send their traffic to you?

MaxBarraclough 3y ago
Why are your concerns specific to LetsEncrypt? They're just another CA, no?

psvz 3y ago

let's encrypt is a public non-profit, who would definitely assist in finding out how domain control verification (DCV) happened - somehow that obvious line of pursuit was missing from the investigation. The mystery of the MAC 90:de:01:49:76:ae is interesting though, but CIA-custom-Made NIC might be a bit superfluous as an idea. Linode (acquired by Akamai) might use something in-house designed and OEM-manufactured, so I wouldn't treat it as strong evidence. Talk to Let's Encrypt - that will likely unveil what happened. Interestingly, search "90:de:01" in this page: https://www.cyberciti.biz/faq/howto-linux-configuring-default-route-with-ipcommand/ https://www.cyberciti.biz/faq/howto-linux-configuring-defaul... ^^ looks like another victim VM of the alleged mysterious interceptor :-)

olegarch 3y ago

Using DANE or so (RRs CAA/TLSA) can help in this case (when provider route traffic to specific port), but does not solve the problem completely. If DNS record will be compromised, then hacker can setup "correct" TLSA/CAA record to trust his fake certificate. As result, problem can be solved, if we will have reliable DNS subsystem, where is impossible to compromise DNS provider or infrastructure. I think, emerDNS can be used in such critical application: https://emercoin.com/en/documentation/blockchain-services/emerdns/emerdns-introduction/ https://emercoin.com/en/documentation/blockchain-services/em...

harrymit907 3y ago

A user reported this on Reddit 3 days ago: https://www.reddit.com/r/hetzner/comments/17ankoh/does_hetzner_run_a_proxy_in_front_of_my_server/ https://www.reddit.com/r/hetzner/comments/17ankoh/does_hetzn...

withinboredom 3y ago

Interesting reddit thread ending here: https://www.reddit.com/r/hetzner/comments/17ccp3i/hetzner_does_run_a_mitm_proxy_in_front_of_my/ https://www.reddit.com/r/hetzner/comments/17ccp3i/hetzner_do...