4 ms·

The summary of the attack from https://notes.valdikss.org.ru/jabber.ru-mitm/ https://notes.valdikss.org.ru/jabber.ru-mitm/ is very interesting: * The attacker

The summary of the attack from https://notes.valdikss.org.ru/jabber.ru-mitm/ https://notes.valdikss.org.ru/jabber.ru-mitm/ is very interesting: * The attacker managed to issue multiple SSL/TLS certificates via Let’s Encrypt for jabber.ru and xmpp.ru domains since 18 Apr 2023 * The Man-in-the-Middle attack for jabber.ru/xmpp.ru client XMPP traffic decryption confirmed to be in place since at least 21 July 2023 for up to 19 Oct 2023, possibly (not confirmed) since 18 Apr 2023, affected 100% of the connections to XMPP STARTTLS port 5222 (not 5223) * The attacker failed to reissue TLS certificate and MiTM proxy started to serve expired certificate on port 5222 for jabber.ru domain (Hetzner) * The MiTM attack stopped shortly after we begun our investigation and network tests on 18 Oct 2023, along with tickets to Hetzner and Linode support team, however passive wiretapping (additional routing hop) is still in place at least on a single Linode server * Neither servers appear to be hacked * Both Hetzner and Linode network appear to be reconfigured specifically for this kind of attack for the XMPP service IP addresses --- Neither that page, nor the page linked from here, mention certificate pinning, maybe because XMPP doesn't support it (I don't know), but if it did, wouldn't that have prevented this kind of attack?

jcims 3y ago
> * The attacker failed to reissue TLS certificate and MiTM proxy started to serve expired certificate on port 5222 for jabber.ru domain (Hetzner) This is gold.
- tptacek 3y ago
  The absolute lack of giving a shit is one of your major clues this was a lawful intercept scenario.
- deeth_starr_v 3y ago
  I mean, I’ve seen the auto renew fail a lot with the certbot. They definitely should have checked it in the renew period to make sure it was working, but I feel for them
- rdtsc 3y ago
  Someone was forced to do it, but they didn’t personally agree with it so they eventually made a “mistake” to tip off the target? There is the plain incompetence explanation: the hosting provider gave control of the operation to the government entity. The underpaid and indifferent government employee did the best they could with their level motivation and skill level.
Nextgrid 3y ago
How would you do certificate pinning if you don't control the clients? My understanding is that certificate pinning is only possible if you control the clients, in which case you can embed which certificates are allowed directly in the client and bypass the whole web PKI. In a situation with general-purpose clients connecting, how would they know which certificates are meant to be allowed? That's what the web PKI is used for. Of course, if you do provide your own clients, this just moves the problem further up the chain - in this case the place where customers would download the custom client software would be compromised and a malicious client served instead.
- LeoPanthera 3y ago
  > How would you do certificate pinning if you don't control the clients? Well you cannot. If you were paranoid, you would perhaps supply a hash through some out-of-band mechanism, which would require manually updating for each new cert. Obviously most people wouldn't ever want to do that.
- dist-epoch 3y ago
  Isn't this what those "key hash pictures" in WhatsApp/Signal are solving? XMPP clients could implement such a mechanism, and if any certificate/domain along the path changes, the users in a conversation would be notified.
  - WhyNotHugo 3y ago
    These are usually to validate the keys used in end-to-end encryption. Both parties must confirm that they see the same details, which confirms that the same keys are being used on both ends.
- KirillPanov 3y ago
  TLS client certificates. Use them. Stop using domains. Stop.
  - callalex 3y ago
    And how do you distribute those to anonymous users?
jacquesm 3y ago
> Both Hetzner and Linode network appear to be reconfigured specifically for this kind of attack for the XMPP service IP addresses This suggests a compromise of Hetzner and Linode network management.
- LeoPanthera 3y ago
  > This suggests a compromise of Hetzner and Linode network management. No it doesn't. It suggests that they both complied with a request from law enforcement.
  - dhbanes 3y ago
    So… compromised.
    - xinayder 3y ago
      Compromised means they got hacked, which would translate to all servers in Linode and Hetzner being unsafe for all customers. Since this affected only one client and it's more likely than not a lawful intercept, it's not a compromise.
      - CommitSyn 3y ago
        You're both using your own internal dictionary differently for the same correct word. In hacking, compromised means hacked. In intelligence, compromised means you have someone doing something on your behalf that they feel forced to do. You seem to understand this so I'm not sure where the confusion lies.
- KirillPanov 3y ago
  More likely the carrier upstream of them.
  - duskwuff 3y ago
    No, that wouldn't insert a hop between the server and the provider's network.
simpaticoder 3y ago
>The attacker managed to issue multiple SSL/TLS certificates via Let’s Encrypt This implies that the attacker was able to validate domain ownership of jabber.ru and xmpp.ru by either a) being able to respond to the ACME protocol on domain(s) hosted http OR b) being able to write to a DNS TXT file for the domain(s). This implies that the victim already lost control of their process server and/or their nameserver.
- tetromino_ 3y ago
  No. Most probably it merely implies someone upstream of them (presumably their hosting provider under compulsion from German law enforcement) intercepted and spoofed their unencrypted, unauthenticated ACME HTTP traffic. Same can happen to you.
  - simpaticoder 3y ago
    Okay, but the part I don't understand is why do it this way? If the hosting provider is a threat, they don't need new certs. They can just read your private keys off your disk and MITM with that.
    - fulafel 3y ago
      Here's a good example why trust is not binary ("you either trust Hetzner or don't"). Quite possibly wiretapping law doesn't require them to facilitate the server breakin.
    - T3OU-736 3y ago
      Your private keys at rest are / should be encrypted, so it would take a bit more than just reading them. The next level of mitigating that sort of a thing is to have keys not be on the hosts at all. Enter HSM - Hardware Security Module. A wildly complex topic I cannot hope to cover in an HN comment, but fundamentally, the private keys are not on the same HW as the server software which needs them. A fundamental property of an HSM is that you, the HSM user, don't actually see the private key. You can ask the HSM to generate one. Derive things from it (in the cryptographic sense of derive). Even prove the provenance of such derived data. But the HSM should not reveal the actual private key. In the cloud world, the HSM equivalent is known as KMS (Key Management Service), and the Good cloud providers all let you manage your own (with the downside being that you now need to manage your own keys).
Bu9818 3y ago
>affected 100% of the connections to XMPP STARTTLS port 5222 (not 5223) Why did they only target the STARTTLS port? On a related note, I would never use the STARTTLS port (opportunistic encryption) if I knew that the server had a regular TLS port...
- KirillPanov 3y ago
  Easier to conceal the attack. The MiTM attacker can pass through a command stream without STARTTLS. If they intercepted 5223 they would have to do their own client-side TLS handshake with the attacked server, which would look really obvious to anybody doing TLS fingerprinting on the server: all of a sudden, 100% of their clients have the exact same TLS fingerprint. Stop outsourcing your PKI to ICANN, folks. Domains are not public keys.
  - MattJ100 3y ago
    They were doing their own TLS handshake - that's how the attack was discovered (the attacker presented a different certicate, which eventually expired, presumably due to negligence). They were decrypting and re-encrypting.
    - KirillPanov 3y ago
      Read it again: > they would have to do their own client-side TLS handshake By intercepting the STARTTLS port the attacker can merely decrypt -- rather than, as you wrote, decrypting and re-encrypting.
      - fuzzy2 3y ago
        This is not what the attackers did however. From the original article (not this one): > Traffic dump on port 5222, the connection is hijacked on application level (L7), the server receives replaced ClientHello message from the client.
      - MattJ100 3y ago
        I don't know what jabber.ru's policy is, it's running a very old version of ejabberd. But you would be hard-pressed to find an XMPP server that would allow authentication without TLS. Starttls makes no difference.
throwaway4good 3y ago
Is it given that this attack was done from within Hetzner? As I understand the techniques applied; this could have also been done at another place on the network route to the targeted server. Ie. by the telecommunications company delivering traffic into Hetzner.
- throwaway4good 3y ago
  Tele communication companies have internal “police groups” (where I am from - I expect it to be the same in Germany) that does nothing but service wiretapping requests from the police. Telcos are required by law to do this. Them expanding into mtm https wiretapping is a new for me but maybe to be expected …
MattJ100 3y ago
XMPP supports channel binding, which is not mentioned in this post but would have prevented this attack. Unfortunately jabber.ru is running server software from 2016, and that old version doesn't support it. Pinning is problematic these days, because certificates are short-lived and renewed frequently. Users would be constantly asked to accept new certificates on a monthly basis, and they wouldn't think twice about clicking 'Accept' on an attacker's certificate then. Channel binding works regardless of the certificate, and ensures the TLS stream is terminated by the entity you exepect. The focus on (lack of) CT/SCT support in XMPP clients puzzles me, because it would not have detected this attack at all, which used valid CT-logged certs from Let's Encrypt. SCT verification would help detect theoretical issuance from another CA if they had a strict CAA record in place (they did not). But even then, channel binding is more privacy-friendly and does not depend on a third party.
- knorker 3y ago
  Pinnig is based on the keypair, not the cert. You can renew and not break pinning, right? Also you can phase in a new cert with pinning.
  - MattJ100 3y ago
    Yes, you can pin the public key instead, which is generally more helpful. But most ACME clients (including the "official" certbot) default to rotating the key too. That can be disabled, but it's a problematic default for this use case which means clients can't just enable pinning.
    - rhn_mk1 3y ago
      How can it be disabled in certbot?
      - natebc 3y ago
        --reuse-key When renewing, use the same private key as the existing certificate. (default: False) From the docs: https://eff-certbot.readthedocs.io/en/latest/using.html#certbot-command-line-options https://eff-certbot.readthedocs.io/en/latest/using.html#cert...

jcims 3y ago

> * The attacker failed to reissue TLS certificate and MiTM proxy started to serve expired certificate on port 5222 for jabber.ru domain (Hetzner) This is gold.

tptacek 3y ago
The absolute lack of giving a shit is one of your major clues this was a lawful intercept scenario.
deeth_starr_v 3y ago
I mean, I’ve seen the auto renew fail a lot with the certbot. They definitely should have checked it in the renew period to make sure it was working, but I feel for them
rdtsc 3y ago
Someone was forced to do it, but they didn’t personally agree with it so they eventually made a “mistake” to tip off the target? There is the plain incompetence explanation: the hosting provider gave control of the operation to the government entity. The underpaid and indifferent government employee did the best they could with their level motivation and skill level.

Nextgrid 3y ago

How would you do certificate pinning if you don't control the clients? My understanding is that certificate pinning is only possible if you control the clients, in which case you can embed which certificates are allowed directly in the client and bypass the whole web PKI. In a situation with general-purpose clients connecting, how would they know which certificates are meant to be allowed? That's what the web PKI is used for. Of course, if you do provide your own clients, this just moves the problem further up the chain - in this case the place where customers would download the custom client software would be compromised and a malicious client served instead.

LeoPanthera 3y ago
> How would you do certificate pinning if you don't control the clients? Well you cannot. If you were paranoid, you would perhaps supply a hash through some out-of-band mechanism, which would require manually updating for each new cert. Obviously most people wouldn't ever want to do that.
dist-epoch 3y ago
Isn't this what those "key hash pictures" in WhatsApp/Signal are solving? XMPP clients could implement such a mechanism, and if any certificate/domain along the path changes, the users in a conversation would be notified.
- WhyNotHugo 3y ago
  These are usually to validate the keys used in end-to-end encryption. Both parties must confirm that they see the same details, which confirms that the same keys are being used on both ends.
KirillPanov 3y ago
TLS client certificates. Use them. Stop using domains. Stop.
- callalex 3y ago
  And how do you distribute those to anonymous users?

jacquesm 3y ago

> Both Hetzner and Linode network appear to be reconfigured specifically for this kind of attack for the XMPP service IP addresses This suggests a compromise of Hetzner and Linode network management.

LeoPanthera 3y ago
> This suggests a compromise of Hetzner and Linode network management. No it doesn't. It suggests that they both complied with a request from law enforcement.
- dhbanes 3y ago
  So… compromised.
  - xinayder 3y ago
    Compromised means they got hacked, which would translate to all servers in Linode and Hetzner being unsafe for all customers. Since this affected only one client and it's more likely than not a lawful intercept, it's not a compromise.
    - CommitSyn 3y ago
      You're both using your own internal dictionary differently for the same correct word. In hacking, compromised means hacked. In intelligence, compromised means you have someone doing something on your behalf that they feel forced to do. You seem to understand this so I'm not sure where the confusion lies.
KirillPanov 3y ago
More likely the carrier upstream of them.
- duskwuff 3y ago
  No, that wouldn't insert a hop between the server and the provider's network.

simpaticoder 3y ago

>The attacker managed to issue multiple SSL/TLS certificates via Let’s Encrypt This implies that the attacker was able to validate domain ownership of jabber.ru and xmpp.ru by either a) being able to respond to the ACME protocol on domain(s) hosted http OR b) being able to write to a DNS TXT file for the domain(s). This implies that the victim already lost control of their process server and/or their nameserver.

tetromino_ 3y ago
No. Most probably it merely implies someone upstream of them (presumably their hosting provider under compulsion from German law enforcement) intercepted and spoofed their unencrypted, unauthenticated ACME HTTP traffic. Same can happen to you.
- simpaticoder 3y ago
  Okay, but the part I don't understand is why do it this way? If the hosting provider is a threat, they don't need new certs. They can just read your private keys off your disk and MITM with that.
  - fulafel 3y ago
    Here's a good example why trust is not binary ("you either trust Hetzner or don't"). Quite possibly wiretapping law doesn't require them to facilitate the server breakin.
  - T3OU-736 3y ago
    Your private keys at rest are / should be encrypted, so it would take a bit more than just reading them. The next level of mitigating that sort of a thing is to have keys not be on the hosts at all. Enter HSM - Hardware Security Module. A wildly complex topic I cannot hope to cover in an HN comment, but fundamentally, the private keys are not on the same HW as the server software which needs them. A fundamental property of an HSM is that you, the HSM user, don't actually see the private key. You can ask the HSM to generate one. Derive things from it (in the cryptographic sense of derive). Even prove the provenance of such derived data. But the HSM should not reveal the actual private key. In the cloud world, the HSM equivalent is known as KMS (Key Management Service), and the Good cloud providers all let you manage your own (with the downside being that you now need to manage your own keys).

Bu9818 3y ago

>affected 100% of the connections to XMPP STARTTLS port 5222 (not 5223) Why did they only target the STARTTLS port? On a related note, I would never use the STARTTLS port (opportunistic encryption) if I knew that the server had a regular TLS port...

KirillPanov 3y ago
Easier to conceal the attack. The MiTM attacker can pass through a command stream without STARTTLS. If they intercepted 5223 they would have to do their own client-side TLS handshake with the attacked server, which would look really obvious to anybody doing TLS fingerprinting on the server: all of a sudden, 100% of their clients have the exact same TLS fingerprint. Stop outsourcing your PKI to ICANN, folks. Domains are not public keys.
- MattJ100 3y ago
  They were doing their own TLS handshake - that's how the attack was discovered (the attacker presented a different certicate, which eventually expired, presumably due to negligence). They were decrypting and re-encrypting.
  - KirillPanov 3y ago
    Read it again: > they would have to do their own client-side TLS handshake By intercepting the STARTTLS port the attacker can merely decrypt -- rather than, as you wrote, decrypting and re-encrypting.
    - fuzzy2 3y ago
      This is not what the attackers did however. From the original article (not this one): > Traffic dump on port 5222, the connection is hijacked on application level (L7), the server receives replaced ClientHello message from the client.
    - MattJ100 3y ago
      I don't know what jabber.ru's policy is, it's running a very old version of ejabberd. But you would be hard-pressed to find an XMPP server that would allow authentication without TLS. Starttls makes no difference.

throwaway4good 3y ago

Is it given that this attack was done from within Hetzner? As I understand the techniques applied; this could have also been done at another place on the network route to the targeted server. Ie. by the telecommunications company delivering traffic into Hetzner.

throwaway4good 3y ago
Tele communication companies have internal “police groups” (where I am from - I expect it to be the same in Germany) that does nothing but service wiretapping requests from the police. Telcos are required by law to do this. Them expanding into mtm https wiretapping is a new for me but maybe to be expected …

MattJ100 3y ago

XMPP supports channel binding, which is not mentioned in this post but would have prevented this attack. Unfortunately jabber.ru is running server software from 2016, and that old version doesn't support it. Pinning is problematic these days, because certificates are short-lived and renewed frequently. Users would be constantly asked to accept new certificates on a monthly basis, and they wouldn't think twice about clicking 'Accept' on an attacker's certificate then. Channel binding works regardless of the certificate, and ensures the TLS stream is terminated by the entity you exepect. The focus on (lack of) CT/SCT support in XMPP clients puzzles me, because it would not have detected this attack at all, which used valid CT-logged certs from Let's Encrypt. SCT verification would help detect theoretical issuance from another CA if they had a strict CAA record in place (they did not). But even then, channel binding is more privacy-friendly and does not depend on a third party.

knorker 3y ago
Pinnig is based on the keypair, not the cert. You can renew and not break pinning, right? Also you can phase in a new cert with pinning.
- MattJ100 3y ago
  Yes, you can pin the public key instead, which is generally more helpful. But most ACME clients (including the "official" certbot) default to rotating the key too. That can be disabled, but it's a problematic default for this use case which means clients can't just enable pinning.
  - rhn_mk1 3y ago
    How can it be disabled in certbot?
    - natebc 3y ago
      --reuse-key When renewing, use the same private key as the existing certificate. (default: False) From the docs: https://eff-certbot.readthedocs.io/en/latest/using.html#certbot-command-line-options https://eff-certbot.readthedocs.io/en/latest/using.html#cert...