13 ms·

Mitigating the Hetzner/Linode XMPP.ru MitM interception incident

341 points by hlandau 3y ago

abigail95 3y ago
> What would a perfect attacker do? If you had physical access to the computer, some sort of bus interception to exfiltrate data from the machine.
- whalesalad 3y ago
  extremely difficult to get physical access in a datacenter
  - sebstefan 3y ago
    Sure but this is the German police and more generally nation states, not only they can, they don't even need to they just ask
    - Beijinger 3y ago
      While the rule of law in Germany is much worse than most people think, it is not so bad as you assume. I doubt that Hetzner would give in to a police request. A court order, yes. But not to a police request. This does not mean, the police won't try it: https://www.dw.com/de/e-mail-firma-kritisiert-ermittler/a-18677560 https://www.dw.com/de/e-mail-firma-kritisiert-ermittler/a-18... Unfortunately can't find the original post. Every idiot police officer thought he has a right to just email them to handle over data :-)
  - LeoPanthera 3y ago
    I would suggest that if you are the police, you can break into a datacenter with a flash of a badge. I can't imagine many would attempt to stop you.
    - mjevans 3y ago
      I would hope they at least: * Require a copy of the badge number, and verify that this officer is assigned and expected to be at this business right now. * Require them to sign into and out of the site. * Annotate which systems / compromises are in place. - That all of the above MIGHT be sealed under a court order; I would hope any such order has an automatic 'sunset' date, and possibly renewal upon review by a different judge.
      - LeoPanthera 3y ago
        Those are some very optimistic hopes!
      - skissane 3y ago
        A business can request visiting law enforcement to do all those things, and hopefully law enforcement complies. However, if they refuse to comply, realistically you just have to let them in anyway. Document their non-compliance and provide it to your lawyers, who can decide what action to take (lodge a formal complaint to the law enforcement agency, apply to a judge for an injunction to compel their compliance, etc) Well, that’s true in countries like Germany or the US. I suspect in somewhere like Russia or China, formal complaints are unlikely to achieve anything except invite government retaliation.
        leosarev 3y ago
        Latter is not correct. It's well known difference in Russia between companies that willingly cooperate with government agencies informally, and those who just provide information upon formal request according to law.
        skissane 3y ago
        How do you know for sure the people who “just provide information upon formal request according to law” aren’t covertly engaging in informal cooperation? If one morning the CEO gets an unexpected visit at home from a group of FSB agents asking for some favours, is the CEO going to say “no”? And if the CEO says “yes”, are you going to hear about it, or are they going to let the CEO continue that pretence? Western CEOs don’t have the same worry about “accidentally” falling out of hospital windows.
  - mike_d 3y ago
    The organization conducting the MitM likely has physical access to the machine already. The original post indicates the link on the network interface went down for 19 seconds, indicating a device was placed in front of the server.
  - jacquesm 3y ago
    All the people working in the datacenter have that level of physical access. Unless they are very closely supervised they can do a lot of damage without anybody being the wiser until they get caught. I've been in (nominally very secure) DCs on behalf of customers and I've seen: - unlocked racks - doors open - temporary network cables and keyboards, monitors and mice attached to running systems - systems logged in left unattended - floor panels raised up and left open unattended exposing cabling - meet-me rooms with interfaces exposed (gear in racks without doors) DC personnel tends to trust each other, and they probably shouldn't. But it's hard to be part of a closely knit crew for a long time without getting into a 'get stuff done' mode where protocol and rules are there in principle but less so in practice because it is seen as an efficiency penalty. It's another instance of the 'normalization of deviation' phenomenon.
    - manxman 3y ago
      Agree re: everything you said but wanted to add datadentre security staff are some of the most interesting characters I’ve encountered. Not sure I sleep as well at night after seeing what I saw.
      - OmarAssadi 3y ago
        Do tell, please; stories about "interesting characters" are often the best.
        jacquesm 3y ago
        GP may well be under NDA and easy to identify.
  - OmarAssadi 3y ago
    Assuming this was done at the government's request, I assume Hetzner is more than willing to comply with a court order mandating they allow them to monitor and physically size a machine. And outside of nation-state requests, even ignoring the fact that someone could probably pay-off an employee, I think ease would depend a lot on the datacenter and target; judging by the awesome and hilarious story behind the Fremont Cabal accidentally becoming an internet exchange by essentially having some dude barely secretly slipping unauthorized cables into the raceways [1], I figure there are a lot of places where if your target is simply renting a couple rack units or single rack rather than an entire locked cage, you can probably get physical access by doing the same. Also, a lot of Deviant Ollam's stories about industrial security and the dozens of ways he's broken into utility companies, server rooms, etc — mostly just by being confident, looking the part, bad doors [2], and badge cloning [3] — don't give me a ton of confidence that someone with skill couldn't feasibly either get direct access to servers they shouldn't, or at the very least, access to an important part of the supply chain for their target. And speaking of supply chain, my processor died recently, so I ordered a brand-new in box replacement Ryzen, and when it arrived last night, out of curiosity, I wanted to see if I could get the CPU out of the box without breaking the tamper-evident authenticity seal... ... and about fifteen minutes later, after borrowing a syringe and hypodermic needle from my mom, a little bit of isopropyl alcohol, a blade from my safety razor, and a quick look at a video from LockPickingLawyer [4] and a couple from datagram at DEFCON's Tamper-Evident Village [5][6], I had the CPU out, put my old one for now, and re-applied the sticker with no visible damage to the box or seal. All I had to do was tip it upside down at about 90-degrees, douse a little bit of the alcohol under the top of the seal, let gravity do most of the work, and then carefully lift the seal with the razor. After that, I just lightly squeezed the box to make the front tab come as forward as possible, and then carefully pushed the ear flaps down to prevent tearing, and then I was in. I've seen others demonstrate it on older AMD boxes that had flexible cardboard in-place of the cooler, allowing them to pull the cardboard to make enough room for tools to get out the CPU without even touching the seal [7]. But in my case, it was a newer box with hard plastic inside where the cooler would've been, so that's why I went for the seal instead. No surprise to me now that counterfeiting is rampant on Amazon, with people returning the box after putting in either random junk, dead Athlons covered by a counterfeit serial-matching IHS, or the cheapest socket-compatible CPU after deluding both and swapping the IHS. I figure with a bit of practice and better tools, like Teflon spudgers and syringes, it'd be significantly easier to get past 99% of tamper-resistant/tamper-evident seals and into boxes you're not supposed to be without risking damage, and then you can intercept a package, compromise something critical, like the server BMC or firmware, reseal everything, and be on your way. And given the relatively recent scare with loads of servers, including Dell and others, being shipped with "AMERICAN MEGATRANDS" labels on their BMC boards, with no one noticing until a YouTube commenter pointed it out during a teardown by ServeTheHome, I think it's totally feasible for an enemy to just compromise the entire physical supply-chain of a company, datacenter, or whatever else [8]. [1]: Oxide's On the Metal: Kenneth Finnegan - https://oxide.computer/podcasts/on-the-metal/kenneth-finnegan https://oxide.computer/podcasts/on-the-metal/kenneth-finnega... [2]: Deviant Ollam @ Shakacon: The Search for the Perfect Door - https://www.youtube.com/watch?v=4YYvBLAF4T8 https://www.youtube.com/watch?v=4YYvBLAF4T8 [3]: Deviant Ollam / Modern Rogue: Getting an RFID Implant - https://youtu.be/SZiRISGdQ4g?t=277 https://youtu.be/SZiRISGdQ4g?t=277 [4]: LockPickingLawyer: Did I Cheat On This Challenge? (Tamper-Sealed Abus) - https://www.youtube.com/watch?v=xUJtqvYDnkg https://www.youtube.com/watch?v=xUJtqvYDnkg [5]: DEFCON 19: Introduction to Tamper Evident Devices - https://www.youtube.com/watch?v=W07ZpEv9Sog https://www.youtube.com/watch?v=W07ZpEv9Sog [6]: DEFCON 30: Tamper Evident Village - https://www.youtube.com/watch?v=slhdowWjSuU https://www.youtube.com/watch?v=slhdowWjSuU [7]: cycurious: How Counterfeiters replace CPU in Sealed Retail Box - https://www.youtube.com/watch?v=Bni8bgGlXDE https://www.youtube.com/watch?v=Bni8bgGlXDE [8]: ServeTheHome: Dude this should NOT be in a Dell Switch… or HPE Supercomputer - https://www.servethehome.com/dude-dell-hpe-ami-american-megatrands/2/ https://www.servethehome.com/dude-dell-hpe-ami-american-mega...
- manxman 3y ago
  Thinking laterally for a moment regarding the big picture here, why do we still rely on data centres. They made sense in a world of dialup and low speed / high latency broadband. But there are lots of places with high speed fibre and not much latency to the peering points. And the more we break away from data centres and clouds, the more the internet infrastructure will have to work the way it was designed instead of having to flow through these crazy aggregation points that are both serious points of failure and major security risks.
  - WhyNotHugo 3y ago
    Regular domestic connections often block ports (especially email). In many countries, ISPs won't sell you a static IP (your dynamic address might not change ever, but you don't control rDNS records). These kind of measures force people to move onto a rented server instead. Often ISPs rent servers themselves. The conflict of interest here is hard to ignore: if ISPs make it easy and convenient to host from home, their business of renting servers suffers.
  - dewey 3y ago
    > They made sense in a world of dialup and low speed / high latency broadband. But there are lots of places with high speed fibre and not much latency to the peering points. Yes, but then you need backup power, someone to replace disks / hardware if things break, proper security for compliance reasons, cooling, noise. Once you set up all these things you just invented a data center again. I don't see how getting rid of data centers makes any sense.
    - chatmasta 3y ago
      You can get all of that in your own business premises with a fiber uplink. But at the point you're staffing IT personnel and managing server racks, I suppose you may as well call the location a datacenter, albeit a private one.
  - immibis 3y ago
    IP addresses, IP addresses, IP addresses, IP addresses.
LeoPanthera 3y ago
The summary of the attack from https://notes.valdikss.org.ru/jabber.ru-mitm/ https://notes.valdikss.org.ru/jabber.ru-mitm/ is very interesting: * The attacker managed to issue multiple SSL/TLS certificates via Let’s Encrypt for jabber.ru and xmpp.ru domains since 18 Apr 2023 * The Man-in-the-Middle attack for jabber.ru/xmpp.ru client XMPP traffic decryption confirmed to be in place since at least 21 July 2023 for up to 19 Oct 2023, possibly (not confirmed) since 18 Apr 2023, affected 100% of the connections to XMPP STARTTLS port 5222 (not 5223) * The attacker failed to reissue TLS certificate and MiTM proxy started to serve expired certificate on port 5222 for jabber.ru domain (Hetzner) * The MiTM attack stopped shortly after we begun our investigation and network tests on 18 Oct 2023, along with tickets to Hetzner and Linode support team, however passive wiretapping (additional routing hop) is still in place at least on a single Linode server * Neither servers appear to be hacked * Both Hetzner and Linode network appear to be reconfigured specifically for this kind of attack for the XMPP service IP addresses --- Neither that page, nor the page linked from here, mention certificate pinning, maybe because XMPP doesn't support it (I don't know), but if it did, wouldn't that have prevented this kind of attack?
- jcims 3y ago
  > * The attacker failed to reissue TLS certificate and MiTM proxy started to serve expired certificate on port 5222 for jabber.ru domain (Hetzner) This is gold.
  - tptacek 3y ago
    The absolute lack of giving a shit is one of your major clues this was a lawful intercept scenario.
  - deeth_starr_v 3y ago
    I mean, I’ve seen the auto renew fail a lot with the certbot. They definitely should have checked it in the renew period to make sure it was working, but I feel for them
  - rdtsc 3y ago
    Someone was forced to do it, but they didn’t personally agree with it so they eventually made a “mistake” to tip off the target? There is the plain incompetence explanation: the hosting provider gave control of the operation to the government entity. The underpaid and indifferent government employee did the best they could with their level motivation and skill level.
tedunangst 3y ago
Run your own CA and choose your roots carefully didn't make the cut.
- fanf2 3y ago
  A bit difficult when providing services to third parties who can use any client software :-/
  - justsomehnguy 3y ago
    Yes, but if you can serve multiple certificates on one endpoint (think SNI) then you can add your own self-signed or private PKI certificate to be able to check if all your requests are being intercepted by a lazy adversary.
  - tedunangst 3y ago
    That's actually probably easier than getting a browser to work with a forbidden cert, how dare you.
liquidk 3y ago
The provider has access to the host, they can just inspect the job from the outside and you won’t be able to tell
- the8472 3y ago
  secure boot + virtualized memory encryption is supposed to prevent that, you'll have to trust intel/amd though.
  - dist-epoch 3y ago
    Only if secure boot was enabled by a trusted party on trusted hardware. If you enable secure boot remotely without physical access to the machine you can't be sure it was actually setup in a non-compromised way. For example the machine could be running a custom backdoor-ed TPM, BIOS settings, ...
  - OmarAssadi 3y ago
    I'm not sure truly verifiable "Secure" Boot is actually realistic in most scenarios, though, right? I'm not super up-to-date on what's being offered right now, but I'm not sure if there is a way to have a proper trusted execution environment on most Intel or AMD offerings; I thought Secure Boot on AMD64 platforms generally rely on TPM or something like SGX for validation, with the former having seemingly a dozen different ways to be tampered with, and the latter being discontinued and being vulnerable to several different attacks, including DOWNFALL. I think EPYC and Sapphire Rapids have some sort of Trusted Execution Environment stuff with SEV-SNP and TDX, maybe? But I don't think either option is really feasible for people paying Hetzner-like prices for hosting; Hetzner's newest Xeon offering is seemingly Cascade Lake, and the only EPYC offered is a single-socket Rome 7502P with 128GB DDR4 for 142 euros, which seems very hard to justify, given they also offer a 7950X3D with 128GB DDR5 for ~25 euros less. Even then, I don't think I could put my confidence in a machine I don't own, didn't setup, can't physically inspect, don't know where it came from, whether the firmware has been tampered with, etc -- especially if it is something as complex as x86, where there is seemingly at least one new horrific hardware-level vulnerability that crops up every generation or two. EDIT: I forgot Hetzner also started offering Ampere Altra servers for 200 euros. I think those have TEE of some sort with the TrustZone stuff? Not sure how secure that really is, though; I haven't really looked into the ARM offerings as much as I should have, mostly since, if you don't want Apple, I'm not aware of a good middle-ground between a cheap SBC and a $3,000+ Ampere server, outside of jerry-rigging some second-hand Gigabyte Cavium ThunderX2 nodes off eBay.
johncolanduoni 3y ago
They point out that "CT is optional", but isn't that ultimately a decision by the XMPP clients? What stops them from requiring CT now that browsers effectively do? What trust store would they be relying on that issues certificates that don't work in the browser?
- hlandau 3y ago
  Right, they could do this and this seems like a good idea to me. The problem this require-CT behaviour isn't switched on by default for basically any TLS library, so an XMPP client developer would have to go out of their way to switch this on, and I assume most or all (currently) don't. That can change of course.
  - jeroenhd 3y ago
    How problematic is that in real life? All CAs worth their salt are already publishing all certificates to CT logs because most TLS servers are browsers and most clients are browsers. Maybe there's a CA somewhere that has opt-in CT logging enabled, but I don't think they're very popular. For example, even the rogue CA certificates were published onto the CT log (because Let's Encrypt does so by default). I think a CT check shouldn't be a very problematic change in practice, as long as there are reasonable exemptions (i.e. don't require CT for certificates imported into the CA store, allow the user to disable the check, etc.). For server<->server communication things become more difficult, but the entire network would be better off if a major server like ejabberd were to require CT.
mike_d 3y ago
Great callout: > Don't use Cloudflare or similar services. See my article here for an explanation on why. If you use a service like this, you're basically already MitMing yourself. I wish more people would realize that when arguing on the internet about CAA, DNSSEC, NSA, etc. that none of it really matters. We willingly allow a government aligned entity to unwrap 20% of all TLS connections on the internet and peak inside.
- worldofmatthew 3y ago
  Cloudflare is horrible for privacy. It is also a bit of a sovereignty issue for European countries to have all their citizens web habits to be MITM by a forging power (no matter how friendly they seam). Edit: not even going in to the sovereignty issue of having an American private company effectively decide your internet regulations.
  - ronsor 3y ago
    The EU regularly de-facto tries to decide regulations for other countries. All is fair in a globally connected world.
    - Sporktacular 3y ago
      Yeah, but it generally goes in the direction of more privacy, environmental and labor standards. Less so from other countries.
    - rini17 3y ago
      That is how American tech monopolies like to paint what the EU does. Lol
      - immibis 3y ago
        dumb question: why do I have the option to downvote your comment, but not many other comments in this thread?
mkj 3y ago
I looked into CAA but the current dns provider doesn't support those records. Is there a reason it had to be a new type not a more common TXT record?
- hlandau 3y ago
  You can try to see if your provider supports "opaque" types in which you can submit the binary encoding of the record without them knowing what it is. Barring that you'll need to request support for it. It's an increasingly popular record type so not supporting it isn't terribly great service nowadays. I'm not familiar with the discussion that went into the design of CAA (it was probably discussed at some point on the relevant IETF mailing list, if you want to go digging).
- rodlette 3y ago
  See RFC5507: "Why Adding a New Resource Record Type Is the Preferred Solution" DNS providers should support a wide range of RR types, and domain owners should vote with their NS records. See https://github.com/StackExchange/dnscontrol/blob/master/documentation/providers.md https://github.com/StackExchange/dnscontrol/blob/master/docu... for a list of DNS providers that support CAA.
- LinuxBender 3y ago
  I have found that some providers that say they don't support it just mean in their web interface. It may be worth testing if you can set up a hidden primary server and have your provider just be a secondary to it. That is how I managed to get around that restriction with Linode.
deleted 3y ago
[deleted]
Sporktacular 3y ago
Who are the end users of xmpp.ru and jabber.ru? Are they hoping to pick up traffic between Russian soldiers? Spies? I hate mass surveillance as much as the next guy but why target Russian domains specifically?
- OmarAssadi 3y ago
  I can't imagine it is war-related, or at least not in any direct sense, like literally trying to intercept soldiers or spies. I think the more likely scenario is that someone was to catch carders/botnet operators, since Jabber/XMPP is still very popular amongst people in that scene in Russia; you'll see often see screenshots or logs containing @exploit.in, @jabber.ru, and various other servers pretty much in any Krebs on Security article, for example. That said, I think mass-surveillance is usually pretty gross regardless of intent. I don't know what the exact ratio of criminal to non-criminal usage of either server is, but just like how Tor simultaneously has a ton of obviously illegal content mixed with whatever percentage of legitimate use [1], they presumably still do have a fair amount of totally normal conversations going on. And so I'm not an exactly a huge fan of invading privacy in order to spy on random cybercriminals likely far out of your jurisdiction in countries unwilling to extradite in the first place. And while I understand if Hetzner was forced to comply, unless they wanted to become another Lavabit, as someone who recommends them often, it'll still be a little disappointing, IMO, if it turns out they were aware and allowed or assisted in the MitM attack, because otherwise, I think they're one of the best hosting companies. Their transparency is great. And I get better uptime, reliability, and support than the overwhelming majority of cloud providers without selling my organs to afford it. And their effort put into turning environmentally-friendly design choices into economically smart choices is commendable. The "reduce" in reduce > reuse > recycle can be seen in their KISS designs—like the raised, angled roof, allowing natural heat escape rather than loads of HVAC [2]; server chassises that are more bare-minimum structural support and air-shrouds than actual chassises; and stripped-down standard motherboard designs with only the most minimal changes, like 90-degree rotated sockets to allow a single fan to cool the VRM, CPU, and memory [3], rather than a dozen screaming, power-slurping Delta fans for every server. And their knowledge of "reuse" is evident by their server auctions, where they do Dutch-auction-style rentals of legacy hardware rather than scrapping everything, avoiding spending a load on new hardware while also avoiding the environmental cost associated with manufacturing new everything each generation. I'd love to be able use them more if only they had dedicated servers outside of Europe. [1]: "legitimate" is hard to define anyway; one country's journalist is another's "foreign agent", and one country's "freedom fighter" is another's "terrorist". [2]: Der8auer: Over 200,000 Servers in One Place! Visiting Hetzner in Falkenstein - https://youtu.be/5eo8nz_niiM?t=259 https://youtu.be/5eo8nz_niiM?t=259 [3]: Der8auer: Hetzner shows Special AM5 Board with 90° Rotated Socket - https://youtu.be/V2P8mjWRqpk https://youtu.be/V2P8mjWRqpk
manxman 3y ago
A quick warning on hetzner. I needed a personal bare metal machine so signed up. I was travelling and on an IP in a distant land so their sign up asked for secondary verification via PayPal. All passed and now it’s should get a server? Nope - next day their support emailed telling me they would not approve my account without… no word of a lie here… either 1: a fax of my passport info page or 2: a scan and email containing the same. I refused reminding them of GDPR and that email is at best opportunistically encrypted and at worst open to interception. They replied stating they believed they were GDPR compliant because all they do is use the passport to verify the account and delete the document. They also said I could hide anything sensitive other than my name and date of birth!! I suggested the process is not GDPR compliant as anyone could intercept unencrypted emails and that they should talk to a lawyer if they did not believe my assertion. Within a short time the server was approved and online. I queried if they would revise their process in light of our interaction. They did not address the question.
- penumbra_3 3y ago
  When was this? Few years ago, something similar happend to me but they provided pub gpg key that i could use..
- petre 3y ago
  They had the option to send it encrypted with PGP. But yes, this reminds me of communist countries where you had to leave your ID at the hotel upon check in. The Stasi mentality lingers on and accomplishes nothing.
  - orhmeh09 3y ago
    What does this have to do with lingering Stasi mentality? Gunzenhausen was in West Germany.
  - schleck8 3y ago
    You don't seem to know anything about VPS administration.
  - OmarAssadi 3y ago
pabs3 3y ago
This article suggests using OTR with XMPP, unfortunately OTRv3 is using obsolete crypto and OTRv4 hasn't been finalised. https://dustri.org/b/time-to-sunset-otr.html https://dustri.org/b/time-to-sunset-otr.html
treffer 3y ago
This list lacks the most obvious one: enable (and ideally enforce) SCRAM-xxxxx-PLUS as the authentication method of choice. The idea of the PLUS variant is both simple and effective: instead of verifying <user,password> with the help of a salt you are verify <user,password,tls session key>. That way the authenticating is only valid on a single TLS connection. This is also called channel binding.
Aerbil313 3y ago
What is the fundamental technical solution for these kinds of attacks from nation-states and other powerful actors? Is there anything which can satisfy the nerdy dream of software which is freely distributable, yet completely safe from censorship, intervention or mutation by anyone other than its author? Are properly built P2P systems over the current internet infrastructure a good enough answer? Or should we rolling out a second low-bandwidth global mesh internet working over radio? I just hope Freenet 2023 evolves to have a zero-friction DX for P2P applications.

341 points by hlandau 3y ago

> What would a perfect attacker do? If you had physical access to the computer, some sort of bus interception to exfiltrate data from the machine.

whalesalad 3y ago
extremely difficult to get physical access in a datacenter
- sebstefan 3y ago
  Sure but this is the German police and more generally nation states, not only they can, they don't even need to they just ask
  - Beijinger 3y ago
    While the rule of law in Germany is much worse than most people think, it is not so bad as you assume. I doubt that Hetzner would give in to a police request. A court order, yes. But not to a police request. This does not mean, the police won't try it: https://www.dw.com/de/e-mail-firma-kritisiert-ermittler/a-18677560 https://www.dw.com/de/e-mail-firma-kritisiert-ermittler/a-18... Unfortunately can't find the original post. Every idiot police officer thought he has a right to just email them to handle over data :-)
- LeoPanthera 3y ago
  I would suggest that if you are the police, you can break into a datacenter with a flash of a badge. I can't imagine many would attempt to stop you.
  - mjevans 3y ago
    I would hope they at least: * Require a copy of the badge number, and verify that this officer is assigned and expected to be at this business right now. * Require them to sign into and out of the site. * Annotate which systems / compromises are in place. - That all of the above MIGHT be sealed under a court order; I would hope any such order has an automatic 'sunset' date, and possibly renewal upon review by a different judge.
    - LeoPanthera 3y ago
      Those are some very optimistic hopes!
    - skissane 3y ago
      A business can request visiting law enforcement to do all those things, and hopefully law enforcement complies. However, if they refuse to comply, realistically you just have to let them in anyway. Document their non-compliance and provide it to your lawyers, who can decide what action to take (lodge a formal complaint to the law enforcement agency, apply to a judge for an injunction to compel their compliance, etc) Well, that’s true in countries like Germany or the US. I suspect in somewhere like Russia or China, formal complaints are unlikely to achieve anything except invite government retaliation.
      - leosarev 3y ago
        Latter is not correct. It's well known difference in Russia between companies that willingly cooperate with government agencies informally, and those who just provide information upon formal request according to law.
        skissane 3y ago
        How do you know for sure the people who “just provide information upon formal request according to law” aren’t covertly engaging in informal cooperation? If one morning the CEO gets an unexpected visit at home from a group of FSB agents asking for some favours, is the CEO going to say “no”? And if the CEO says “yes”, are you going to hear about it, or are they going to let the CEO continue that pretence? Western CEOs don’t have the same worry about “accidentally” falling out of hospital windows.
- mike_d 3y ago
  The organization conducting the MitM likely has physical access to the machine already. The original post indicates the link on the network interface went down for 19 seconds, indicating a device was placed in front of the server.
- jacquesm 3y ago
  All the people working in the datacenter have that level of physical access. Unless they are very closely supervised they can do a lot of damage without anybody being the wiser until they get caught. I've been in (nominally very secure) DCs on behalf of customers and I've seen: - unlocked racks - doors open - temporary network cables and keyboards, monitors and mice attached to running systems - systems logged in left unattended - floor panels raised up and left open unattended exposing cabling - meet-me rooms with interfaces exposed (gear in racks without doors) DC personnel tends to trust each other, and they probably shouldn't. But it's hard to be part of a closely knit crew for a long time without getting into a 'get stuff done' mode where protocol and rules are there in principle but less so in practice because it is seen as an efficiency penalty. It's another instance of the 'normalization of deviation' phenomenon.
  - manxman 3y ago
    Agree re: everything you said but wanted to add datadentre security staff are some of the most interesting characters I’ve encountered. Not sure I sleep as well at night after seeing what I saw.
    - OmarAssadi 3y ago
      Do tell, please; stories about "interesting characters" are often the best.
      - jacquesm 3y ago
        GP may well be under NDA and easy to identify.
- OmarAssadi 3y ago
  Assuming this was done at the government's request, I assume Hetzner is more than willing to comply with a court order mandating they allow them to monitor and physically size a machine. And outside of nation-state requests, even ignoring the fact that someone could probably pay-off an employee, I think ease would depend a lot on the datacenter and target; judging by the awesome and hilarious story behind the Fremont Cabal accidentally becoming an internet exchange by essentially having some dude barely secretly slipping unauthorized cables into the raceways [1], I figure there are a lot of places where if your target is simply renting a couple rack units or single rack rather than an entire locked cage, you can probably get physical access by doing the same. Also, a lot of Deviant Ollam's stories about industrial security and the dozens of ways he's broken into utility companies, server rooms, etc — mostly just by being confident, looking the part, bad doors [2], and badge cloning [3] — don't give me a ton of confidence that someone with skill couldn't feasibly either get direct access to servers they shouldn't, or at the very least, access to an important part of the supply chain for their target. And speaking of supply chain, my processor died recently, so I ordered a brand-new in box replacement Ryzen, and when it arrived last night, out of curiosity, I wanted to see if I could get the CPU out of the box without breaking the tamper-evident authenticity seal... ... and about fifteen minutes later, after borrowing a syringe and hypodermic needle from my mom, a little bit of isopropyl alcohol, a blade from my safety razor, and a quick look at a video from LockPickingLawyer [4] and a couple from datagram at DEFCON's Tamper-Evident Village [5][6], I had the CPU out, put my old one for now, and re-applied the sticker with no visible damage to the box or seal. All I had to do was tip it upside down at about 90-degrees, douse a little bit of the alcohol under the top of the seal, let gravity do most of the work, and then carefully lift the seal with the razor. After that, I just lightly squeezed the box to make the front tab come as forward as possible, and then carefully pushed the ear flaps down to prevent tearing, and then I was in. I've seen others demonstrate it on older AMD boxes that had flexible cardboard in-place of the cooler, allowing them to pull the cardboard to make enough room for tools to get out the CPU without even touching the seal [7]. But in my case, it was a newer box with hard plastic inside where the cooler would've been, so that's why I went for the seal instead. No surprise to me now that counterfeiting is rampant on Amazon, with people returning the box after putting in either random junk, dead Athlons covered by a counterfeit serial-matching IHS, or the cheapest socket-compatible CPU after deluding both and swapping the IHS. I figure with a bit of practice and better tools, like Teflon spudgers and syringes, it'd be significantly easier to get past 99% of tamper-resistant/tamper-evident seals and into boxes you're not supposed to be without risking damage, and then you can intercept a package, compromise something critical, like the server BMC or firmware, reseal everything, and be on your way. And given the relatively recent scare with loads of servers, including Dell and others, being shipped with "AMERICAN MEGATRANDS" labels on their BMC boards, with no one noticing until a YouTube commenter pointed it out during a teardown by ServeTheHome, I think it's totally feasible for an enemy to just compromise the entire physical supply-chain of a company, datacenter, or whatever else [8]. [1]: Oxide's On the Metal: Kenneth Finnegan - https://oxide.computer/podcasts/on-the-metal/kenneth-finnegan https://oxide.computer/podcasts/on-the-metal/kenneth-finnega... [2]: Deviant Ollam @ Shakacon: The Search for the Perfect Door - https://www.youtube.com/watch?v=4YYvBLAF4T8 https://www.youtube.com/watch?v=4YYvBLAF4T8 [3]: Deviant Ollam / Modern Rogue: Getting an RFID Implant - https://youtu.be/SZiRISGdQ4g?t=277 https://youtu.be/SZiRISGdQ4g?t=277 [4]: LockPickingLawyer: Did I Cheat On This Challenge? (Tamper-Sealed Abus) - https://www.youtube.com/watch?v=xUJtqvYDnkg https://www.youtube.com/watch?v=xUJtqvYDnkg [5]: DEFCON 19: Introduction to Tamper Evident Devices - https://www.youtube.com/watch?v=W07ZpEv9Sog https://www.youtube.com/watch?v=W07ZpEv9Sog [6]: DEFCON 30: Tamper Evident Village - https://www.youtube.com/watch?v=slhdowWjSuU https://www.youtube.com/watch?v=slhdowWjSuU [7]: cycurious: How Counterfeiters replace CPU in Sealed Retail Box - https://www.youtube.com/watch?v=Bni8bgGlXDE https://www.youtube.com/watch?v=Bni8bgGlXDE [8]: ServeTheHome: Dude this should NOT be in a Dell Switch… or HPE Supercomputer - https://www.servethehome.com/dude-dell-hpe-ami-american-megatrands/2/ https://www.servethehome.com/dude-dell-hpe-ami-american-mega...
manxman 3y ago
Thinking laterally for a moment regarding the big picture here, why do we still rely on data centres. They made sense in a world of dialup and low speed / high latency broadband. But there are lots of places with high speed fibre and not much latency to the peering points. And the more we break away from data centres and clouds, the more the internet infrastructure will have to work the way it was designed instead of having to flow through these crazy aggregation points that are both serious points of failure and major security risks.
- WhyNotHugo 3y ago
  Regular domestic connections often block ports (especially email). In many countries, ISPs won't sell you a static IP (your dynamic address might not change ever, but you don't control rDNS records). These kind of measures force people to move onto a rented server instead. Often ISPs rent servers themselves. The conflict of interest here is hard to ignore: if ISPs make it easy and convenient to host from home, their business of renting servers suffers.
- dewey 3y ago
  > They made sense in a world of dialup and low speed / high latency broadband. But there are lots of places with high speed fibre and not much latency to the peering points. Yes, but then you need backup power, someone to replace disks / hardware if things break, proper security for compliance reasons, cooling, noise. Once you set up all these things you just invented a data center again. I don't see how getting rid of data centers makes any sense.
  - chatmasta 3y ago
    You can get all of that in your own business premises with a fiber uplink. But at the point you're staffing IT personnel and managing server racks, I suppose you may as well call the location a datacenter, albeit a private one.
- immibis 3y ago
  IP addresses, IP addresses, IP addresses, IP addresses.

LeoPanthera 3y ago

The summary of the attack from https://notes.valdikss.org.ru/jabber.ru-mitm/ https://notes.valdikss.org.ru/jabber.ru-mitm/ is very interesting: * The attacker managed to issue multiple SSL/TLS certificates via Let’s Encrypt for jabber.ru and xmpp.ru domains since 18 Apr 2023 * The Man-in-the-Middle attack for jabber.ru/xmpp.ru client XMPP traffic decryption confirmed to be in place since at least 21 July 2023 for up to 19 Oct 2023, possibly (not confirmed) since 18 Apr 2023, affected 100% of the connections to XMPP STARTTLS port 5222 (not 5223) * The attacker failed to reissue TLS certificate and MiTM proxy started to serve expired certificate on port 5222 for jabber.ru domain (Hetzner) * The MiTM attack stopped shortly after we begun our investigation and network tests on 18 Oct 2023, along with tickets to Hetzner and Linode support team, however passive wiretapping (additional routing hop) is still in place at least on a single Linode server * Neither servers appear to be hacked * Both Hetzner and Linode network appear to be reconfigured specifically for this kind of attack for the XMPP service IP addresses --- Neither that page, nor the page linked from here, mention certificate pinning, maybe because XMPP doesn't support it (I don't know), but if it did, wouldn't that have prevented this kind of attack?

jcims 3y ago
> * The attacker failed to reissue TLS certificate and MiTM proxy started to serve expired certificate on port 5222 for jabber.ru domain (Hetzner) This is gold.
- tptacek 3y ago
  The absolute lack of giving a shit is one of your major clues this was a lawful intercept scenario.
- deeth_starr_v 3y ago
  I mean, I’ve seen the auto renew fail a lot with the certbot. They definitely should have checked it in the renew period to make sure it was working, but I feel for them
- rdtsc 3y ago
  Someone was forced to do it, but they didn’t personally agree with it so they eventually made a “mistake” to tip off the target? There is the plain incompetence explanation: the hosting provider gave control of the operation to the government entity. The underpaid and indifferent government employee did the best they could with their level motivation and skill level.

tedunangst 3y ago

Run your own CA and choose your roots carefully didn't make the cut.

fanf2 3y ago
A bit difficult when providing services to third parties who can use any client software :-/
- justsomehnguy 3y ago
  Yes, but if you can serve multiple certificates on one endpoint (think SNI) then you can add your own self-signed or private PKI certificate to be able to check if all your requests are being intercepted by a lazy adversary.
- tedunangst 3y ago
  That's actually probably easier than getting a browser to work with a forbidden cert, how dare you.

liquidk 3y ago

The provider has access to the host, they can just inspect the job from the outside and you won’t be able to tell

the8472 3y ago
secure boot + virtualized memory encryption is supposed to prevent that, you'll have to trust intel/amd though.
- dist-epoch 3y ago
  Only if secure boot was enabled by a trusted party on trusted hardware. If you enable secure boot remotely without physical access to the machine you can't be sure it was actually setup in a non-compromised way. For example the machine could be running a custom backdoor-ed TPM, BIOS settings, ...
- OmarAssadi 3y ago
  I'm not sure truly verifiable "Secure" Boot is actually realistic in most scenarios, though, right? I'm not super up-to-date on what's being offered right now, but I'm not sure if there is a way to have a proper trusted execution environment on most Intel or AMD offerings; I thought Secure Boot on AMD64 platforms generally rely on TPM or something like SGX for validation, with the former having seemingly a dozen different ways to be tampered with, and the latter being discontinued and being vulnerable to several different attacks, including DOWNFALL. I think EPYC and Sapphire Rapids have some sort of Trusted Execution Environment stuff with SEV-SNP and TDX, maybe? But I don't think either option is really feasible for people paying Hetzner-like prices for hosting; Hetzner's newest Xeon offering is seemingly Cascade Lake, and the only EPYC offered is a single-socket Rome 7502P with 128GB DDR4 for 142 euros, which seems very hard to justify, given they also offer a 7950X3D with 128GB DDR5 for ~25 euros less. Even then, I don't think I could put my confidence in a machine I don't own, didn't setup, can't physically inspect, don't know where it came from, whether the firmware has been tampered with, etc -- especially if it is something as complex as x86, where there is seemingly at least one new horrific hardware-level vulnerability that crops up every generation or two. EDIT: I forgot Hetzner also started offering Ampere Altra servers for 200 euros. I think those have TEE of some sort with the TrustZone stuff? Not sure how secure that really is, though; I haven't really looked into the ARM offerings as much as I should have, mostly since, if you don't want Apple, I'm not aware of a good middle-ground between a cheap SBC and a $3,000+ Ampere server, outside of jerry-rigging some second-hand Gigabyte Cavium ThunderX2 nodes off eBay.

johncolanduoni 3y ago

They point out that "CT is optional", but isn't that ultimately a decision by the XMPP clients? What stops them from requiring CT now that browsers effectively do? What trust store would they be relying on that issues certificates that don't work in the browser?

hlandau 3y ago
Right, they could do this and this seems like a good idea to me. The problem this require-CT behaviour isn't switched on by default for basically any TLS library, so an XMPP client developer would have to go out of their way to switch this on, and I assume most or all (currently) don't. That can change of course.
- jeroenhd 3y ago
  How problematic is that in real life? All CAs worth their salt are already publishing all certificates to CT logs because most TLS servers are browsers and most clients are browsers. Maybe there's a CA somewhere that has opt-in CT logging enabled, but I don't think they're very popular. For example, even the rogue CA certificates were published onto the CT log (because Let's Encrypt does so by default). I think a CT check shouldn't be a very problematic change in practice, as long as there are reasonable exemptions (i.e. don't require CT for certificates imported into the CA store, allow the user to disable the check, etc.). For server<->server communication things become more difficult, but the entire network would be better off if a major server like ejabberd were to require CT.

mike_d 3y ago

Great callout: > Don't use Cloudflare or similar services. See my article here for an explanation on why. If you use a service like this, you're basically already MitMing yourself. I wish more people would realize that when arguing on the internet about CAA, DNSSEC, NSA, etc. that none of it really matters. We willingly allow a government aligned entity to unwrap 20% of all TLS connections on the internet and peak inside.

worldofmatthew 3y ago
Cloudflare is horrible for privacy. It is also a bit of a sovereignty issue for European countries to have all their citizens web habits to be MITM by a forging power (no matter how friendly they seam). Edit: not even going in to the sovereignty issue of having an American private company effectively decide your internet regulations.
- ronsor 3y ago
  The EU regularly de-facto tries to decide regulations for other countries. All is fair in a globally connected world.
  - Sporktacular 3y ago
    Yeah, but it generally goes in the direction of more privacy, environmental and labor standards. Less so from other countries.
  - rini17 3y ago
    That is how American tech monopolies like to paint what the EU does. Lol
    - immibis 3y ago
      dumb question: why do I have the option to downvote your comment, but not many other comments in this thread?

mkj 3y ago

I looked into CAA but the current dns provider doesn't support those records. Is there a reason it had to be a new type not a more common TXT record?

hlandau 3y ago
You can try to see if your provider supports "opaque" types in which you can submit the binary encoding of the record without them knowing what it is. Barring that you'll need to request support for it. It's an increasingly popular record type so not supporting it isn't terribly great service nowadays. I'm not familiar with the discussion that went into the design of CAA (it was probably discussed at some point on the relevant IETF mailing list, if you want to go digging).
rodlette 3y ago
See RFC5507: "Why Adding a New Resource Record Type Is the Preferred Solution" DNS providers should support a wide range of RR types, and domain owners should vote with their NS records. See https://github.com/StackExchange/dnscontrol/blob/master/documentation/providers.md https://github.com/StackExchange/dnscontrol/blob/master/docu... for a list of DNS providers that support CAA.
LinuxBender 3y ago
I have found that some providers that say they don't support it just mean in their web interface. It may be worth testing if you can set up a hidden primary server and have your provider just be a secondary to it. That is how I managed to get around that restriction with Linode.

deleted 3y ago

[deleted]

Sporktacular 3y ago

Who are the end users of xmpp.ru and jabber.ru? Are they hoping to pick up traffic between Russian soldiers? Spies? I hate mass surveillance as much as the next guy but why target Russian domains specifically?

OmarAssadi 3y ago
I can't imagine it is war-related, or at least not in any direct sense, like literally trying to intercept soldiers or spies. I think the more likely scenario is that someone was to catch carders/botnet operators, since Jabber/XMPP is still very popular amongst people in that scene in Russia; you'll see often see screenshots or logs containing @exploit.in, @jabber.ru, and various other servers pretty much in any Krebs on Security article, for example. That said, I think mass-surveillance is usually pretty gross regardless of intent. I don't know what the exact ratio of criminal to non-criminal usage of either server is, but just like how Tor simultaneously has a ton of obviously illegal content mixed with whatever percentage of legitimate use [1], they presumably still do have a fair amount of totally normal conversations going on. And so I'm not an exactly a huge fan of invading privacy in order to spy on random cybercriminals likely far out of your jurisdiction in countries unwilling to extradite in the first place. And while I understand if Hetzner was forced to comply, unless they wanted to become another Lavabit, as someone who recommends them often, it'll still be a little disappointing, IMO, if it turns out they were aware and allowed or assisted in the MitM attack, because otherwise, I think they're one of the best hosting companies. Their transparency is great. And I get better uptime, reliability, and support than the overwhelming majority of cloud providers without selling my organs to afford it. And their effort put into turning environmentally-friendly design choices into economically smart choices is commendable. The "reduce" in reduce > reuse > recycle can be seen in their KISS designs—like the raised, angled roof, allowing natural heat escape rather than loads of HVAC [2]; server chassises that are more bare-minimum structural support and air-shrouds than actual chassises; and stripped-down standard motherboard designs with only the most minimal changes, like 90-degree rotated sockets to allow a single fan to cool the VRM, CPU, and memory [3], rather than a dozen screaming, power-slurping Delta fans for every server. And their knowledge of "reuse" is evident by their server auctions, where they do Dutch-auction-style rentals of legacy hardware rather than scrapping everything, avoiding spending a load on new hardware while also avoiding the environmental cost associated with manufacturing new everything each generation. I'd love to be able use them more if only they had dedicated servers outside of Europe. [1]: "legitimate" is hard to define anyway; one country's journalist is another's "foreign agent", and one country's "freedom fighter" is another's "terrorist". [2]: Der8auer: Over 200,000 Servers in One Place! Visiting Hetzner in Falkenstein - https://youtu.be/5eo8nz_niiM?t=259 https://youtu.be/5eo8nz_niiM?t=259 [3]: Der8auer: Hetzner shows Special AM5 Board with 90° Rotated Socket - https://youtu.be/V2P8mjWRqpk https://youtu.be/V2P8mjWRqpk

manxman 3y ago

A quick warning on hetzner. I needed a personal bare metal machine so signed up. I was travelling and on an IP in a distant land so their sign up asked for secondary verification via PayPal. All passed and now it’s should get a server? Nope - next day their support emailed telling me they would not approve my account without… no word of a lie here… either 1: a fax of my passport info page or 2: a scan and email containing the same. I refused reminding them of GDPR and that email is at best opportunistically encrypted and at worst open to interception. They replied stating they believed they were GDPR compliant because all they do is use the passport to verify the account and delete the document. They also said I could hide anything sensitive other than my name and date of birth!! I suggested the process is not GDPR compliant as anyone could intercept unencrypted emails and that they should talk to a lawyer if they did not believe my assertion. Within a short time the server was approved and online. I queried if they would revise their process in light of our interaction. They did not address the question.

penumbra_3 3y ago
When was this? Few years ago, something similar happend to me but they provided pub gpg key that i could use..
petre 3y ago
They had the option to send it encrypted with PGP. But yes, this reminds me of communist countries where you had to leave your ID at the hotel upon check in. The Stasi mentality lingers on and accomplishes nothing.
- orhmeh09 3y ago
  What does this have to do with lingering Stasi mentality? Gunzenhausen was in West Germany.
- schleck8 3y ago
  You don't seem to know anything about VPS administration.
- OmarAssadi 3y ago

pabs3 3y ago

This article suggests using OTR with XMPP, unfortunately OTRv3 is using obsolete crypto and OTRv4 hasn't been finalised. https://dustri.org/b/time-to-sunset-otr.html https://dustri.org/b/time-to-sunset-otr.html

treffer 3y ago

This list lacks the most obvious one: enable (and ideally enforce) SCRAM-xxxxx-PLUS as the authentication method of choice. The idea of the PLUS variant is both simple and effective: instead of verifying <user,password> with the help of a salt you are verify <user,password,tls session key>. That way the authenticating is only valid on a single TLS connection. This is also called channel binding.

Aerbil313 3y ago

What is the fundamental technical solution for these kinds of attacks from nation-states and other powerful actors? Is there anything which can satisfy the nerdy dream of software which is freely distributable, yet completely safe from censorship, intervention or mutation by anyone other than its author? Are properly built P2P systems over the current internet infrastructure a good enough answer? Or should we rolling out a second low-bandwidth global mesh internet working over radio? I just hope Freenet 2023 evolves to have a zero-friction DX for P2P applications.