5 ms·

> What would a perfect attacker do? If you had physical access to the computer, some sort of bus interception to exfiltrate data from the machine.

whalesalad 3y ago
extremely difficult to get physical access in a datacenter
- sebstefan 3y ago
  Sure but this is the German police and more generally nation states, not only they can, they don't even need to they just ask
  - Beijinger 3y ago
    While the rule of law in Germany is much worse than most people think, it is not so bad as you assume. I doubt that Hetzner would give in to a police request. A court order, yes. But not to a police request. This does not mean, the police won't try it: https://www.dw.com/de/e-mail-firma-kritisiert-ermittler/a-18677560 https://www.dw.com/de/e-mail-firma-kritisiert-ermittler/a-18... Unfortunately can't find the original post. Every idiot police officer thought he has a right to just email them to handle over data :-)
- LeoPanthera 3y ago
  I would suggest that if you are the police, you can break into a datacenter with a flash of a badge. I can't imagine many would attempt to stop you.
  - mjevans 3y ago
    I would hope they at least: * Require a copy of the badge number, and verify that this officer is assigned and expected to be at this business right now. * Require them to sign into and out of the site. * Annotate which systems / compromises are in place. - That all of the above MIGHT be sealed under a court order; I would hope any such order has an automatic 'sunset' date, and possibly renewal upon review by a different judge.
    - LeoPanthera 3y ago
      Those are some very optimistic hopes!
    - skissane 3y ago
      A business can request visiting law enforcement to do all those things, and hopefully law enforcement complies. However, if they refuse to comply, realistically you just have to let them in anyway. Document their non-compliance and provide it to your lawyers, who can decide what action to take (lodge a formal complaint to the law enforcement agency, apply to a judge for an injunction to compel their compliance, etc) Well, that’s true in countries like Germany or the US. I suspect in somewhere like Russia or China, formal complaints are unlikely to achieve anything except invite government retaliation.
      - leosarev 3y ago
        Latter is not correct. It's well known difference in Russia between companies that willingly cooperate with government agencies informally, and those who just provide information upon formal request according to law.
        skissane 3y ago
        How do you know for sure the people who “just provide information upon formal request according to law” aren’t covertly engaging in informal cooperation? If one morning the CEO gets an unexpected visit at home from a group of FSB agents asking for some favours, is the CEO going to say “no”? And if the CEO says “yes”, are you going to hear about it, or are they going to let the CEO continue that pretence? Western CEOs don’t have the same worry about “accidentally” falling out of hospital windows.
        leosarev 3y ago
        Actually, that's happen to some people that I know. Roem.ru site (small but ifluential at time) recieved official, but illegal request from high level FSB agent to disclose commentators identities. They send formal complaint to a FSB own security and to public prosecutor office. Former officially warned FSB to stoppes illegal actions. Funny thing: 7 years later FSB agent was convicted for being CIA asset.
  - jacquesm 3y ago
    I highly doubt it is that simple for LE to enter a DC without a warrant signed by a judge, but insiders have all of that access and plants in DCs can and do happen. I was present when Dutch LE seized a bunch of servers on behalf of an FBI liaison officer in NL and everything went 'by the book', there is no way an LE officer without a signed order from a judge would have been granted access.
  - bsder 3y ago
    > I can't imagine many would attempt to stop you. You would be 99% wrong. Even if law enforcment presented proper paperwork, every colo I have ever used would call and verify the paperwork. They might not call me, but they sure as hell would call their own lawyers. Once law enforcement is on the other side of the cage, important customers who pay real money could get compromised. There is a massive difference between getting physical access to your server in a data center and coughing up everything about your server by simply emailing a minion in a cloud provider.
- mike_d 3y ago
  The organization conducting the MitM likely has physical access to the machine already. The original post indicates the link on the network interface went down for 19 seconds, indicating a device was placed in front of the server.
- jacquesm 3y ago
  All the people working in the datacenter have that level of physical access. Unless they are very closely supervised they can do a lot of damage without anybody being the wiser until they get caught. I've been in (nominally very secure) DCs on behalf of customers and I've seen: - unlocked racks - doors open - temporary network cables and keyboards, monitors and mice attached to running systems - systems logged in left unattended - floor panels raised up and left open unattended exposing cabling - meet-me rooms with interfaces exposed (gear in racks without doors) DC personnel tends to trust each other, and they probably shouldn't. But it's hard to be part of a closely knit crew for a long time without getting into a 'get stuff done' mode where protocol and rules are there in principle but less so in practice because it is seen as an efficiency penalty. It's another instance of the 'normalization of deviation' phenomenon.
  - manxman 3y ago
    Agree re: everything you said but wanted to add datadentre security staff are some of the most interesting characters I’ve encountered. Not sure I sleep as well at night after seeing what I saw.
    - OmarAssadi 3y ago
      Do tell, please; stories about "interesting characters" are often the best.
      - jacquesm 3y ago
        GP may well be under NDA and easy to identify.
- OmarAssadi 3y ago
  Assuming this was done at the government's request, I assume Hetzner is more than willing to comply with a court order mandating they allow them to monitor and physically size a machine. And outside of nation-state requests, even ignoring the fact that someone could probably pay-off an employee, I think ease would depend a lot on the datacenter and target; judging by the awesome and hilarious story behind the Fremont Cabal accidentally becoming an internet exchange by essentially having some dude barely secretly slipping unauthorized cables into the raceways [1], I figure there are a lot of places where if your target is simply renting a couple rack units or single rack rather than an entire locked cage, you can probably get physical access by doing the same. Also, a lot of Deviant Ollam's stories about industrial security and the dozens of ways he's broken into utility companies, server rooms, etc — mostly just by being confident, looking the part, bad doors [2], and badge cloning [3] — don't give me a ton of confidence that someone with skill couldn't feasibly either get direct access to servers they shouldn't, or at the very least, access to an important part of the supply chain for their target. And speaking of supply chain, my processor died recently, so I ordered a brand-new in box replacement Ryzen, and when it arrived last night, out of curiosity, I wanted to see if I could get the CPU out of the box without breaking the tamper-evident authenticity seal... ... and about fifteen minutes later, after borrowing a syringe and hypodermic needle from my mom, a little bit of isopropyl alcohol, a blade from my safety razor, and a quick look at a video from LockPickingLawyer [4] and a couple from datagram at DEFCON's Tamper-Evident Village [5][6], I had the CPU out, put my old one for now, and re-applied the sticker with no visible damage to the box or seal. All I had to do was tip it upside down at about 90-degrees, douse a little bit of the alcohol under the top of the seal, let gravity do most of the work, and then carefully lift the seal with the razor. After that, I just lightly squeezed the box to make the front tab come as forward as possible, and then carefully pushed the ear flaps down to prevent tearing, and then I was in. I've seen others demonstrate it on older AMD boxes that had flexible cardboard in-place of the cooler, allowing them to pull the cardboard to make enough room for tools to get out the CPU without even touching the seal [7]. But in my case, it was a newer box with hard plastic inside where the cooler would've been, so that's why I went for the seal instead. No surprise to me now that counterfeiting is rampant on Amazon, with people returning the box after putting in either random junk, dead Athlons covered by a counterfeit serial-matching IHS, or the cheapest socket-compatible CPU after deluding both and swapping the IHS. I figure with a bit of practice and better tools, like Teflon spudgers and syringes, it'd be significantly easier to get past 99% of tamper-resistant/tamper-evident seals and into boxes you're not supposed to be without risking damage, and then you can intercept a package, compromise something critical, like the server BMC or firmware, reseal everything, and be on your way. And given the relatively recent scare with loads of servers, including Dell and others, being shipped with "AMERICAN MEGATRANDS" labels on their BMC boards, with no one noticing until a YouTube commenter pointed it out during a teardown by ServeTheHome, I think it's totally feasible for an enemy to just compromise the entire physical supply-chain of a company, datacenter, or whatever else [8]. [1]: Oxide's On the Metal: Kenneth Finnegan - https://oxide.computer/podcasts/on-the-metal/kenneth-finnegan https://oxide.computer/podcasts/on-the-metal/kenneth-finnega... [2]: Deviant Ollam @ Shakacon: The Search for the Perfect Door - https://www.youtube.com/watch?v=4YYvBLAF4T8 https://www.youtube.com/watch?v=4YYvBLAF4T8 [3]: Deviant Ollam / Modern Rogue: Getting an RFID Implant - https://youtu.be/SZiRISGdQ4g?t=277 https://youtu.be/SZiRISGdQ4g?t=277 [4]: LockPickingLawyer: Did I Cheat On This Challenge? (Tamper-Sealed Abus) - https://www.youtube.com/watch?v=xUJtqvYDnkg https://www.youtube.com/watch?v=xUJtqvYDnkg [5]: DEFCON 19: Introduction to Tamper Evident Devices - https://www.youtube.com/watch?v=W07ZpEv9Sog https://www.youtube.com/watch?v=W07ZpEv9Sog [6]: DEFCON 30: Tamper Evident Village - https://www.youtube.com/watch?v=slhdowWjSuU https://www.youtube.com/watch?v=slhdowWjSuU [7]: cycurious: How Counterfeiters replace CPU in Sealed Retail Box - https://www.youtube.com/watch?v=Bni8bgGlXDE https://www.youtube.com/watch?v=Bni8bgGlXDE [8]: ServeTheHome: Dude this should NOT be in a Dell Switch… or HPE Supercomputer - https://www.servethehome.com/dude-dell-hpe-ami-american-megatrands/2/ https://www.servethehome.com/dude-dell-hpe-ami-american-mega...
manxman 3y ago
Thinking laterally for a moment regarding the big picture here, why do we still rely on data centres. They made sense in a world of dialup and low speed / high latency broadband. But there are lots of places with high speed fibre and not much latency to the peering points. And the more we break away from data centres and clouds, the more the internet infrastructure will have to work the way it was designed instead of having to flow through these crazy aggregation points that are both serious points of failure and major security risks.
- WhyNotHugo 3y ago
  Regular domestic connections often block ports (especially email). In many countries, ISPs won't sell you a static IP (your dynamic address might not change ever, but you don't control rDNS records). These kind of measures force people to move onto a rented server instead. Often ISPs rent servers themselves. The conflict of interest here is hard to ignore: if ISPs make it easy and convenient to host from home, their business of renting servers suffers.
- dewey 3y ago
  > They made sense in a world of dialup and low speed / high latency broadband. But there are lots of places with high speed fibre and not much latency to the peering points. Yes, but then you need backup power, someone to replace disks / hardware if things break, proper security for compliance reasons, cooling, noise. Once you set up all these things you just invented a data center again. I don't see how getting rid of data centers makes any sense.
  - chatmasta 3y ago
    You can get all of that in your own business premises with a fiber uplink. But at the point you're staffing IT personnel and managing server racks, I suppose you may as well call the location a datacenter, albeit a private one.
- immibis 3y ago
  IP addresses, IP addresses, IP addresses, IP addresses.

whalesalad 3y ago

extremely difficult to get physical access in a datacenter

sebstefan 3y ago
Sure but this is the German police and more generally nation states, not only they can, they don't even need to they just ask
- Beijinger 3y ago
  While the rule of law in Germany is much worse than most people think, it is not so bad as you assume. I doubt that Hetzner would give in to a police request. A court order, yes. But not to a police request. This does not mean, the police won't try it: https://www.dw.com/de/e-mail-firma-kritisiert-ermittler/a-18677560 https://www.dw.com/de/e-mail-firma-kritisiert-ermittler/a-18... Unfortunately can't find the original post. Every idiot police officer thought he has a right to just email them to handle over data :-)
LeoPanthera 3y ago
I would suggest that if you are the police, you can break into a datacenter with a flash of a badge. I can't imagine many would attempt to stop you.
- mjevans 3y ago
  I would hope they at least: * Require a copy of the badge number, and verify that this officer is assigned and expected to be at this business right now. * Require them to sign into and out of the site. * Annotate which systems / compromises are in place. - That all of the above MIGHT be sealed under a court order; I would hope any such order has an automatic 'sunset' date, and possibly renewal upon review by a different judge.
  - LeoPanthera 3y ago
    Those are some very optimistic hopes!
  - skissane 3y ago
    A business can request visiting law enforcement to do all those things, and hopefully law enforcement complies. However, if they refuse to comply, realistically you just have to let them in anyway. Document their non-compliance and provide it to your lawyers, who can decide what action to take (lodge a formal complaint to the law enforcement agency, apply to a judge for an injunction to compel their compliance, etc) Well, that’s true in countries like Germany or the US. I suspect in somewhere like Russia or China, formal complaints are unlikely to achieve anything except invite government retaliation.
    - leosarev 3y ago
      Latter is not correct. It's well known difference in Russia between companies that willingly cooperate with government agencies informally, and those who just provide information upon formal request according to law.
      - skissane 3y ago
        How do you know for sure the people who “just provide information upon formal request according to law” aren’t covertly engaging in informal cooperation? If one morning the CEO gets an unexpected visit at home from a group of FSB agents asking for some favours, is the CEO going to say “no”? And if the CEO says “yes”, are you going to hear about it, or are they going to let the CEO continue that pretence? Western CEOs don’t have the same worry about “accidentally” falling out of hospital windows.
        leosarev 3y ago
        Actually, that's happen to some people that I know. Roem.ru site (small but ifluential at time) recieved official, but illegal request from high level FSB agent to disclose commentators identities. They send formal complaint to a FSB own security and to public prosecutor office. Former officially warned FSB to stoppes illegal actions. Funny thing: 7 years later FSB agent was convicted for being CIA asset.
- jacquesm 3y ago
  I highly doubt it is that simple for LE to enter a DC without a warrant signed by a judge, but insiders have all of that access and plants in DCs can and do happen. I was present when Dutch LE seized a bunch of servers on behalf of an FBI liaison officer in NL and everything went 'by the book', there is no way an LE officer without a signed order from a judge would have been granted access.
- bsder 3y ago
  > I can't imagine many would attempt to stop you. You would be 99% wrong. Even if law enforcment presented proper paperwork, every colo I have ever used would call and verify the paperwork. They might not call me, but they sure as hell would call their own lawyers. Once law enforcement is on the other side of the cage, important customers who pay real money could get compromised. There is a massive difference between getting physical access to your server in a data center and coughing up everything about your server by simply emailing a minion in a cloud provider.
mike_d 3y ago
The organization conducting the MitM likely has physical access to the machine already. The original post indicates the link on the network interface went down for 19 seconds, indicating a device was placed in front of the server.
jacquesm 3y ago
All the people working in the datacenter have that level of physical access. Unless they are very closely supervised they can do a lot of damage without anybody being the wiser until they get caught. I've been in (nominally very secure) DCs on behalf of customers and I've seen: - unlocked racks - doors open - temporary network cables and keyboards, monitors and mice attached to running systems - systems logged in left unattended - floor panels raised up and left open unattended exposing cabling - meet-me rooms with interfaces exposed (gear in racks without doors) DC personnel tends to trust each other, and they probably shouldn't. But it's hard to be part of a closely knit crew for a long time without getting into a 'get stuff done' mode where protocol and rules are there in principle but less so in practice because it is seen as an efficiency penalty. It's another instance of the 'normalization of deviation' phenomenon.
- manxman 3y ago
  Agree re: everything you said but wanted to add datadentre security staff are some of the most interesting characters I’ve encountered. Not sure I sleep as well at night after seeing what I saw.
  - OmarAssadi 3y ago
    Do tell, please; stories about "interesting characters" are often the best.
    - jacquesm 3y ago
      GP may well be under NDA and easy to identify.
OmarAssadi 3y ago
Assuming this was done at the government's request, I assume Hetzner is more than willing to comply with a court order mandating they allow them to monitor and physically size a machine. And outside of nation-state requests, even ignoring the fact that someone could probably pay-off an employee, I think ease would depend a lot on the datacenter and target; judging by the awesome and hilarious story behind the Fremont Cabal accidentally becoming an internet exchange by essentially having some dude barely secretly slipping unauthorized cables into the raceways [1], I figure there are a lot of places where if your target is simply renting a couple rack units or single rack rather than an entire locked cage, you can probably get physical access by doing the same. Also, a lot of Deviant Ollam's stories about industrial security and the dozens of ways he's broken into utility companies, server rooms, etc — mostly just by being confident, looking the part, bad doors [2], and badge cloning [3] — don't give me a ton of confidence that someone with skill couldn't feasibly either get direct access to servers they shouldn't, or at the very least, access to an important part of the supply chain for their target. And speaking of supply chain, my processor died recently, so I ordered a brand-new in box replacement Ryzen, and when it arrived last night, out of curiosity, I wanted to see if I could get the CPU out of the box without breaking the tamper-evident authenticity seal... ... and about fifteen minutes later, after borrowing a syringe and hypodermic needle from my mom, a little bit of isopropyl alcohol, a blade from my safety razor, and a quick look at a video from LockPickingLawyer [4] and a couple from datagram at DEFCON's Tamper-Evident Village [5][6], I had the CPU out, put my old one for now, and re-applied the sticker with no visible damage to the box or seal. All I had to do was tip it upside down at about 90-degrees, douse a little bit of the alcohol under the top of the seal, let gravity do most of the work, and then carefully lift the seal with the razor. After that, I just lightly squeezed the box to make the front tab come as forward as possible, and then carefully pushed the ear flaps down to prevent tearing, and then I was in. I've seen others demonstrate it on older AMD boxes that had flexible cardboard in-place of the cooler, allowing them to pull the cardboard to make enough room for tools to get out the CPU without even touching the seal [7]. But in my case, it was a newer box with hard plastic inside where the cooler would've been, so that's why I went for the seal instead. No surprise to me now that counterfeiting is rampant on Amazon, with people returning the box after putting in either random junk, dead Athlons covered by a counterfeit serial-matching IHS, or the cheapest socket-compatible CPU after deluding both and swapping the IHS. I figure with a bit of practice and better tools, like Teflon spudgers and syringes, it'd be significantly easier to get past 99% of tamper-resistant/tamper-evident seals and into boxes you're not supposed to be without risking damage, and then you can intercept a package, compromise something critical, like the server BMC or firmware, reseal everything, and be on your way. And given the relatively recent scare with loads of servers, including Dell and others, being shipped with "AMERICAN MEGATRANDS" labels on their BMC boards, with no one noticing until a YouTube commenter pointed it out during a teardown by ServeTheHome, I think it's totally feasible for an enemy to just compromise the entire physical supply-chain of a company, datacenter, or whatever else [8]. [1]: Oxide's On the Metal: Kenneth Finnegan - https://oxide.computer/podcasts/on-the-metal/kenneth-finnegan https://oxide.computer/podcasts/on-the-metal/kenneth-finnega... [2]: Deviant Ollam @ Shakacon: The Search for the Perfect Door - https://www.youtube.com/watch?v=4YYvBLAF4T8 https://www.youtube.com/watch?v=4YYvBLAF4T8 [3]: Deviant Ollam / Modern Rogue: Getting an RFID Implant - https://youtu.be/SZiRISGdQ4g?t=277 https://youtu.be/SZiRISGdQ4g?t=277 [4]: LockPickingLawyer: Did I Cheat On This Challenge? (Tamper-Sealed Abus) - https://www.youtube.com/watch?v=xUJtqvYDnkg https://www.youtube.com/watch?v=xUJtqvYDnkg [5]: DEFCON 19: Introduction to Tamper Evident Devices - https://www.youtube.com/watch?v=W07ZpEv9Sog https://www.youtube.com/watch?v=W07ZpEv9Sog [6]: DEFCON 30: Tamper Evident Village - https://www.youtube.com/watch?v=slhdowWjSuU https://www.youtube.com/watch?v=slhdowWjSuU [7]: cycurious: How Counterfeiters replace CPU in Sealed Retail Box - https://www.youtube.com/watch?v=Bni8bgGlXDE https://www.youtube.com/watch?v=Bni8bgGlXDE [8]: ServeTheHome: Dude this should NOT be in a Dell Switch… or HPE Supercomputer - https://www.servethehome.com/dude-dell-hpe-ami-american-megatrands/2/ https://www.servethehome.com/dude-dell-hpe-ami-american-mega...

manxman 3y ago

Thinking laterally for a moment regarding the big picture here, why do we still rely on data centres. They made sense in a world of dialup and low speed / high latency broadband. But there are lots of places with high speed fibre and not much latency to the peering points. And the more we break away from data centres and clouds, the more the internet infrastructure will have to work the way it was designed instead of having to flow through these crazy aggregation points that are both serious points of failure and major security risks.

WhyNotHugo 3y ago
Regular domestic connections often block ports (especially email). In many countries, ISPs won't sell you a static IP (your dynamic address might not change ever, but you don't control rDNS records). These kind of measures force people to move onto a rented server instead. Often ISPs rent servers themselves. The conflict of interest here is hard to ignore: if ISPs make it easy and convenient to host from home, their business of renting servers suffers.
dewey 3y ago
> They made sense in a world of dialup and low speed / high latency broadband. But there are lots of places with high speed fibre and not much latency to the peering points. Yes, but then you need backup power, someone to replace disks / hardware if things break, proper security for compliance reasons, cooling, noise. Once you set up all these things you just invented a data center again. I don't see how getting rid of data centers makes any sense.
- chatmasta 3y ago
  You can get all of that in your own business premises with a fiber uplink. But at the point you're staffing IT personnel and managing server racks, I suppose you may as well call the location a datacenter, albeit a private one.
immibis 3y ago
IP addresses, IP addresses, IP addresses, IP addresses.