4 ms·

>Also I wonder how they got a server image without him noticing since it is typically something you'd need to shut down the machine for. Was the whole thing run

by lsc 13y ago

>Also I wonder how they got a server image without him noticing since it is typically something you'd need to shut down the machine for. Was the whole thing running off a hosting service? How often does one of your servers crash? I mean, it happens. I estimate maybe once a year/server, on average, assuming a 5 year lifecycle. (well, usually it's more like 'no crashes for the first three years, several crashes a year after' - hardware ages.) Hell, whole racks lose power at times. Doesn't happen all that often, but it happens often enough that if your provider says "We blew breaker X" well, more often than not, it's a honest problem, and not the FBI yanking power to image a drive. Or hell... what if it's a server with a mirrored drive? It'd be easy enough to pull half the mirror (the drive 'failed' right? Hell, you can say you let the salesguy into the co-lo and he bumped the hard drive release catch, or you sent in the new kid to swap a drive and they pulled the wrong one. These things aren't common, but they are way more common than the FBI.) Hell, a drive could have legitimately failed and been sent back to seagate/wd by the provider (assuming he was renting servers) for warranty repair. The FBI could have intercepted the drive (or gotten it from the manufacturer) and run their own analysis. So yeah. I totally believe that the FBI could get a reasonable image without DPR or anyone being the wiser.

bigiain 13y ago
"Hell, whole racks lose power at times. Doesn't happen all that often, but it happens often enough that if your provider says "We blew breaker X" well, more often than not, it's a honest problem, and not the FBI yanking power to image a drive." Now you've got me wondering whether the apparent disparity between manufacturers claimed MTBF and what we see in failure rates in the real world, might plausibly be attributed to mysterious government agencies coercing data center owners into unexpected-but-plausible downtime. (four or five nines of power uptime might just mean the FBI/NSA need to batch server imaging and grab a whole bunch in a particular data center at once)
- lsc 13y ago
  >Now you've got me wondering whether the apparent disparity between manufacturers claimed MTBF and what we see in failure rates in the real world, might plausibly be attributed to mysterious government agencies coercing data center owners into unexpected-but-plausible downtime. (four or five nines of power uptime might just mean the FBI/NSA need to batch server imaging and grab a whole bunch in a particular data center at once) It's far more likely that people are idiots. How many hardware techs do you know who even own an ESD wrist strap? I get actively ridiculed when I pull mine out. Next, the SLAs claimed by datacenters are usually bullshit on multiple levels. First, the penalty is usually "we will refund you for the time you were down, if you ask." - which is fine, but a 5 minute power outage can be brutal to clean up after, while 5 minutes of your monthly bill is hardly worth asking for. I'd be happy to give people a 100% sla on those terms. I mean, obviously, the service isn't going to be up 100%, but the penalties are so low that who cares? Then, well, even if the facility doesn't lose power, there are a hundred different ways a server or a rack can lose power. Hell, even I let a guy into my co-lo who plugged in one of those ancient computers with a manual 110-240v switch. (everything made in the last decade auto-switches.) He plugged it into my 208v power, with the switch on 110, causing the fuses on my PDU to blow (and taking out the whole rack) And power cords. Especially if you don't have dual power supplies, power cords get bumped. The mark of a honest sysadmin is that s/he admits it when they bump the cord[1] So yeah, while it /could/ be the FBI, the vast majority of the time, well, someone fucked up. [1]http://blog.prgmr.com/xenophilia/2013/06/more-downtime-on-javier.html http://blog.prgmr.com/xenophilia/2013/06/more-downtime-on-ja...
  - bigiain 13y ago
    (Adds lsc to the list of likely NSA collaborators…) ;-) And yeah, you're right about hosting SLAs - I've got a hosting account which proudly advertises "100% uptime guarantee", which in the fineprint/t&cs offers "pro rata refunds for _twice_ your costs of any downtime!" – on a $48/year invoice - so if they go down for an entire _week_, they'll owe me not quite two whole dollars. Thanks... Even the much more expensive/professional hosting I arrange for other clients always includes something like: Limitation of Damages Recovery of damages from $hostingCompany may not exceed the amount of fees it has collected on the account.
    - lsc 13y ago
      >(Adds lsc to the list of likely NSA collaborators…) ;-) The interesting thing is that I haven't ever been served with a warrant. Which is weird, as I know much smaller competitors who have. Of course, there's no reason why you should believe that statement.

by lsc 13y ago

bigiain 13y ago

"Hell, whole racks lose power at times. Doesn't happen all that often, but it happens often enough that if your provider says "We blew breaker X" well, more often than not, it's a honest problem, and not the FBI yanking power to image a drive." Now you've got me wondering whether the apparent disparity between manufacturers claimed MTBF and what we see in failure rates in the real world, might plausibly be attributed to mysterious government agencies coercing data center owners into unexpected-but-plausible downtime. (four or five nines of power uptime might just mean the FBI/NSA need to batch server imaging and grab a whole bunch in a particular data center at once)

lsc 13y ago
>Now you've got me wondering whether the apparent disparity between manufacturers claimed MTBF and what we see in failure rates in the real world, might plausibly be attributed to mysterious government agencies coercing data center owners into unexpected-but-plausible downtime. (four or five nines of power uptime might just mean the FBI/NSA need to batch server imaging and grab a whole bunch in a particular data center at once) It's far more likely that people are idiots. How many hardware techs do you know who even own an ESD wrist strap? I get actively ridiculed when I pull mine out. Next, the SLAs claimed by datacenters are usually bullshit on multiple levels. First, the penalty is usually "we will refund you for the time you were down, if you ask." - which is fine, but a 5 minute power outage can be brutal to clean up after, while 5 minutes of your monthly bill is hardly worth asking for. I'd be happy to give people a 100% sla on those terms. I mean, obviously, the service isn't going to be up 100%, but the penalties are so low that who cares? Then, well, even if the facility doesn't lose power, there are a hundred different ways a server or a rack can lose power. Hell, even I let a guy into my co-lo who plugged in one of those ancient computers with a manual 110-240v switch. (everything made in the last decade auto-switches.) He plugged it into my 208v power, with the switch on 110, causing the fuses on my PDU to blow (and taking out the whole rack) And power cords. Especially if you don't have dual power supplies, power cords get bumped. The mark of a honest sysadmin is that s/he admits it when they bump the cord[1] So yeah, while it /could/ be the FBI, the vast majority of the time, well, someone fucked up. [1]http://blog.prgmr.com/xenophilia/2013/06/more-downtime-on-javier.html http://blog.prgmr.com/xenophilia/2013/06/more-downtime-on-ja...
- bigiain 13y ago
  (Adds lsc to the list of likely NSA collaborators…) ;-) And yeah, you're right about hosting SLAs - I've got a hosting account which proudly advertises "100% uptime guarantee", which in the fineprint/t&cs offers "pro rata refunds for _twice_ your costs of any downtime!" – on a $48/year invoice - so if they go down for an entire _week_, they'll owe me not quite two whole dollars. Thanks... Even the much more expensive/professional hosting I arrange for other clients always includes something like: Limitation of Damages Recovery of damages from $hostingCompany may not exceed the amount of fees it has collected on the account.
  - lsc 13y ago
    >(Adds lsc to the list of likely NSA collaborators…) ;-) The interesting thing is that I haven't ever been served with a warrant. Which is weird, as I know much smaller competitors who have. Of course, there's no reason why you should believe that statement.