7 ms·

Instapaper Server Update

163 points by psychotik 15y ago

ck2 15y ago
Assume all data on the drives has been cloned. There's little chance they didn't.
- m0nastic 15y ago
  He should absolutely assume that the data has been cloned, but there is about a 99% chance that it wasn't. If the data was cloned on-site, then we wouldn't have heard about the search warrant (there'd have been a gag order). And as this happened yesterday, there's almost no chance that the server could have been taken to the lab, imaged, and then returned to be reconnected today. I would bet 100 dollars that the actual target server of the raid is still sitting on a dolly in the forensics cage while a tech is waiting for someone to tell him what he's supposed to be looking for.
  - Skroob 15y ago
    You'd be surprised. I've worked with forensic law enforcement in the past, and they were equipped to clone drives quickly and easily. The idea is to create a clean image of the drive that they can use for evidence, and they don't boot the machine so if there's any trapdoors or whatever they won't hit them. And this was a local police force over 5 years ago, I can't imagine what the FBI would be capable of these days.
    - m0nastic 15y ago
      The FBI has the capacity to clone drives quickly, absolutely (although amusingly, the process is actually slower now than it was then, as drives are larger). However, field cloning is absolutely not the desired procedure to use (for a lot of reasons). You only do field cloning if you are either: 1) Under a time crunch, or 2) Don't want the target to know that the clone has occurred. If at all possible, you just take the equipment and process it in the lab. I've done about 2 dozen of these for the FBI. We did field-cloning maybe three times.
    - amorphid 15y ago
      Some should build a hard drive that logs being powered up and powered down!
      - m0nastic 15y ago
        I wouldn't be surprised if SSD's somehow had information like that stored in them (or at least the ability for it to be stored). Thankfully I got out of the Forensics racket before they were mainstream.
      - duskwuff 15y ago
        SMART power cycle count?
  - mike-cardwell 15y ago
    99%? I'd assume that the first thing they do on seizing a machine is pull the hard drive out and drop it in a disk cloning machine?
- ConceitedCode 15y ago
  I like to always follow the saying "Hope for the best. Plan for the worst". Assume the FBI has a copy of everything on the drives and assume that they will see every file on the drive, even though odds are they don't and they won't.
- TheCondor 15y ago
  What do you base this upon? He should contact the FBI and continue to beat on Digital One and determine if it was seized or not. It very well may have simply been turned off, is that not a valid possibility?
ichilton 15y ago
I hope he securely erased the data rather than just deleting it, otherwise it might not be just the FBI who have potentially got a copy...
- nettdata 15y ago
  Would there not also be backups at DigitalOne?
  - SeoxyS 15y ago
    With dedicated servers, the responsibility is on the client to back up what matters to him. Ditto with VPSs.
cheald 15y ago
Out of curiosity, does anyone know how volume-level encryption (like dm-crypt) holds up against this sort of thing? I find myself wondering what I'd tell customers if my server were seized as collateral damage like Instapaper's were. Will that sort of encryption serve as a plausible safeguard of customer data, or is it more of a padlock (easily broken with the right tools)?
- psychotik 15y ago
  You tell them what Marco told them.
- Tharkun 15y ago
  It would certainly make it much harder to steal the data -- and at the same time it would make it much harder to boot the machine, because at some point you'd have to enter a passphrase. It's possible to do so over SSH, but it's still a bit tricky. Rebooting the server while you're away from an internet connexion will leave it in a useless state. That being said, even with volume encryption, the key is in memory somewhere. RAM isn't wiped as quickly as you might think, and it's apparently possible to extract keys from memory after the machine has been powered down. So be warned, and be paranoid!
  - JoachimSchipper 15y ago
    Indeed, the encryption keys can be recovered from memory fairly easily. http://www.usenix.org/events/sec08/tech/full_papers/halderman/halderman.pdf http://www.usenix.org/events/sec08/tech/full_papers/halderma... is very readable; http://www.youtube.com/watch?v=JDaicPIgn9U http://www.youtube.com/watch?v=JDaicPIgn9U is a well-made 5-minute introduction by the authors.
    - mike-cardwell 15y ago
      I thought this issue had been partially addressed by overwriting the memory containing the keys on unmount. Of course, that doesn't work if somebody just pulls the plug on the machine. There was an article I read recently about a patch for the Linux kernel to store the key in CPU registers instead of memory as well...
tswicegood 15y ago
+1 to the attitude. It's all about getting the service back up and making it better. We (collectively) tend to spend too much time trying to figure out who to blame instead of just doing the cool shit that needs to be done. Kudos!
- SeoxyS 15y ago
  I wish more people thought the way you do. You, sir, are an inspiration.
- Gorbzel 15y ago
  Sadly, I simply cannot agree. I absolutely understand Marco's desire to focus on developing Instapaper, but there's quite a bit more at stake here, and his willingness to move on without even a bit of protest is concerning. He claims that there's nothing that can be done, but there are definitely legal actions one can take in this scenario. Sure, they might not ever fully compensate him, nor convince the skeptics, but it will at least reveal the details behind the seizure, and it will send a message to both the FBI and to others like DigitalOne that could prevent further "seize first and ask questions later" style operations.
  - statictype 15y ago
    So for a one-man software shop, given the choice between: a) Spend money and time on legal action against the FBI for holding onto his servers for a couple of days while in the process of an investigation, in order to hopefully try and make a larger point. and b) Spending his time and no money by actually improving his bottom line by enhancing the features of his software, you expect any rational person to choose (a) ?
    - guelo 15y ago
      The attitude might make sense for marco but we should all be thankful for the irrational stubborn assholes that say screw it and fight and effect change and make things better for the rest of us. The martyr game is usually a long thankless ordeal. You need to be an idealist dreamer, have your heart 110% in it and be prepared for the long slog after the initial passion cools off. Even someone like geohot who had the passion, the financial resources, tons of support and nothing to lose, eventually got cold feet once he saw the daunting task ahead.
gojomo 15y ago
Should services consider storing all user data on encrypted volumes? Pro: after a powerdown, seizure/theft/cloning of the volume won' reveal user data Con: need a secure way to supply decryption passphrase on each reboot
Supermighty 15y ago
I think a neat idea would be to create 1U - 4U lock doors that could be bolted in front of a server in a rack. I wonder if this would be enough to give each server it's own legal "separate space" insomuch as to prevent a whole rack from being seized due to warrants like this.
- jsatok 15y ago
  The server that was taken from Marco was a blade server, and he just rented a single node in the blade. Locking the single blade he was on wouldn't have prevented the situation that occurred, as the FBI seized a number of enclosures, rather than servers. Most likely, the agents who took the servers didn't understand a blade server, and took the entire enclosure, rather than pulling a couple nodes out of the blade.
patio11 15y ago
I think you're more than justified in assuming no compromise happened. Someone with legitimate authority had physical access to your server. That has happened before and will happen again. Your server's cage opens several times a month, probably, as techs and whatnot do their icky atom-touching sorcery. Physical access to your server by the adversary means untraceable compromise, sure, but if that is true, hasn't it happened before? (Sure, the techs might be more trustworthy than the FBI... Unless they were the FBI, which is paranoid squared, but if you think that agents executing a warrant routinely grab servers in the vicinity because, hey, computers tend to have juicy stuff on them and we have way too much free time, then assuming infiltration seems almost reasonable by comparison.) In a threat environment this bad, you would personally be scrutinizing sign-in lists every month. You have better things to do, right? Right. Because this is really not Instapaper's threat environment. You got mildly inconvenienced as a side-effect of routine government operations. You can pick whether you want to escalate that to A Full Crisis. That seems like a bad call, which you recognize on the legal side. Do it on the security side, too. You may think that vocal parties will disagree with "This was a non-event for us.". They might. Screw them. You're a CEO; dealing with paranoid ramblings does not generate business value like two months of normal operations does.
- IgorPartola 15y ago
  Isn't there a way to get information from HDD's about how long they have operated from when they were manufactured using SMART? Not that anyone keeps those records on hand once the HDD is no longer in their possession, but if you are paranoid, you could start offloading those records every minute to a different server. That way if yours is seized, you could at least know if someone had turned on your HDD's and for how long. It'd be cool if HDD's/SSD's started compiling rough stats about numbers of reads and writes for this purpose.
richardblair 15y ago
Man, you have an awesome service. It served me greatly through my post secondary education. I am glad you got your server back, and you can keep on keepin on. Again, thank you for the services you offer.
nanoanderson 15y ago
Tangent alert! Marco makes me feel great. One of my favorite things about Marco is that he gives me confidence that there is not a gaping wide chasm between the skills of the best engineer-entrpreneurs and me. This time, it was his frank admission that he was taught by commenters and emailers than his current password encryption was unsafe and that there were better options out there. When you realize that someone as successful as Marco has the same learning to do that "the rest of us" do, it makes me feel great.
- sneak 15y ago
  News flash: Security breaches happen because almost everyone in the industry sucks at this stuff, including (most importantly) those who SHOULD KNOW BETTER, yourself and Mr. Arment included.
  - nanoanderson 15y ago
    And now we do. :-)
- absconditus 15y ago
  Why do you assume that Marco is one of "the best engineer-entrepreneurs"?
cosgroveb 15y ago
"I’m not convinced that they did everything they could to prevent the seizure of non-targeted servers, and their lack of proactive communication with the affected customers is beneath the level of service I expect from a host." I can't speak to the level of service you did or did not receive but I seriously doubt DigitalOne could have done more to prevent "non-targeted" servers from being seized. If it was in the same rack as the "targeted" server I'm sure that 100% of the time LE would take it and find out whether it was related later. It's their job to take every step necessary to preserve evidence first and foremost when executing a search warrant like this. I agree with you that the data was almost certainly not compromised and I think it's great that you're so proactive about protecting your customers by moving to bcrypt.
suking 15y ago
This is the exact opposite approach dropbox takes - well done with good communication. Props.
smackfu 15y ago
This is all just assumptions, since amazingly the hosting company still has no comments at all. Who exactly rebooted the server? Not the FBI, for damn sure.

163 points by psychotik 15y ago

ck2 15y ago

Assume all data on the drives has been cloned. There's little chance they didn't.

m0nastic 15y ago
He should absolutely assume that the data has been cloned, but there is about a 99% chance that it wasn't. If the data was cloned on-site, then we wouldn't have heard about the search warrant (there'd have been a gag order). And as this happened yesterday, there's almost no chance that the server could have been taken to the lab, imaged, and then returned to be reconnected today. I would bet 100 dollars that the actual target server of the raid is still sitting on a dolly in the forensics cage while a tech is waiting for someone to tell him what he's supposed to be looking for.
- Skroob 15y ago
  You'd be surprised. I've worked with forensic law enforcement in the past, and they were equipped to clone drives quickly and easily. The idea is to create a clean image of the drive that they can use for evidence, and they don't boot the machine so if there's any trapdoors or whatever they won't hit them. And this was a local police force over 5 years ago, I can't imagine what the FBI would be capable of these days.
  - m0nastic 15y ago
    The FBI has the capacity to clone drives quickly, absolutely (although amusingly, the process is actually slower now than it was then, as drives are larger). However, field cloning is absolutely not the desired procedure to use (for a lot of reasons). You only do field cloning if you are either: 1) Under a time crunch, or 2) Don't want the target to know that the clone has occurred. If at all possible, you just take the equipment and process it in the lab. I've done about 2 dozen of these for the FBI. We did field-cloning maybe three times.
  - amorphid 15y ago
    Some should build a hard drive that logs being powered up and powered down!
    - m0nastic 15y ago
      I wouldn't be surprised if SSD's somehow had information like that stored in them (or at least the ability for it to be stored). Thankfully I got out of the Forensics racket before they were mainstream.
    - duskwuff 15y ago
      SMART power cycle count?
- mike-cardwell 15y ago
  99%? I'd assume that the first thing they do on seizing a machine is pull the hard drive out and drop it in a disk cloning machine?
ConceitedCode 15y ago
I like to always follow the saying "Hope for the best. Plan for the worst". Assume the FBI has a copy of everything on the drives and assume that they will see every file on the drive, even though odds are they don't and they won't.
TheCondor 15y ago
What do you base this upon? He should contact the FBI and continue to beat on Digital One and determine if it was seized or not. It very well may have simply been turned off, is that not a valid possibility?

ichilton 15y ago

I hope he securely erased the data rather than just deleting it, otherwise it might not be just the FBI who have potentially got a copy...

nettdata 15y ago
Would there not also be backups at DigitalOne?
- SeoxyS 15y ago
  With dedicated servers, the responsibility is on the client to back up what matters to him. Ditto with VPSs.

cheald 15y ago

Out of curiosity, does anyone know how volume-level encryption (like dm-crypt) holds up against this sort of thing? I find myself wondering what I'd tell customers if my server were seized as collateral damage like Instapaper's were. Will that sort of encryption serve as a plausible safeguard of customer data, or is it more of a padlock (easily broken with the right tools)?

psychotik 15y ago
You tell them what Marco told them.
Tharkun 15y ago
It would certainly make it much harder to steal the data -- and at the same time it would make it much harder to boot the machine, because at some point you'd have to enter a passphrase. It's possible to do so over SSH, but it's still a bit tricky. Rebooting the server while you're away from an internet connexion will leave it in a useless state. That being said, even with volume encryption, the key is in memory somewhere. RAM isn't wiped as quickly as you might think, and it's apparently possible to extract keys from memory after the machine has been powered down. So be warned, and be paranoid!
- JoachimSchipper 15y ago
  Indeed, the encryption keys can be recovered from memory fairly easily. http://www.usenix.org/events/sec08/tech/full_papers/halderman/halderman.pdf http://www.usenix.org/events/sec08/tech/full_papers/halderma... is very readable; http://www.youtube.com/watch?v=JDaicPIgn9U http://www.youtube.com/watch?v=JDaicPIgn9U is a well-made 5-minute introduction by the authors.
  - mike-cardwell 15y ago
    I thought this issue had been partially addressed by overwriting the memory containing the keys on unmount. Of course, that doesn't work if somebody just pulls the plug on the machine. There was an article I read recently about a patch for the Linux kernel to store the key in CPU registers instead of memory as well...

tswicegood 15y ago

+1 to the attitude. It's all about getting the service back up and making it better. We (collectively) tend to spend too much time trying to figure out who to blame instead of just doing the cool shit that needs to be done. Kudos!

SeoxyS 15y ago
I wish more people thought the way you do. You, sir, are an inspiration.
Gorbzel 15y ago
Sadly, I simply cannot agree. I absolutely understand Marco's desire to focus on developing Instapaper, but there's quite a bit more at stake here, and his willingness to move on without even a bit of protest is concerning. He claims that there's nothing that can be done, but there are definitely legal actions one can take in this scenario. Sure, they might not ever fully compensate him, nor convince the skeptics, but it will at least reveal the details behind the seizure, and it will send a message to both the FBI and to others like DigitalOne that could prevent further "seize first and ask questions later" style operations.
- statictype 15y ago
  So for a one-man software shop, given the choice between: a) Spend money and time on legal action against the FBI for holding onto his servers for a couple of days while in the process of an investigation, in order to hopefully try and make a larger point. and b) Spending his time and no money by actually improving his bottom line by enhancing the features of his software, you expect any rational person to choose (a) ?
  - guelo 15y ago
    The attitude might make sense for marco but we should all be thankful for the irrational stubborn assholes that say screw it and fight and effect change and make things better for the rest of us. The martyr game is usually a long thankless ordeal. You need to be an idealist dreamer, have your heart 110% in it and be prepared for the long slog after the initial passion cools off. Even someone like geohot who had the passion, the financial resources, tons of support and nothing to lose, eventually got cold feet once he saw the daunting task ahead.

gojomo 15y ago

Should services consider storing all user data on encrypted volumes? Pro: after a powerdown, seizure/theft/cloning of the volume won' reveal user data Con: need a secure way to supply decryption passphrase on each reboot

Supermighty 15y ago

I think a neat idea would be to create 1U - 4U lock doors that could be bolted in front of a server in a rack. I wonder if this would be enough to give each server it's own legal "separate space" insomuch as to prevent a whole rack from being seized due to warrants like this.

jsatok 15y ago
The server that was taken from Marco was a blade server, and he just rented a single node in the blade. Locking the single blade he was on wouldn't have prevented the situation that occurred, as the FBI seized a number of enclosures, rather than servers. Most likely, the agents who took the servers didn't understand a blade server, and took the entire enclosure, rather than pulling a couple nodes out of the blade.

patio11 15y ago

I think you're more than justified in assuming no compromise happened. Someone with legitimate authority had physical access to your server. That has happened before and will happen again. Your server's cage opens several times a month, probably, as techs and whatnot do their icky atom-touching sorcery. Physical access to your server by the adversary means untraceable compromise, sure, but if that is true, hasn't it happened before? (Sure, the techs might be more trustworthy than the FBI... Unless they were the FBI, which is paranoid squared, but if you think that agents executing a warrant routinely grab servers in the vicinity because, hey, computers tend to have juicy stuff on them and we have way too much free time, then assuming infiltration seems almost reasonable by comparison.) In a threat environment this bad, you would personally be scrutinizing sign-in lists every month. You have better things to do, right? Right. Because this is really not Instapaper's threat environment. You got mildly inconvenienced as a side-effect of routine government operations. You can pick whether you want to escalate that to A Full Crisis. That seems like a bad call, which you recognize on the legal side. Do it on the security side, too. You may think that vocal parties will disagree with "This was a non-event for us.". They might. Screw them. You're a CEO; dealing with paranoid ramblings does not generate business value like two months of normal operations does.

IgorPartola 15y ago
Isn't there a way to get information from HDD's about how long they have operated from when they were manufactured using SMART? Not that anyone keeps those records on hand once the HDD is no longer in their possession, but if you are paranoid, you could start offloading those records every minute to a different server. That way if yours is seized, you could at least know if someone had turned on your HDD's and for how long. It'd be cool if HDD's/SSD's started compiling rough stats about numbers of reads and writes for this purpose.

richardblair 15y ago

Man, you have an awesome service. It served me greatly through my post secondary education. I am glad you got your server back, and you can keep on keepin on. Again, thank you for the services you offer.

nanoanderson 15y ago

Tangent alert! Marco makes me feel great. One of my favorite things about Marco is that he gives me confidence that there is not a gaping wide chasm between the skills of the best engineer-entrpreneurs and me. This time, it was his frank admission that he was taught by commenters and emailers than his current password encryption was unsafe and that there were better options out there. When you realize that someone as successful as Marco has the same learning to do that "the rest of us" do, it makes me feel great.

sneak 15y ago
News flash: Security breaches happen because almost everyone in the industry sucks at this stuff, including (most importantly) those who SHOULD KNOW BETTER, yourself and Mr. Arment included.
- nanoanderson 15y ago
  And now we do. :-)
absconditus 15y ago
Why do you assume that Marco is one of "the best engineer-entrepreneurs"?

cosgroveb 15y ago

"I’m not convinced that they did everything they could to prevent the seizure of non-targeted servers, and their lack of proactive communication with the affected customers is beneath the level of service I expect from a host." I can't speak to the level of service you did or did not receive but I seriously doubt DigitalOne could have done more to prevent "non-targeted" servers from being seized. If it was in the same rack as the "targeted" server I'm sure that 100% of the time LE would take it and find out whether it was related later. It's their job to take every step necessary to preserve evidence first and foremost when executing a search warrant like this. I agree with you that the data was almost certainly not compromised and I think it's great that you're so proactive about protecting your customers by moving to bcrypt.

suking 15y ago

This is the exact opposite approach dropbox takes - well done with good communication. Props.

smackfu 15y ago

This is all just assumptions, since amazingly the hosting company still has no comments at all. Who exactly rebooted the server? Not the FBI, for damn sure.