6 ms·
Pinboard.in service limited - FBI raided hosting company and pulled equpiment
- idlewords 15y agoWe're not down. Our main DB server is unreachable and there is limited service (no API, search or feeds) while we run off a smaller backup server. See http://status.pinboard.in/ http://status.pinboard.in/ or @pinboard on Twitter for updates.
- buro9 15y agoThanks, one question though... I've just been making some more bookmarks... is your backup server database now your master? So when you get your main databases back (or a replacement), will my bookmarks remain when you switch back?
- idlewords 15y agoIt's a master-master setup with only one accepting writes at any time. So it should catch up first, and then I'll be able to switch back over to the original DB without data loss.
- jvehent 15y agoGood architecture. And nice of you to explain clearly and completely the problem and to invite your users to backup. I'd like to see that more often, but more companies are too scared to lose customers to do that.
- ojilles 15y agoWhich is exactly why I'm staying with pinboard!
- agilo 15y agoBut they'll win new ones when this story and the likes come to their attention, and when they learn how the company handled it with honesty and aplomb. I for one was unaware of this service and this story has made me consider signing-up.
- rufibarbatus 15y ago@idlewords: would you have stats at hand? Maybe a temporary loss of functionality earned you a few users above the daily average.
- reustle 15y agoIt seems civilian causalities will always exist.
- marceldegraaf 15y agoI love how Pinboard deals with this issue: site remains online (albeit with limited abilities), all data is secured and backed up, users are encouraged to use the export tool if they feel the need and there are status updates on Twitter. Keep it up Pinboard!
- wmeredith 15y agoThis is why I love Pinboard. They seem like they always do it right by the customer. Not to mention it's dead-simple to use.
- bgraves 15y agoOne thing that does work is our export page, and if you're feeling nervous about your data I encourage you to use it. We have fresh backups (from one day ago) safely stored on S3, but in these matters you can't be too careful. +1 Maciej. I am glad I paid for an account.
- product50 15y agoDropbox's Arash should learn something from this.
- ldayley 15y agoThe brave new world of overzealous law enforcement data warehouse searches just continues to wreak havoc on "cloud" services and content. It's akin to digging up the neighborhood to search somebody's house. What can the Feds/providers do differently to prevent this from happening so often?
- owenmarshall 15y ago>What can the Feds/providers do differently to prevent this from happening so often? The feds could do plenty, but won't. The legal system is slow to react, and the feds likely fear that just taking virtual disks would lead to evidentiary issues at trial. Hosting providers might be able to provision in a way that the servers hosting evil.site could be taken by the feds but backups existed of the other sites, but it would be an extremely hard -- if not impossible -- task. In the end, cloud computing doesn't change the rules of safely deploying your site to the world -- if you care about uptime & your data, your site must be running on at least two different hosts.
- SoftwareMaven 15y agoTwo different hosts separated by a distance.
- jacques_chester 15y agoTwo different hosts separated by a border would be better.
- seabee 15y ago> and the feds likely fear that just taking virtual disks would lead to evidentiary issues at trial. Honest question: why are physical hard disks more valid as evidence than virtual ones? Aren't they equally easy to tamper with?
- wmf 15y agoYou probably can't take a virtual disk; you could copy it, which answers your question: a copy is considered less valid than the original.
- ltamake 15y agoA site I visit frequently got its server raided by the FBI (the host was rewiredHost; it was unrelated to the site I visit). The site had to start from scratch except for their frontend code. I wonder why the FBI is raiding all these servers. Another comment mentioned a few hosts they've hit.
- illumin8 15y agoA popular method used by hackers is to sign up for a virtual server with a stolen credit card. If they are careful and only access it through a proxy, their hacking attempts are virtually untraceable. With the amount of hacking going on lately by Lulzsec and other groups, there is bound to be a lot of collateral damage.
- pbhjpbhj 15y agoSounds like a very black hat way of taking out your unprepared competition. I'm sure you can buy this sort of service somewhere - get a hit on a datacentre/rack by security services in response to highly illegal activity perpetrated in the open in order to cause the most possible disruption.
- illumin8 15y agoI didn't mean to imply they would intentionally do that to take out competition, although that is indeed a possibility. The main purpose of having a stolen virtual server would be to launch attacks from. The truth is even if you rented a virtual server in the same data center, your chances of being in the same rack of hardware are pretty slim.
- buzzmunk 15y agoWhat I find awesome is that instead of being disgruntled or disappointed, the way in which pinboard has handled this situation has reinforced my confidence and appreciation for it's service. They are truly a case study in customer awareness and communication for the tech industry.
- jarek 15y agoCan't wait to read the write-up of that.
- rubergly 15y agoI just started using Pinboard a couple days ago (mostly because of integration with Instapaper), and so far it looks great. The fact that, even with all this chaos going on, the main functionality of the service is still up and running is really great.
- noonespecial 15y agoThe FBI is just another in the list of bad things that can happen to your servers. Flood, fire, theft, law-enforcement... its all the same thing to a hosted service. Off site backups, executed masterfully here saves the day. There are a few services that I run on servers in separate countries. Failure of reasonable rule of law within a single country is a failure mode we consider.
- boredguy8 15y agoSadly this isn't implied, so let me suggest: "Off site backups with a tested restore process saves the day"
- idlewords 15y agoThis was a failover, not a restore. And we were just lucky that the failover machines (in the same datacenter) were not affected.
- dorian-graph 15y agoDo you think you will change hosts and/or have secondary services elsewhere?
- redthrowaway 15y agoWhy not spin up some EC2 instances if your main datacenter fails? You can pay minimal costs to send a diff of your database a couple times a day in order to keep the backup current, and only pay for real bandwidth if your main server goes down. Seems to be a pretty decent backup plan.
- sudonim 15y agoIs it possible to insure / sue for careless seizures by government agencies? Pinboard is (probably?) not the one they are after. What recourse do they have?
- 15y ago
- deleted 15y ago[deleted]
- EwanToo 15y agoIt's interesting to think through what would happen with AWS in the same situation. Would the FBI turn up and say "Where's the 100 servers for customer X", then seize up to 100 different physical servers, depending on the distribution? Or even 100 racks worth of physical servers...
- jvandenbroeck 15y agoI think they just asked for a server on an IP and the datacenter staff (!= not the DigitalOne staff) said the IP was in the range of DigitalOne - so the FBI took them all. I think the datacenter will rent IP ranges to their clients. I can imagine in an Amazone datacenter they would be able to point to the exact server they're looking for.
- Maxious 15y agoI think the OP's point is that Amazon EC2 can run 100s of instances for one customer which could be on up to 100 different physical servers and when not using EBS roots, any one of those servers might still contain the evidence the FBI wants. Mentioning the specifics of Amazon further, what about if your account for the highly illegal operation was using/paying for EBS/S3/SimpleDB/RDS/SQS/SNS/SES (or all at the same time). Any could contain forensic evidence of a crime as they hold data in some sense. EBS especially is likely to run on some kind of SAN; would they have to crack open the racks and take out individual drives? would they have to take whole arrays of discs because of RAID-esque striping? It's like a LEO denial of service (both on amazon and the forensic analysis). Pragmatically, they might trust amazon enough to consolidate/quarantine the data into the smallest surface area first.
- newobj 15y agoReally impressed by your ability to weather this storm. Definitely going to take a closer look at the service and see if it's useful to me (I'm a kind of 'write-only' Instapaper user... could Pinboard help improve my read rate at all?)
- iphoneedbot 15y agoWow! I wonder if Amazon would ever get hit like this? How can a hosting company protect itself from being effectively shutdown? theres a PRODUCT in this debacle somewhere! Hat tip to Pinboard for doing a great job dealing with this crisis!
- jvandenbroeck 15y agoLet this be a lesson to chose your hosting company wisely. You can't always predict such things, but I think if you pick a big name, with a good business reputation, you'll minimize the risk. They might do checks for abuse and have plans in place to keep everything up when something goes wrong. Eg. I can't imagine that if you host with http://www.rackspace.com/ http://www.rackspace.com/ your website would be down if you didn't do anything illegal. DigitalOne's site is also down http://www.digitalone.com/ http://www.digitalone.com/ so they don't even have a backup server for their own website..
- oscardelben 15y agoAll big names were once small. I doubt pinboard didn't choose their hosting company wisely, although I get what you're saying. Troubles and down times happen to anyone, it's how good you can deal with them that makes a difference, not the hosting company you choose.
- jvandenbroeck 15y agoOfcourse, there's no blame on pinboard, they handled it amazing! And the webhosting company was around for 7 years if I googled it correctly - but I do think there's a big difference between webhosting company's. Suppose DigitalOne rents space in a datacenter and has no one on-site (exept for staff from the datacenter). I can imagine that if the FBI enters there "we need servers from IP x.x.x.x and y.y.y.y immediately - that the people from the datacenter just point to the servers from DigitalOne "oh that's in their range".. Ofcourse that's just speculation if you don't know how it really went down - never the less, I would chose webhosting carefully if your business depends on it:)
- idlewords 15y agohttp://www.eweek.com/c/a/Cloud-Computing/Final-Thoughts-on-the-FiveDay-AWS-Outage-236462/ http://www.eweek.com/c/a/Cloud-Computing/Final-Thoughts-on-t... Shit happens.
- JeffDClark 15y agoI wonder if the law enforcement folks look at the "collateral damage", i.e., all the innocent servers (virtual or otherwise), as a windfall of extra data to mine. If my neighbor gets his house raided by the SWAT team they do not have access to my house only his, but with hosting this is not necessarily true. Further, I at least know who my neighbors are and can choose to not live in a bad neighborhood or even move away if I want, however with hosting this is not the case.
- falcolas 15y agoSuddenly, the PCI compliance requirement to encrypt all sensitive "data at rest" makes a lot more sense. Probably not the case with Pinboard, but interesting nontheless.
- deleted 15y ago[deleted]
- toddh 15y agoIf the data was replicated in N places would they serve all N places? Just curious how that works.
- mikiem 15y agoI am the founder and current CEO of a dedicated server host. The FBI and other law enforcement agencies do often contact us regarding activity emanating from or related to our network. Its the same for any host and the bigger you are, the more often it happens. Rarely, a local law enforcement officer from within or outside of the US get cranky when you tell ask them to get a subpoena (makes work for them). However, in my experience the FBI and other three letter agencies are very professional, fair, and genuinely want to do it right. Getting a subpoena is no big deal at all for them. Usually the US Attorney is right down the hall from them, in the same building. They do officially have policies and programs designed to befriend local businesses in the area of a field office. They just want to catch the bad guys and not mess with the innocent. They want people to like them and trust them. They count on the cooperation of businesses, especially hosting and access providers. There are civilized processes for everyone to get what they want. There are the cases of people trying to get information who are not actually law enforcement, and that is one of the many reasons you must ask for a subpoena... To protect yourself and your customers privacy. The FBI can get one in a very short time. Where this all goes bad is when you do not respond to subpoenas for subscriber information, when you don't hand over the disks, etc. If you do not comply, then what alternative does the FBI have but to come and get it? My guess?... The host didn't play nice with the FBI.
- davidu 15y agoOr the FBI believed the host was aware of, sanctioning, in cahoots with, or participating in the alleged activity and thus could not tip their cards. This happens more often than you might think.
- thaumaturgy 15y agoCan a law-knowledgeable individual shed some light on what this means for individuals who weren't a part of the warrant? i.e., I'm hosted on the same server as Joe, who the FBI are investigating. They seize the equipment we share. During the course of their investigation, they naturally also examine my data. Do I then have a viable lawsuit or claim towards unlawful search & seizure, or invasion of privacy? Or, if I happened to also have illegal content on the same system, would they be able to use the evidence they encountered there in a case against me?
- faboo 15y agoIn general, law enforcement has to get very specific warrants accurately indicating where they're going to search and what they hope to seize. This includes both physical locations and, usually, computers. Additionally, case law makes a distinction between property that you personally own and/or control (e.g. your house that you live in), and property that is shared (e.g. the apartment you rent with a buddy), and recognizes that the part of something that is "yours" (your room in your shared apartment, for instance) is separate from you buddy's (their room). (I imagine this is because the 4th amendment protects your papers and effects from search and seizure, not just your real estate.) So, all things being equal, the FBI is supposed to take pains not to even look at the data of people not named on the warrant (automated processes have been exempted from this, IIRC, as they aren't actual people), even though your data is housed on the save server as the person(s) named on the warrant. And if they did by chance see something they weren't supposed to, it wouldn't be admissible in court (in a case against you - they might be able to use it against someone else). IANAL, I just took a couple of law courses in college.
- thaumaturgy 15y ago> ...they might be able to use it against someone else... That's an interesting point. So, if I'm running a hosting company, and they come across data from one of my customers...? Also, putting aside for a moment IT best practices and all that, they potentially are crippling my business by seizing machines which I share with someone else. Is there any recourse for that?
- shrike 15y agoI've been a Pinboard customer for a long time, I think it is an example of a well executed, very well designed service that just blends into the background even when I use it a dozen times a day.
- chrisjsmith 15y agoHere's where hosting on EC2 might actually be worth it.