8 ms·
F.B.I. Seizes Web Servers, Knocking Sites Offline
- ChuckMcM 15y agoCollateral damage in a digital world. If this gets out of hand it could play havoc with the 'Cloud' business model.
- cheez 15y agoOnly the little guys will get hurt in this instance.
- pointillistic 15y agoAnd by little guy you mean all the multinational corporation that got hacked?
- deleted 15y ago[deleted]
- cheez 15y agoSorry, I was under the impression that random small businesses got damaged from the the government investigation into the inconveniencing of big multinational corporations. But you may also read it like that if you like.
- pointillistic 15y agoIf hackers get out of hand it already playing havoc in all business.
- pavel_lishin 15y ago"Won't somebody think of the hackers", eh? Thank you, Mrs. Flanders.
- deleted 15y ago[deleted]
- phlux 15y agoSo, in thinking about how to mitigate unforeseen disasters while trying to maintain anonymity - I think the following solution to such a problem would be really easy (not free, but easy) Host your site/content across multiple cloud providers. Then host VPS web servers with other providers and have them balance between the backend cloud servers. Then put cloudflare on top of that. Ideally, you would want to have multiple service instances with the cloud providers which are silo'd from each other - but exact duplicates (e.g. they are under fully different accounts with the cloud provider) Your front-end web servers keep a cache of content from the cloud servers - and they are pushed to. For even lower-end content needs, you can pull from a drop box account or something. Any of the front end gear gets seized - You can bring it up with a VPS image really quickly.
- RyanKearney 15y agoEven easier, you use Softlayer and use their global load balancer.
- mechanical_fish 15y agoThen host VPS web servers with other providers and have them balance between the backend cloud servers. If you don't know what the word latency means, this setup will teach you all about it. Another problem is that your various providers will probably charge you for WAN bandwidth. So you will pay three times for every request: Twice from backend to balancer (one charge from provider A, one from provider B) plus another charge to send the same data back out from the balancer to the customer. Plus you will be miserable trying to keep your site up 100% of the time across two cloud providers. Have a problem on either one, and 50% of your capacity will go offline. Might be better to realize that "my servers were seized by the FBI" is a rare occurrence and you can probably afford a few hours' worth of downtime and/or data loss. Make offsite backups from your primary provider on a relatively long timescale (once per day, maybe once per hour if you're more sensitive; live database replication for the crazy-sensitive) and have a procedure for spinning those up at a secondary provider. Test that procedure every month or so. The beautiful thing about cloud services is that you can pay for your emergency-backup servers by the hour and only when you are using them, or testing them.
- pbz 15y agoPlease tell me this is illegal and FBI will be sued over this...
- deleted 15y ago[deleted]
- camiller 15y agoIf you are small enough to have to colo, you probably can't afford to sue the FBI.
- ugh 15y agoCouldn’t the web host sue?
- there 15y agosmall enough to colo? what world do you live in?
- phlux 15y agoHe is <24 years old and his primary world view has been formed whilst "cloud" was the pinnacle of technical obviousness.
- camiller 15y ago>40 and primary world view formed working in fortune 100 corporate environment. Basically, every company I have ever worked for has had it's own multiple large data centers. I guess the point I didn't try very hard to make was, do you think a company the size of pinboard has the financial resources to engage in a prolonged legal battle with the FBI? I'm thinking not, but I could be wrong.
- kragen 15y agoThe US court system is not nearly as utterly corrupt as you seem to think. Even individual people often fight law enforcement agencies in court there, and win. Certainly a Fortune 100 company would stand a better chance. (Maciej might be at a bit of a disadvantage, being a foreigner, and one who's written publicly about a run-in with law enforcement in China, at that.) There's a pretty large range between Pinboard (10 000 customers, maybe? Two employees? Annual revenue in the range of $100k? I'm guessing here) and Rite Aid (109000 "associates", maybe 20 million customers, $26 billion revenue). More than five orders of magnitude, actually.
- ghshephard 15y agoPro-Tip – when you are using a CoLo for your servers: * Make sure you failover to servers in a separate cabinet. * Better yet, make sure you failover to servers in a different data center. * Best Possible – Configure a task to switch your primary and failover environment from data center to data center once a week at random times. Or eliminate the concept of primary and secondary altogether. Who knows what others servers are located in the cabinet you are sharing. The good news is your mitigation process also becomes a first class disaster recovery plan as well.
- RyanKearney 15y agoOr the FBI could just go through the proper channels like everyone else and contact the datacenter and request it's immediate takedown (after providing a warrant, of course). This isn't the first time the FBI busted into a data center all willy-nilly knocking innocent peoples hardware offline. Not only does this adversely effect the innocent users, it also almost destroys the datacenters reputation and it's a loss for nearly every party.
- tlb 15y agoFBI seizures are only a tiny fraction of outages. Lots of things knock out servers: power failures, water leaks, A/C failures, theft, fire, explosions, zombie outbreaks, runaway forklifts, mice chewing on cables. The only solution is backup hosting in a separate data center.
- RyanKearney 15y agoHonestly, the only legit reason for an outage is some natural disaster took down the entire building. I really wouldn't want my server hosted at a mice infested building anyway, or one without backup power, cheap A/C units, etc. To me, it looks much worse that your server went offline because the FBI busted into the datacenter and knocked shit around than say an earthquake took out our fiber lines. Personally, if I heard the FBI raided a datacenter and knocked a bunch of servers offline which had nothing to do with what they were after, then I'd seriously question the security, legal team, and response team of that place. To me, it sounds like the FBI just showed up to one guy sitting in a chair watching TV and let the FBI roam free.
- ericd 15y agoCan anyone tell me why they take whole cabinets of servers rather than just yanking the offending ones?
- code_duck 15y agoMy evaluation is that their priority is preservation of evidence, and they consider a few random websites being down for a couple of days to be relatively unimportant.
- ericd 15y agoTrue, but unless they think someone at the colo is going to tip off the owners, taking 10 minutes to figure out which machine hosts which site shouldn't be an issue. With a warrant, I imagine the colo would be willing to look up the mapping from company -> rack/server being leased.
- preinheimer 15y agoif anyone associated with the colo tipped someone off (or there was cameras in the colo streaming) 10 minutes is more then enough to start deleting things.
- learc83 15y agoJust power down the whole rack then decide.
- smallblacksun 15y ago>unless they think someone at the colo is going to tip off the owners, That is exactly what they are worried about happening.
- jleader 15y agoKeep in mind that the FBI agent-in-charge's perceived cost for pissing off some small web companies is pretty low; their perceived cost for tipping off whoever they're after, and blowing the entire operation, is much, much higher.
- preinheimer 15y agoI lost my server in Estonia for over a week when the local police working with the FBI shut everything down looking for a command and control node for a bot net. Fun times.
- antihero 15y agoWhich countries are fairly immune to the FBI?
- killerdark 15y agoI've seen very little FBI activity in Antarctica lately.
- nkassis 15y agochina?
- gabaix 15y agoDoes anyone know who the target was? If it's LulzSec, as the article assumes, I would hope they have good reasons to do so. LulzSec is a group of hackers searching for recognition. By chasing them they are giving them more publicity.
- ryoshu 15y agoHow expensive is it to spin up a VPS and start poisoning data centers? Three racks to take down one site? It would be a nightmare of a whack-a-mole game on a global scale.
- jackfoxy 15y agoI've been in the biz long enough to see several paradigms touted as the end-all solution. Today it's the cloud, but it's not too hard to envision something the size of your smart-phone, or maybe even your smart-phone, as the server. There are a lot of good reasons to keep the server in your physical possession. The network infrastructure to do that is just about there.
- skimbrel 15y ago> The network infrastructure to do that is just about there. There's a long way to go from having enough bandwidth to serve your personal blog or small-time web app out of your {apartment, house, office} to getting what you'd need for a large-scale web app (at a minimum, redundant 100 Mbps fiber connections with five-nines SLAs) routed there. That and getting a nominal 100 Mbps connection from your friendly neighborhood telco is a lot different from getting a 100 Mbps connection you're expecting to saturate 24/7. The telco won't put up with that for very long, because they overcommit their subscriber bandwidth in the (correct) expectation that most of their customers will not use all of it.
- jackfoxy 15y ago"Large-scale web app", true.
- Goladus 15y agoI wonder if whoever was operating the target server knew that it was 25 miles from FBI headquarters.
- pavel_lishin 15y agoDo you think that made any difference?
- antihero 15y agoThanks, FBI.
- hsmyers 15y agoI'm probably missing something, by the Times article doesn't mention the word 'warrant'---that being the case shouldn't that ruffle a feather or two?
- revscat 15y agoI suspect the magic spell "national security" was cast. Hackers seem to be becoming the new terrorists.
- killerdark 15y agoHackers seem to be becoming the new terrorists AGAIN. Second time for me that they turned the cassette over. Makes me wonder how many times this has happened since Gutenberg started this mess?
- deleted 15y ago[deleted]
- Nelson69 15y agoSo you were sharing "virtual servers" with other companies and the FBI seized the actual servers? Or something to that effect? That raises some interesting questions, the impression every seems to be operating on is that they stormed in and took 3 cabinets or full racks with computers owned by other companies. All things being equal, this sounds like the normal operating procedure for the FBI.
- zoowar 15y agoI wish the FBI was as diligent at pursuing corporate malfeasance.
- andreyf 15y agoI'm surprised there aren't laws against this. Can a business sue the FBI for ruining them like this? Surely, they could put together a team which can go in, copy the hard drives they need, and be out in a couple of hours. Even then, they should evaluate and repay whatever damage they caused?
- mmaunder 15y agoThe impact of this clear case of incompetence by the FBI is minor compared to what it could have been. A 1U Dell R610 can fit 96G of memory and two very fast 64 bit processors. You can fit 48 of these servers in a standard rack. Running VM's, that configuration could easily support 20 virtual machines per server (4GB/machine + OS) which gives you 960 websites per rack. And that assumes you need an entire machine for each website (no virtual hosts). So the next time the FBI decides to knock over someone's rack, the impact could be a lot wider than a handful of sites. US security agencies are becoming increasingly cavalier when it comes to seizing domains and hardware. This is becoming a significant risk for online startups like my company and the companies of other HN members. The suggestions here to mitigate this risk are not cheap. Setting up a fast enough link between data centers to have real-time replication is prohibitively expensive for most small companies. I'd like to see legislation that lets us know what our rights are and that lays out a standard procedure for these kinds of data center incursions. I'd hate to see a cloud provider's rack get taken down in one of these raids, and I don't think any of us are happy about the government using our tax dollars to settle costly lawsuits caused by their own incompetence.
- bcl 15y agoAccording to the 4th Amendment a warrant is supposed to: "...and no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized." When they seize unrelated hardware they are clearly overstepping their bounds. I'd like to see the innocent parties take this to court and put a stop to this. We are far past the time when the FBI can claim ignorance as to how a colocation site operates.
- patio11 15y agoThis is becoming a significant risk for online startups like my company and the companies of other HN members. It's like a meteor strike: well reported, flashy, but very very not likely to actually happen to your site. How many websites do HNers collectively operate? How many have ever gone down because the FBI took their hardware? How many have ever gone down because, oh, the hard drive crashed? Because they pushed bad code live? Because they misconfigured a firewall? Because the hosting company had a network or power issue? Those are real risks for your business. (Bonus points: many interventions for these and similar issues fixes your FBI problem, too!) In the event of a totally freak incident like this, you're probably going to have downtime and a day's worth of data loss, but recovery for most folks here is likely "Reimage the VPS(s) from the latest backup, hit the on switch, and change DNS records."
- eurohacker 15y agolooks like some american oligarchs really dont like the existence of the internet , http://www.youtube.com/watch?v=Ct9xzXUQLuY http://www.youtube.com/watch?v=Ct9xzXUQLuY
- ams6110 15y agoA government official who declined to be named said earlier in the day that the F.B.I. was actively investigating the Lulz Security group and any affiliated hackers. So these cracker groups have finally woken the sleeping giant? And the bleary-eyed half-awake giant is swinging his arms around wildly, demolishing everything in reach? Maybe after some coffee and donuts they'll be a bit more delicate.
- dendory 15y agoIf you dont keep redundant servers in other countries nowadays you never know when you're going to be left in the cold by some random government action.
- meow 15y agoThis is like sending elephants to catch mice. They cause more trampling than all the mice combined.
- mikiem 15y agoI am the founder and current CEO of a dedicated server host. The FBI and other law enforcement agencies do often contact us regarding activity emanating from or related to our network. Its the same for any host and the bigger you are, the more often it happens. Rarely, a local law enforcement officer from within or outside of the US get cranky when you tell ask them to get a subpoena (makes work for them). However, in my experience the FBI and other three letter agencies are very professional, fair, and genuinely want to do it right. Getting a subpoena is no big deal at all for them. Usually the US Attorney is right down the hall from them, in the same building. They do officially have policies and programs designed to befriend local businesses in the area of a field office. They just want to catch the bad guys and not mess with the innocent. They want people to like them and trust them. They count on the cooperation of businesses, especially hosting and access providers. There are civilized processes for everyone to get what they want. There are the cases of people trying to get information who are not actually law enforcement, and that is one of the many reasons you must ask for a subpoena... To protect yourself and your customers privacy. The FBI can get one in a very short time. Where this all goes bad is when you do not respond to subpoenas for subscriber information, when you don't hand over the disks, etc. If you do not comply, then what alternative does the FBI have but to come and get it? My guess?... The host didn't play nice with or respond promptly to the FBI.
- vegasbrianc 15y agoA little bit of devil advocate here. If the server rack was full of virtual hosting that would mean one server could ideally host multiple websites and if all the servers shared 1 or 2 SAN's it would be difficult from an outsider looking in to know which server hosted what website especially with Graveyard shift. Even if you knew node 1 server hosted the site it would still be shared on the SAN and could easily be started on another server.
- nphase 15y agoI wonder what happens when they decide to hit an AWS EC2 datacenter cabinet like this.