20 ms·
How We Got Owned by a Few Teenagers (and Why It Will Never Happen Again)
- pdenya 16y agoWhat a crazy story. If the timelines are accurate there was an extremely small chance of this happening. Bad luck all around. My site is still down, guess i'm in the unlucky 1%.
- ReinH 16y agoCan you give us more details at help.phpfog.com please? (apologies if I'm breaking HN etiquette)
- jacques_chester 16y agoMurphy's extended laws: * It will go wrong at the worst possible time. * If there are 4 possible ways for it to go wrong and you prevent all of them, it will go wrong in a 5th way.
- hitonagashi 16y agoYou don't hear about the 99% of the attempts that were successfully foiled by good coding practices/security audits, so given that we are hearing about a security breach at a competent firm, the likelyhood of it being a black swan event is probably fairly high.
- Legion 16y agoYou also don't hear about the attempts that were successful and unbeknownst to the hacked service provider.
- greendestiny 16y agoYeah but the problem with 'things planned for the near future' is that they have a tendency to stay in the future until something like this happens.
- 16s 16y agoYes, and after the fact, they sound like excuses (even when they are true and they are not meant to be excuses as in this case).
- Aaronontheweb 16y agoGoes to show you why the DRY principle (I might be stretching that analogy here, but bear with me) is important here - if you have old stuff lying around in production that was cloned a long time ago, you might forget about it and open yourself up to unfortunate incidents like this. PHP Fog is doing great work to make the PHP ecosystem easier to work with, and I hope they didn't suffer too much from this mistake.
- tjarratt 16y agoStill stretching the analogy, but the same could be said of their password reuse.
- uxp 16y agoThat's exactly what I thought of. DRYP - Don't repeat your passwords.
- smokestack 16y agoI don't think DRY applies here unless you really stretch it. The "fix it if you see it, don't put it off" principle fits a lot better.
- sriramk 16y agoI feel really bad for the phpfog guys. But given the situation, I think they handled it admirably well - kudos to them. No software is secure and this could have happened to anyone. Especially startups who have to take shortcuts at the very beginning. I know the attackers were just kids but I have to admit pursuing legal action sounds very tempting - even to just act as a deterrent to others. If they had just put up phpfogsucks.com, it might have been ok. But tweeting trash from their twitter account, redirecting their root domain to phpfogsucks, etc - are all not cool at all and should have some consequences.
- BasDirks 16y agoSecurity has nothing to do with feelings. Given the amount of should-haves in their explanation this should be a welcome wake-up call to them, a free audit. Other than playing the victim card, they handled it quite well. Nothing better to sharpen your skills and higher your standards than a _good crisis_
- Apocryphon 16y agoHire the hackers. It's what the CIA would do.
- sriramk 16y agoOnly if they demonstrated great skill - which I don't think happened in this case (but I'm no security expert either)
- sliverstorm 16y agoAnd great restraint/maturity. Which didn't happen either.
- jarin 16y agoHiring the hackers is a terrible idea. If they had done it ethically, yes. But do you really want to hire someone who has already displayed highly unethical behavior, and is likely a ticking timebomb? There are plenty of smart, ethical hackers out there.
- deleted 16y ago[deleted]
- mikelbring 16y agoActually phpfog is built on Ruby
- jonursenbach 16y agoThey're actually a Ruby shop according to the leaked codebase.
- dashr 16y agogreat to hear all the details so quickly so that others building similar systems aren't in the same situation. as fellow PHP'ers its also great to hear that you are not blaming it on PHP somehow (no fuel for the php haters).
- svlla 16y agophp... a language by amateurs, for amateurs. phpfog... a service by amateurs, for amateurs.
- mikey_p 16y agoYour argument could also apply to the Linux kernel.
- skeletonjelly 16y agoI know the point you're trying to make, but it's a weak one. 75% of the kernel is written by corporate employees. http://apcmag.com/linux-now-75-corporate.htm http://apcmag.com/linux-now-75-corporate.htm
- mikey_p 16y agoTrue, I don't know what percentage of contributions to PHP are made while in employee of a corporation funding said contributions, but there are commercial supporters of PHP out there. I'm surprised how few people don't know that Zend exists. The original comment just struck me oddly because I've come to assume that a large portion of the software that I use regularly was originally developed as an amateur project, or by amateurs.
- zaidf 16y ago75% of the kernel is written by corporate employees. And that implies what about the kernel?
- deleted 16y ago[deleted]
- skeletonjelly 16y agoI'm not sure where you're heading with this. I was trying to tell the parent comment that arguing against 'amateur' coders writing php is the same as the Linux kernel was factually incorrect.
- citricsquid 16y agoI mentioned this last time, but I don't think anyone was interested, but the "John" guy is compwhizii (same handle on Twitter) who runs the forums (facepunch.com) for garrysmod, a very popular game. I will be curious to see how garry (owner person) responds to this, or if he already has. Elliot is apparently VERY scared and blames John (compwhizii) (edit: not john, he blames someone else called supersnail1): http://www.facepunch.com/threads/1071855-A-member-of-Facepunch-may-cause-me-to-be-sued http://www.facepunch.com/threads/1071855-A-member-of-Facepun... Here is (compwhizii) Johns reply: http://www.facepunch.com/threads/1071855-A-member-of-Facepunch-may-cause-me-to-be-sued?p=28754506&highlight=#post28754506 http://www.facepunch.com/threads/1071855-A-member-of-Facepun...
- sriramk 16y agoReading through those forum posts, I have less and less sympathy for these guys.
- deleted 16y ago[deleted]
- nbpoole 16y agoAnd here's Elliot's "official statement": http://elliotspeck.com/phpfog.html http://elliotspeck.com/phpfog.html And for anyone who missed it, here's what Elliot posted in the previous HN discussion about the phpFog breach: http://news.ycombinator.com/item?id=2346161 http://news.ycombinator.com/item?id=2346161
- va_coder 16y agoAs soon he says "I don't believe I did a bad thing" I thought this boy needs to be prosecuted.
- nbpoole 16y agoPersonally, I'm a fan of some of his post-hoc justification: "Following this, I took a hold of their Twitter account and posted a couple of bits to draw attention to the fact. This did two things. One, it showed people the system was insecure, but on the other hand people always subconsciously root for the underdog; I drew attention to the company and the product. I know a number of people have actually registered (or intend to register, registration is closed) for phpFog since the incident thanks to the attention drawn to it by myself."
- geekfactor 16y ago"We have hired professional white hat hackers with government level security experience to attempt regular pen tests on our system..." I guess whenever I read this kind of statement from now on I'll be thinking of HBGary and chuckling a bit inside.
- bmelton 16y agoAt the risk of that comment being taken as a joke, I've done a lot of work with the federal government, and I can assure you that while the level of hilarity that HBGary has generated, the typical level of talent in government cleared individuals is not necessarily great. I don't mean to impune the capabilities of the people involved (I don't know who they are,) and it isn't to say that you can't find some AMAZING talent in the government realm, but as in all fields, it's the exception, not the rule.
- jarin 16y agoReminds me of when I was in the Navy and I went to Navy Security and Vulnerabilities Technician school. I was all excited, so I went out and bought a copy of Hackers Exposed and read through the whole thing, learning everything from how to determine what family and version of operating system a computer is running by what ports are open, to how a buffer overflow attack actually works. Fast-forward to the class, and we're sitting there running tools like BackOrifice that exploit vulnerabilities that had been patched for years, and learning that a SYN flood is "a malicious attack". That's it, just "a malicious attack". When I asked about the difference between a SYN flood and a Christmas tree attack, I got a blank stare and "they're both malicious attacks". I spent the rest of the class in the back of the room, reading the Armadillo Book. Also, I did not once in my brief Navy career get to hack an enemy computer. Hugely disappointing.
- bmelton 16y agoThat's exactly it. I've worked for agencies where the Chief IT Security Officer (if they have one, or an equivalent role) got that exact training, and nothing more, and was considered the site expert. Like I said, it isn't all bad. Two of the best security guys I know work in the government, and one of them was actually ex-Navy. But the hurdle for finding people that can get top secret clearances AND tie their shoes often proves too high to hire anybody, much less somebody qualified.
- tjarratt 16y agoThe phpfog guys really deserve praise for being so open on this issue. As a fellow engineer, being able to learn from their mistakes and see exactly what they could have done ahead of time to avoid the disaster is priceless. Just goes to show that those with the time to spend are the most likely to break your stuff, even if you pay "professional white hat hackers" to test your system.
- mtogo 16y agoOn the contrary, they knew of security vulnerabilities and intentionally left them unpatched, then blamed it on chance and timing when they got owned because of it. Avoid phpfog if at all possible, in my opinion.
- tjarratt 16y agoIf I lost $1 for each time I left a known vulnerability unpatched because I was convinced I had more important work to do, I would be a very poor man indeed. Honestly, there are very very few developers that fix security problems in beta environments before anything else. In my experience, it's more likely that you're fighting fires, handling outages, and dealing with problems of scale than fixing security vulnerabilities. Besides, isn't a beta the correct time to find these security issues? (Design / Alpha would be the ideal time, granted, but sometimes that's not possible.)
- jschuur 16y agoNever? I would be cautious about issuing a challenge like that.
- biot 16y agoThey might still have a security hole big enough to drive a freight train through, but that specific attack will never happen again given that they've shut down the shared failover server.
- deleted 16y ago[deleted]
- tzs 16y agoWait...their model is an EC2 instance per customer? The normal limits Amazon imposes are 20 reserved or on-demand instances and 100 spot instances per region. You can request more, but will Amazon really accommodate a one instance per customer model?
- jasonwatkinspdx 16y agoAmazon is happy to. The limits you cite are merely the point at which you need to have a conversation with Amazon staff. They are quite happy to accomodate _much_ heavier usage from customers.
- zmmmmm 16y agoWhen I asked for a raise to my limit they denied me, on the basis that my usage was insufficient. Of course, my usage was low because I hadn't launched my product fully because I didn't have enough instances to serve a lot of customers ... catch 22. So I had to build out a rather convoluted architecture that used the loophole of deploying to multiple regions and failing over to whichever region would give me an instance ... which gives me up to about 80 instances ... just barely enough for me to get going with a trial beta program. Which is all just to say, it is slightly more than just a "conversation" that you need to have to get a higher limit.
- smokestack 16y ago"Can I please speak to your supervisor" works sooo often in these situations.
- jasonwatkinspdx 16y agoSorry you had a bad experience. I have gotten nothing but superb support from Amazon staff. I'm surprised they weren't willing to accommodate you, doubly so if you were ready to pay. One thing that surprises me is when people talk about utilizing multiple availability zones in EC2 as some sort of burden. It's very clear from their documentation and architecture that you need to be capable running in at least 2 availability zones regardless if you want any sort of availability.
- Popcorned23 16y agoHere's an interesting tweet from one of their developers. http://twitter.com/ReinH/status/50348989366796288 http://twitter.com/ReinH/status/50348989366796288 > Your password in the database is SHA512 encrypted, but we're not taking chances. I hope he knows what he's talking about and is just tired from the past few days.
- 16s 16y agoLet's hope they are salted and iterated.
- jarin 16y agoOr swapped over to Bcrypt
- lyonheart 16y agowe've cleared the old SHA512-salted passwords out of our production database and have upgraded the password hashing to bcrypt, with a cost of 10.
- jarin 16y agoGood call :) Just in case anyone doesn't know why Bcrypt is so awesome, it's because it actually takes longer to hash (based on the difficulty level you set, and you can bump up the difficulty level as hardware gets more powerful). For other applications, you want hashing to be fast. But for passwords, you want hashing to be as slow as possible without compromising user experience.
- lyonheart 16y agoI wanted Lucas to link to Coda Hale's post on bcrypt (found by googling "Coda Hale bcrypt") in the blog post, but he edited that out. So it goes.
- nodoubt 16y agoThe blog post is riddled with the words "luck" and "timing" which brings doubt into my mind that the team can actually take full responsibility for their actions. "aware of the potential security threat " but they left it for the next week, who honestly here would do that? I have also seen comments around the web of migrating to Php Fog because of how they handled the situation. If you are one of these people please enlighten my mind as to how you came to such a logical decision or how much you get paid per year. Also if Php Fog could enlighten us on how their terms of agreement will work in the case where our intellectual property is stolen on no fault of our own. Save your sympathy for the sites that are still down, four days and counting
- jammur 16y agoExactly. This post tries very hard to trivialize the security holes, and blame their problems on bad luck. They had problems because they decided not to fix glaring security holes immediately.
- teyc 16y agoMy understanding is that this is all caused by an unsecured failover server. Hopefully, we get a bit more details of how this came to be and learn what they intend to do with future server deployments.
- wooter 16y agoI couldn't agree more. The phpFog team cut corners to deliver quickly. We (devs) all do it. The important part is to clean up after yourself. The whole blog post seems a bit melodramatic. I mean seriously, who here hasn't spent 3 all nighters in a row fixing a mistake? sack up and do what you should've done before deploying other people's data. ...and who would seriously sue these kids? they handled it poorly but they're smart (definitely smarter than i was at 16) you're lucky it was curious kids, rather than malicious (and experienced) hackers that would've been harder to catch. Do you really want to burden them with a criminal record for life?
- Kudos 16y ago
- benatkin 16y ago> Eliminate shared hosting failover server – We may never do shared hosting failover again if we can not guarantee its security. We might do a non-realtime failover to automatically launch a new instance for you, but this experience taught us what a bad idea this can be. What does realtime mean in this case? Anyway, this isn't the only option. They could keep a few bare instances of their php stack online and simply run the deploy script instead of the image creation script. That ought to be able to run in under ten seconds I think.
- jasonwatkinspdx 16y agoRealtime means a request in flight is retried if there's a failure. Said a different way, the load balancer already knows about a spare at all times. This is compared to a ~5 minute downtime if spawning a replacement EC2 instance is required. Nice idea on the hot spare instances however.
- dfheer 16y agoDear Friend: --------- http://www.fullmalls.com http://www.fullmalls.com --------- --------- http://www.fullmalls.com http://www.fullmalls.com --------- We can supply all kind jersey with good quality and low price. contact me, let’s talk details. Delivery time: 5-6days, Shipping Method: EMS Newest stock list: every 3-5days updated Air jordan (1-24) shoes $33 UGG BOOT $50 Nike Shox (R4,NZ,OZ,TL1,TL2,TL3) $35 Handbags (Coach ,LV ,Fendi ,D&G) $35 Tshirts (Polo ,ED Hardy,Lacoste) $16 Jean (True Religion,ED Hardy,Coogi) $30 Sunglasses (Oakey,Coach,Gucci,A rmaini) $16 New era cap $15 Bikini (Ed hardy,polo) $25 FREE SHIPPING --------- http://www.fullmalls.com http://www.fullmalls.com --------- --------- http://www.fullmalls.com http://www.fullmalls.com ---------
- brisance 16y agoWhile it is admirable and good that they have learned from their mistakes and are taking steps to reduce the likelihood of getting hacked in future, to say "never again" is to paint a big red bullseye on yourself.
- noonespecial 16y agoIt seems like incredible coincidence that allowed this to happen but when I think back to all of the security incidents I've been involved in, it always seems this way. I guess the best way to think of it is that badness on the internet is like water. It will flow into every tiny crack in your wall you haven't sealed up tight. A crack in a dam doesn't leak less because its in an "obscure" location.
- hinathan 16y agoThis feels like a business model where the lean/MVP approach isn't quite appropriate. A lot of things fall out of that decision, not the least of which is that the exposure surface area you get from an environment that allows user-sourced code on purpose is enormous. I feel for the guys going through this but there were a lot of errors in the wild all at once to allow this to happen.
- gsharma 16y agoIMO, in this sort of business it is important to define the right MVP.
- nethsix 16y agoGreat to see disclosure. This can happen to anyone, and more so for startups, where labor is short, focus is on developing features. Using the phrase "Never Happen Again" is a bit strong though. Security is risk management; spend until you can accept the remaining risk while still maintaining profit and avoid being a hacker's low-hanging fruit.
- teyc 16y agoI feel for the people at phpfog.com, but this is a bigger blow to cloud computing. Customers who are already pretty risk averse to their data being stored in the cloud would see this as another reason not to take the risk. The cloud computing consortium needs to work on a stable stack as well as figure out how to audit that it works properly. In addition, it calls for security ahead of features. Given that phpfog is funded, they'll need to implement the equivalent of a bleeding edge stack and a locked down stack.
- eel 16y agoI am bothered by some of the language in this post: - we were aware of the potential security threat behind post-deploy hooks and were about to disable them [...] but... - we were days away from replacing this server - They were a short-term stopgap measure we had been planning to replace To me, it sounds like the real problem could have been stated as "We were lax on security," but almost worse than that is the lack of accountability that I sense from company. Yeah, maybe it won't happen again, but it's hard to be full of confidence to buy into a service like that.
- CGamesPlay 16y agoThe article starts out in that tone, but it changes to be pretty remorseful after that. For example the "why it won't happen again" part.
- deleted 16y ago[deleted]
- krobertson 16y agoI agree with parfe's comment below. They sound incredibly laxed on security and the "we were days away from fixing it" could be complete bull. To Lucas, it probably sounds better to say they were close to fixing it instead of admitting they were unaware of these exploits. I find the disclosure in the blog post great, but the conditions they had leading up to the hack very disappointing. If they were aware of the exploits, they should have taken quicker action. They'll probably be focusing on security big time now... they have no other choice.
- ahi 16y agoThey seemed to be blaming it on "bad timing" as if these things were ever excusable. These are also things that you either do or don't do. Your systems are either secure or they're not. "They were going to be secure tomorrow" does no one any good. It doesn't look like any of the parties involved learned much of anything from this episode.
- 16y ago
- getsat 16y ago2:56:45 AM Elliot : then I used the method detailed by turby 2:56:46 AM Elliot : to gain root Has anything been said about what this method was?
- nbpoole 16y agoIf it's what was on the PHP Fog Sucks website, it worked something like this (I may be misremembering a step): 1. Use the post-deploy hook to chmod /home/ubuntu/.ssh so that it could be written to. 2. Upload a PHP shell, use it to write your public key into /home/ubuntu/.ssh/authorized_keys, and get the public IP of the EC2 instance. 3. SSH into the box, sudo su will get you root.
- getsat 16y agoOk, thanks. I was wondering if there was some other root exploit at play, not just bad system configuration.
- pdaviesa 16y agoSo, shouldn't the first thing you learn as a hacker include how to mask your physical location so as not to have the FBI knocking on your door?
- Stormbringer 16y agoWow, that is quite the list of security measures that they had almost but not completely/correctly implemented, or hadn't got around to yet. I guess the real moral of the story is to finish what you begin, or don't keep putting security off until it is convenient for you.
- drivingmenuts 16y agoLeaving the doors to your house wide open does not grant every passerby the right to enter. So, yeah, PHPFog screwed up and did that. Then these kids went in, threw paint on the walls, smashed some windows, etc. PHPFog was stupid - they admitted that. The kids were criminal. The first is not illegal - the second is.
- skbohra123 16y agoI am sure, many of the HN users here would have found at least a loophole in similar systems in the course of time. What I do in such situation is letting the service know about the flaw. Isn't that the ideal behaviour ?
- intended 16y agoTheir response and abilty to turn the situation around is a case study in dealing with a difficult situation. Kudos! I'm saving their response and will use it when dealing with things. Being able to have a counter party to identify has definitely helped in handling the situation. I didn't realize how powerful that can be until I saw this, I learnt something new. Its a brilliant piece and a great start/way to restore faith and recover from what must be a pretty grueling ordeal. Good job.
- djcapelis 16y agoUgh, you shouldn't try writing an apology after not sleeping for days. Sleep on it first, always sleep on it. Talking about prosecution and explaining this with a framing that it was all a fluke caused by the only person who was silly enough to IM you with a confession... add one more person who will never be a customer of yours with an apology like that. Now I know you're irresponsible. Seriously don't write official blog posts for your company while you're experiencing "I was just in the field for days trying to fix this stuff" emotions. Calm down, then try and be graceful about the fact that you were hacked by a few clueless kids. (Clueful kids don't let you know who they are.) Then try and figure out how to protect yourself against people with a clue.
- zaidf 16y agoI remain in two minds about idea of charging the kids. There is no doubt they did some things they should not have. And I don't doubt there can be a decent case built against them. But as someone who actually had something from his teen years come to bite years later, it's not pleasant. At least in my case it was a MAJOR maturing moment(also the worst day of my life). May be it will take a lawsuit to get these kids to mature up...to that extent anything that gets em to mature up before they really get screwed would be fair. I'm not merely advocating another chance but actually something that gets these kids to be a tad more thoughtful about their actions. It's not always easy to do that when you are 16 and full of adrenaline.
- samjohanssen 16y agoCongratulations to PHPFog. They've managed to direct the attention to the 16 year old kids rather than their own incompetence. Is it me or no one mentions the lack of expertise of the PHPFog team in PHP and Systems Administrations. Sure kids broke in and the way they published their findings was despicable. The fact remains that PHPFog was utterly broken to pieces and the exact essence of the problem is simply the lack of knowledge in their field. I am very disappointed by the tone of the blog post and think PHPFog don't really have a notion of what they are doing. I would much rather seem them where they belong, in the Ruby world where their experience is.
- rexreed 16y agoThis post convinced me not to use PHPFog. They reveal more in their lack of foresight and security prevention measures than their response to what was otherwise a fairly trivial exploit. I am not sure this blog post was helpful in convincing customers like me that want to feel that their infrastructure providers are on top of things.
- RobMcCullough 16y agoThere is no such thing as bad publicity! Kudo's for turning lemons into a viral blog post! Although, if I understand correctly, you were reusing passwords and storing them in plain text! This is an ABC123 computer security nono. Thank goodness it was just some young script kiddies and not someone with malicious intent!