7 ms·
How I found a Remote Code Execution bug affecting Facebook's servers
- reginaldo 13y agoHi. I'm the one who found the bug. Facebook's side of this story is at https://www.facebook.com/BugBounty https://www.facebook.com/BugBounty
- jwcrux 13y agoFantastic bug, and great writeup. Of course, everyone will be curious about the payout ;)
- reginaldo 13y agoWell, it's already disclosed, but I really wanted to know how much people would think this kind of bug is worth.
- onestone 13y agoAfter Ryan McGeehan's comment about the "million dollar bug" (cited in your writeup), I'd say your bug is worth at least $100k.
- jonknee 13y agoA bug that lets you execute code on Facebook's servers is worth millions if not billions of dollars. You'll be rewarded with much less than that, but considering Facebook's market cap it is extraordinarily valuable.
- grugq 13y agoNo, it is not worth "millions or billions". It is worth whatever anyone is willing to pay for it. Since Facebook has very aggressive monitoring and will shutdown hacks quite rapidly, the ROI for a bug like this would have to be realised very quickly. Say in the order of days, (or maybe even hours), rather than months. How would you monetise 1 week of running code on facebook? Injecting malware would get the whole thing shutdown even faster, so you'd have to either go passive or operate in a reduced window of opportunity. There are no legal entities that would buy the bug, the USG can access any data w/ a warrant (thats free) vs. "millions or billions". Any other law enforcement agency could do the same thing. There is really no value there to them. So it would have to be blackhats, and that means some idiotic Russians mass owning everyone with old Java bugs. Again - not worth much. This sort of bug has very little value, except to facebook.
- reginaldo 13y agoI quoted that as a joke. I'm too familiar with bug bounties to ever expect one million dollars as reward for a bug. Let's hope people don't take it seriously. Lesson learned: since I'm not a native speaker, I shouldn't joke unless the joke is obvious.
- famousactress 13y agoComments on the FB post suggest the amount was 33,500$.
- oneeyedpigeon 13y agoI'm curious: how much time would you say you worked on researching and identifying this bug? BTW, I don't begrudge you the payout one little bit, no matter how long you spent on it; such an amount is change down the back of the sofa for facebook, and the potential impact of the bug means they got a great deal!
- reginaldo 13y agoWell, I originally found the OpenID bug in 2012, but hadn't noticed Facebook was vulnerable until very recently. After I found their OpenID endpoint, the hardest part was getting them to make me a Yadis discover request. Then I had to squash a little bug in the exploit. Most of the time was spent re-reading the OpenID spec. I'd say total amount of work (including the time it took me to write the post) was about 2 days. As I said in the post, I already had a strong suspicion that, once I could read files, escalating to RCE would be easy. But I decided not to do it without permission and they fixed the bug very quickly. As much as I'd loved to actually see the output of an ls or something like that, I think I made the right call.
- danoc 13y agoHere's a permanent link: https://www.facebook.com/BugBounty/posts/778897822124446?stream_ref=10 https://www.facebook.com/BugBounty/posts/778897822124446?str...
- chmars 13y agoThe payment was apparently USD 33'500. Not being in the bug bounty business, I had expected a higher payment.
- hueving 13y agoNot bad for 2 days of work.
- nikcub 13y agoThe part of the work you don't see is the hours, days and months spent, usually unpaid, spent auditing code to find the bugs. It is like the anecdote of Tesla and Ford and knowing where to put the X[0], you aren't paying for time or manual labour - bug value is derived from how much damage it can cause, what its worth to Facebook to not be exploited and what the exploit is worth to the bad guys on the black market. [0] http://www.snopes.com/business/genius/where.asp http://www.snopes.com/business/genius/where.asp
- skittles 13y agoSo would you rather pay a master $33,500 for 2 days of work or a novice $33,500 for a year of work (and possibly no results)?
- chmars 13y agoYou pay for results, not for time. And you can use an existing market to define a price. Time is often used to measure compensation – lawyers are well known for being paid in hourly fees –, however, in the long run, only results counts. And concerning the market, for how much could this bug have been sold, for example to the NSA?
- lijman 13y agoFb is so stingy with payouts.Bugs of a website with over a billion users can be sold for millions.Is fb ignoring this fact?
- sachinag 13y agoAs discussed in the FB comments on the FB post, Google's standard RCE payout is $20,000. So FB was certainly not being stingy here.
- skj 13y agoSo, both Google AND Facebook are stingy with these payouts?
- rschmitty 13y agoIs there some basis to your "fact" of bugs being bought for millions for a social networking site? I could understand if someone found a remote execution bug on Big Bank Corp's website allowing you to transfer anyone's funds to your personal BTC wallet.
- billyhoffman 13y agoPeople here seem to have a strongly misplaced expectations about what bug bounties pay. Vulnerabilities in web apps/servers tend to be worth less than vulnerabilities in client computers for a few reasons First, web app vulns are usually specific to a single site. (Unless obviously you find an issue in a common underlining framework, say, a session fixation attack in how PHP or ASP.NET handles sessions). Second, and much more importantly, the vast majority of site's don't have financially actionably information. Unless you handle banking/credit card info, I am limited in what I can do to extract value from the server (compared to a compromised client). There aren't that many vectors to extract value. -Dump their list of usernames/passwords? Ok, maybe some of those will also be used on other banking or commerce sites, but I have challenges/risks actually getting money out. And if I want stolen credit card numbers I can just buy them in card forums. -Serve sleazy advertising? Ok, possible, but ad's are a crappy business to be in and its definitely a high volume/long time approach (ask how well Huffpo pays its writers). You can try affiliate spamming/stuffing, but again, not huge value. Both ads and affiliate approaches are dependent on how much traffic the server you hacked gets. Low traffic, you make no money. The more traffic, the larger the site, the smarter/better equipped the IT/security team. How long do you really think an Alexa top 10 or top 100 site won't notice an IFRAME pointing to .ru or .cn? -Mining cryptocurrency? Not financially viable What usually happens when a server is compromised is that an exploit kit is installed and it's used to attack the visitors (specifically exploit a vulnerability in the client). And so we are back to attacking clients to extract value over attacking (most) websites. Why do this? Simple: - There are orders of magnitude more desktops/browsers than web servers. - They are running tons of diverse plugins and software so the attack surface is much larger. - Most of that software will be out of date and have known vulnerabilities. - Very few of these clients are "managed" by a personal IT person like a web server. The user is far less likely to notice anything bad. All of that mean I can reach more targets, compromise a larger number of them, and hold them for longer. Why is this better than pwning a server? Because lots of scenarios to extract value that don't work on a handful of web servers do work when I have thousands and thousands of compromised clients: -Show them ads -Stuff affiliate links -Changing their DNS settings and MitM all their traffic (bank.com? Why that's right over here!) -Keylog them to actually steal financial data, credits cards, bank logins, etc -Use them to send spam -Use them as a botnet to DDoS people and get paid protection money -Mining? perhaps? To put this in prospective, very smart hackers doing crazy stuff to break out of Chrome's sandbox and exploit clients are getting $50-$100k in public contests like Pwn2Own. Getting $35,000 for a RCE is pretty awesome
- sekasi 13y agoExtremely interesting (and open) write-up reginaldo. Congratulations on the payout.
- rigelt 13y agoCongratulations Reginaldo - great read, great story from A-Z. Don't worry about the payout. Apparently you got brains and a white hat, that's all you need for a carefree life.
- MrGando 13y agoGreat work dude, keep em' coming.
- 14th 13y agoThis was very interesting. What are good reasorces to learn more about this kind of stuff?
- ladzoppelin 13y agoIts hard and I wish I had more info. One thing I do know is that blogs are the worst form of information. I would think being a whiz at Javascript and PHP would be a prerequisite but maybe you just need a good understanding of different specs and protocols.
- hueving 13y agoYou don't really need to be a master of a specific language. You just have to understand general programming concepts. The hard part is getting the right approach to searching for vulnerabilities. Learning to recognize everywhere the target system is taking input and estimating what it is doing with that input is where the skill is at.
- markshepard 13y agoJust curious. Would you be interested in checking (for a fee ofcourse) other web server products that use similar technology. If so please let me know and I would love to connect with you.
- hundchenkatze 13y agoFrom the article, he says just shoot him an email. If you find this interesting and want to hire me to do a security focused review or penetration testing in your own (or your company's) code, don't hesitate to send me an email at reginaldo@ubercomp.com. *edit: formatting
- markshepard 13y agoThanks. I had not noticed it. I will definitely ping him!
- nradov 13y agoFor those using the Java stack we have found HP's Fortify static analysis tool helpful in automatically detecting XXE vulnerabilities. http://h30499.www3.hp.com/t5/HP-Security-Research-Blog/Q2-2013-HP-Fortify-Software-Security-Content-Update/ba-p/6119297 http://h30499.www3.hp.com/t5/HP-Security-Research-Blog/Q2-20...
- billyhoffman 13y agoHP's WebInspect, a blackbox testing tool, also can find XXE's. However, as the OP shows, XXE's can be tricky and involve a lot of nuance to cox them out. General dynamic testing tools aren't as good at uncovering XXE's as static analysis tools. Disclaimer: I used to work on WebInspect's audit engines
- cheald 13y agoSince this sounds like it affects a lot of people in a lot of places, I went about auditing my own code and found that if you're using libxml2 >= 2.9.0, you should be safe, unless you're explicitly requesting entity expansion: https://mail.gnome.org/archives/xml/2012-October/msg00045.html https://mail.gnome.org/archives/xml/2012-October/msg00045.ht... For the nokogiri users, there are a couple of proofs-of-concept in this ticket: https://github.com/sparklemotion/nokogiri/issues/693 https://github.com/sparklemotion/nokogiri/issues/693 - they should be patched with modern versions of nokogiri and libxml2, but if you're running older versions, you might want to verify their behavior before someone else does it for you.
- laxmanm1 13y agoIt should be 100,000USD. Read this article of microsoft bug bounty program. http://blogs.technet.com/b/bluehat/archive/2013/10/08/congratulations-to-james-forshaw-recipient-of-our-first-100-000-bounty-for-new-mitigation-bypass-techniques.aspx http://blogs.technet.com/b/bluehat/archive/2013/10/08/congra...
- billyhoffman 13y agoNot at all. Your link is for someone finding an entirely new way to bypass protections/sandboxing in IE. That is an enormously impactful issue because it affects hundreds of millions of desktop PCs. Comparing that to a RCE affect a few web servers, even at a site as large as FB, is misplaced.
- Thaxll 13y agoI'm very surprised that the php code isn't running in a chroot / jail...
- meshko 13y agoWait, so how do you escalate this to RCE?
- meowface 13y agoDon't think either party has disclosed that. >and due to a valid scenario he theorized involving an administrative feature we are scheduled to deprecate soon, we decided to re-classify the issue as a potential RCE bug. I imagine it might be some feature that could maybe be triggered internally through a file:// or http://localhost/ http://localhost/ URL, and in doing so gain access to an interface that can issue shell commands. That's pure speculation though, and I'm probably way off.
- reginaldo 13y agoYou're actually pretty close.
- adrenalinup 13y agoI had the same question. After researching a bit, I found that you have multiple wrappers that you can use. One of them is file:// another is php://. I wonder if the php:// one is available in HipHop. http://www.php.net/manual/en/wrappers.php.php http://www.php.net/manual/en/wrappers.php.php
- silasb 13y agoIf he can do remote network calls couldn't he download a file, like netcat?
- shabble 13y agoMy initial understanding is that the XXE flaw allows the attacker to read local files, or make network requests via the remote host (essentially, proxying them), but still only delivered to his client, rather than actually modifying or creating files on the remote host itself. remote read access is much more limited than remote write access, but even write access will be limited by file permissions, and doesn't necessarily translate to code execution. injecting some code into some of the web-app source that gets triggered by an additional request would probably be hte easiest way, but you might also look for system binaries that get called by cron or similar. Sounds like he didn't use any of these, and it was actually some sort of local web-accessible (but externally firewalled) admin interface that a suitable request could exploit, and I'm very curious how that part of it would work (especially how you'd know/find out about it as an outsider)
- raverbashing 13y agoSo, why is it possible to have XML files reading arbitrary stuff from whatever place again? XML is data. If your application needs to send a request for reading a file through XML this should really be explicit, not relying in a "permission happy" XML library, no?
- billyhoffman 13y agoYes, XML is data, but it allows you to specify where other data is located that should be included when processing it. These links can be to addition data, or to definitions of how to process the data in the document (XML entities). It's a simplification, but it's as if XML can have #includes, where the source of the #include is a URL, and can even be a file:/// URL. So the attack looks like this: Server takes input from evil user, inserts it into an XML document in memory. The input is malicious, and contains not only XML data, but XML directives to include other documents, specifically /etc/passwd on the location machine. The XML document is processed, the contents of /etc/passwd are automatically read by the XML parser/processor. However the data is not in the correct format, and the XML parser/processor spits out a detailed error message, showing the data that could not be processed/parsed, which is the contents of /etc/passwd. Make sense?
- raverbashing 13y agoSure it makes sense, what doesn't make sense is the library reading anything it is thrown at it. I'm thinking there should be a "root path" for the library to be able to access files. Sure, you want to include "base.xml" it's in a specific directory and the library is allowed to read only that, and no "../../../../etc/password" tricks.
- D3_4dl1N3 13y agoVery interesting, but there is a good chance that other people have stumbled upon, but they preferred to wear a black hat. Anyway, hats off.
- cji 13y agofrom the FB post "In parallel, other members of the Security team investigated the logs corresponding to this issue and confirmed that it had not been previously exploited or used maliciously."
- zobzu 13y agoA lot of people think its not enough money. In this case it appears to be about 30K USD. Here's an issue with paying more: the security engineers employes are paid around 100 to 150K a year. Imagine you'd get paid 100K for a big exploit. It wont be valuable to be a security engineer anymore (since you can't be paid additionally for finding the bugs), it's better to be unemployed and spend your time finding the bugs. I think that's one of the main issue, at least until bugs are extremely, extremely rare (which really, they aren't - these news are really the tip of the iceberg).