17 ms·
Remote Code Execution on a Facebook server
- Areading314 8y agoThis is a great concrete example why you should never run debug mode on a public server. Django can only do so much for redacting private info. This is also a great example of how insecure pickle is!
- ssebastianj 8y agoLuckily, Django provides checks to avoid this kind of leakage before hitting production with https://docs.djangoproject.com/en/2.1/howto/deployment/checklist/#run-manage-py-check-deploy https://docs.djangoproject.com/en/2.1/howto/deployment/check...
- misiti3780 8y agowow thanks - just ran this on some production servers and caught some intersting stuff! had no clue this command existed.
- 5h 8y agoThey were running django 1.6 ....
- js2 8y agoSentry still requires 1.6: https://github.com/getsentry/sentry/blob/ea8fe10d117f5325f9e457f4ca727c82c4d63557/requirements-base.txt#L14 https://github.com/getsentry/sentry/blob/ea8fe10d117f5325f9e...
- deusofnull 8y agoAnd to be fair, the docs are littered with warnings about not using debug mode in production. Debug mode is open for.... debugging. not production.
- Someone1234 8y agoI know this is going to get some jeering, but that's one nice thing about .Net's machine.config <deployment retail="true" />, you designate the machine itself as a non-development environment and tracing, debug output, and so on are disabled for all .Net/ASP.Net applications. That might not work for all edge cases, but broadly there's a lot of machines which are only for non-development/production code, and a system-wide setting makes you a lot safer.
- bpicolo 8y agoYou can use env vars to do similarly for all the big frameworks (django, rails, etc)
- gandreani 8y agoJust a thought. This might actually be trivial to implement in a unix environment from the administration side of things. I'm not 100% sure but all child processes inherit the env vars from the parent correct? So setting `environment=production` high up in the process tree should make it available to all processes. There's still chances of this getting overridden down the line and all apps have to conform to one style but at least it's possible?
- Someone1234 8y agoAbsolutely. But none of the major players are looking for it, and they all have unique ways of designating the machine that way. There's no specific technological challenges, this is entirely political, getting half a dozen or more different projects with different priorities to check the same variable for the same purpose.
- phinnaeus 8y agoPlenty of production apps in my line of work are executed with `env -i` for consistency.
- kawsper 8y agoFunnily enough, it is mostly .NET applications running in production that I see stack-traces from these days.
- buster 8y agoExactly.. i think you often see a warning about pickle when it comes to safety. For example, the current documentation has a big, red warning right at the top: https://docs.python.org/3/library/pickle.html https://docs.python.org/3/library/pickle.html The Django project also has a lot of warnings about the pickle serializer: https://docs.djangoproject.com/en/2.1/topics/http/sessions/ https://docs.djangoproject.com/en/2.1/topics/http/sessions/
- rrcaptain 8y agoI think maybe frameworks should stop calling it debug mode and start calling it "danger mode". Because clearly people aren't paying attention to what it actually does.
- exikyut 8y agoSo, this was simply taking advantage of a crash-prone webapp running on a debug-enabled Django instance using Pickle session serialization, and more specifically this was only possible because _Django didn't redact the stored secret key used to sign serialized inputs out of the crashdump information!_ Did the author tell Django about this yet, or is this a (possibly unintentional) 0-day? Besides the above interestingness, the morals of this story I get are - Stay persistent and leave your scanners running; you never know what new things will turn up. - Crashdumps _are_ interesting - Yay, $5,000! - Middleware and frameworks will always clash in useful and interesting ways?
- ch0wn 8y agoIf I remember correctly, Django only shows these information if left in debug mode. Needless to say, this should never be used in production.
- acidburnNSA 8y agoDjango is pretty serious about warning you of the risks of this [1]. I think in this case the key was in some third party options variable so the bug is there. The sentry thing sounds like an internal bug to Facebook rather than a django issue. [1] https://docs.djangoproject.com/en/2.1/topics/http/sessions/#using-cookie-based-sessions https://docs.djangoproject.com/en/2.1/topics/http/sessions/#...
- chatmasta 8y agoMaybe I misunderstood the article, but I thought it said that Django does strip this information, and the Sentry app went out of its way to store the secret key in the SENTRY_OPTIONS payload. This custom, non-Django code effectively circumvents Django’s protections, making the bug the responsibility of Sentry, not Django.
- exikyut 8y agoAh. You're right, I completely got this bit wrong. Thanks. Wait, so that means Sentry kind of has a vulnerability.
- Alex3917 8y agoIn contrast, I submitted a bad vulnerability in Facebook’s password reset feature yesterday that lets attacker’s send password reset PIN numbers to email addresses the user doesn’t necessarily control. The security team said to works as designed so they’re not going to fix it. Basically if someone requests a password reset on your account then the PIN number gets sent to all email addresses associated with your account, not only the primary one. This is an issue because many people have one locked down email address for things like registering accounts, but others they use to talk with people, delegate to their staff, use with CRM apps, etc. (But you still need your everyday email addresses linked to your account so that people can find you by email, see your email on your profile, etc.) The FB security team just says that delegating your email address isn’t secure so it’s not their problem. Like no shit, that’s why it’s a vulnerability. But for some reason the FB security team thinks it’s a good idea to let anyone immediately bypass 2FA and hijack your account.
- rileymat2 8y agoIf you have two factor authentication enabled, this will allow you to reset the password, but won't they still need the other factor? SMS? Presuming the other factor is not also an email. So the primary exploit might be stolen phone where the sms and email go to same device?
- oihoaihsfoiahsf 8y agoI tend to agree w/ the FB security team here. Don't list email addresses owned by an adversary in your account. :-/
- Alex3917 8y ago> Don't list email addresses owned by an adversary in your account. I mean if you're delegating your email address, your staff aren't adversaries. But they shouldn't be able to, say, drain all your retirement accounts either. Just because I want people who search for alex.krupp@gmail.com to be able to find my Facebook account doesn't mean I want password reset requests sent there. It wouldn't be at all unreasonable to send them there if that was explained in the UI, but just immediately sending a password reset pin to a non-primary email address without any warning is crazy. At least wait a few days if the user doesn't take any action after it's sent to their primary address.
- gandreani 8y agoNice job! I also really appreciate the lack of memes and very concise format of this blog post
- deleted 8y ago[deleted]
- konraditurbe 8y agoAlso the title. Clear and concise without being click-bait (such as Facebook RCE for fun and profit, all your Facebook belong to us, etc...)
- alrs 8y agoThose kinds of titles are really tired, but clickbait is unfair. "X For Fun and Profit" can be found in g-files back to the late '80s, a time well before clicking. http://www.textfiles.com/phreak/ http://www.textfiles.com/phreak/
- kashyapc 8y agoThe point is, "for fun and profit" is such an overused and utterly boring cliché. Meaningful titles are pleasant to read and shows that the writer has put some effort to bring clarity into what they're trying to convey. (As someone who sits on a major open source conference talk panel, I cringe when I see one of these clichés slapped into the title without much thought. I politely suggest to rephrase to convey more "signal" in the title.)
- BFatts 8y agoGreat article... One suggestion: You use "However" quite a bit. Not sure if you intended to show your thought process as it evolved, but that is the feeling I got.
- cc-d 8y agoFacebook joins Patreon in the "why somebody should make sure our python web framework debug mode isn't enabled in prod" club.
- makomk 8y agoPatreon's screw-up was a lot more embarassing though - they apparently left an actual Python shell exposed to the web for at least a week after someone warned them about it, and their entire user database was exfiltrated and posted on the net as a result.
- meowface 8y agoYep. Every company will face security issues; it's unavoidable. But what happened to Patreon should make people seriously question trusting them with your personal information or money. (They probably use a 3rd party payment processor who has much better security practices, but still. Also, that 3rd party doesn't do you much good if an attacker with control of your production web/application servers or CDNs is intercepting credit card form data before it's sent off.) Facebook has had vulnerabilities and exposures, but nothing like that.
- Cthulhu_ 8y agoWow, a fix in <24 hours, that's pretty impressive.
- jrowley 8y agoI mean the fix is toggling a single environmental variable from True to False, on a system that isn't normally accessed by customers, so the risk is really small in rolling out the change.
- lathiat 8y agoSometimes you’re lucky if a company reads your report in this time but of course I would expect and we generally see much better from the likes of Facebook etc
- jrowley 8y agoYou're right, being able to read, triage and act on something in such a massive system is quiet the accomplishment.
- mandeepj 8y agoFacebook deploy updates in their prod 10-20 or even more, times a day
- Scoundreller 8y agoThey could deploy continuously, but deploying to PROD doesn't include minutes/hours/days/weeks/months of investigation, testing, documentation, verification...
- jwilk 8y agoHopefully they also changed the compromised key.
- Thaxll 8y agoIt took them more than 10 days actually... They just shutdown the instance until they could find a solution. 30.07.2018 00:00 CEST : initial disclosure with every details. 09.08.2018 18:10 CEST : patch in place.
- srcmap 8y agoOther than fixing the Django source code , is there any OS level mitigation techniques that can detect and prevent such security vulnerabilities? I am thinking something like selinux, docker or chroot - a bit like internal firewall for Django (or any other webapp). Any suggestions on the links to latest best practices?
- Someone1234 8y agoDepends on your threat model. For all we know chroot, docker, and SELinux could have been in active usage on this machine. But Facebook may view a compromise on their edge network and potentially one of their trusted servers as serious even if actual damage on that server itself is limited.
- jrowley 8y agoTo be clear, this isn't an issue with the Django source, as much it was a miss configured server - having debug mode on enabled the stack trace to be leaked which yielded the secret key. Debug mode shouldn't be used in production, and django probably shouldn't be responsible for snipping every possible value out of debug traces.
- jrockway 8y agoIt seems unlikely. The application has a feature to use a secret key to secure everything. The same application then prints out the secret key to anyone that asks. Your OS can't do much about that. Maybe if you tell your OS "never let any data containing this string leave the machine" you can kind of mitigate this, but it's unlikely to work. The HTTP response is probably compressed. Someone probably base64-encoded the thing and stuck it inside some JSON, which is then base64-encoded again. Ultimately, it's about managing complexity. Django and the extension punted: Django says it will print anything and everything it has access to, and the extension has a documentation caveat about how bad that would be. One could imagine an API that tries to be resistant to this sort of thing; when the extension is initialized, it deletes the key from the environment dictionary (only partially possible), stores it in a private attribute, and only provides public EncryptAndSignCookie and DecryptAndVerifyCookie methods. This will be better than a documentation caveat, maybe, but the truly ambitious debug mode will probably get the key and print it out. (I would also point out that I'm not a fan of storing state in encrypted+signed cookies, if only because there is no way to revoke a stolen cookie without revoking every cookie ever. If you have the state on the server to store a revocation list, you might as well just store everything there and never have this problem.)
- mandeepj 8y ago> scanning an IP range that belongs to Facebook (199.201.65.0/24) ping -4 facebook.com results in 157.240.18.35. Maybe, author used some other way to get those IPs. Can anyone throw a light on this?
- strictnein 8y agoA side note. By scanning, he probably just means a Shodan search: https://www.shodan.io/search?query=net%3A199.201.65.0%2F24&language=en https://www.shodan.io/search?query=net%3A199.201.65.0%2F24&l... Login required
- sauravt 8y agoYes, I'd like to know this as well.
- mirimir 8y agoUsing whois and BGP tools, I suspect. Edit: https://bgp.he.net/search?search%5Bsearch%5D=facebook&commit=Search https://bgp.he.net/search?search%5Bsearch%5D=facebook&commit... ... 157.240.18.0/24 ... Also see https://bgp.he.net/net/157.240.18.0/24#_whois https://bgp.he.net/net/157.240.18.0/24#_whois
- deleted 8y ago[deleted]
- craigmi 8y agoPlenty of organisations (especially one of Facebook's size) tend to have their own autonomous system numbers, pretty trivial to get the ranges from BGP announcements for any given ASN.
- jelly 8y agoFacebook, like many large internet companies, buys their IP blocks outright, so they show up under their AS number [0]. Facebook seems to have 3 AS numbers [1,2,3] and that IP appears in [3] [0] https://en.wikipedia.org/wiki/Autonomous_system_%28Internet%29 https://en.wikipedia.org/wiki/Autonomous_system_%28Internet%... [1] https://bgp.he.net/AS32934 https://bgp.he.net/AS32934 [2] https://bgp.he.net/AS63293 https://bgp.he.net/AS63293 [3] https://bgp.he.net/AS54115 https://bgp.he.net/AS54115
- deleted 8y ago[deleted]
- jpmoyn 8y agoGreat, concise article! But the real question is how much did they pay you for the bounty??
- lvh 8y agoArticle has a timeline specifying $5000.
- tenfold 8y agoTagged as 'vulnérabilité'
- tenfold 8y agoHow does someone who leaves debug mode enabled on a public server get a job at Facebook?
- beefhash 8y ago> Quoting the Sentry documentation, system.secret-key is “a secret key used for session signing. If this becomes compromised it’s important to regenerate it as otherwise its much easier to hijack user sessions.“; wow, it looks like it’s a sort of Django SECRET-KEY override! One wonders why that is even there. Was Django's own session code not good enough?
- deleted 8y ago[deleted]
- the_mitsuhiko 8y ago`system.secret-key` is a value from the options system that gets propagated to different parts. It actually just sets the `SECRET_KEY` value in the settings file for all intends and purposes which has different consumers. This abstraction exists because some options can be set from the admin UIO.
- green_on_black 8y agoHe got $5k for an arbitrary remote execution bug? What a rip-off.
- 0x8BADF00D 8y agoHe should have gone to the black market, better yet sat on it. How long did it take Facebook to come forward with its user privacy violations?
- tyleraldrich 8y agoYou didn't read the article. "09.08.2018 20:10 CEST : a 5000$ bounty is awarded – the server was in a separate VLAN with no users’ specific data."
- jwilk 8y agoFrom the HN guidelines: Please don't insinuate that someone hasn't read an article.
- Kiro 8y agoNot applicable when it's obvious that the poster hasn't read the article, like in this case.
- dang 8y agoIt's applicable precisely in that case.
- Sohcahtoa82 8y agoTo be fair, it's a hard rule to follow sometimes when someone makes it painfully clear that they didn't actually read the article.
- pvg 8y ago
- mandeepj 8y ago> I found a Sentry service hosted on 199.201.65.36 I do not remember on top of my head now but I think there are few scanning software to find all the running apps on a remote machine. If you are aware then please share
- bluntfang 8y agohttps://builtwith.com/ https://builtwith.com/
- mandeepj 8y agoShoot. It did not strike me. I thought it was good only for web apps and not in-background running processes like Pickle (a binary protocol )
- Jwarder 8y agoI've used nmap to check for open ports. Not 1:1 for running applications, but an easy way to check for mistakes when making a machine public. I've also seen plenty of references to shodan.io, but I have no experience with it.
- ericnyamu 8y agoNmap -sV
- amelius 8y agoThe fact that the machine has a hostname "*.thefacebook.com" doesn't imply that it also runs software of the "Facebook" social media software. So not sure how much impact this exploit would have had.
- bluntfang 8y agoI don't think anyone will ever know how much impact, but it implies that Facebook is not good at security.
- shawn 8y agoI have bad news for you. No one is good at security.
- tetha 8y agoNo one talks about the working parts of security. Like, in this case, having the application on a separate box, and having that box separated by vlans from important things.
- bluntfang 8y agoI don't think the solution is to put things you don't care about behind vlans and having applications on separate boxes. This box is/was an attack vector. It was holding secrets. Secrets provide other attack vectors.
- matharmin 8y agoThis is why you (looking at frameworks) should never use a format that may contain code to store data, especially when the client has control over that data (even if signed). The same vulnerability has occurred in almost every language/framework that does this, including Rails and Java-based ones. Just use something like JSON, which completely avoids code execution vulnerabilities like this. Except of course for the early JavaScript JSON parsers that just used eval for parsing...
- rmetzler 8y agoAnd "something like JSON" could mean YAML, which had it's own share of RCE bugs. It's still better than these object deserialization bugs one can find in Java or Python Pickle, which seems to be even more permissive.
- whatshisface 8y agoPython Pickle RCE is hardly even a bug - because it is meant to deserialize objects (including functions!) for a language that is almost completely dynamic. Indeed, "discovering" that your unpickling is vulnerable to remote code execution is not too far from discovering RCE in a plan to compress data to the Kolmogorov limit by allowing users to send you arbitrary x86 binaries.
- someguy101010 8y agoI agree with the sentiment, and it is very easy to avoid this in Django through the use of either Environment Variables, or importing the relevant setting data with a JSON file. As long as you follow best practices (easier said than done) its easy to mitigate these sort of risks. Also debug mode in production is just lazy.
- jrochkind1 8y agoEven without the pickle-related vulnerability, exposing your secret key that is securing cookies seems pretty bad and likely to lead to other vulnerabilities, although they'd take longer to find. And if the secret key were secure, the pickle use would not be vulnerable. Still, multiple layers of security, yadda yadda, sure. But this is beyond the pickle issue. I'm not sure I'm completely convinced you should not use pickle for browser cookies that are appropriately cryptographically verified. (although fwiw I believe Rails changed it's default cookie serialization to use json instead of the ruby equivalent to pickle which suffered from the same issues).
- tekromancr 8y agoSome surprising takeaways for me; Facebook uses a Django app in their infrastructure. That app was in debug mode, revealing server secrets. That app was also configured to use the Pickle based session storage, leading to one of the few serious RCE vulnerabilities in Django. I'll have to remember this next time I think an exploit scenario is too unlikely.
- antoncohen 8y agoFacebook runs the largest deployment of Django in the world -- Instagram. https://instagram-engineering.com/web-service-efficiency-at-instagram-with-python-4976d078e366 https://instagram-engineering.com/web-service-efficiency-at-...
- ericnyamu 8y agoWhat did you use to scan facebook ?how did you land on that particular ip?
- QuadrupleA 8y agoGreat article. Big frameworks like Django are nice to get an app started quickly, but if you use them long-term it really pays to study how they work under the hood, read some source code, etc. Although "don't run debug mode on production" is probably in the first page of the tutorial :)
- bayesian_horse 8y agoI can get remote code execution on an Amazon Server (AWS). Do I get a cookie?
- Grollicus 8y agoThis is a good example why you need regular pentests in big companies. Everyone (should) know that using pickle is insecure and everyone (should) know that django debug should be False in production. Still, if the numbers get large enough someone will miss something.
- Polycryptus 8y agoThe use of Pickle isn't uncommon for session cookies in Python apps, from what I've seen. Pickle isn't really a problem unless you end up unserializing untrusted data... which a sign+encrypt scheme is supposed to ensure doesn't happen. You just can't leak the secret key or you're in trouble. Though, there's no excuse for leaving Django debug on in production.
- smsm42 8y agoI'd say it's a bad idea anyway - why you need to trust the user with anything that needs pickle (as opposed to much more primitive format) to unserialize? If you ever have a reason for non-opaque-id cookies at all, it should be very simple. If you stuff very complex objects that require native serialization into user-side storage, it's probably bad idea regardless of security implications.
- arachnids 8y agoFacebook does pentests all the time, but they don't find everything. This is why you should also run a bug bounty program.
- smsm42 8y agoSo, summarily we have: 1. Enabling debug mode in production 2. Running publicly-accessible app with publicly-accessible crash screens without any monitoring system noticing it's happening 3. Relying on auto-cleaning in debug facilities to sanitize security information (never works) 4. Using over-powered serialization protocol which allows for code execution for storing user-accessible data 5. Thinking that merely signing content prevents abuse of (4) Not bad.
- TheDong 8y agoNote that 5 is true though. Using a secret to sign or encrypt a cookie does normally work, and it's a common practice. Usually the impact of the secret leaking is that you can impersonate anyone, not that you can run arbitrary code, but the practice of using a session secret is common and not a bad practice nor broken inherently. 3 as well I think is unfair. That isn't something facebook implemented or is relying on; it's just the default behavior of django's debug stack. That's entirely on django to do that and lull people into a false sense of security in some cases (though it also probably helps in many cases too, so it might be okay). The real issues are 1 (leaving debug mode on by accident), 2 (not noticing 1), and 4 (which is a django issue I think, not a facebook one).
- smsm42 8y ago> Using a secret to sign or encrypt a cookie does normally work If the secret key is not compromised. So you have to ask yourself - why you send to the user some info that is so sensitive that needs signing? Why not just keep this info to yourself and send an opaque ID instead? Yes, I know there are issues with it too, but at least this issue is not there. > 3 as well I think is unfair. That isn't something facebook implemented I didn't say it's Facebook fault - though ultimately, of course, it is as much as if you run certain software on your servers and do not configure it properly, it's your fault. So there's a fail in having security key in a place that's so easily accessible that debug mode dumps it without even asking. Not necessarily a direct Facebook fail, but a fail.
- nijave 8y ago
- protondonor 8y agoGreat hunt!
- nickthemagicman 8y agoSo they were pickeling EXECUTABLE objects in the session and storing it in the users browser cookie? Interesting. Nice find.
- ebikelaw 8y agoThere is no such thing as non-executable pickle. Pickle is not safe and must not be used for anything, ever.
- nickthemagicman 8y agoWow. I have to look into it. That sounds wildy unsafe
- Kiro 8y ago> Pickle is a Python module used to serialize data, but contrarily to JSON or YAML, it allows to serialize objects properties but also methods. In most of the cases, this is not a problem, but one can also serialize an object with code in the __reduce__() method, which is called when the object is unpickled. Why does it run __reduce__?
- detaro 8y agoif it exists, __reduce__ is run when pickling the object and returns code that will be run when unpickling. It allows to completely customize how the object is re-created on the other side, which might be needed e.g. when the type is defined in a native extension (at least the docs name this use case).
- rhacker 8y agoCongrats on the bounty payment! That has got to be one of the most concise blog posts too.
- mattrobenolt 8y agoHey, developer of Sentry here. We're submitting a patch that prevents showing any settings on this DEBUG page. I'd like to mention that we never suggest running Sentry in DEBUG mode, nor do we document how to do this. Sentry does use Django, so it's pretty easy to put pieces together and figure out how to do it. So while we can help obfuscate this specific issue, running in DEBUG mode always has the potential to surface insecure information due to the nature of it. Our patch to suppress this information entirely can be found here: https://github.com/getsentry/sentry/pull/9516 https://github.com/getsentry/sentry/pull/9516