8 ms·
I think tumblr has a huge security hole
- adrinavarro 16y agoThey were throwing this when opening a tumblr blog. A twitter search reveals some people have seen this message too.
- radq 16y agoHave you sent them an email about this?
- sudhirj 16y agoJust did.
- adrinavarro 16y agoThanks, I really didn't know where to email them (nor I'm sober enough right now to do that, IMHO)
- sudhirj 16y agoThey've got their Twitter / Facebook / oAuth secret keys in there. Doesn't that mean everyone who sees this can act as Tumblr post to those services on behalf of users? I hope they've changed them.
- xuki 16y agoNope, you need the users' token to perform that (given that they gave post to wall permission). However using Facebook secret key, you can ban users from using your app. But then again, you need user id.
- jimktrains2 16y agoBut you could have users log into your site and then act as tumblre. If they already OK'd the tumblr app, they would never know. Otherwise they'd just have to ignore the screen saying what App you're OKing (and most people, I'd wager, would, and just think you were using tumblr for something). right?
- timerickson 16y agoI sure hope they realize they just broadcasted the pass for the "tumblr3" database user, as well as their Twitter, Facebook, Recaptcha and other secret keys.
- sudhirj 16y agoWell, at least they're using strong passwords. Fat lot of good it's doing them, though.
- smbwrs 16y agoIf they're clever, that password isn't the actual password, it's a salted hash that the Database class breaks down to the real password before connecting. In theory, that hash alone shouldn't be enough for a breach, unless someone is able to figure out how it's encrypted and salted.
- xentronium 16y agoIf you can "break down" something to real password, it's not hashing but rather encryption. Anyways, I don't see how it helps if the actual php code is exposed. Adding layers of super-duper-secret php code (Database class) is not going to help if you display them in browser.
- smbwrs 16y agoMy point is more that there's at least one layer of protection between that page and the actual password, which I'd say is more important than displaying some code. It would take more than one mistake to really do damage, which is better than nothing.
- deleted 16y ago[deleted]
- blazer 16y agoIts not HUGE.
- angadsg 16y agoThere was a similar "gaping" hole 2 years back. http://news.ycombinator.com/item?id=164422 http://news.ycombinator.com/item?id=164422 Better email them.
- deleted 16y ago[deleted]
- mrspeaker 16y agoI know it's easy to criticize, but far out Tumbler, you guys really have to get your act together - the downtime and general laggy-ness is at least understandable, but there is no excuse for absolute newbie foul-ups like this. Although, on the plus side, having a site that mashes up tumbler as a content provider certainly has given us plenty of opportunity to fine tune and improve our caching strategy.
- yuvadam 16y agoMirrored to http://codepad.org/UP3FxRNv http://codepad.org/UP3FxRNv Just in case. edit: oh man. Check out what Google has on this[1] edit: in readable form on github gist[2] [1] - http://www.google.com/webhp?hl=en#sclient=psy&hl=en&site=webhp&source=hp&q=site%3Atumblr.com+m3MpH1C0Koh39AQD83TFhsBPlOM1Rx9eW55Z8YWStbgTmcgQWJvFt4 http://www.google.com/webhp?hl=en#sclient=psy&hl=en&... [2] - https://gist.github.com/29c5c5970d1f3313abd1 https://gist.github.com/29c5c5970d1f3313abd1
- erikpukinskis 16y agoAnd it looks like the hole has been there for at least a month: http://www.google.com/search?q=site%3Atumblr.com+m3MpH1C0Koh39AQD83TFhsBPlOM1Rx9eW55Z8YWStbgTmcgQWJvFt4&hl=en&safe=off&sa=X&ei=kCCFTYW2L4uWsgPPvdiEAg&ved=0CAkQpwUoBg&source=lnt&tbs=cdr%3A1%2Ccd_min%3A2%2F1%2F2011%2Ccd_max%3A3%2F1%2F2011&tbm=#q=site:tumblr.com+m3MpH1C0Koh39AQD83TFhsBPlOM1Rx9eW55Z8YWStbgTmcgQWJvFt4&hl=en&safe=off&prmd=ivns&sa=X&ei=xiCFTfatN5D4swOYmcXlAQ&ved=0CA0QpwUoBg&source=lnt&tbs=cdr:1%2Ccd_min%3A2%2F18%2F2011%2Ccd_max%3A2%2F18%2F2011&tbm=&bav=on.2,or.r_gc.r_pw.&fp=9fb592477e9d4777 http://www.google.com/search?q=site%3Atumblr.com+m3MpH1C0Koh...
- zaidf 16y agoWe saw this from facebook few years ago. Now with tumblr. Is there something at core of php that makes this inevitable? I ask this as a concerned php dev(and not out of snark).
- tjogin 16y agoI think PHP has so many gotchas, so many ways to trip up even seasoned developers, that it's inevitable to miss something.
- zaidf 16y agoI understand that view. But I am talking specifically about something that would just dump the code of the php file to the browser.
- yrashk 16y agothey just had a single typo i?php instead of <?php I guess somebody's using vim
- jarin 16y agoThat's why I always git diff and run tests before committing, I do that all the time :-/
- drdaeman 16y agoAnd this is the reason one should separate production and development environments and NEVER edit any files on production servers (let `git push` or something similiar handle it)
- russss 16y agoAt Last.fm we had a Subversion pre-commit hook which ensured that all .php files began with <?. That would have easily caught this problem, no drama.
- FirstHopSystems 16y agoI always use a include for any hashes or passwords in a separate file. When I started learning PHP I exposed my MySQL database password more times then I could keep track of. It does hammer home the point of staging before deploying. Also the point of making sure you vary your passwords between sites.
- jasonlotito 16y agoDo not store configuration in code. Store it in files that aren't part of the software. Store this file outside the web root.
- oscardelben 16y agoFound a link with better formatting on twitter: https://gist.github.com/29c5c5970d1f3313abd1 https://gist.github.com/29c5c5970d1f3313abd1
- holdenc 16y agoCan someone please explain this a little more? My basic understanding is that they incorrectly opened a PHP tag and exposed the code. If that's the case, wouldn't the page have appeared broken in development? Or have been found during testing?
- deleted 16y ago[deleted]
- oscardelben 16y agoBy looking at the code !quality, I wouldn't be surprised to learn that they have no tests at all.
- pilif 16y agoHow can you look at this code and make any statement about general code quality? The stuff that was exposed was pure initialization code that would look the same anywhere you look. Granted. This should move outside of the web root, but just looking at this one file that just sets up configuration values provides no ground to say anything about general code quality
- jasonlotito 16y ago> How can you look at this code and make any statement about general code quality? The stuff that was exposed was pure initialization code that would look the same anywhere you look. No it doesn't. Having to modify live code just to change configuration options is bad. If I want to add a new slave, or change some configuration option, it should have no affect on running code. I shouldn't be changing any software logic, and that's exactly what editing this file does. Their are standard ways to avoid this. But no, if changing your config file means changing what essentially amounts to changing the executable of your code, you're asking for trouble.
- bluesnowmonkey 16y agoIt doesn't really matter if the configuration is in an executable form or not. Without testing, you can break it either way. With testing, you probably won't. They're equivalent.
- csears 16y agoAUTHORIZE_ID and AUTHORIZE_SECRET_KEY... anyone know if those are for Authorize.net? Yikes.
- chuhnk 16y agoI hope their firewalls are good because this may instigate an attack. Tumblr, 80,443,established,related accept, all else DROP.
- smallwords 16y agoWhilst I hope tumblr correct the problem rather quickly as it is a major problem, I find those jumping to blame are forgetting one small problem. No programmer is perfect, typos are easy to mistake on any keyboard and it will happen to everyone no matter how much of a ‘ninja rockstar poodle’ they think they are. I hate to see someone else work in the clear like this. It’s like popping a zit before your first date. It’s painful and will show up for day/s afterwards. Now I know what will be today’s headline I can bypass techmeme. Yes its a big boo boo. It’s a massive security risk and to some it may feel like the end of the world but by then it will be tomorrow. Passwords will be reset, keys will be replaced and the valley will be talking about something else. Hopefully it won’t be someone else’s mistake. P.S Don’t forget to test your code before deploying – now you know why.
- elliottcarlson 16y agoTypos are very easy to make - but that's why you need to first test your code locally, test your code in a development environment, have others test and approve your code in a staging environment before a small typo gets to production where something like this can happen.
- joebananas 16y agoWhich is why you have test servers and never ever make live edits to deployed code. I find it exceedingly easy to say that this was kind of incompetent.
- nikcub 16y agothe attacker left their IP address in the dump, and their LAN interface address Edit: it's whoever dumped this as opposed to an attacker I don't think tumblr have acted on this yet. The other exposed pages have S3 API secret keys, facebook api secret keys, the username and passwords for vimeo, clickatell (whatever that is), twitter oauth secret key, etc. they are going to have to revoke and re-setup each of these.
- mattkirman 16y agoIt doesn't really look like there was any attack. One of the developers made a mistake with the PHP opening tag in the config file. Anyone visiting the page would have seen this output.
- bl4k 16y agothere are dozens of pages that have been exposed on this one server. I have no idea how the opening tag can get mangled in so many different files
- elliottcarlson 16y agoBy having a central config file that is included throughout the pages. Once included, it's not parsing it as PHP because it has the typo in it - thus showing it on all pages that include that file...
- jarin 16y agoIf the typo is in a top-level layout file or something that would happen.
- deleted 16y ago[deleted]
- rmoriz 16y agoI tought tumblr.com was build with Ruby on Rails.
- nsfmc 16y agoIn the abstract, i understand, why anyone would post this. I get it, and maybe i'm getting older or something, but part of me equates posting this sort of thing with pointing at an acquaintance on the street with his fly down and having a laugh with your friends rather than telling him and having a chuckle after he's zipped up. But the whole self-righteous and incredibly passive-aggressive "man, i think these guys have a huge problem..." followed by the "man, these guys need to shape up" and "n00b mistake!" are unproductive to the extreme. I mean, find a bug, report it, move on. Maybe i've got some unrealistic notion of karma or general human benevolence or something, but it seems hard to believe that this is such a difficult path to take especially when nearly everyone commenting has to deal with bs of this sort day-in/out.
- InclinedPlane 16y agoThis isn't a bug, it's a failure of basic security principles. Imagine if the super to your apartment complex accidentally mailed a box full of duplicate keys to a local methadone clinic. That's not an embarrassing mistake, it's a catastrophic error bordering on criminal negligence. Drawing attention to it is meant to not only deepen the embarrassment and thus encourage fixing the underlying problem so that it never happens again but also to let other people know (such as tumblr users) about the amateurishness of the tumblr operation and finally to encourage other developers out there to avoid making the same mistakes.
- smbwrs 16y agoYou lost me at "bordering on criminal negligence". They gave away a lot of details about how their internal systems are structured, but surprisingly little as far as actual usable data. Passwords can be changed, API keys can be disabled and regenerated, local IP addresses can be switched up. No user data was revealed. How is this even close to criminal, let alone catastrophic? This is pants-down embarrassing, at worst.
- deleted 16y ago[deleted]
- 16y ago
- joshfng 16y agoIt's been taken care of http://staff.tumblr.com/post/3959106211/update-regarding-security-issue http://staff.tumblr.com/post/3959106211/update-regarding-sec...
- KevBurnsJr 16y agoYou can clearly see all the routes in the app. 400+ routes and only 11 controllers. Most routes are concentrated on 3-4 controllers. Each of those controllers has got to be 10,000-20,000 lines apiece. The dashboard controller alone has approximately 120 actions.
- Maxious 16y agoOfficial update from tumblr: http://staff.tumblr.com/post/3959106211/update-regarding-security-issue http://staff.tumblr.com/post/3959106211/update-regarding-sec...
- nikcub 16y agoIf you have a PHP app, place nothing inside of web root other than an include to your front controller. This is all that should ever get exposed: include '../app/front_controller.php';
- russss 16y agoThen, if you accidentally edit your front controller file so that it starts with 'i?php' instead of '<?php', it will get exposed. This is what happened with Tumblr.
- CWIZO 16y agoSure but you will expose only that line, which tells you almost nothing.
- nikcub 16y agoYou are right, it wil expose the front controller, but it wont expose the includes from there. If you screw up the open tag on the config file, even if it is outside of webroot it will be exposed I just tested a few variations on one of my old projects and the reason why I didn't see it is because I had ob_start() and was killing the buffer in my templating class (which is a good idea) not to mention that the db passwords were in an external yaml file anyway (most of the frameworks do this) which would get loaded and cached
- 16y ago
- stowaway 16y agoTL;DR: Amateurish PHP developers at Tumblr fuck up; HN developers who don't know PHP that well make wildly incorrect assumptions about PHP. People, I know it's en vogue to bash PHP (just wait, in a few years it'll be Ruby and Python - remember, PHP was once hyped, too, and now it's going in the other direction) - but if you criticize PHP, could you at least try to sound like you've actually developed in PHP for more than a week? Because most of the negative comments here about PHP have absolutely nothing to do with PHP as such - the Tumblr error in question has to do with incompetent programmers. If you read The Daily WTF you'll know that incompetent programmers can screw up no matter what language they're using.
- nbpoole 16y agoAmateurish? Incompetent? It seems a bit extreme to make those generalizations because someone made a typo in a PHP file that managed to hit production.
- stowaway 16y agoMaybe I went a little overboard with those words. However, it's not so much the typo; it's the fact that the typo caused passwords to be visible.
- nbpoole 16y agoAgain though, PHP apps storing configuration info in PHP is a fairly common practice. If that's the way you store that information, any typo that takes your configuration file out of "PHP mode" is going to leak data. Before today, I'm sure a lot of PHP coders didn't consider that when dealing with configuration files for applications: they chose to design that way because it's simple to deal with.
- stowaway 16y agoSure, but if they'd made just a tiny effort to test their code before releasing it they'd have caught that bug instantly. We're not talking about time-consuming tests here - we talking about loading the page in the browser once. If the PHP code is showing, you're doing it wrong.
- apgwoz 16y agoI feel really bad for them right now, but I can't stop reading it.
- jjm 16y agoI rarely go off like this but man, look at this PHP file! It's a mess! Very long, not dry, no wonder the dev fat fingered it... Is the rest of the framework like this? No headless output testing? Manual output checking (at the least)? For the record, as much as I hate to admit it now I was a big long time PHP developer. My code is used round the web today...
- datasink 16y agoIt's a config file. How can you make dozens upon dozens of unique memcache, database, and defined constants DRY?
- mickeyben 16y agoI pretty much agree with jjm comment. I never have ONE config file but a directory of config files, everything is separated : database, memcache, heywatch, s3, routes, ... It's way easier for the developer, the sysadmin, the deployment, ... Of course it would multiply the risk of this happening by n files.
- jjm 16y agoThats the point. Who says it has to be a config file? Think outside the box. Here is how Flask can do configuration (a Python micro framework): <code> class Config(object): DEBUG = False TESTING = False DATABASE_URI = 'sqlite://:memory:' class ProductionConfig(Config): DATABASE_URI = 'mysql://user@localhost/foo' class DevelopmentConfig(Config): DEBUG = True class TestinConfig(Config): TESTING = True </code>
- datasink 16y agoZend Framework also handles configs with inheritance. It's an elegant way to handle overrides, no doubt. They do one better by allowing the definition to be in ini files, which would have circumvented the Tumblr bug. Still, I think config files are often best when they're verbose and as straightforward as possible. There are fewer surprises this way. If I'm working on a project by myself or with a very small team of people I trust, I'd go the inheritance route. If the team is larger, I'd go the verbose route.
- nikcub 16y ago"this is not what I had in mind when I said we should open source the backend"
- KevBurnsJr 16y agoHow to prevent a leak like Tumblr's - http://news.ycombinator.com/item?id=2343561 http://news.ycombinator.com/item?id=2343561
- reustle 16y agoDon't edit files on the production server?
- adolph 16y agoIf one accepts that errors like these happen, I guess it would be a good idea to have an automated way to quickly change passwords on all the services that are used. Does anyone have some citations for literature on how to deal with that?
- troels 16y agoGood point. Just like you'd want to have a well described (preferably automated) way to restore from backups, you should also have one for resetting all passwords. Such a process is also useful for protecting against disgruntled employees.
- nearix 16y agoThis looks like a vim switching mode mistake to me. Oh humans...
- nikcub 16y agothat means the edit was done on the production server?
- virtualroot 16y agohttp://staff.tumblr.com/post/3959106211/update-regarding-security-issue http://staff.tumblr.com/post/3959106211/update-regarding-sec...
- virtualroot 16y agoA human error caused some sensitive server configuration information to be exposed this morning. Our technicians took immediate measures to protect from any issues that may come as a result. We’re triple checking everything and bringing in outside auditors to confirm, but we have no reason to believe that anything was compromised. We’re certain that none of your personal information (passwords, etc.) was exposed, and your blog is backed up and safe as always. This was an embarrassing error, but something we were prepared for. The fact that this occurred at all is still unacceptable, and we’ll be seriously evaluating and adjusting our processes to ensure an error like this can never happen again.
- Jarred 16y agoI bet the tumblr developers are happy 100+ people are debugging their mistake.
- glesperance 16y agoThey could make their app completely open source then ; in that way they'd probably have a better quality assurance process than "suffer, apologize & fix".
- jentulman 16y agoRegardless of how it occurred, and given that this isn't a language issue, I have two questions.... Does anyone know how long was it actually in this state? (There's a heck of a lot of entries in the Google search quoted in another comment, but then how often does Google index tumblr?) Did no-one at least press F5 or CMD-R after making the edit, let alone run tests? Quality control is the real issue. I can easily imagine myself making this mistake, typo's are the source of the majority of my bugs, but I find it hard to imagine taking more than 10 seconds to notice it.
- mattmight 16y agoIf we want security, programming languages must either make secure code easy to write or insecure code impossible to write. (Or both!) PHP doesn't do well on either count. Writing secure PHP code isn't impossible, but it's tedious even for seasoned developers.
- deleted 16y ago[deleted]
- tsigo 16y agoI haven't used PHP heavily in about 2 years, but it surprises me that at this point there's not a configuration setting that negates the need for the opening PHP tag and instead just treats everything in a .php file as php. I doubt anyone using PHP at Tumblr's level is mixing raw HTML into their PHP files and thus has no need for the php tags.
- mkramlich 16y agoClicked on link, just saw a huge ad and no obvious way to dismiss it.
- didip 16y agoAll this could have been avoided by using: if(getenv('tumblr_env') == 'production') { //See: http://php.net/manual/en/function.register-shutdown-function.php register_shutdown_function('five_hundred_err_func') }