10 ms·

Drupal Core – Highly Critical Public Service announcement

222 points by ohashi 12y ago

syntheticnature 12y ago
Note that the "within hours" is in the past -- if you didn't patch rapidly on 10/15, you should assume you're compromised.
- Xorlev 12y ago
  You should assume you're compromised regardless. More than researchers could have known about the vulnerability.
daviddede 12y ago
That's as big as it can be. We started seeing attacks hours after the initial disclosure and shared some of them here: http://blog.sucuri.net/2014/10/drupal-sql-injection-attempts-in-the-wild.html?updated http://blog.sucuri.net/2014/10/drupal-sql-injection-attempts... This is a lot worse than Heartbleed, Poodle and others. Full database / server take over for 700,000+ sites that use Drupal.
- ohashi 12y ago
  Is it becoming more common to see these type of attacks directly after patches are released?
  - hawkice 12y ago
    "More common" might be a bit tricky, because it's always been a race once patches go out. But modern systems can hit every IP address and quite a ridiculous number of domains extremely easily. There are more targets, so while it might be just as easy to get N% of vulnerable machines, the rewards for a hacker doing that are much higher, so the incentives put you at substantially more risk. There's also greater danger and incentive for attackers now that more websites are run by primarily non-technical people, as they are less likely to patch immediately.
    - sbarre 12y ago
      I'm sure there are also indexers out there who catalogue known Drupal/Wordpress/RoR/etc sites in anticipation of quickly hitting them with an exploit once a new one is released.
      - mixologic 12y ago
        This definitely happened. Major hosting providers were seeing attacks against all of their sites, in alphabetical order.
  - daviddede 12y ago
    It depends on the complexity of the attack. This Drupal one took our team less than an hour to have a working proof of concept (just based on the diffs). The exploit is very simple, and doesn't require any interaction with the remote site. Explaining why they started on it very fast. On other more complex exploits, we generally see days (if not weeks), before the attacks start.
  - x0x0 12y ago
    If I were an attacker, I'd fingerprint sites. It should be pretty straightforward to maintain a relatively up-to-date database of web server + version, and app stack + version. Then, when a vuln hits, you know exactly whom to attack...
  - acveilleux 12y ago
    This is just the lowest hanging fruit possible: 1. Widely deployed software. 2. Internet facing on purpose. 3. Trivial to fingerprint. 4. Fairly robust exploit (doesn't crash the target or require version specific offsets, magic numbers, etc.) And yes, when similar bug happens that meet these characteristics, mass scale exploitation will follow quickly after for various purpose (spamming, google ranking scam, DDOS, etc.)
- jerf 12y ago
  "This is a lot worse than Heartbleed, Poodle and others." There are many security vulnerabilities that have permitted full machine takeovers in an automated fashion for a long time now. Generally speaking though such attacks only work against very small fractions of the Internet, and, no matter how big Drupal may be in absolute terms, in relative terms, it is not that large. Heartbleed was so bad because it was so widespread. So many things use OpenSSL. That you could lose private keys was just icing on the cake; the arbitrary memory read was bad enough on its own, and the difficulty of detecting it also factored into its high rating. So, yes, I'd still rate Heartbleed far, far above this. Or the Ruby YAML attack. (POODLE was by contrast well-named; annoying, yappy, ultimately not significant enough to warrant its own entry in the Great Security Vulnerability list. Enough to worry about and mitigate and it if helps bury SSLv3, hey, great, but not a thing for universal panic the way Heartbleed or Shellshock were.)
  - daviddede 12y ago
    You have a good point, but I was looking at these two points: 1- Extent of the damage 2- Number of points vulnerable Heartbleed had (has) a lot more servers vulnerable, but the impact is a lot lower and it is a lot harder to exploit to extract valuable data. In fact, I doubt you will see a compromise or a major issue because of heartbleed (despite the mass drama). Compared to this problem with Drupal, that is used by the many of the top sites online, the overall damage can be a lot bigger. Time will tell.
    - SCHiM 12y ago
      I disagree, the type of information that was potentially leaked by services that use openssl is much more critical than the assets you can obtain by hacking servers via the Drupal vulnerability. My reasoning is that it's obvious (at leas I hope it is!) to your system admin that the system has been compromised when he's actively looking for indicators of compromise. This is not the case with heartbleed, so yes you can steal keys if you hack the cms and you control the server for a brief while. But this is obvious, the keys are going to rescinded, the users are going to be alerted and your access to the server is going to be severed again. In contrast the consequences of heartbleed may not be completely known even now. What if the private keys of a linux kernel dev were compromised? The attack surface was huge, and the sensitive information covers more than only cryptographic keys. There could have been all kinds of stuff in the memory of the vulnerable servers.
- x0x0 12y ago
  wow. Drupal took a month to turn this around. 16. Sep. 2014 - Notified the Drupal devs via security contact form 15. Okt. 2014 - Relase of Bugfix by Drupal core Developers [1] [1] https://www.sektioneins.de/en/advisories/advisory-012014-drupal-pre-auth-sql-injection-vulnerability.html https://www.sektioneins.de/en/advisories/advisory-012014-dru...
  - TacticalMalice 12y ago
    The date was scheduled to allow people a quick update. It could have been released oob, but would have caught many people by surprise.
    - tie 12y ago
      The surprise of an OOB patch is nothing compared to the surprise of a defaced site.
  - geoka9 12y ago
    """The Drupal Security Team was informed of this issue in the third week of September of 2014. Given the severity of the issue, we debated about releasing it early. Our main concern was when people would have the time to perform the upgrade. Drupalcon Amsterdam started on September 29th meaning that many of our community members were busy preparing for that event. The week after Drupalcon is typically busy catching up from being at Drupalcon and then October 15th was the first regularly planned security release Wednesday. We felt that it would be better to use the regularly scheduled date which also happened to be the first date when the Drupal community would be likely to have time to focus on the upgrade.""" https://www.drupal.org/node/2357241 https://www.drupal.org/node/2357241
    - zhte415 12y ago
      This is terrible. --- We didn't want to disrupt the busy schedule of Drupalcon Attendees attended Drupalcon, a event to engage in discussion of the platform we, the organisers, know currently has a critical vulnerability. We also assume attendees are uninterested in critical vulnerabilities while attending Drupalcon. We assume attendees will be unable to return to their regular roles due to the excitement / insight / general awesomeness / other affairs unrelated to Drupal for a full week after attending out event. Non-attendees implicitly missed out on our fun We have now issued a fix, which is one line of code altering a database query string. Please be noted in our security advisory that you have almost no way to know whether your site was compromised and if it remains compromised. --- It is more than terrible. It is arrogant, negligent and contempt.
untog 12y ago
Wow. This is about as big as it can get - your site has probably been compromised, data could have been stolen, and you will have no idea if you were hit or not.
at-fates-hands 12y ago
This is going to be a nightmare for a lot of smaller shops I know who have hundreds of Drupal clients. They must be going crazy right now. I stopped using Drupal and WordPress about a year ago and am glad I did. Myself and several clients just dodged a MASSIVE bullet!
- pgrote 12y ago
  What are you using in place of those? Wouldn't your clients normally pay for maintenance? The killer here is: "Consider obtaining a new server, or otherwise remove all the website’s files and database from the server. (Keep a copy safe for later analysis.)"
  - walterbell 12y ago
    Statically-generated site with client-side customization?
    - geerlingguy 12y ago
      You still have server-level vulnerabilities to contend with. It might be a simpler task to maintain server-layer security and not have to worry as much about application-layer, but security is still something that needs to be dealt with, if you have a server that is powered on and connected to a network.
      - walterbell 12y ago
        Host with a private repo on Github Pages.
  - zippergz 12y ago
    I replaced Wordpress with a home-built solution that is drastically simpler. It retains most of the URL compatibility so links wouldn't break, but it has only a tiny fraction of the functionality of Wordpress (most of which we didn't use anyway). It's entirely possible that our solution has vulnerabilities (though we designed it with security in mind, and the code base is much easier to audit due to its simplicity). But at least it's not going to get compromised due to a generic Wordpress exploit.
amingilani 12y ago
I'm concerned about all the critical websites powered by drupal, including Whitehouse.gov.
- mkempe 12y ago
  Actually, the White House was hacked two weeks ago. [1] Not much is officially known, some think that the hackers managed to penetrate a lot of their systems (some employees leaked info before the announcement). [1] http://www.usatoday.com/story/theoval/2014/10/29/white-house-computer-hacks-russia/18104231/ http://www.usatoday.com/story/theoval/2014/10/29/white-house...
  - snowwrestler 12y ago
    That's their internal network (LAN), not their web servers.
    - mkempe 12y ago
      Surely you are not claiming that there is no possible connection? if/once the Drupal Web server was hacked, the hackers could not possibly have served malware to admin users, or reached into other machines?
      - eli 12y ago
        Anything is possible, but I think it would be unwise to assume they are related.
- geerlingguy 12y ago
  I presume most of the bigger Drupal sites are run by teams that care about security enough to follow security lists closely and have release management in place that would allow hotfixing within hours, if not minutes, of a critical vulnerability being announced.
- snowwrestler 12y ago
  Whitehouse.gov is still on Drupal 6 and not affected by this vulnerability. The petition site is D7. But last I checked they have some pretty heavy hitters on that project. They probably knew about and patched the vulnerability before it was even publically released.
abhishekmdb 12y ago
yeah they said drupal 7 was vulnerable to sql injection however it looks bigger http://www.techworm.net/2014/10/drupal-7-vulnerable-sql-injection-can-leave-site-open-attacks.html http://www.techworm.net/2014/10/drupal-7-vulnerable-sql-inje...
- TacticalMalice 12y ago
  It is a SQL-injection bug. The consequences of such a bug are pretty grave, and include remote code execution.
0x0 12y ago
Why is this posted now, more than 2 weeks later? Even back then, within hours, people had publicly published working examples of remote code execution via simple single HTTP POST requests? For example: https://twitter.com/i0n1c/status/522495098630987777 https://twitter.com/i0n1c/status/522495098630987777
- geerlingguy 12y ago
  The Drupal Security Team decided to post a PSA today reinforcing the criticality of this bug. Most of the big sites, including all the sites hosted on popular Drupal PaaS services like Acquia Cloud and Pantheon, were updated or otherwise protected promptly when the original vulnerability was released. Many other sites were patched within hours. However, there's a long tail of sites still running Drupal <7.32, and all those site's owners should assume, at this point, they've been compromised. As such, this PSA gives instructions for how to avert total disaster.
  - mixologic 12y ago
    and by long tail we mean 75% of drupal 7 sites.
- acomjean 12y ago
  As someone who was learning drupal I was wondering this. My test site came up with a friendly notice to "upgrade core". I figured I should since it seemed like a good exercise. But trying to figure out how was not intuitive. The drupal web site was mum on the issue (you figured they post something on the site telling people to upgrade asap) I figured it out eventually. Was disappointed how it was handled.
  - TacticalMalice 12y ago
    It's best to follow the security announcement list. Such announcements are also posted to https://drupal.org/security https://drupal.org/security and the release notes (click through from the upgrade warning).
  - 12y ago
geerlingguy 12y ago
The takeaway: if you run a site or application that is accessible on the Internet, you are responsible for ensuring the site, servers, and infrastructure is maintained after the initial build is complete. If you're helping a client or company build a new site/app, and they are not ready and willing to either maintain the site themselves, or pay for a decent maintenance contract, or otherwise guarantee that security patches and hot fixes are applied in a timely manner, they don't yet value their website/app enough. It's your job—as someone who knows the consequences—to convince people of this. This applies all the more when a project is built on a widely-used platform like Drupal, Wordpress, RoR, Django, etc. Though bespoke/lesser-used frameworks may provide a tiny amount of protection through obscurity, that's still no excuse for not educating people on the importance of maintenance in any software project.
- yuhong 12y ago
  Don't forget providing a way for security researchers to report XSS and similar bugs.
- Nickoladze 12y ago
  If I'm using something like Composer to manage dependencies, should I be running an update every week on every past website I've created? I feel like having this run as a cron job would be a bad idea. If I'm not, is there a way to be notified about critical patches? Should I be signed up for a mailing list for every Github project I've incorporated? Where I work, it seems like the standard practice is to finish the project, deploy it, and let it sit until the client requests any changes.
  - sbarre 12y ago
    It would definitely be a bad idea to run Composer update via cron because unless you know for sure that future versions of packages won't break any functionality you built on top of them, your website could stop working (or worse start working in unexpected ways) without your knowledge. Maintain a staging environment (even in a temporary virtual machine if necessary) and run your updates in that environment first, then check it, and then deploy to production once you've confirmed everything is ok.
andyhnj 12y ago
We did a lot of Drupal work at my last job, and I sent my ex-boss a friendly warning about this when it was first announced, but I don't think he patched all of their Drupal sites. I could be wrong, but I just checked the CHANGELOG.txt on a couple of them, and they're still on 7.27.
- JohnTHaller 12y ago
  Why on earth is CHANGELOG.txt included on a production server and publicly accessible?
  - Fannon 12y ago
    I had to try to believe it.
- dageshi 12y ago
  Probably did the one line patch? There was a one line patch that fixed this particular bug. Probably a lot easier to apply that than to do a full core update to a bunch of sites.
jhgg 12y ago
Looking at the patch that fixed the vulnerability [0], I think it's a pretty safe bet to say that having a hybrid array/key-value store [1] is a generally terrible idea. [0]: https://www.drupal.org/files/issues/SA-CORE-2014-005-D7.patch https://www.drupal.org/files/issues/SA-CORE-2014-005-D7.patc... [1]: http://php.net/manual/en/language.types.array.php http://php.net/manual/en/language.types.array.php
- las_cases 12y ago
  I don't agree with you, I believe it is wrong to blame the tool for your mistakes. I agree that it may be debatable how much of the screw ups in the PHP world is actually inherited behavior from how easy and permissive PHP is to work with (perhaps a truism though and nothing to debate about).
thrillgore 12y ago
I am doing an in-development project, we patched to 7.32 minutes after the initial SA was sent. We do have some components sitting out on the net (not public) and they were not updated (brought in sync with development code) for about three hours, so I am going to go ahead and conduct an audit.
raesene4 12y ago
This is a really good example of why, if you have something important to your organisation on the Internet, you need to really be doing defence in depth, to either reduce the risk of compromise, and/or increase your chance of noticing when you're attacked and reacting accordingly. So things like IDS (network and host), perhaps looking at adding a WAF to catch basic attacks, shipping all your logs off your front end servers so they can't be easily destroyed by attackers etc. This kind of disclosure--> attack timeline now looks to be the norm, so this will become a theme and companies are either going to have to spend more on defense, or spend more on incident clean-up and then spend more on defense....
- dc2447 12y ago
  WAF are a bad idea imvho. They can give a false sense of security and the feeling that security is being taken care of. The reality is theta they are generally not very good at catching anything other than the most basic of attacks. I much prefer - external security monitoring (there are many vendors) - automated testing in pre production using skipfish/w3af/whatever - static code analysis - penetration testing - responsible disclosure programmes - hackdays
  - raesene4 12y ago
    I don't see it as an either/or decision TBH. I wouldn't suggest that WAFs are a panacea, but that doesn't mean that they can't be a useful defensive layer. A lot of companies have difficulties getting app. patches applied quickly due to test cycles, so applying a WAF rule to block known issues (this one for example) can be a fast, low risk way of reducing the risk.
  - muppetman 12y ago
    You'd be surprised at how many robot attacks don't have a user agent, or similar things you can trigger on. I agree, a direct targetted attack against your site they won't help (much). But for stopping a lot of robotic, automated hacking attempts they certainly have their place.
  - daviddede 12y ago
sam_lowry_ 12y ago
To worsen the things, Drupal did not make the new version available through its usual update mechanism immediately after the announcement. `drush up` did not bring up the 7.32 version for many hours if not a day.
- TacticalMalice 12y ago
  I believe that to be a drush issue. drush cc drush should have fixed that.
  - idbehold 12y ago
    Why on Earth is the list of available Drupal updates cached?
    - TacticalMalice 12y ago
      Beats me. This is a third-party tool. The release XML was updated immediately after release, which is the normal procedure.
ThinkBeat 12y ago
The premise that the attacks occurred after the announcement was made, and thus can be be blamed on the announcement itself is in error. The article details how it can be practically impossible to tell if a site has been hacked. There is no reason to believe that your site has not been exploited prior to the announcement. Whilst the post might have increased the volume of such attacks, I strongly doubt that this exploit was completely unknown prior to announcement. In other words, if you run a Drupal site, that was vulnerable to this attack prior to the announcement, there is a risk that your site was exploited before the announcement. This is a much more realistic scenario and also a more frightening one.
- mjhoy 12y ago
  All the more realistic given that the issue -- or a big fat hint at the issue, anyway -- was sitting in the public issue queue for nearly a year. https://www.drupal.org/node/2146839 https://www.drupal.org/node/2146839
- mixologic 12y ago
  Yes, except that none of the major hosting providers who host on behalf of thousands of clients (pantheon, acquia, etc) were able to find any examples in their logs of an attack signature prior to the disclosure. Just like shellshock has been sitting around for 25 years until somebody found it, it is highly unlikely that this vulnerabilty has been exploited prior to that, despite the fact that it was 'possible'.
- thejosh 12y ago
  Apart from it is very obvious when an attack is made when looking at logs. We saw the attacks start hours after the announcement.
currysausage 12y ago
Whenever I read about the latest vulnerability in a popular WCMS, I wonder why static HTML export still doesn't seem to be a prioritized feature in popular systems. After all, most sites out there probably don't need server-side dynamic preprocessing for every request. The CMS directory could be locked using HTTP auth (implemented by the HTTP server); this way, not every little CMS bug would allow the world to compromise the whole server. Do we really expect every parish choir with a web site to hire a CMS specialist who installs updates within hours of the release and fixes all compatibility quirks that occur with new major releases? This is an unworldly approach that bestows thousands of zombie machines on us. And what happens if the CMS for some old site stops being maintained? A responsible admin would shut the site down, resulting in a loss of potentially valuable information. This issue would be solved by using static HTML export, too. Are there any well-maintained open-source CMS out there where static HTML export is an integral part of the architecture, ideally with good usability and written in PHP (not that I like the language, but that's what is available everywhere)? (I'm not talking about command line static site generators without a user-friendly backend - those are only an option for techies.)
- martinml 12y ago
  I've been using Stacey for that purpose. It doesn't exactly export HTML but the surface area of attack is very reduced as it doesn't interact too much with the outside world: http://www.staceyapp.com/ http://www.staceyapp.com/
  - currysausage 12y ago
    It doesn't have a backend for editing content, does it? This would make it unsuitable for the proverbial parish choir.
    - wonderyak 12y ago
      Stacey is like Kirby without a web interface. You manage content by dropping text files into directories. It's not inelegant. Admin interfaces can be just as poor for representing content as anything else (see: TinyMCE/WordPress) so once you get a client used to the workflow it isn't much of a hassle. It's not like you're forcing them to build jekyll.
deleted 12y ago
[deleted]
hberg 12y ago
So glad I stopped using Drupal to build websites long ago.
drawkbox 12y ago
Those developers still using Drupal must really not be having fun. Drupal is an EOL still sticking around from back in the day of monolithic php cms frameworks that were broken at the core (Joomla, PHPNuke, etc). They make developers lives worse and lead to very little in solid products. It was good for a time, that time has passed and there are better ways. Keep this in mind when you pick up new monolithic/take-over frameworks, they bloat and die, microframeworks are the way to go. I am sad for developers lost in Drupal dead zones. Drupal man walking!
- mixologic 12y ago
  Have fun trying to get your microframework to satisfy the requirements of massive governmental organizations(http://australia.gov.au/misc/drupal.js http://australia.gov.au/misc/drupal.js) , universities (http://www.harvard.edu/misc/drupal.js http://www.harvard.edu/misc/drupal.js), and gigantic media sites (http://www.nbc.com/misc/drupal.js http://www.nbc.com/misc/drupal.js). I think you haven't a clue as to what drupal's capabilities are if you've got it even in the same ballpark as Joomla and PHPNuke.
  - jonahx 12y ago
    The parent's claim was that Drupal is monolithic, bloated, and behind the times -- not that it was lacking in functionality. If anything, linking to massive governmental organizations and universities (both famous for their bureaucracy and resistance to change) that use Drupal would seem to support his claim.
    - jtreminio 12y ago
      Drupal 8 aims to cut down on the NIH syndrome by using high-quality Symfony 2 components instead of in house libraries.
      - shkkmo 12y ago
        I've been waiting to see a Drupal 8 RC for years...
sarciszewski 12y ago
There is a lesson to be learned here, I believe: Thoroughly vet any platform (i.e. audit as much code as you can) before you deploy it, even if you want to believe that there are a lot of well-meaning white-hats looking at the product. (Linus's Law, etc.) The SQL Injection vulnerability here was a little bit more clever than the ones you'd normally see. ($query = "SELECT * FROM table WHERE id = $tainted"; mysql_query($query);) All the more reason to take a closer look at the code. And if no one else will, I'm certainly up to the task. But I make no promises I will publish my findings.
jrochkind1 12y ago
I'm feeling like it's a whole new age of security, these days. The amount of time/resources it takes to keep your apps or sites secure today is so much greater than it was even a few years ago. Development and maintenance practices that seemed reasonable to people only a few years ago now seem impossible. Delivering an app or site based on Drupal, WordPress, Rails, etc. as a finished product to a client that does not have sufficient in-house IT staff -- you can almost guarantee they're going to run into security trouble. And what is required for 'sufficient in-house IT staff' is way more than we thought a few years ago -- even if not everyone has realized it yet (those who have not will get burned).
- coldtea 12y ago
  >Delivering an app or site based on Drupal, WordPress, Rails, etc. as a finished product to a client that does not have sufficient in-house IT staff -- you can almost guarantee they're going to run into security trouble. And what is required for 'sufficient in-house IT staff' is way more than we thought a few years ago -- even if not everyone has realized it yet (those who have not will get burned). And that for 99% of the cases (unless they process credit cards and transactions), it won't matter much, if at all.
  - jrochkind1 12y ago
    Well, it matters to the customer if their WordPress site goes down because it was infected by malware that sends out spam or makes clicks on Google Adwords. This happened with someone I was working with, to try and rescue their WordPress. Ironically, the site went down only because the malware that I'm guessing was scraping Google or making clicks on google adwords or something (I just skimmed the malicious code, it wasn't entirely clear to me what it did) -- had a bug in it that brought down their site. If it had been bug free, it could have kept using their site for it's malicious purposes for years without them ever noticing.
lectrick 12y ago
My 1-project experience with Drupal is that Drupal is a clusterfuck that any experienced developer will quickly become frustrated with (much like its underlying PHP).
cdnsteve 12y ago
Yes Drupal waiting too long to send out this public notice, yes it seems there were exploits written very quickly. The thing is, no one can manually exploit nearly every Drupal site out there without assistance of search engines. It's my opinion that search engines are the primary tool in a mass malicious exploit attempt such as this. If search engines had restrictions in place for this obvious type of malicious search activity, we'd be much better off and would see a massive reduction in the huge number of infected/exploited sites/apps. They are using them as a database of targets and must be stopped. I feel this is a wake up call to petition search engine providers to implement and restrict these types of queries that are obviously malicious. We can't prevent direct attacks but these robots that farm search engines for mass infection, something can be done. The other item is these software vendors need to restrict and minimize anything that puts a giant flag up saying "Look at me, I'm a Drupal site!" EG: CHANGELOG.txt shouldn't be in www root, etc. Even some simple htaccess rules can provide a mountain of help.

222 points by ohashi 12y ago

Note that the "within hours" is in the past -- if you didn't patch rapidly on 10/15, you should assume you're compromised.

Xorlev 12y ago
You should assume you're compromised regardless. More than researchers could have known about the vulnerability.

daviddede 12y ago

That's as big as it can be. We started seeing attacks hours after the initial disclosure and shared some of them here: http://blog.sucuri.net/2014/10/drupal-sql-injection-attempts-in-the-wild.html?updated http://blog.sucuri.net/2014/10/drupal-sql-injection-attempts... This is a lot worse than Heartbleed, Poodle and others. Full database / server take over for 700,000+ sites that use Drupal.

ohashi 12y ago
Is it becoming more common to see these type of attacks directly after patches are released?
- hawkice 12y ago
  "More common" might be a bit tricky, because it's always been a race once patches go out. But modern systems can hit every IP address and quite a ridiculous number of domains extremely easily. There are more targets, so while it might be just as easy to get N% of vulnerable machines, the rewards for a hacker doing that are much higher, so the incentives put you at substantially more risk. There's also greater danger and incentive for attackers now that more websites are run by primarily non-technical people, as they are less likely to patch immediately.
  - sbarre 12y ago
    I'm sure there are also indexers out there who catalogue known Drupal/Wordpress/RoR/etc sites in anticipation of quickly hitting them with an exploit once a new one is released.
    - mixologic 12y ago
      This definitely happened. Major hosting providers were seeing attacks against all of their sites, in alphabetical order.
- daviddede 12y ago
  It depends on the complexity of the attack. This Drupal one took our team less than an hour to have a working proof of concept (just based on the diffs). The exploit is very simple, and doesn't require any interaction with the remote site. Explaining why they started on it very fast. On other more complex exploits, we generally see days (if not weeks), before the attacks start.
- x0x0 12y ago
  If I were an attacker, I'd fingerprint sites. It should be pretty straightforward to maintain a relatively up-to-date database of web server + version, and app stack + version. Then, when a vuln hits, you know exactly whom to attack...
- acveilleux 12y ago
  This is just the lowest hanging fruit possible: 1. Widely deployed software. 2. Internet facing on purpose. 3. Trivial to fingerprint. 4. Fairly robust exploit (doesn't crash the target or require version specific offsets, magic numbers, etc.) And yes, when similar bug happens that meet these characteristics, mass scale exploitation will follow quickly after for various purpose (spamming, google ranking scam, DDOS, etc.)
jerf 12y ago
"This is a lot worse than Heartbleed, Poodle and others." There are many security vulnerabilities that have permitted full machine takeovers in an automated fashion for a long time now. Generally speaking though such attacks only work against very small fractions of the Internet, and, no matter how big Drupal may be in absolute terms, in relative terms, it is not that large. Heartbleed was so bad because it was so widespread. So many things use OpenSSL. That you could lose private keys was just icing on the cake; the arbitrary memory read was bad enough on its own, and the difficulty of detecting it also factored into its high rating. So, yes, I'd still rate Heartbleed far, far above this. Or the Ruby YAML attack. (POODLE was by contrast well-named; annoying, yappy, ultimately not significant enough to warrant its own entry in the Great Security Vulnerability list. Enough to worry about and mitigate and it if helps bury SSLv3, hey, great, but not a thing for universal panic the way Heartbleed or Shellshock were.)
- daviddede 12y ago
  You have a good point, but I was looking at these two points: 1- Extent of the damage 2- Number of points vulnerable Heartbleed had (has) a lot more servers vulnerable, but the impact is a lot lower and it is a lot harder to exploit to extract valuable data. In fact, I doubt you will see a compromise or a major issue because of heartbleed (despite the mass drama). Compared to this problem with Drupal, that is used by the many of the top sites online, the overall damage can be a lot bigger. Time will tell.
  - SCHiM 12y ago
    I disagree, the type of information that was potentially leaked by services that use openssl is much more critical than the assets you can obtain by hacking servers via the Drupal vulnerability. My reasoning is that it's obvious (at leas I hope it is!) to your system admin that the system has been compromised when he's actively looking for indicators of compromise. This is not the case with heartbleed, so yes you can steal keys if you hack the cms and you control the server for a brief while. But this is obvious, the keys are going to rescinded, the users are going to be alerted and your access to the server is going to be severed again. In contrast the consequences of heartbleed may not be completely known even now. What if the private keys of a linux kernel dev were compromised? The attack surface was huge, and the sensitive information covers more than only cryptographic keys. There could have been all kinds of stuff in the memory of the vulnerable servers.
x0x0 12y ago
wow. Drupal took a month to turn this around. 16. Sep. 2014 - Notified the Drupal devs via security contact form 15. Okt. 2014 - Relase of Bugfix by Drupal core Developers [1] [1] https://www.sektioneins.de/en/advisories/advisory-012014-drupal-pre-auth-sql-injection-vulnerability.html https://www.sektioneins.de/en/advisories/advisory-012014-dru...
- TacticalMalice 12y ago
  The date was scheduled to allow people a quick update. It could have been released oob, but would have caught many people by surprise.
  - tie 12y ago
    The surprise of an OOB patch is nothing compared to the surprise of a defaced site.
- geoka9 12y ago
  """The Drupal Security Team was informed of this issue in the third week of September of 2014. Given the severity of the issue, we debated about releasing it early. Our main concern was when people would have the time to perform the upgrade. Drupalcon Amsterdam started on September 29th meaning that many of our community members were busy preparing for that event. The week after Drupalcon is typically busy catching up from being at Drupalcon and then October 15th was the first regularly planned security release Wednesday. We felt that it would be better to use the regularly scheduled date which also happened to be the first date when the Drupal community would be likely to have time to focus on the upgrade.""" https://www.drupal.org/node/2357241 https://www.drupal.org/node/2357241
  - zhte415 12y ago
    This is terrible. --- We didn't want to disrupt the busy schedule of Drupalcon Attendees attended Drupalcon, a event to engage in discussion of the platform we, the organisers, know currently has a critical vulnerability. We also assume attendees are uninterested in critical vulnerabilities while attending Drupalcon. We assume attendees will be unable to return to their regular roles due to the excitement / insight / general awesomeness / other affairs unrelated to Drupal for a full week after attending out event. Non-attendees implicitly missed out on our fun We have now issued a fix, which is one line of code altering a database query string. Please be noted in our security advisory that you have almost no way to know whether your site was compromised and if it remains compromised. --- It is more than terrible. It is arrogant, negligent and contempt.

untog 12y ago

Wow. This is about as big as it can get - your site has probably been compromised, data could have been stolen, and you will have no idea if you were hit or not.

at-fates-hands 12y ago

This is going to be a nightmare for a lot of smaller shops I know who have hundreds of Drupal clients. They must be going crazy right now. I stopped using Drupal and WordPress about a year ago and am glad I did. Myself and several clients just dodged a MASSIVE bullet!

pgrote 12y ago
What are you using in place of those? Wouldn't your clients normally pay for maintenance? The killer here is: "Consider obtaining a new server, or otherwise remove all the website’s files and database from the server. (Keep a copy safe for later analysis.)"
- walterbell 12y ago
  Statically-generated site with client-side customization?
  - geerlingguy 12y ago
    You still have server-level vulnerabilities to contend with. It might be a simpler task to maintain server-layer security and not have to worry as much about application-layer, but security is still something that needs to be dealt with, if you have a server that is powered on and connected to a network.
    - walterbell 12y ago
      Host with a private repo on Github Pages.
- zippergz 12y ago
  I replaced Wordpress with a home-built solution that is drastically simpler. It retains most of the URL compatibility so links wouldn't break, but it has only a tiny fraction of the functionality of Wordpress (most of which we didn't use anyway). It's entirely possible that our solution has vulnerabilities (though we designed it with security in mind, and the code base is much easier to audit due to its simplicity). But at least it's not going to get compromised due to a generic Wordpress exploit.

amingilani 12y ago

I'm concerned about all the critical websites powered by drupal, including Whitehouse.gov.

mkempe 12y ago
Actually, the White House was hacked two weeks ago. [1] Not much is officially known, some think that the hackers managed to penetrate a lot of their systems (some employees leaked info before the announcement). [1] http://www.usatoday.com/story/theoval/2014/10/29/white-house-computer-hacks-russia/18104231/ http://www.usatoday.com/story/theoval/2014/10/29/white-house...
- snowwrestler 12y ago
  That's their internal network (LAN), not their web servers.
  - mkempe 12y ago
    Surely you are not claiming that there is no possible connection? if/once the Drupal Web server was hacked, the hackers could not possibly have served malware to admin users, or reached into other machines?
    - eli 12y ago
      Anything is possible, but I think it would be unwise to assume they are related.
geerlingguy 12y ago
I presume most of the bigger Drupal sites are run by teams that care about security enough to follow security lists closely and have release management in place that would allow hotfixing within hours, if not minutes, of a critical vulnerability being announced.
snowwrestler 12y ago
Whitehouse.gov is still on Drupal 6 and not affected by this vulnerability. The petition site is D7. But last I checked they have some pretty heavy hitters on that project. They probably knew about and patched the vulnerability before it was even publically released.

abhishekmdb 12y ago

yeah they said drupal 7 was vulnerable to sql injection however it looks bigger http://www.techworm.net/2014/10/drupal-7-vulnerable-sql-injection-can-leave-site-open-attacks.html http://www.techworm.net/2014/10/drupal-7-vulnerable-sql-inje...

TacticalMalice 12y ago
It is a SQL-injection bug. The consequences of such a bug are pretty grave, and include remote code execution.

0x0 12y ago

Why is this posted now, more than 2 weeks later? Even back then, within hours, people had publicly published working examples of remote code execution via simple single HTTP POST requests? For example: https://twitter.com/i0n1c/status/522495098630987777 https://twitter.com/i0n1c/status/522495098630987777

geerlingguy 12y ago
The Drupal Security Team decided to post a PSA today reinforcing the criticality of this bug. Most of the big sites, including all the sites hosted on popular Drupal PaaS services like Acquia Cloud and Pantheon, were updated or otherwise protected promptly when the original vulnerability was released. Many other sites were patched within hours. However, there's a long tail of sites still running Drupal <7.32, and all those site's owners should assume, at this point, they've been compromised. As such, this PSA gives instructions for how to avert total disaster.
- mixologic 12y ago
  and by long tail we mean 75% of drupal 7 sites.
acomjean 12y ago
As someone who was learning drupal I was wondering this. My test site came up with a friendly notice to "upgrade core". I figured I should since it seemed like a good exercise. But trying to figure out how was not intuitive. The drupal web site was mum on the issue (you figured they post something on the site telling people to upgrade asap) I figured it out eventually. Was disappointed how it was handled.
- TacticalMalice 12y ago
  It's best to follow the security announcement list. Such announcements are also posted to https://drupal.org/security https://drupal.org/security and the release notes (click through from the upgrade warning).
- 12y ago

geerlingguy 12y ago

The takeaway: if you run a site or application that is accessible on the Internet, you are responsible for ensuring the site, servers, and infrastructure is maintained after the initial build is complete. If you're helping a client or company build a new site/app, and they are not ready and willing to either maintain the site themselves, or pay for a decent maintenance contract, or otherwise guarantee that security patches and hot fixes are applied in a timely manner, they don't yet value their website/app enough. It's your job—as someone who knows the consequences—to convince people of this. This applies all the more when a project is built on a widely-used platform like Drupal, Wordpress, RoR, Django, etc. Though bespoke/lesser-used frameworks may provide a tiny amount of protection through obscurity, that's still no excuse for not educating people on the importance of maintenance in any software project.

yuhong 12y ago
Don't forget providing a way for security researchers to report XSS and similar bugs.
Nickoladze 12y ago
If I'm using something like Composer to manage dependencies, should I be running an update every week on every past website I've created? I feel like having this run as a cron job would be a bad idea. If I'm not, is there a way to be notified about critical patches? Should I be signed up for a mailing list for every Github project I've incorporated? Where I work, it seems like the standard practice is to finish the project, deploy it, and let it sit until the client requests any changes.
- sbarre 12y ago
  It would definitely be a bad idea to run Composer update via cron because unless you know for sure that future versions of packages won't break any functionality you built on top of them, your website could stop working (or worse start working in unexpected ways) without your knowledge. Maintain a staging environment (even in a temporary virtual machine if necessary) and run your updates in that environment first, then check it, and then deploy to production once you've confirmed everything is ok.

andyhnj 12y ago

We did a lot of Drupal work at my last job, and I sent my ex-boss a friendly warning about this when it was first announced, but I don't think he patched all of their Drupal sites. I could be wrong, but I just checked the CHANGELOG.txt on a couple of them, and they're still on 7.27.

JohnTHaller 12y ago
Why on earth is CHANGELOG.txt included on a production server and publicly accessible?
- Fannon 12y ago
  I had to try to believe it.
dageshi 12y ago
Probably did the one line patch? There was a one line patch that fixed this particular bug. Probably a lot easier to apply that than to do a full core update to a bunch of sites.

jhgg 12y ago

Looking at the patch that fixed the vulnerability [0], I think it's a pretty safe bet to say that having a hybrid array/key-value store [1] is a generally terrible idea. [0]: https://www.drupal.org/files/issues/SA-CORE-2014-005-D7.patch https://www.drupal.org/files/issues/SA-CORE-2014-005-D7.patc... [1]: http://php.net/manual/en/language.types.array.php http://php.net/manual/en/language.types.array.php

las_cases 12y ago
I don't agree with you, I believe it is wrong to blame the tool for your mistakes. I agree that it may be debatable how much of the screw ups in the PHP world is actually inherited behavior from how easy and permissive PHP is to work with (perhaps a truism though and nothing to debate about).

thrillgore 12y ago

I am doing an in-development project, we patched to 7.32 minutes after the initial SA was sent. We do have some components sitting out on the net (not public) and they were not updated (brought in sync with development code) for about three hours, so I am going to go ahead and conduct an audit.

raesene4 12y ago

This is a really good example of why, if you have something important to your organisation on the Internet, you need to really be doing defence in depth, to either reduce the risk of compromise, and/or increase your chance of noticing when you're attacked and reacting accordingly. So things like IDS (network and host), perhaps looking at adding a WAF to catch basic attacks, shipping all your logs off your front end servers so they can't be easily destroyed by attackers etc. This kind of disclosure--> attack timeline now looks to be the norm, so this will become a theme and companies are either going to have to spend more on defense, or spend more on incident clean-up and then spend more on defense....

dc2447 12y ago
WAF are a bad idea imvho. They can give a false sense of security and the feeling that security is being taken care of. The reality is theta they are generally not very good at catching anything other than the most basic of attacks. I much prefer - external security monitoring (there are many vendors) - automated testing in pre production using skipfish/w3af/whatever - static code analysis - penetration testing - responsible disclosure programmes - hackdays
- raesene4 12y ago
  I don't see it as an either/or decision TBH. I wouldn't suggest that WAFs are a panacea, but that doesn't mean that they can't be a useful defensive layer. A lot of companies have difficulties getting app. patches applied quickly due to test cycles, so applying a WAF rule to block known issues (this one for example) can be a fast, low risk way of reducing the risk.
- muppetman 12y ago
  You'd be surprised at how many robot attacks don't have a user agent, or similar things you can trigger on. I agree, a direct targetted attack against your site they won't help (much). But for stopping a lot of robotic, automated hacking attempts they certainly have their place.
- daviddede 12y ago

sam_lowry_ 12y ago

To worsen the things, Drupal did not make the new version available through its usual update mechanism immediately after the announcement. `drush up` did not bring up the 7.32 version for many hours if not a day.

TacticalMalice 12y ago
I believe that to be a drush issue. drush cc drush should have fixed that.
- idbehold 12y ago
  Why on Earth is the list of available Drupal updates cached?
  - TacticalMalice 12y ago
    Beats me. This is a third-party tool. The release XML was updated immediately after release, which is the normal procedure.

ThinkBeat 12y ago

The premise that the attacks occurred after the announcement was made, and thus can be be blamed on the announcement itself is in error. The article details how it can be practically impossible to tell if a site has been hacked. There is no reason to believe that your site has not been exploited prior to the announcement. Whilst the post might have increased the volume of such attacks, I strongly doubt that this exploit was completely unknown prior to announcement. In other words, if you run a Drupal site, that was vulnerable to this attack prior to the announcement, there is a risk that your site was exploited before the announcement. This is a much more realistic scenario and also a more frightening one.

mjhoy 12y ago
All the more realistic given that the issue -- or a big fat hint at the issue, anyway -- was sitting in the public issue queue for nearly a year. https://www.drupal.org/node/2146839 https://www.drupal.org/node/2146839
mixologic 12y ago
Yes, except that none of the major hosting providers who host on behalf of thousands of clients (pantheon, acquia, etc) were able to find any examples in their logs of an attack signature prior to the disclosure. Just like shellshock has been sitting around for 25 years until somebody found it, it is highly unlikely that this vulnerabilty has been exploited prior to that, despite the fact that it was 'possible'.
thejosh 12y ago
Apart from it is very obvious when an attack is made when looking at logs. We saw the attacks start hours after the announcement.

currysausage 12y ago

Whenever I read about the latest vulnerability in a popular WCMS, I wonder why static HTML export still doesn't seem to be a prioritized feature in popular systems. After all, most sites out there probably don't need server-side dynamic preprocessing for every request. The CMS directory could be locked using HTTP auth (implemented by the HTTP server); this way, not every little CMS bug would allow the world to compromise the whole server. Do we really expect every parish choir with a web site to hire a CMS specialist who installs updates within hours of the release and fixes all compatibility quirks that occur with new major releases? This is an unworldly approach that bestows thousands of zombie machines on us. And what happens if the CMS for some old site stops being maintained? A responsible admin would shut the site down, resulting in a loss of potentially valuable information. This issue would be solved by using static HTML export, too. Are there any well-maintained open-source CMS out there where static HTML export is an integral part of the architecture, ideally with good usability and written in PHP (not that I like the language, but that's what is available everywhere)? (I'm not talking about command line static site generators without a user-friendly backend - those are only an option for techies.)

martinml 12y ago
I've been using Stacey for that purpose. It doesn't exactly export HTML but the surface area of attack is very reduced as it doesn't interact too much with the outside world: http://www.staceyapp.com/ http://www.staceyapp.com/
- currysausage 12y ago
  It doesn't have a backend for editing content, does it? This would make it unsuitable for the proverbial parish choir.
  - wonderyak 12y ago
    Stacey is like Kirby without a web interface. You manage content by dropping text files into directories. It's not inelegant. Admin interfaces can be just as poor for representing content as anything else (see: TinyMCE/WordPress) so once you get a client used to the workflow it isn't much of a hassle. It's not like you're forcing them to build jekyll.

deleted 12y ago

[deleted]

hberg 12y ago

So glad I stopped using Drupal to build websites long ago.

drawkbox 12y ago

Those developers still using Drupal must really not be having fun. Drupal is an EOL still sticking around from back in the day of monolithic php cms frameworks that were broken at the core (Joomla, PHPNuke, etc). They make developers lives worse and lead to very little in solid products. It was good for a time, that time has passed and there are better ways. Keep this in mind when you pick up new monolithic/take-over frameworks, they bloat and die, microframeworks are the way to go. I am sad for developers lost in Drupal dead zones. Drupal man walking!

mixologic 12y ago
Have fun trying to get your microframework to satisfy the requirements of massive governmental organizations(http://australia.gov.au/misc/drupal.js http://australia.gov.au/misc/drupal.js) , universities (http://www.harvard.edu/misc/drupal.js http://www.harvard.edu/misc/drupal.js), and gigantic media sites (http://www.nbc.com/misc/drupal.js http://www.nbc.com/misc/drupal.js). I think you haven't a clue as to what drupal's capabilities are if you've got it even in the same ballpark as Joomla and PHPNuke.
- jonahx 12y ago
  The parent's claim was that Drupal is monolithic, bloated, and behind the times -- not that it was lacking in functionality. If anything, linking to massive governmental organizations and universities (both famous for their bureaucracy and resistance to change) that use Drupal would seem to support his claim.
  - jtreminio 12y ago
    Drupal 8 aims to cut down on the NIH syndrome by using high-quality Symfony 2 components instead of in house libraries.
    - shkkmo 12y ago
      I've been waiting to see a Drupal 8 RC for years...

sarciszewski 12y ago

There is a lesson to be learned here, I believe: Thoroughly vet any platform (i.e. audit as much code as you can) before you deploy it, even if you want to believe that there are a lot of well-meaning white-hats looking at the product. (Linus's Law, etc.) The SQL Injection vulnerability here was a little bit more clever than the ones you'd normally see. ($query = "SELECT * FROM table WHERE id = $tainted"; mysql_query($query);) All the more reason to take a closer look at the code. And if no one else will, I'm certainly up to the task. But I make no promises I will publish my findings.

jrochkind1 12y ago

I'm feeling like it's a whole new age of security, these days. The amount of time/resources it takes to keep your apps or sites secure today is so much greater than it was even a few years ago. Development and maintenance practices that seemed reasonable to people only a few years ago now seem impossible. Delivering an app or site based on Drupal, WordPress, Rails, etc. as a finished product to a client that does not have sufficient in-house IT staff -- you can almost guarantee they're going to run into security trouble. And what is required for 'sufficient in-house IT staff' is way more than we thought a few years ago -- even if not everyone has realized it yet (those who have not will get burned).

coldtea 12y ago
>Delivering an app or site based on Drupal, WordPress, Rails, etc. as a finished product to a client that does not have sufficient in-house IT staff -- you can almost guarantee they're going to run into security trouble. And what is required for 'sufficient in-house IT staff' is way more than we thought a few years ago -- even if not everyone has realized it yet (those who have not will get burned). And that for 99% of the cases (unless they process credit cards and transactions), it won't matter much, if at all.
- jrochkind1 12y ago
  Well, it matters to the customer if their WordPress site goes down because it was infected by malware that sends out spam or makes clicks on Google Adwords. This happened with someone I was working with, to try and rescue their WordPress. Ironically, the site went down only because the malware that I'm guessing was scraping Google or making clicks on google adwords or something (I just skimmed the malicious code, it wasn't entirely clear to me what it did) -- had a bug in it that brought down their site. If it had been bug free, it could have kept using their site for it's malicious purposes for years without them ever noticing.

lectrick 12y ago

My 1-project experience with Drupal is that Drupal is a clusterfuck that any experienced developer will quickly become frustrated with (much like its underlying PHP).

cdnsteve 12y ago

Yes Drupal waiting too long to send out this public notice, yes it seems there were exploits written very quickly. The thing is, no one can manually exploit nearly every Drupal site out there without assistance of search engines. It's my opinion that search engines are the primary tool in a mass malicious exploit attempt such as this. If search engines had restrictions in place for this obvious type of malicious search activity, we'd be much better off and would see a massive reduction in the huge number of infected/exploited sites/apps. They are using them as a database of targets and must be stopped. I feel this is a wake up call to petition search engine providers to implement and restrict these types of queries that are obviously malicious. We can't prevent direct attacks but these robots that farm search engines for mass infection, something can be done. The other item is these software vendors need to restrict and minimize anything that puts a giant flag up saying "Look at me, I'm a Drupal site!" EG: CHANGELOG.txt shouldn't be in www root, etc. Even some simple htaccess rules can provide a mountain of help.