5 ms·

My sympathies are with the Drupal community, it may never fully recover from a crisis of this magnitude despite whatever. At times like this, I really wish the

by aleem 12y ago

My sympathies are with the Drupal community, it may never fully recover from a crisis of this magnitude despite whatever. At times like this, I really wish there were more best practices to mitigate against these type of vulnerabilities. One idea is using the remote-execution vulnerability itself to remotely patch the servers by passing the antidote as the payload. I am not sure if this would be legal. The other is something what CloudFlare already does but doing it at the hosting-provider or ISP layer. It need not be as proactive as CloudFlare which can inspect HTTP requests for known exploits and blocks them. It could be something as simple as running scanners and routinely informing customers that their websites are vulnerable. Or better still, temporarily disabling the account or putting it into read-only mode, thereby forcing users to take action.

drzaiusapelord 12y ago
Or, autopatching of security patches without human intervention. This stuff isn't that hard too figure out. The Drupal community needs to stop pretending that its 1996 anymore and start taking security seriously. I think the CMS world needs a Microsoft circa 1999 moment.
- ayrx 12y ago
  That is a absolutely horrible idea for frameworks that Drupal. That essentially implies that someone has arbitrary write access to your servers and is free to modify things as he or she wishes.
geerlingguy 12y ago
> At times like this, I really wish there were more best practices to mitigate against these type of vulnerabilities. There are many best practices: - Use version control. - Have remote backups for the past day, week, month, etc. - Apply security patches immediately (and have test infrastructure so you can make sure they don't break anything). - Be able to rebuild a server from scratch w/ backups (configuration management really helps here). - Subscribe to security mailing lists/RSS feeds/Twitter accounts. This vulnerability is pretty bad, of course, but there have been many like it before, and there will be many like it in years to come. If nothing else, use this situation to prepare for the worst on your own site; how would you respond if your site was hacked, your database in an unknown state, and your codebase potentially backdoored?
- dageshi 12y ago
  To be honest, it's not actually as dire as it may seem in my opinion. Consider, if you're on shared hosting, which a lot of drupal sites will be then the server itself isn't your problem only the site is. Copy the site off and have your host reinitialise a clean account, that's taken care of. In terms of the site you're looking for code backdoors, there are a relatively limited number of places these can live. You can completely overwrite drupal core + contrib modules with good clean versions, that really limits the places where bad code can live to the files directory and the themes directory. Files for the most part shouldn't contain any executable scripts so locating backdoors there is relatively easy. The theme will contain php files but their number is relatively limited and is primarily templating related, it should be possible to find bad code here if you know what you're looking for, after all the theme doesn't handle authentication so can't exactly introduce subtle 1 character tweaks here to make the site vulnerable. The database is more complicated unfortunately, besides simple privilege escalation and permission alteration which are relatively easy to find, drupal often allows php to be stored and executed from within the db. I can see if an attacker is particularly sophisticated a backdoor in there being harder to find, but I imagine there will be people working on tools to hunt that kind of thing in the imminent future.
- eropple 12y ago
  Configuration management is an enormous pain for systems like Drupal that are essentially built ad-hoc, though. It's not something that's practically baked into something like Chef unless you're willing to spend a ton--a ton--of time baking your assumptions into code. And that (along with stuff like a test infrastructure) assume the capability of even doing it; Bob's Bait Shop does not have the technical staff necessary to act according to best practices but definitely needs a website. ("Use SquareSpace" is rarely an answer, though depending on the project it can be; anything remotely clever is going to need to go outside of the usual SaaS providers and that's why there are so many rickety Drupal sites out there. I used to write them, I know.) Once again, software is harder than it needs to be. Gary Bernhardt has been going into this in depth on Twitter lately, and I agree with him more and more as I pay more attention to the shitshow around us.
  - 12y ago
YZF 12y ago
IMO a very important practice is designing to block escalation of privileges. For example, if your web server never stores a list of all e-mail accounts but instead queries a different service for an authenticated user's e-mail then even a complete compromise of the web server would not allow an attacker to get a list of all user's e-mails. Perhaps the canonical example is the handling of passwords in Unix/Linux. Most competent developers know how to handle passwords but it seems not necessarily the underlying lessons/principles. Even if you manage to gain root privileges you can not retrieve user's passwords. Another way of thinking about this is defense in layers while assuming the exterior layers can be completely broken. I like the idea of deploying an antidote but that won't always work and I can see people not liking it. CloudFlare would be an example of defense by layers. Another observation is about how the vulnerability is disclosed. I don't think this was handled properly here. Ideally you want to see a disclosure that allows users to take action without actually revealing the specific vulnerability. An extreme example would simply be to tell all Drupal users to take their sites down and give them enough time to do this before disclosing anything. Possibly a less broad action could be taken while still hiding the specific exploit. To release a patch revealing the exploit while many users are still open to an exploit is extremely problematic.
- pizzeys 12y ago
  I don't think this would work. The people getting caught by this would also miss the notice to take their sites down.
  - YZF 12y ago
    It's a matter of controlling that window and minimizing the number of affected sites. Scenario #1: Release patch with no advance notice. Takes time for people to take notice and apply. In that window lots (12 million?!) sites get hacked. That doesn't sound like a very happy outcome. Scenario #2: Announce a patch will be released in advance. Get the word out. Give people time to hear it. You're not giving any specifics other than there is some vulnerability. That's doesn't help any hacker, every product has some vulnerability. There'll probably be a lot less than 12M sites hacked. Also consider closing the vulnerability in ways that don't directly touch the affected code to make it more difficult to reverse engineer. Obviously if everyone knows of the exploit you don't have the time but often the way hackers hear about the exploit is through your announcement... [EDIT: Thanks to aryx above I learnt there was a 5 day window in this specific case (An Oct 10th announcement for Oct 15th security update). I don't think that announcement made it clear enough what the consequences of not applying the patch immediately would be. I think this is something that needs more careful consideration in the future.]
pizzeys 12y ago
Many web hosts do this. At <redacted UK-based LAMP host> we do both the WAF stuff (which Cloudflare don't actually do, unless you pay them), and your latter suggestion of read-only'ing accounts on discovery of vulnerabilities. The unfortunate reality of the latter one is that it's going to break their site, probably, since we don't know what they run, and you can't just read only an entire account blindly without breaking something. We could do better there, but it will never be perfect in this environment.

by aleem 12y ago

drzaiusapelord 12y ago

Or, autopatching of security patches without human intervention. This stuff isn't that hard too figure out. The Drupal community needs to stop pretending that its 1996 anymore and start taking security seriously. I think the CMS world needs a Microsoft circa 1999 moment.

ayrx 12y ago
That is a absolutely horrible idea for frameworks that Drupal. That essentially implies that someone has arbitrary write access to your servers and is free to modify things as he or she wishes.

geerlingguy 12y ago

> At times like this, I really wish there were more best practices to mitigate against these type of vulnerabilities. There are many best practices: - Use version control. - Have remote backups for the past day, week, month, etc. - Apply security patches immediately (and have test infrastructure so you can make sure they don't break anything). - Be able to rebuild a server from scratch w/ backups (configuration management really helps here). - Subscribe to security mailing lists/RSS feeds/Twitter accounts. This vulnerability is pretty bad, of course, but there have been many like it before, and there will be many like it in years to come. If nothing else, use this situation to prepare for the worst on your own site; how would you respond if your site was hacked, your database in an unknown state, and your codebase potentially backdoored?

dageshi 12y ago
To be honest, it's not actually as dire as it may seem in my opinion. Consider, if you're on shared hosting, which a lot of drupal sites will be then the server itself isn't your problem only the site is. Copy the site off and have your host reinitialise a clean account, that's taken care of. In terms of the site you're looking for code backdoors, there are a relatively limited number of places these can live. You can completely overwrite drupal core + contrib modules with good clean versions, that really limits the places where bad code can live to the files directory and the themes directory. Files for the most part shouldn't contain any executable scripts so locating backdoors there is relatively easy. The theme will contain php files but their number is relatively limited and is primarily templating related, it should be possible to find bad code here if you know what you're looking for, after all the theme doesn't handle authentication so can't exactly introduce subtle 1 character tweaks here to make the site vulnerable. The database is more complicated unfortunately, besides simple privilege escalation and permission alteration which are relatively easy to find, drupal often allows php to be stored and executed from within the db. I can see if an attacker is particularly sophisticated a backdoor in there being harder to find, but I imagine there will be people working on tools to hunt that kind of thing in the imminent future.
eropple 12y ago
Configuration management is an enormous pain for systems like Drupal that are essentially built ad-hoc, though. It's not something that's practically baked into something like Chef unless you're willing to spend a ton--a ton--of time baking your assumptions into code. And that (along with stuff like a test infrastructure) assume the capability of even doing it; Bob's Bait Shop does not have the technical staff necessary to act according to best practices but definitely needs a website. ("Use SquareSpace" is rarely an answer, though depending on the project it can be; anything remotely clever is going to need to go outside of the usual SaaS providers and that's why there are so many rickety Drupal sites out there. I used to write them, I know.) Once again, software is harder than it needs to be. Gary Bernhardt has been going into this in depth on Twitter lately, and I agree with him more and more as I pay more attention to the shitshow around us.
- 12y ago

YZF 12y ago

IMO a very important practice is designing to block escalation of privileges. For example, if your web server never stores a list of all e-mail accounts but instead queries a different service for an authenticated user's e-mail then even a complete compromise of the web server would not allow an attacker to get a list of all user's e-mails. Perhaps the canonical example is the handling of passwords in Unix/Linux. Most competent developers know how to handle passwords but it seems not necessarily the underlying lessons/principles. Even if you manage to gain root privileges you can not retrieve user's passwords. Another way of thinking about this is defense in layers while assuming the exterior layers can be completely broken. I like the idea of deploying an antidote but that won't always work and I can see people not liking it. CloudFlare would be an example of defense by layers. Another observation is about how the vulnerability is disclosed. I don't think this was handled properly here. Ideally you want to see a disclosure that allows users to take action without actually revealing the specific vulnerability. An extreme example would simply be to tell all Drupal users to take their sites down and give them enough time to do this before disclosing anything. Possibly a less broad action could be taken while still hiding the specific exploit. To release a patch revealing the exploit while many users are still open to an exploit is extremely problematic.

pizzeys 12y ago
I don't think this would work. The people getting caught by this would also miss the notice to take their sites down.
- YZF 12y ago
  It's a matter of controlling that window and minimizing the number of affected sites. Scenario #1: Release patch with no advance notice. Takes time for people to take notice and apply. In that window lots (12 million?!) sites get hacked. That doesn't sound like a very happy outcome. Scenario #2: Announce a patch will be released in advance. Get the word out. Give people time to hear it. You're not giving any specifics other than there is some vulnerability. That's doesn't help any hacker, every product has some vulnerability. There'll probably be a lot less than 12M sites hacked. Also consider closing the vulnerability in ways that don't directly touch the affected code to make it more difficult to reverse engineer. Obviously if everyone knows of the exploit you don't have the time but often the way hackers hear about the exploit is through your announcement... [EDIT: Thanks to aryx above I learnt there was a 5 day window in this specific case (An Oct 10th announcement for Oct 15th security update). I don't think that announcement made it clear enough what the consequences of not applying the patch immediately would be. I think this is something that needs more careful consideration in the future.]

pizzeys 12y ago

Many web hosts do this. At <redacted UK-based LAMP host> we do both the WAF stuff (which Cloudflare don't actually do, unless you pay them), and your latter suggestion of read-only'ing accounts on discovery of vulnerabilities. The unfortunate reality of the latter one is that it's going to break their site, probably, since we don't know what they run, and you can't just read only an entire account blindly without breaking something. We could do better there, but it will never be perfect in this environment.