10 ms·

Apache web server bug grants root access on shared hosting environments

117 points by praveenscience 8y ago

josteink 8y ago
> Because on most Unix systems Apache httpd runs under the root user, any threat actor who has planted a malicious CGI script on an Apache server can use CVE-2019-0211 to take over the underlying system running the Apache httpd process, and inherently control the entire machine. Maybe I am conflating things or mixing something up, but I was under the impression that it only used root-privileges to obtain access to restricted ports and immediately afterwords lowered its privileges lever to something sane/non-risky. And if that is how it operates, then this exploit should not be effective. Is that wrong? Have I been misled?
- netsec_burn 8y ago
  The parent process continues to run as root.
  - vbezhenar 8y ago
    Why is it necessary?
    - mjw1007 8y ago
      The usual reason for this sort of design is so you can support loading a new configuration without having to restart the whole thing, if the new config might need you to do additional things that require privilege (maybe bind to new low ports, or read key files).
      - vbezhenar 8y ago
        I would expect restart to bind to new ports. It's not something that should be supported on-the fly (it's good thing, of course, but not at expense of keeping root process). Reloading key files is useful, but I think that it could be implemented with other means (e.g. reload program which runs under root, could read those files and pass them via some kind of IPC).
    - dspillett 8y ago
      Because, depending on your configured concurrency model, it may need to be able to spawn new processes/threads as other users, and this is not possible unless you have at least one process running with some privileges beyond that of the normal users. This usually means root. Even if not root, it needs to be a user/group privileged enough to be able to impersonate the other users, or it can't do its job. The code running with greater privilege is kept to a minimum but at least some needs to be there, and this exploit potentially gives a route through to manipulate it. FastCGI and similar can help here - it can push the creation of user specific processes away from Apache, making it harder to cross the barrier.
      - merb 8y ago
        you can't bind to port 80/443 without being root unless you have cap_net_bind_service. so instead of using nat/caps you just spawn as root to bind.
        kelnos 8y ago
        Right, but there's no reason for the parent process to keep root privileges after binding to the privileged ports.
      - vbezhenar 8y ago
        Thanks. I never used that feature, nor I ever heard of anyone using it. IMO there should be some configuration option to turn that feature off, which will allow to run all processes as a designated user. Or, if those users could be enumerated at startup, it could spawn one process for each user.
- ams6110 8y ago
  I still don't get it. CGI scripts would run with reduced privileges. The CVE description says "code executing in less-privileged child processes or threads (including scripts executed by an in-process scripting interpreter) could execute arbitrary code with the privileges of the parent process (usually root) by manipulating the scoreboard" What is the scoreboard?
  - chrismeller 8y ago
    It’s basically the list the master process keeps of the worker processes it has spun up. The workers can report back some stats that the master tracks about each. Apparently something in that report back process could be exploited to run arbitrary code.
    - yrro 8y ago
      Who on earth doesn't run CGI scripts, etc., as a separate user to the one that their apache workers run as?
      - tedunangst 8y ago
        Probably most people. Who uses setuid CGI scripts?
        yrro 8y ago
        You can't really have a setuid "script" anyway. But you can, at the bare minimum, launch CGI scripts via suEXEC. This prevents them from being able to attack the httpd worker processes, since they won't be running as the same user.
      - arbitrage 8y ago
        Welcome to reality. Please leave your principles at the door, you won't be needing them here.
      - rkeene2 8y ago
        I think it's been very common for 15 years to use Apache's "suexec" to run each CGI process as the individual user rather than as a user related to Apache. There's also suPHP which allows you to do the same thing for PHP processes.
- chrismeller 8y ago
  The worker processes that actually do all the rendering run as the user and group you’ve configured. The parent continues as root.
- SEJeff 8y ago
  > Because on most Unix systems Apache httpd runs under the root user This hasn't been true for what, 10+ years? Apache runs as user www-data on Ubuntu/Debian and as user httpd on Redhat / SUSE based distributions. You're entirely right, but this is a bit of embellishment on the part of the exploit author (even though it is serious!)
  - icedchai 8y ago
    It actually starts as root, then drops privileges / setuids to www-data.
  - tyingq 8y ago
    "this is a bit of embellishment on the part of the exploit author" Apache is saying the same thing. "In Apache HTTP Server 2.4 releases 2.4.17 to 2.4.38, ...code executing in less-privileged child processes or threads (including scripts executed by an in-process scripting interpreter) could execute arbitrary code with the privileges of the parent process (usually root)" https://httpd.apache.org/security/vulnerabilities_24.html#CVE-2019-0211 https://httpd.apache.org/security/vulnerabilities_24.html#CV...
  - mchristen 8y ago
    Lots of people install from source and just sudo everything because that's all they know.
    - tannhaeuser 8y ago
      The vanilla APACHE_RUN_USER/_GROUP is daemon/daemon so they're fine. daemon and nobody are users that by definition have no permissions on the host.
  - Vogtinator 8y ago
    It runs as wwwrun on SUSE based distros.
- 0x0 8y ago
  This exploit lets an unprivileged child worker (for example running mod_php on regular shared hosting clients' .php files) hack back into the root parent process and thus escalate from unprivileged to root.
ArtemZ 8y ago
I thought that shared hosting is not a thing anymore in age of clouds
3ch0 8y ago
Found this, looks like a more detailed writeup of whats going on. https://cfreal.github.io/carpe-diem-cve-2019-0211-apache-local-root.html https://cfreal.github.io/carpe-diem-cve-2019-0211-apache-loc...
- ajackfox 8y ago
  Probably a stupid question, but are the version #'s mentioned here ( "From version 2.4.17 (Oct 9, 2015) to version 2.4.38 (Apr 1, 2019), Apache HTTP suffers from a local root privilege escalation vulnerability due to an out-of-bounds array access leading to an arbitrary function call." ) the only ones affected, or is it possible earlier versions are affected as well? Also, I just want to throw out there that the name of this one is great: "Why the name ? CARPE: stands for CVE-2019-0211 Apache Root Privilege Escalation DIEM: the exploit triggers once a day I had to."
  - dspillett 8y ago
    > but are the version #'s mentioned here ... the only ones affected Usually when a range like that is given, yes. > From version 2.4.17 (Oct 9, 2015) to version 2.4.38 (Apr 1, 2019) This case implies that they know the bug was introduced in a particular change, which went public with version 2.4.17 and was either fixed or otherwise mitigated in 2.4.38. The only earlier or other versions that I would expect to see affected are dev/alpha/beta branches.
- ec109685 8y ago
  This part is concerning: > Apache's team has been prompt to respond and patch, and nice as hell. Really good experience. PHP never answered regarding the UAF.
  - Operyl 8y ago
    That’s been my experience reporting any kind of bug with the PHP core team. It really is a pain in the neck.
  - hannob 8y ago
6xggd7 8y ago
Who's still using Apache?
- sccxy 8y ago
  40% of web?
  - spurgu 8y ago
    "Apache is used by 43.8% of all the websites whose web server we know." according to https://w3techs.com/technologies/details/ws-apache/all/all https://w3techs.com/technologies/details/ws-apache/all/all I would venture a guess and say it's largely because all the shared hosting providers use it. Less and less people actually choose it given a choice.
    - 6xggd7 8y ago
      Yeah sorry, I meant on purpose. Most shared hostings still use Apache for compatibility with .htaccess and rubbish like that.
      - justinph 8y ago
        Rubbish? You mean like keeping the web working and making sure URLs never die? Not everything needs to be upgraded to the latest hot thing. HTTPD is proven & stable; this is a rare bug.
- rkeene2 8y ago
  I use it. On purpose. It's a good general purpose web server and does a good job.
  - neurobashing 8y ago
    It's also copiously documented and extremely well-understood. There will always be bugs in software but you can usually fix problems quickly, there's few surprises or gotchas. I trust it.
btown 8y ago
For something this severe, there doesn’t appear to have been coordination with large shared hosting providers to ensure the patch was applied before public disclosure of a PoC. Is there a system in place for this?
- foobarbazetc 8y ago
  It doesn’t effect CentOS 7, which is what most hosting providers run on.
  - equalunique 8y ago
    How do you know it doesn't affect CentOS 7? Please provide a citation for your claim. :)
    - ihattendorf 8y ago
      I'm not the person you're replying to, but: https://access.redhat.com/security/cve/cve-2019-0211 https://access.redhat.com/security/cve/cve-2019-0211 My guess is because they're on an older version (2.4.6) that isn't affected. But that all depends on what patches have been applied to the RHEL/CentOS version.
  - tannhaeuser 8y ago
    You're sure? From what I've experienced, shared hosting is owned by Debian/Devuan and Ubuntu, typically on a Debian-style Apache installation with modular config file directories and enmod/ensite utilties for management.
  - regecks 8y ago
    Well, most shared hosting providers are also running cPanel on CentOS 7 and are getting their httpd packages from EasyApache 3/4, not from the CentOS base repo. cPanel definitely responded to this CVE: https://github.com/CpanelInc/ea-apache2/blob/386800043a02e88a490596f11373a5fc0e66a8fd/SPECS/ea-apache24.spec#L1959 https://github.com/CpanelInc/ea-apache2/blob/386800043a02e88...
jondubois 8y ago
>> less-privileged Apache child processes (such as CGI scripts) can execute malicious code with the privileges of the parent process. Vulnerabilities always seem to be related to CGI scripts somehow.
- tyingq 8y ago
  Something that makes the cgi process not a child of the Apache process wouldn't be subject to this. Like a fcgi process.
  - rbritton 8y ago
    I'm surprised FastCGI isn't more popular for that reason. Using PHP as an example, Apache + mod_php executes all code under Apache's user/group. If, instead, you use PHP-FPM, you can lock down the execution environment much more. PHP-FPM supports an arbitrary number of named pools, each of which may be configured with its own user/group pair. Ideally, each virtual host gets its own pool. e.g., [poolname] user = poolowner group = poolgroup listen = /run/php/php7.2-fpm-poolname.sock listen.owner = www-data listen.group = www-data ...
    - dspillett 8y ago
      > I'm surprised FastCGI isn't more popular for that reason. FastCGI is more faf to configure where mod_php is more likely to be configured out of the box. Most shared hosting environment are configured entirely out-of-the-box as the market is so saturated and margins so low there is no way to justify anything else. Potential customers who care much about security will most likely be looking for their own dedicated server or VM instead so there isn't really much of a market for a more secure shared host. Also if packing as many users as possible into as little hardware as possible, which is the only way to make any margin in shared hosting these days, you'll find mod_php more efficient by this measure. You don't have a pool of processes that only specific users can take advantage of, taking up an amount of RAM (however small) each even when not actively in use. So FastCGI tends to be used less. When I ran PHP I used it, usually as you suggest one pool per vhost, but I was only hosting my own projects and some bits for friends & family, so I didn't need to justify the setup effort against any "bottom line".
raintrees 8y ago
Wouldn't SELinux disallow writing to the cgi folder, unless that attribute were explicitly set?
- LinuxBender 8y ago
  Yes. Sadly, very few people use it. Even less so in shared hosting environments.
- giobox 8y ago
  Almost everywhere I’ve worked ends up turning off Selinux rather than learn how to actually configure/use it properly, which of course is ridiculous when it’s not actually that hard to learn or configure at all generally, but such is the reality of many places it seems.

> Because on most Unix systems Apache httpd runs under the root user, any threat actor who has planted a malicious CGI script on an Apache server can use CVE-2019-0211 to take over the underlying system running the Apache httpd process, and inherently control the entire machine. Maybe I am conflating things or mixing something up, but I was under the impression that it only used root-privileges to obtain access to restricted ports and immediately afterwords lowered its privileges lever to something sane/non-risky. And if that is how it operates, then this exploit should not be effective. Is that wrong? Have I been misled?

netsec_burn 8y ago
The parent process continues to run as root.
- vbezhenar 8y ago
  Why is it necessary?
  - mjw1007 8y ago
    The usual reason for this sort of design is so you can support loading a new configuration without having to restart the whole thing, if the new config might need you to do additional things that require privilege (maybe bind to new low ports, or read key files).
    - vbezhenar 8y ago
      I would expect restart to bind to new ports. It's not something that should be supported on-the fly (it's good thing, of course, but not at expense of keeping root process). Reloading key files is useful, but I think that it could be implemented with other means (e.g. reload program which runs under root, could read those files and pass them via some kind of IPC).
  - dspillett 8y ago
    Because, depending on your configured concurrency model, it may need to be able to spawn new processes/threads as other users, and this is not possible unless you have at least one process running with some privileges beyond that of the normal users. This usually means root. Even if not root, it needs to be a user/group privileged enough to be able to impersonate the other users, or it can't do its job. The code running with greater privilege is kept to a minimum but at least some needs to be there, and this exploit potentially gives a route through to manipulate it. FastCGI and similar can help here - it can push the creation of user specific processes away from Apache, making it harder to cross the barrier.
    - merb 8y ago
      you can't bind to port 80/443 without being root unless you have cap_net_bind_service. so instead of using nat/caps you just spawn as root to bind.
      - kelnos 8y ago
        Right, but there's no reason for the parent process to keep root privileges after binding to the privileged ports.
    - vbezhenar 8y ago
      Thanks. I never used that feature, nor I ever heard of anyone using it. IMO there should be some configuration option to turn that feature off, which will allow to run all processes as a designated user. Or, if those users could be enumerated at startup, it could spawn one process for each user.
ams6110 8y ago
I still don't get it. CGI scripts would run with reduced privileges. The CVE description says "code executing in less-privileged child processes or threads (including scripts executed by an in-process scripting interpreter) could execute arbitrary code with the privileges of the parent process (usually root) by manipulating the scoreboard" What is the scoreboard?
- chrismeller 8y ago
  It’s basically the list the master process keeps of the worker processes it has spun up. The workers can report back some stats that the master tracks about each. Apparently something in that report back process could be exploited to run arbitrary code.
  - yrro 8y ago
    Who on earth doesn't run CGI scripts, etc., as a separate user to the one that their apache workers run as?
    - tedunangst 8y ago
      Probably most people. Who uses setuid CGI scripts?
      - yrro 8y ago
        You can't really have a setuid "script" anyway. But you can, at the bare minimum, launch CGI scripts via suEXEC. This prevents them from being able to attack the httpd worker processes, since they won't be running as the same user.
    - arbitrage 8y ago
      Welcome to reality. Please leave your principles at the door, you won't be needing them here.
    - rkeene2 8y ago
      I think it's been very common for 15 years to use Apache's "suexec" to run each CGI process as the individual user rather than as a user related to Apache. There's also suPHP which allows you to do the same thing for PHP processes.
chrismeller 8y ago
The worker processes that actually do all the rendering run as the user and group you’ve configured. The parent continues as root.
SEJeff 8y ago
> Because on most Unix systems Apache httpd runs under the root user This hasn't been true for what, 10+ years? Apache runs as user www-data on Ubuntu/Debian and as user httpd on Redhat / SUSE based distributions. You're entirely right, but this is a bit of embellishment on the part of the exploit author (even though it is serious!)
- icedchai 8y ago
  It actually starts as root, then drops privileges / setuids to www-data.
- tyingq 8y ago
  "this is a bit of embellishment on the part of the exploit author" Apache is saying the same thing. "In Apache HTTP Server 2.4 releases 2.4.17 to 2.4.38, ...code executing in less-privileged child processes or threads (including scripts executed by an in-process scripting interpreter) could execute arbitrary code with the privileges of the parent process (usually root)" https://httpd.apache.org/security/vulnerabilities_24.html#CVE-2019-0211 https://httpd.apache.org/security/vulnerabilities_24.html#CV...
- mchristen 8y ago
  Lots of people install from source and just sudo everything because that's all they know.
  - tannhaeuser 8y ago
    The vanilla APACHE_RUN_USER/_GROUP is daemon/daemon so they're fine. daemon and nobody are users that by definition have no permissions on the host.
- Vogtinator 8y ago
  It runs as wwwrun on SUSE based distros.
0x0 8y ago
This exploit lets an unprivileged child worker (for example running mod_php on regular shared hosting clients' .php files) hack back into the root parent process and thus escalate from unprivileged to root.

ArtemZ 8y ago

I thought that shared hosting is not a thing anymore in age of clouds

3ch0 8y ago

Found this, looks like a more detailed writeup of whats going on. https://cfreal.github.io/carpe-diem-cve-2019-0211-apache-local-root.html https://cfreal.github.io/carpe-diem-cve-2019-0211-apache-loc...

ajackfox 8y ago
Probably a stupid question, but are the version #'s mentioned here ( "From version 2.4.17 (Oct 9, 2015) to version 2.4.38 (Apr 1, 2019), Apache HTTP suffers from a local root privilege escalation vulnerability due to an out-of-bounds array access leading to an arbitrary function call." ) the only ones affected, or is it possible earlier versions are affected as well? Also, I just want to throw out there that the name of this one is great: "Why the name ? CARPE: stands for CVE-2019-0211 Apache Root Privilege Escalation DIEM: the exploit triggers once a day I had to."
- dspillett 8y ago
  > but are the version #'s mentioned here ... the only ones affected Usually when a range like that is given, yes. > From version 2.4.17 (Oct 9, 2015) to version 2.4.38 (Apr 1, 2019) This case implies that they know the bug was introduced in a particular change, which went public with version 2.4.17 and was either fixed or otherwise mitigated in 2.4.38. The only earlier or other versions that I would expect to see affected are dev/alpha/beta branches.
ec109685 8y ago
This part is concerning: > Apache's team has been prompt to respond and patch, and nice as hell. Really good experience. PHP never answered regarding the UAF.
- Operyl 8y ago
  That’s been my experience reporting any kind of bug with the PHP core team. It really is a pain in the neck.
- hannob 8y ago

6xggd7 8y ago

Who's still using Apache?

sccxy 8y ago
40% of web?
- spurgu 8y ago
  "Apache is used by 43.8% of all the websites whose web server we know." according to https://w3techs.com/technologies/details/ws-apache/all/all https://w3techs.com/technologies/details/ws-apache/all/all I would venture a guess and say it's largely because all the shared hosting providers use it. Less and less people actually choose it given a choice.
  - 6xggd7 8y ago
    Yeah sorry, I meant on purpose. Most shared hostings still use Apache for compatibility with .htaccess and rubbish like that.
    - justinph 8y ago
      Rubbish? You mean like keeping the web working and making sure URLs never die? Not everything needs to be upgraded to the latest hot thing. HTTPD is proven & stable; this is a rare bug.
rkeene2 8y ago
I use it. On purpose. It's a good general purpose web server and does a good job.
- neurobashing 8y ago
  It's also copiously documented and extremely well-understood. There will always be bugs in software but you can usually fix problems quickly, there's few surprises or gotchas. I trust it.

btown 8y ago

For something this severe, there doesn’t appear to have been coordination with large shared hosting providers to ensure the patch was applied before public disclosure of a PoC. Is there a system in place for this?

foobarbazetc 8y ago
It doesn’t effect CentOS 7, which is what most hosting providers run on.
- equalunique 8y ago
  How do you know it doesn't affect CentOS 7? Please provide a citation for your claim. :)
  - ihattendorf 8y ago
    I'm not the person you're replying to, but: https://access.redhat.com/security/cve/cve-2019-0211 https://access.redhat.com/security/cve/cve-2019-0211 My guess is because they're on an older version (2.4.6) that isn't affected. But that all depends on what patches have been applied to the RHEL/CentOS version.
- tannhaeuser 8y ago
  You're sure? From what I've experienced, shared hosting is owned by Debian/Devuan and Ubuntu, typically on a Debian-style Apache installation with modular config file directories and enmod/ensite utilties for management.
- regecks 8y ago
  Well, most shared hosting providers are also running cPanel on CentOS 7 and are getting their httpd packages from EasyApache 3/4, not from the CentOS base repo. cPanel definitely responded to this CVE: https://github.com/CpanelInc/ea-apache2/blob/386800043a02e88a490596f11373a5fc0e66a8fd/SPECS/ea-apache24.spec#L1959 https://github.com/CpanelInc/ea-apache2/blob/386800043a02e88...

jondubois 8y ago

>> less-privileged Apache child processes (such as CGI scripts) can execute malicious code with the privileges of the parent process. Vulnerabilities always seem to be related to CGI scripts somehow.

tyingq 8y ago
Something that makes the cgi process not a child of the Apache process wouldn't be subject to this. Like a fcgi process.
- rbritton 8y ago
  I'm surprised FastCGI isn't more popular for that reason. Using PHP as an example, Apache + mod_php executes all code under Apache's user/group. If, instead, you use PHP-FPM, you can lock down the execution environment much more. PHP-FPM supports an arbitrary number of named pools, each of which may be configured with its own user/group pair. Ideally, each virtual host gets its own pool. e.g., [poolname] user = poolowner group = poolgroup listen = /run/php/php7.2-fpm-poolname.sock listen.owner = www-data listen.group = www-data ...
  - dspillett 8y ago
    > I'm surprised FastCGI isn't more popular for that reason. FastCGI is more faf to configure where mod_php is more likely to be configured out of the box. Most shared hosting environment are configured entirely out-of-the-box as the market is so saturated and margins so low there is no way to justify anything else. Potential customers who care much about security will most likely be looking for their own dedicated server or VM instead so there isn't really much of a market for a more secure shared host. Also if packing as many users as possible into as little hardware as possible, which is the only way to make any margin in shared hosting these days, you'll find mod_php more efficient by this measure. You don't have a pool of processes that only specific users can take advantage of, taking up an amount of RAM (however small) each even when not actively in use. So FastCGI tends to be used less. When I ran PHP I used it, usually as you suggest one pool per vhost, but I was only hosting my own projects and some bits for friends & family, so I didn't need to justify the setup effort against any "bottom line".

raintrees 8y ago

Wouldn't SELinux disallow writing to the cgi folder, unless that attribute were explicitly set?

LinuxBender 8y ago
Yes. Sadly, very few people use it. Even less so in shared hosting environments.
giobox 8y ago
Almost everywhere I’ve worked ends up turning off Selinux rather than learn how to actually configure/use it properly, which of course is ridiculous when it’s not actually that hard to learn or configure at all generally, but such is the reality of many places it seems.