9 ms·

Extra security measures for next week's releases

197 points by chanks 14y ago

overshard 14y ago
As much as I don't like stuff being hidden from me I think this is a good move. The title made me think it was a permanent move but it's just till this update is completed. The bad part of this is, that it's obviously a very serious vulnerability...
- thomasvendetta 14y ago
  And now the bad guys know there is a very serious vulnerability, somewhere.
  - pavel_lishin 14y ago
    They'd know it was there as soon as a patch was released, anyway.
  - dan_manges 14y ago
    Knowing that there is a vulnerability might motivate them to look for it, but given the size of the software, I doubt they'll be able to find it without knowing more.
    - jfim 14y ago
      You'd be surprised; on Windows, at least, there are people who reverse engineer the security patches from Microsoft in order to determine the initial vulnerability[1]. [1] http://www.phreedom.org/presentations/reverse-engineering-and-security/reverse-engineering-and-security.pdf http://www.phreedom.org/presentations/reverse-engineering-an... Edit: Misinterpreted your post. You're right, it's unlikely that they'll guess where it is until a patch comes out.
      - rosser 14y ago
        Because there are enough people running Windows who haven't applied the patch that figuring out how to exploit it is a worthwhile undertaking. Then again, IME of many years as a PostgreSQL DBA, the vast, overwhelming majority of postgres shops aren't running anywhere near the latest release, so depending on how far back this vulnerability goes, there could be a very large number of exploitable targets...
    - mixedbit 14y ago
      The knowledge may also motivate them to prepare for attacks to be executed once the vulnerability is public but most instances do not have it patched. Scan the Internet for PG backed applications, identify high profile ones, prepare automatic scripts, etc.
      - zalew 14y ago
        > Scan the Internet for PG backed applications, identify high profile ones, prepare automatic scripts, etc. it rather works like: - take exploit - spread it over the whole internet and calls home where it sticks
  - sophacles 14y ago
    The bad guys already assumed that. Seriously - the entire premise of IT security (no matter the color of your hat) is the assumption that there is no such thing as a secure computer.
sophacles 14y ago
This is an interesting tradeoff between responsible security and open source transparentness that the Postgres team is facing. I personally think this is a good way to handle a situation with a serious bug, but there are some questions it raises... Is Postgres working with downstream teams to have everything in place to do a coordinated security release? For instance, are they working with the likes of Debian's security team (for example) to not only make the direct source pullable, but also have releases available to as many users as possible in the platform preferred formats? If they are, how do they do keep this under wraps? It seems like the kind of thing that would require a fairly wide "pre-disclosure", and managing trust in a large network gets hard.
- danielweber 14y ago
  A lot of the people they need help from don't need to know the specifics. They can tell Debian what is basically in this mail, and Debian can be ready to accept a new package.
  - sounds 14y ago
    I agree. As long as the debian infrastructure is fully automated (it is), the actual time delay from pgsql's announcement to it hitting debian servers would be only an hour or so.
- ffk 14y ago
  Those are some good observations. Most likely they have given the information to Debian security. With something like this, there is a degree of trust that is maintained. The Debian security team has access to other zero-days on a regular basis, so ideally they aren't compromised. It wouldn't surprise me if AB tests were performed on security experts on a regular basis. E.g. two exploits discovered, one sent to half the team, the other sent to the other half. After a log(n) number of iterations, potential leaks are exposed. Diving further though, at what level do you say you trust the system though? Do you trust your compilers to not inject malicious code? (see http://c2.com/cgi/wiki?TheKenThompsonHack http://c2.com/cgi/wiki?TheKenThompsonHack) Do you trust peripheral devices? It's very easy to install a physical key logger into a system. Do you trust your chipsets? Compromised chipsets exist and can be used against you. (http://blogs.scientificamerican.com/observations/2011/07/11/hacked-hardware-has-been-sold-in-the-u-s/ http://blogs.scientificamerican.com/observations/2011/07/11/...) It's a tough situation to deal with. This is part of the reason layered security solutions are typically employed. Even if one system has a zero-day, ideally multiple layers should increase the overall complexity of triggering it. One of those layers are security teams and blackout periods where information is not released to the general public, even if they aren't always effective.
georgemcbay 14y ago
Wouldn't it make somewhat more sense to branch to a private repo without telling the public, make the required changes there, create the packages from that branch, and then later push the changes into the public repo? The way they are doing it now entices hackers who don't know the exploit but happen to have a recent clone of the repo to look for the big hole in hopes of finding it ahead of the fix. Granted, hackers are probably already doing that sort of thing on high profile services like Postgresql to begin with, but in my experience it is easier to find something exploitable when you already know something exploitable exists than it is when you're just randomly poking around. At the very least it makes it easier to stay motivated and focused.
- ibejoeb 14y ago
  Yes, folks are already attempting to find exploitable weakness in these projects. We can assume they exist. Just mentioning that one is confirmed doesn't really lend any insight. The surface area is pretty huge on that project. If I had to guess where it is, though, I'd bet it was in a PL module. I'm sure there is quite a bit of activity around finding NativeHelper-like situations.
  - brokenparser 14y ago
    It has to be something severe for this scenario to come into play. A broken procedure can only be exploited if such a procedure exists and can be invoked as the definer. This model is well understood by all, to the point a vulnerable PL may not be a critical issue for most users. Given the precautions that have been implemented, my bets are on authentication. This would mostly affect TCP/IP enabled hosts, which is fortunately not a default configuration (tested on Ubuntu).
  - shardling 14y ago
    >Yes, folks are already attempting to find exploitable weakness in these projects. We can assume they exist. Just mentioning that one is confirmed doesn't really lend any insight. You say that now, then one day, you wake up and all the blue-eyed islanders are gone!
danielweber 14y ago
How far back does Postgres maintain old versions? Will, say, Postgres 8 be updated with this fix (assuming the problem is in Postgres 8)? EDIT: Here is the answer: http://www.postgresql.org/support/versioning/ http://www.postgresql.org/support/versioning/ 8.0 was EOL'd in 2010, but 8.4 will go through July 2014.
selenamarie 14y ago
We've now posted a general announcement about the availability of packages on April 4, 2013: http://www.postgresql.org/message-id/CAN1EF+x0dmwMFuJGWuXMiRQtyT1s=Pe95f9gaF=uVCEa=V61fQ@mail.gmail.com http://www.postgresql.org/message-id/CAN1EF+x0dmwMFuJGWuXMiR... and http://www.postgresql.org/about/news/1454/ http://www.postgresql.org/about/news/1454/ As stated in the announcement and Tom's email to -hackers, the reasons for advanced notifications are as follows: * People watching for vulnerabilities and contributors are going to notice that we've stopped automatic updates -- it's better for our project to just tell them all why * Upgrading relational databases is often not trivial -- we want to give our users time to schedule an upgrade rather than just dropping an important update suddenly
jimrandomh 14y ago
What sort of security vulnerability would justify this extra paranoia? The worst case is that it's something which affects the very-common case of postgres servers that only talk to local services, like a unicode or quoting error that made sites which nominally quote their queries correctly vulnerable to SQL injection. That would be as serious as the recent Rails vulnerabilities: drop everything, patch everything everywhere, or definitely be rooted. Be ready to patch as soon as it's out; this could be a big deal.
- zer01 14y ago
  From what I hear this is pre-auth access to the DB, though it's not from the most reliable of sources.
willvarfar 14y ago
I wonder what the worst possible bug might be? A bug in query parameter parsing that would allow SQL injection attacks?
- onedognight 14y ago
  As conjectured above, the worst case is pre authentication remote code execution. i.e. anyone can just connect, send magic packets, and get a shell.
  - ef4 14y ago
    While that would be bad, if it required a magic packet it would have limited impact -- lots of postgres databases don't talk to public networks. Worse would be a vulnerability that you could trigger just by manipulating query parameters. Then almost every postgres-backed website would be vulnerable.
tiedemann 14y ago
I, for one, welcome our new secure PostgreSQL overlords!
lawnchair_larry 14y ago
This bug shouldn't be a huge deal because if you are treating your sensitive database server as anything but exploitable from any machine with network access, you've already lost. Even if your DB server is properly restricted, you should still patch quickly, but there is no way that it should be reachable unless you're already heavily compromised.
- tptacek 14y ago
  Unless it's something that compromises query/statement integrity from normal user input; a character set problem, for instance.
  - lawnchair_larry 14y ago
    Yep, there are a few potential ways that could be exceptions, depending on the bug. But let's not kid ourselves, nobody in the real world properly quarantines their DB servers anyway ;)
- fulafel 14y ago
  This is a pretty firewall happy mindset. Some of us don't have any "internal networks" as a matter of principle.
  - lawnchair_larry 14y ago
    There are varying degrees of "firewall happiness" and reasonable minds can disagree as to how far you go to balance convenience/security, but...you don't do any network segmentation as a matter of principle? Either I don't understand what you are saying, or you need to make a case for the immediate termination of everybody in charge of your network.

197 points by chanks 14y ago

As much as I don't like stuff being hidden from me I think this is a good move. The title made me think it was a permanent move but it's just till this update is completed. The bad part of this is, that it's obviously a very serious vulnerability...

thomasvendetta 14y ago
And now the bad guys know there is a very serious vulnerability, somewhere.
- pavel_lishin 14y ago
  They'd know it was there as soon as a patch was released, anyway.
- dan_manges 14y ago
  Knowing that there is a vulnerability might motivate them to look for it, but given the size of the software, I doubt they'll be able to find it without knowing more.
  - jfim 14y ago
    You'd be surprised; on Windows, at least, there are people who reverse engineer the security patches from Microsoft in order to determine the initial vulnerability[1]. [1] http://www.phreedom.org/presentations/reverse-engineering-and-security/reverse-engineering-and-security.pdf http://www.phreedom.org/presentations/reverse-engineering-an... Edit: Misinterpreted your post. You're right, it's unlikely that they'll guess where it is until a patch comes out.
    - rosser 14y ago
      Because there are enough people running Windows who haven't applied the patch that figuring out how to exploit it is a worthwhile undertaking. Then again, IME of many years as a PostgreSQL DBA, the vast, overwhelming majority of postgres shops aren't running anywhere near the latest release, so depending on how far back this vulnerability goes, there could be a very large number of exploitable targets...
  - mixedbit 14y ago
    The knowledge may also motivate them to prepare for attacks to be executed once the vulnerability is public but most instances do not have it patched. Scan the Internet for PG backed applications, identify high profile ones, prepare automatic scripts, etc.
    - zalew 14y ago
      > Scan the Internet for PG backed applications, identify high profile ones, prepare automatic scripts, etc. it rather works like: - take exploit - spread it over the whole internet and calls home where it sticks
- sophacles 14y ago
  The bad guys already assumed that. Seriously - the entire premise of IT security (no matter the color of your hat) is the assumption that there is no such thing as a secure computer.

sophacles 14y ago

This is an interesting tradeoff between responsible security and open source transparentness that the Postgres team is facing. I personally think this is a good way to handle a situation with a serious bug, but there are some questions it raises... Is Postgres working with downstream teams to have everything in place to do a coordinated security release? For instance, are they working with the likes of Debian's security team (for example) to not only make the direct source pullable, but also have releases available to as many users as possible in the platform preferred formats? If they are, how do they do keep this under wraps? It seems like the kind of thing that would require a fairly wide "pre-disclosure", and managing trust in a large network gets hard.

danielweber 14y ago
A lot of the people they need help from don't need to know the specifics. They can tell Debian what is basically in this mail, and Debian can be ready to accept a new package.
- sounds 14y ago
  I agree. As long as the debian infrastructure is fully automated (it is), the actual time delay from pgsql's announcement to it hitting debian servers would be only an hour or so.
ffk 14y ago
Those are some good observations. Most likely they have given the information to Debian security. With something like this, there is a degree of trust that is maintained. The Debian security team has access to other zero-days on a regular basis, so ideally they aren't compromised. It wouldn't surprise me if AB tests were performed on security experts on a regular basis. E.g. two exploits discovered, one sent to half the team, the other sent to the other half. After a log(n) number of iterations, potential leaks are exposed. Diving further though, at what level do you say you trust the system though? Do you trust your compilers to not inject malicious code? (see http://c2.com/cgi/wiki?TheKenThompsonHack http://c2.com/cgi/wiki?TheKenThompsonHack) Do you trust peripheral devices? It's very easy to install a physical key logger into a system. Do you trust your chipsets? Compromised chipsets exist and can be used against you. (http://blogs.scientificamerican.com/observations/2011/07/11/hacked-hardware-has-been-sold-in-the-u-s/ http://blogs.scientificamerican.com/observations/2011/07/11/...) It's a tough situation to deal with. This is part of the reason layered security solutions are typically employed. Even if one system has a zero-day, ideally multiple layers should increase the overall complexity of triggering it. One of those layers are security teams and blackout periods where information is not released to the general public, even if they aren't always effective.

georgemcbay 14y ago

Wouldn't it make somewhat more sense to branch to a private repo without telling the public, make the required changes there, create the packages from that branch, and then later push the changes into the public repo? The way they are doing it now entices hackers who don't know the exploit but happen to have a recent clone of the repo to look for the big hole in hopes of finding it ahead of the fix. Granted, hackers are probably already doing that sort of thing on high profile services like Postgresql to begin with, but in my experience it is easier to find something exploitable when you already know something exploitable exists than it is when you're just randomly poking around. At the very least it makes it easier to stay motivated and focused.

ibejoeb 14y ago
Yes, folks are already attempting to find exploitable weakness in these projects. We can assume they exist. Just mentioning that one is confirmed doesn't really lend any insight. The surface area is pretty huge on that project. If I had to guess where it is, though, I'd bet it was in a PL module. I'm sure there is quite a bit of activity around finding NativeHelper-like situations.
- brokenparser 14y ago
  It has to be something severe for this scenario to come into play. A broken procedure can only be exploited if such a procedure exists and can be invoked as the definer. This model is well understood by all, to the point a vulnerable PL may not be a critical issue for most users. Given the precautions that have been implemented, my bets are on authentication. This would mostly affect TCP/IP enabled hosts, which is fortunately not a default configuration (tested on Ubuntu).
- shardling 14y ago
  >Yes, folks are already attempting to find exploitable weakness in these projects. We can assume they exist. Just mentioning that one is confirmed doesn't really lend any insight. You say that now, then one day, you wake up and all the blue-eyed islanders are gone!

danielweber 14y ago

How far back does Postgres maintain old versions? Will, say, Postgres 8 be updated with this fix (assuming the problem is in Postgres 8)? EDIT: Here is the answer: http://www.postgresql.org/support/versioning/ http://www.postgresql.org/support/versioning/ 8.0 was EOL'd in 2010, but 8.4 will go through July 2014.

selenamarie 14y ago

We've now posted a general announcement about the availability of packages on April 4, 2013: http://www.postgresql.org/message-id/CAN1EF+x0dmwMFuJGWuXMiRQtyT1s=Pe95f9gaF=uVCEa=V61fQ@mail.gmail.com http://www.postgresql.org/message-id/CAN1EF+x0dmwMFuJGWuXMiR... and http://www.postgresql.org/about/news/1454/ http://www.postgresql.org/about/news/1454/ As stated in the announcement and Tom's email to -hackers, the reasons for advanced notifications are as follows: * People watching for vulnerabilities and contributors are going to notice that we've stopped automatic updates -- it's better for our project to just tell them all why * Upgrading relational databases is often not trivial -- we want to give our users time to schedule an upgrade rather than just dropping an important update suddenly

jimrandomh 14y ago

What sort of security vulnerability would justify this extra paranoia? The worst case is that it's something which affects the very-common case of postgres servers that only talk to local services, like a unicode or quoting error that made sites which nominally quote their queries correctly vulnerable to SQL injection. That would be as serious as the recent Rails vulnerabilities: drop everything, patch everything everywhere, or definitely be rooted. Be ready to patch as soon as it's out; this could be a big deal.

zer01 14y ago
From what I hear this is pre-auth access to the DB, though it's not from the most reliable of sources.

willvarfar 14y ago

I wonder what the worst possible bug might be? A bug in query parameter parsing that would allow SQL injection attacks?

onedognight 14y ago
As conjectured above, the worst case is pre authentication remote code execution. i.e. anyone can just connect, send magic packets, and get a shell.
- ef4 14y ago
  While that would be bad, if it required a magic packet it would have limited impact -- lots of postgres databases don't talk to public networks. Worse would be a vulnerability that you could trigger just by manipulating query parameters. Then almost every postgres-backed website would be vulnerable.

tiedemann 14y ago

I, for one, welcome our new secure PostgreSQL overlords!

lawnchair_larry 14y ago

This bug shouldn't be a huge deal because if you are treating your sensitive database server as anything but exploitable from any machine with network access, you've already lost. Even if your DB server is properly restricted, you should still patch quickly, but there is no way that it should be reachable unless you're already heavily compromised.

tptacek 14y ago
Unless it's something that compromises query/statement integrity from normal user input; a character set problem, for instance.
- lawnchair_larry 14y ago
  Yep, there are a few potential ways that could be exceptions, depending on the bug. But let's not kid ourselves, nobody in the real world properly quarantines their DB servers anyway ;)
fulafel 14y ago
This is a pretty firewall happy mindset. Some of us don't have any "internal networks" as a matter of principle.
- lawnchair_larry 14y ago
  There are varying degrees of "firewall happiness" and reasonable minds can disagree as to how far you go to balance convenience/security, but...you don't do any network segmentation as a matter of principle? Either I don't understand what you are saying, or you need to make a case for the immediate termination of everybody in charge of your network.