7 ms·

Main GNU source repository server compromised

135 points by there 16y ago

Sandman 16y ago
I see a lot of angry comments on their webpage, people asking why somebody would do this. Well, the real answer is probably - simply because he/she could. And to all of those wondering how anybody could do this to a project that hosts free software, well, the attackers don't really care. They saw the exploit and used it. Personally, I expected that a project such as Savannah wouldn't be vulnerable to an attack as simple as SQL injection.
- deleted 16y ago
  [deleted]
- dkarl 16y ago
  Maybe someone had an idea for putting a vulnerability into a GNU utility. Personally, I expected that a project such as Savannah wouldn't be vulnerable to an attack as simple as SQL injection. Seconded. But then again, the GNU source code is mostly C, isn't it? They're used to doing things that most developers would consider intractable and therefore impossible to do safely: such as comprehensively sanitizing inputs so you can safely construct a query string out of them. "Hey, those web guys say sanitizing inputs is effectively impossible. Should we really be doing this?" "Snort. That's what they say about manually deallocating memory, too." "Oh yeah! What a bunch of wimps! I'll get right to work on the query builder."
  - deleted 16y ago
    [deleted]
  - rryyan 16y ago
    Going further: how about a vulnerability in every program compiled with gcc? I'll just leave this here: http://cm.bell-labs.com/who/ken/trust.html http://cm.bell-labs.com/who/ken/trust.html
  - riffraff 16y ago
    I'm pretty sure savannah is php, forked from the original *forge code, so not really a fault of the people who wrote ls and gcc.
- tptacek 16y ago
  People in 1997 never expected that <insert project here> could be vulnerable to an attack as simple as a buffer overflow; after all, all it takes to not be vulnerable to overflows is "counting", and who has problems counting with a computer? Tens of billions of dollars in losses later: do you feel safe hitting arbitrary pages with your web browser?
- prodigal_erik 16y ago
  The code might be much worse than anything they would have written themselves. http://en.wikipedia.org/wiki/GNU_Savannah http://en.wikipedia.org/wiki/GNU_Savannah explains they started with a fork of the SourceForge code, and they seem to still be on it (the Python/Django rewrite isn't complete). It's not obvious what SourceForge was written in, but http://sourceforge.net/apps/trac/sourceforge/wiki/Open%20Source%20software%20used%20at%20SourceForge.net http://sourceforge.net/apps/trac/sourceforge/wiki/Open%20Sou... shows an awful lot of PHP dependencies, and the site dates back to 1999, a time when the worst of PHP's myriad design flaws (such as register_globals and magic_quotes) were widely embraced.
raphman 16y ago
From the comments on the linked page: Date: Tue Nov 30 01:34:15 2010 Pseudo: mjflick - Savannah Hacker Comment: re: HappyCrow, Only one project was targeted in this attack. Second, a postmortem will be forthcoming, as well as more information shortly from the FSF staff, who are planning on making an official announcement about this.
viraptor 16y ago
Ok... I hope I misunderstood this, but does anyone see a problem in the order they're doing this in? [X] Reset passwords [/] Fix SQL injection and look for potential others [ ] Implement crypt-md5 support (like /etc/shadow, strong and LDAP-compatible) hashes [ ] Implement password strength enforcement So they're basically forcing password resets, even though people can use "password123" in plaintext on a site that's not completely audited for vulnerabilities?
- lukeschlather 16y ago
  You eliminated step one, which clearly states: [X] Put services online using backup, except for password-based ones (e.g. the web interface) The other step you omitted is the one where they allow people to log in after fixing the vulnerability. As for the audit, there's no such thing as a complete security audit, and there's no reason to keep the site down in fear of hypothetical holes.
  - viraptor 16y ago
    Sorry, my mistake. I got the website<->other services restoration time backwards. Thanks for correcting :) Re. audit - of course you cannot be sure, but once you get taken over and your system is offline anyways, it might be a good idea to at least grep the sources for queries and quickly check for obvious stuff. Which is what I assume they're doing.
konad 16y ago
Headline missed a word : "again" Original issue date: August 13, 2003 http://www.cert.org/advisories/CA-2003-21.html http://www.cert.org/advisories/CA-2003-21.html ftp://ftp.gnu.org/MISSING-FILES.README Moving Forward All releases after the 2003-08-01 date will have checksums GPG-signed by the GNU maintainer who prepared the release. This assures automatic certification of the integrity of all GNU source from that date onward.
- tjr 16y ago
  Slightly different, as that compromise was of the FTP servers, which store packaged releases. Savannah is the development system.
pluies 16y ago
Ouch. This is bad. It illustrates how much of a pain SQL injections are, if they can affect the GNU project which has some of the most incredibly talented hackers worlwide... Props to them for having a working backup strategy though.
- petrilli 16y ago
  Actually, no... you have to be pretty ignorant to be subject to them. Use bind variables in your queries and you are done. That it's still one of the top 2-3 exploits tells you how horrible most developers really are.
  - stcredzero 16y ago
    Not all talented developers are talented or even good at security.
  - tptacek 16y ago
    No. You are not done with SQL injection just because you use "bind variables". Plenty of things can't be parameterized that are nonetheless subject to user influence. Click the top of any reverse-sortable column in a web user interface to see one obvious example. Parameterized queries are a good thing, and you should use them, but I feel like I've had to be a broken record about this over the last week: they are not magic anti- SQL injection fairy dust. Our whole team spends most of its time looking at smart people's code, and we routinely find SQL injection. Plenty of people who aren't ignorant of SQL injection manage to let SQL injection slip through. Knowing how SQL injection works and being able to devise and implement engineering procedures to reliably prevent them are two very different things.
    - btilly 16y ago
      Click the top of any reverse-sortable column in a web user interface to see one obvious example. People don't do those in JavaScript? That's what I did a decade ago. Well, then I moved it back to the server after we got tired of the performance problems that JavaScript had back then. But today it wouldn't be an issue. And when we moved it back, we were careful not to have an SQL injection attack. If memory serves we actually did the resort in Perl. (In our defense, much of the data we were serving lived in flat files, or were generated on the fly from a compute server, instead of coming from a database.) However at another company I had the same problem, and I did the obvious "process CGI parameter, insert appropriate ORDER BY statement". Where the definition of appropriate was by column position, from which I worked out the field to sort by, so I didn't have to trust the client for the name of that column. I should back up. We didn't have any SQL injection attacks that I knew of and were reasonably careful. But that code base did not get audited, so I can't really know that. However after the next company that I worked for got bought by eBay, they did a penetration test on us. The worst thing that they found was an open redirect that could be used to let a spammer construct a link to any web page with us as the referer. I'm happy to use this as a testament that programmers really can avoid SQL injections. However their surprise that they didn't find any SQL injections in our code supports your claim that most teams fail to do so successfully.
linuxaddict 16y ago
I thought more eyes on the source means that OSS is impenetrable. Guess not.
- technomancy 16y ago
  Nobody really cares about helping out with Savannah. On the emacs-devel mailing list every time Savannah is mentioned, it's in the context of "Yeah, we would like to do that but lack the manpower. Any volunteers?" followed by crickets.
  - linuxaddict 16y ago
    That is no excuse for the software getting hacked. It is poorly developed, and that is the bottom line. If no one is working on it, it should be turned off, now it is just a flawed, insecure tool thats insecurity will be used against OSS. Stop making excuses, and start fixing the sources.
SriniK 16y ago
What a moran whoever did this.
- konad 16y ago
  “But he has nothing on at all,” said a little child at last. “Good heavens! listen to the voice of an innocent child,” said the father, and one whispered to the other what the child had said. “But he has nothing on at all,” cried at last the whole people. That made a deep impression upon the emperor, for it seemed to him that they were right; but he thought to himself, “Now I must bear up to the end.” And the chamberlains walked with still greater dignity, as if they carried the train which did not exist.
  - konad 16y ago
    point missed
linuxaddict 16y ago
GNU hackers fail again.
sandGorgon 16y ago
Question - if they had been using a DVCS which checksums every commit, would they have been safer ? Presumably, you could build a tool that stores a list of checksums and if you had to do a restore, that tool could validate the backup. Or is it that, with a little effort, DVCS' can be hacked/corrupted fundamentally ?
- JoachimSchipper 16y ago
  I don't think DVCS would have helped in any way. It's not the repository server itself that was compromised, AFAIK, but "only" the web interface. This allows password recovery and thus allows people to e.g. make malicious commits from a trusted account, but that would be no less true if there were a DVCS under it all.
  - sandGorgon 16y ago
    agreed - but my question still stands: Take SVN for instance - if I managed to hack into a SVN server, can I change the source code (under SVN management) in a way that would be undetectable ? Repeat the same question for git, mercurial and CVS.
    - JoachimSchipper 16y ago
      For CVS and svn: you can hide it until someone actually looks at the relevant code. For git: not really (as long as you assume you change only the data - clever attackers could serve one repository to the developers and another to .mil/the rest of the world/...) I don't know about mercurial, but I'd wager it's the same as git in this respect. In either case, an attacker could add arbitrary commits, and the system administrator would roll back to the last known-good backup as soon as the compromise was detected. I don't think DVCSes help much here.
      - sandGorgon 16y ago
        For CVS and svn: you can hide it until someone actually looks at the relevant code. For git: not really I don't think DVCSes help much here. Those statements sound contradictory. Let me try to understand - the distributed aspects of DVCSes would not have helped here. But the incidental fact of checksums in DVCSes (maybe necessitated by the nature of "distributed") does help. Right ?

135 points by there 16y ago

I see a lot of angry comments on their webpage, people asking why somebody would do this. Well, the real answer is probably - simply because he/she could. And to all of those wondering how anybody could do this to a project that hosts free software, well, the attackers don't really care. They saw the exploit and used it. Personally, I expected that a project such as Savannah wouldn't be vulnerable to an attack as simple as SQL injection.

deleted 16y ago
[deleted]
dkarl 16y ago
Maybe someone had an idea for putting a vulnerability into a GNU utility. Personally, I expected that a project such as Savannah wouldn't be vulnerable to an attack as simple as SQL injection. Seconded. But then again, the GNU source code is mostly C, isn't it? They're used to doing things that most developers would consider intractable and therefore impossible to do safely: such as comprehensively sanitizing inputs so you can safely construct a query string out of them. "Hey, those web guys say sanitizing inputs is effectively impossible. Should we really be doing this?" "Snort. That's what they say about manually deallocating memory, too." "Oh yeah! What a bunch of wimps! I'll get right to work on the query builder."
- deleted 16y ago
  [deleted]
- rryyan 16y ago
  Going further: how about a vulnerability in every program compiled with gcc? I'll just leave this here: http://cm.bell-labs.com/who/ken/trust.html http://cm.bell-labs.com/who/ken/trust.html
- riffraff 16y ago
  I'm pretty sure savannah is php, forked from the original *forge code, so not really a fault of the people who wrote ls and gcc.
tptacek 16y ago
People in 1997 never expected that <insert project here> could be vulnerable to an attack as simple as a buffer overflow; after all, all it takes to not be vulnerable to overflows is "counting", and who has problems counting with a computer? Tens of billions of dollars in losses later: do you feel safe hitting arbitrary pages with your web browser?
prodigal_erik 16y ago
The code might be much worse than anything they would have written themselves. http://en.wikipedia.org/wiki/GNU_Savannah http://en.wikipedia.org/wiki/GNU_Savannah explains they started with a fork of the SourceForge code, and they seem to still be on it (the Python/Django rewrite isn't complete). It's not obvious what SourceForge was written in, but http://sourceforge.net/apps/trac/sourceforge/wiki/Open%20Source%20software%20used%20at%20SourceForge.net http://sourceforge.net/apps/trac/sourceforge/wiki/Open%20Sou... shows an awful lot of PHP dependencies, and the site dates back to 1999, a time when the worst of PHP's myriad design flaws (such as register_globals and magic_quotes) were widely embraced.

raphman 16y ago

From the comments on the linked page: Date: Tue Nov 30 01:34:15 2010 Pseudo: mjflick - Savannah Hacker Comment: re: HappyCrow, Only one project was targeted in this attack. Second, a postmortem will be forthcoming, as well as more information shortly from the FSF staff, who are planning on making an official announcement about this.

viraptor 16y ago

Ok... I hope I misunderstood this, but does anyone see a problem in the order they're doing this in? [X] Reset passwords [/] Fix SQL injection and look for potential others [ ] Implement crypt-md5 support (like /etc/shadow, strong and LDAP-compatible) hashes [ ] Implement password strength enforcement So they're basically forcing password resets, even though people can use "password123" in plaintext on a site that's not completely audited for vulnerabilities?

lukeschlather 16y ago
You eliminated step one, which clearly states: [X] Put services online using backup, except for password-based ones (e.g. the web interface) The other step you omitted is the one where they allow people to log in after fixing the vulnerability. As for the audit, there's no such thing as a complete security audit, and there's no reason to keep the site down in fear of hypothetical holes.
- viraptor 16y ago
  Sorry, my mistake. I got the website<->other services restoration time backwards. Thanks for correcting :) Re. audit - of course you cannot be sure, but once you get taken over and your system is offline anyways, it might be a good idea to at least grep the sources for queries and quickly check for obvious stuff. Which is what I assume they're doing.

konad 16y ago

Headline missed a word : "again" Original issue date: August 13, 2003 http://www.cert.org/advisories/CA-2003-21.html http://www.cert.org/advisories/CA-2003-21.html ftp://ftp.gnu.org/MISSING-FILES.README Moving Forward All releases after the 2003-08-01 date will have checksums GPG-signed by the GNU maintainer who prepared the release. This assures automatic certification of the integrity of all GNU source from that date onward.

tjr 16y ago
Slightly different, as that compromise was of the FTP servers, which store packaged releases. Savannah is the development system.

pluies 16y ago

Ouch. This is bad. It illustrates how much of a pain SQL injections are, if they can affect the GNU project which has some of the most incredibly talented hackers worlwide... Props to them for having a working backup strategy though.

petrilli 16y ago
Actually, no... you have to be pretty ignorant to be subject to them. Use bind variables in your queries and you are done. That it's still one of the top 2-3 exploits tells you how horrible most developers really are.
- stcredzero 16y ago
  Not all talented developers are talented or even good at security.
- tptacek 16y ago
  No. You are not done with SQL injection just because you use "bind variables". Plenty of things can't be parameterized that are nonetheless subject to user influence. Click the top of any reverse-sortable column in a web user interface to see one obvious example. Parameterized queries are a good thing, and you should use them, but I feel like I've had to be a broken record about this over the last week: they are not magic anti- SQL injection fairy dust. Our whole team spends most of its time looking at smart people's code, and we routinely find SQL injection. Plenty of people who aren't ignorant of SQL injection manage to let SQL injection slip through. Knowing how SQL injection works and being able to devise and implement engineering procedures to reliably prevent them are two very different things.
  - btilly 16y ago
    Click the top of any reverse-sortable column in a web user interface to see one obvious example. People don't do those in JavaScript? That's what I did a decade ago. Well, then I moved it back to the server after we got tired of the performance problems that JavaScript had back then. But today it wouldn't be an issue. And when we moved it back, we were careful not to have an SQL injection attack. If memory serves we actually did the resort in Perl. (In our defense, much of the data we were serving lived in flat files, or were generated on the fly from a compute server, instead of coming from a database.) However at another company I had the same problem, and I did the obvious "process CGI parameter, insert appropriate ORDER BY statement". Where the definition of appropriate was by column position, from which I worked out the field to sort by, so I didn't have to trust the client for the name of that column. I should back up. We didn't have any SQL injection attacks that I knew of and were reasonably careful. But that code base did not get audited, so I can't really know that. However after the next company that I worked for got bought by eBay, they did a penetration test on us. The worst thing that they found was an open redirect that could be used to let a spammer construct a link to any web page with us as the referer. I'm happy to use this as a testament that programmers really can avoid SQL injections. However their surprise that they didn't find any SQL injections in our code supports your claim that most teams fail to do so successfully.

linuxaddict 16y ago

I thought more eyes on the source means that OSS is impenetrable. Guess not.

technomancy 16y ago
Nobody really cares about helping out with Savannah. On the emacs-devel mailing list every time Savannah is mentioned, it's in the context of "Yeah, we would like to do that but lack the manpower. Any volunteers?" followed by crickets.
- linuxaddict 16y ago
  That is no excuse for the software getting hacked. It is poorly developed, and that is the bottom line. If no one is working on it, it should be turned off, now it is just a flawed, insecure tool thats insecurity will be used against OSS. Stop making excuses, and start fixing the sources.

SriniK 16y ago

What a moran whoever did this.

konad 16y ago
“But he has nothing on at all,” said a little child at last. “Good heavens! listen to the voice of an innocent child,” said the father, and one whispered to the other what the child had said. “But he has nothing on at all,” cried at last the whole people. That made a deep impression upon the emperor, for it seemed to him that they were right; but he thought to himself, “Now I must bear up to the end.” And the chamberlains walked with still greater dignity, as if they carried the train which did not exist.
- konad 16y ago
  point missed

linuxaddict 16y ago

GNU hackers fail again.

sandGorgon 16y ago

Question - if they had been using a DVCS which checksums every commit, would they have been safer ? Presumably, you could build a tool that stores a list of checksums and if you had to do a restore, that tool could validate the backup. Or is it that, with a little effort, DVCS' can be hacked/corrupted fundamentally ?

JoachimSchipper 16y ago
I don't think DVCS would have helped in any way. It's not the repository server itself that was compromised, AFAIK, but "only" the web interface. This allows password recovery and thus allows people to e.g. make malicious commits from a trusted account, but that would be no less true if there were a DVCS under it all.
- sandGorgon 16y ago
  agreed - but my question still stands: Take SVN for instance - if I managed to hack into a SVN server, can I change the source code (under SVN management) in a way that would be undetectable ? Repeat the same question for git, mercurial and CVS.
  - JoachimSchipper 16y ago
    For CVS and svn: you can hide it until someone actually looks at the relevant code. For git: not really (as long as you assume you change only the data - clever attackers could serve one repository to the developers and another to .mil/the rest of the world/...) I don't know about mercurial, but I'd wager it's the same as git in this respect. In either case, an attacker could add arbitrary commits, and the system administrator would roll back to the last known-good backup as soon as the compromise was detected. I don't think DVCSes help much here.
    - sandGorgon 16y ago
      For CVS and svn: you can hide it until someone actually looks at the relevant code. For git: not really I don't think DVCSes help much here. Those statements sound contradictory. Let me try to understand - the distributed aspects of DVCSes would not have helped here. But the incidental fact of checksums in DVCSes (maybe necessitated by the nature of "distributed") does help. Right ?