6 ms·

Security Incident on FreeBSD Infrastructure

126 points by dous 14y ago

darkf 14y ago
FreeBSD reports are always extremely professional, I love it.
0x0 14y ago
Interesting choice that some machines will not be reinstalled, only "thoroughly audited".
- cperciva 14y ago
  There are some systems (generally speaking, ones which were installed in the past few weeks) for which we know exactly what files should be installed and what their SHA256 hashes are. Thoroughly audited means "every single bit is correct".
  - 0x0 14y ago
    Thanks for clarifying. I assume if they had root though, they could theoretically install a rootkit in the MBR. Did you SHA256 verify the MBR too? :)
    - cperciva 14y ago
      I didn't (I'm not involved in cluster management) but I suspect that it was done by someone.
  - deleted 14y ago
    [deleted]
- batgaijin 14y ago
  I'm surprised they weren't more transparent about that reasoning... maybe those machines were running everything within jails?
- 3amOpsGuy 14y ago
  They're probably one of the few projects with such a track record that we can take them at their word. Thoroughly audited will mean exactly that. They're most definitely not amateurs, see the back catalogue for examples of 'how to do it right'
DrCatbox 14y ago
They use SVN still?
- cperciva 14y ago
  FreeBSD only finished switching from CVS to SVN for the core repositories a few months ago. (And there's still some legacy systems which rely on CVS.)
- pyrotechnick 14y ago
  http://www.wikivs.com/wiki/Git_vs_Subversion#Licensing http://www.wikivs.com/wiki/Git_vs_Subversion#Licensing
- wazoox 14y ago
  What's wrong with svn?
  - zdw 14y ago
    Nothing, if you don't mind using a centralized development model. The BSD's tend to be ultra-conservative in regard to version control systems - for example OpenBSD is still on CVS and developed their own version: http://www.opencvs.org http://www.opencvs.org
    - jimktrains2 14y ago
      I also find the ability to branch and stash easily two super-useful features. Also, I like not having to go to the server to do diffs, merge, or pull files from other branches. bisect is nice, as well. I use git-svn at work, so I have everything but easy (svn) branching and I find it super useful.
  - harshreality 14y ago
    Lacking cryptographic hashes used by most DVCS software, SVN repositories need some external mechanism to be pretty sure the source repository hasn't been compromised. For instance, one might have snapshots of the SVN tree taken every week. It's much easier with git or hg, even in the single-master-repo model, even if all developers are using the CVS gateway so they don't have local copies of commit history. All such a project would need in order to protect against compromise (ignoring physical redundancy) is a backup git or hg repo, kept offline except for periodic pulls. As long as there are no warnings about upstream rebases, and you trust all visible commits, then everything's fine[1]. With SVN, as long as there's no built-in cryptographically secure way to connect one snapshot to the next when backing up SVN repos, determining the existence of a backdoor requires comparing entire backup SVN repos against the live main repo. Even if sufficient backups exist, the process is slow and requires taking the main SVN repo offline if there's any chance of a root compromise. [1] Caveat 1: in any RCS, there could be backdoors masquerading as legitimate-looking commits if the attacker had commit access. Caveat 2: The security is subject to the cryptographic security of SHA-1. In this FreeBSD case, if they're certain the svn repo wasn't compromised, all they have to do is validate the checkout on the two compromised machines against the svn repo.
Zenst 14y ago
This is how you tell people about a security breach. Inform them soon as you know with what you know and assume the worst with your appraoch to restoring things. Much respect and defineing the word professional for many.
lhm 14y ago
I'm a bit suprised that the affected machines were powered off instead of just disconnected. Would that not make an audit more complicated?
- dietrichepp 14y ago
  On the contrary -- you have to at least admit that if you disconnect a compromised machine, the attacker could have installed a script that detects that the machine is disconnected and erases evidence. If you power off, you can always mount the hard drive read-only and do a forensic analysis.
  - fhars 14y ago
    On the contrary - you have to at least admit that if you power off a compromised machine, the attacker could have installed all his code in RAM only so that powering off erases evidence. If you disconnect, you can still try to examine the current memory content and do a forensic analysis. Of course what you really want is a memory dump via a trusted channel while the CPU is halted (hardware hypervisor or something like that) and then immediately power down. This is usually not supported on COTS hardware, so you have to choose the strategy that will erase the least evidence (power off, disconnect, suspend to disk, VM snapshot, whatever) depending on what you suspect the attack to be.
- hendi_ 14y ago
  Yes, it's more complicated than just SSH'ing into the server. But on a compromised machine you can't trust anybody, not even the kernel. Assuming the worst, the attacker could have gained root privileges and modified the kernel or the base tools like ls and grep. You also can't trust the log files if they're not stored off-site. The modified kernel or ls could hide the attacker's traces from you. Thus, the only possibility to really make sure nothing is hidden from you is to (power off the machine and) attach its hard disks to a trusted computer where they're mounted and investigated.
  - rdl 14y ago
    Live forensics are preferred to just reading disk; lots of problems with that and solutions to those problems on various types of machine.
lifeguard 14y ago
Take note: "We unfortunately cannot guarantee the integrity of any packages available for installation between 19th September 2012 and 11th November 2012, or of any ports compiled from trees obtained via any means other than through svn.freebsd.org or one of its mirrors. Although we have no evidence to suggest any tampering took place and believe such interference is unlikely, we have to recommend you consider reinstalling any machine from scratch, using trusted sources."
bulibuta 14y ago
Scary stuff. Please use passwords for your keys and allow key access only to a small set of known IP addresses. Also do share other security techniques you're using besides the ones above.
- danielpal 14y ago
  Passwords are useful, but impossible to enforce. If anyone decides not to use the passwords for their key you are in the same spot. You need better tools, something you can enforce server side. We use two-factor: (note that I founded Authy) http://blog.authy.com/two-factor-ssh-in-thirty-seconds http://blog.authy.com/two-factor-ssh-in-thirty-seconds
meaty 14y ago
I have a couple of FreeBSD machines which pulled binary packages between those dates. I'm not overly worried. The packages have been removed and installed again from ports after a fresh portsnap dump and the systems have been verified with "freebsd-update IDS" against known good signatures. Any modified files were manually checked. I use MAC on each machine and pf up front on firewalls so I know what is going in and out as well. The fact that these mechanisms are available is the reason I use such a system. Also, if you consider any problems like this happening to a closed source vendor, you may never know it's happened. And don't tell me they don't do it as I've worked for a couple of companies that felt that burying security fuck ups was acceptable practice. It's why I don't work for them any more.
- zobzu 14y ago
  Actually, even with opensource you may never know it happened. Does not matter where it comes from: - They have to find it out first. - Then they've to be willing to disclose the incident - Even if, you still trust the source of the packages, the developers, etc. There's a zillion bugs that look like "just an error" which can also be "just a backdoor". For these reasons, running mac, checking modified files, etc is ALWAYS good practice (that you seem to follow, don't get me wrong - but that's pretty rare)
  - meaty 14y ago
    I only follow them after I got owned in '97 on the end of a dialup for running an open telnet server (on FreeBSD) with a crappy root password :)
    - olefoo 14y ago
      Nobody takes security seriously until they've learned to distrust a computer they know intimately. It's a fact of life.
chmike 14y ago
Excuse the naive question, but how does one detect intrusion when using bi-key authentication ?
ladzoppelin 14y ago
How does one know the offsite repository's are clean if svnsync runs at set intervals? What if part of the attack was to make it look like happened at much late date/time after the malicious code was mirrored and backed up to the offsite repositories?
niels_olson 14y ago
End-user question: any word on how this affects my pc-bsd laptop recenttly updated to 9.1 RC-2?

126 points by dous 14y ago

darkf 14y ago

FreeBSD reports are always extremely professional, I love it.

0x0 14y ago

Interesting choice that some machines will not be reinstalled, only "thoroughly audited".

cperciva 14y ago
There are some systems (generally speaking, ones which were installed in the past few weeks) for which we know exactly what files should be installed and what their SHA256 hashes are. Thoroughly audited means "every single bit is correct".
- 0x0 14y ago
  Thanks for clarifying. I assume if they had root though, they could theoretically install a rootkit in the MBR. Did you SHA256 verify the MBR too? :)
  - cperciva 14y ago
    I didn't (I'm not involved in cluster management) but I suspect that it was done by someone.
- deleted 14y ago
  [deleted]
batgaijin 14y ago
I'm surprised they weren't more transparent about that reasoning... maybe those machines were running everything within jails?
3amOpsGuy 14y ago
They're probably one of the few projects with such a track record that we can take them at their word. Thoroughly audited will mean exactly that. They're most definitely not amateurs, see the back catalogue for examples of 'how to do it right'

DrCatbox 14y ago

They use SVN still?

cperciva 14y ago
FreeBSD only finished switching from CVS to SVN for the core repositories a few months ago. (And there's still some legacy systems which rely on CVS.)
pyrotechnick 14y ago
http://www.wikivs.com/wiki/Git_vs_Subversion#Licensing http://www.wikivs.com/wiki/Git_vs_Subversion#Licensing
wazoox 14y ago
What's wrong with svn?
- zdw 14y ago
  Nothing, if you don't mind using a centralized development model. The BSD's tend to be ultra-conservative in regard to version control systems - for example OpenBSD is still on CVS and developed their own version: http://www.opencvs.org http://www.opencvs.org
  - jimktrains2 14y ago
    I also find the ability to branch and stash easily two super-useful features. Also, I like not having to go to the server to do diffs, merge, or pull files from other branches. bisect is nice, as well. I use git-svn at work, so I have everything but easy (svn) branching and I find it super useful.
- harshreality 14y ago
  Lacking cryptographic hashes used by most DVCS software, SVN repositories need some external mechanism to be pretty sure the source repository hasn't been compromised. For instance, one might have snapshots of the SVN tree taken every week. It's much easier with git or hg, even in the single-master-repo model, even if all developers are using the CVS gateway so they don't have local copies of commit history. All such a project would need in order to protect against compromise (ignoring physical redundancy) is a backup git or hg repo, kept offline except for periodic pulls. As long as there are no warnings about upstream rebases, and you trust all visible commits, then everything's fine[1]. With SVN, as long as there's no built-in cryptographically secure way to connect one snapshot to the next when backing up SVN repos, determining the existence of a backdoor requires comparing entire backup SVN repos against the live main repo. Even if sufficient backups exist, the process is slow and requires taking the main SVN repo offline if there's any chance of a root compromise. [1] Caveat 1: in any RCS, there could be backdoors masquerading as legitimate-looking commits if the attacker had commit access. Caveat 2: The security is subject to the cryptographic security of SHA-1. In this FreeBSD case, if they're certain the svn repo wasn't compromised, all they have to do is validate the checkout on the two compromised machines against the svn repo.

Zenst 14y ago

This is how you tell people about a security breach. Inform them soon as you know with what you know and assume the worst with your appraoch to restoring things. Much respect and defineing the word professional for many.

lhm 14y ago

I'm a bit suprised that the affected machines were powered off instead of just disconnected. Would that not make an audit more complicated?

dietrichepp 14y ago
On the contrary -- you have to at least admit that if you disconnect a compromised machine, the attacker could have installed a script that detects that the machine is disconnected and erases evidence. If you power off, you can always mount the hard drive read-only and do a forensic analysis.
- fhars 14y ago
  On the contrary - you have to at least admit that if you power off a compromised machine, the attacker could have installed all his code in RAM only so that powering off erases evidence. If you disconnect, you can still try to examine the current memory content and do a forensic analysis. Of course what you really want is a memory dump via a trusted channel while the CPU is halted (hardware hypervisor or something like that) and then immediately power down. This is usually not supported on COTS hardware, so you have to choose the strategy that will erase the least evidence (power off, disconnect, suspend to disk, VM snapshot, whatever) depending on what you suspect the attack to be.
hendi_ 14y ago
Yes, it's more complicated than just SSH'ing into the server. But on a compromised machine you can't trust anybody, not even the kernel. Assuming the worst, the attacker could have gained root privileges and modified the kernel or the base tools like ls and grep. You also can't trust the log files if they're not stored off-site. The modified kernel or ls could hide the attacker's traces from you. Thus, the only possibility to really make sure nothing is hidden from you is to (power off the machine and) attach its hard disks to a trusted computer where they're mounted and investigated.
- rdl 14y ago
  Live forensics are preferred to just reading disk; lots of problems with that and solutions to those problems on various types of machine.

lifeguard 14y ago

Take note: "We unfortunately cannot guarantee the integrity of any packages available for installation between 19th September 2012 and 11th November 2012, or of any ports compiled from trees obtained via any means other than through svn.freebsd.org or one of its mirrors. Although we have no evidence to suggest any tampering took place and believe such interference is unlikely, we have to recommend you consider reinstalling any machine from scratch, using trusted sources."

bulibuta 14y ago

Scary stuff. Please use passwords for your keys and allow key access only to a small set of known IP addresses. Also do share other security techniques you're using besides the ones above.

danielpal 14y ago
Passwords are useful, but impossible to enforce. If anyone decides not to use the passwords for their key you are in the same spot. You need better tools, something you can enforce server side. We use two-factor: (note that I founded Authy) http://blog.authy.com/two-factor-ssh-in-thirty-seconds http://blog.authy.com/two-factor-ssh-in-thirty-seconds

meaty 14y ago

I have a couple of FreeBSD machines which pulled binary packages between those dates. I'm not overly worried. The packages have been removed and installed again from ports after a fresh portsnap dump and the systems have been verified with "freebsd-update IDS" against known good signatures. Any modified files were manually checked. I use MAC on each machine and pf up front on firewalls so I know what is going in and out as well. The fact that these mechanisms are available is the reason I use such a system. Also, if you consider any problems like this happening to a closed source vendor, you may never know it's happened. And don't tell me they don't do it as I've worked for a couple of companies that felt that burying security fuck ups was acceptable practice. It's why I don't work for them any more.

zobzu 14y ago
Actually, even with opensource you may never know it happened. Does not matter where it comes from: - They have to find it out first. - Then they've to be willing to disclose the incident - Even if, you still trust the source of the packages, the developers, etc. There's a zillion bugs that look like "just an error" which can also be "just a backdoor". For these reasons, running mac, checking modified files, etc is ALWAYS good practice (that you seem to follow, don't get me wrong - but that's pretty rare)
- meaty 14y ago
  I only follow them after I got owned in '97 on the end of a dialup for running an open telnet server (on FreeBSD) with a crappy root password :)
  - olefoo 14y ago
    Nobody takes security seriously until they've learned to distrust a computer they know intimately. It's a fact of life.

chmike 14y ago

Excuse the naive question, but how does one detect intrusion when using bi-key authentication ?

ladzoppelin 14y ago

How does one know the offsite repository's are clean if svnsync runs at set intervals? What if part of the attack was to make it look like happened at much late date/time after the malicious code was mirrored and backed up to the offsite repositories?

niels_olson 14y ago

End-user question: any word on how this affects my pc-bsd laptop recenttly updated to 9.1 RC-2?