7 ms·

Deb.haskell.org Security Breach

114 points by corndoge 12y ago

49i4 12y ago
Curious what monitoring system they have to detect things like that.
- meowface 12y ago
  Looks like the hosting provider picked up on it. Likely an IDS such as Snort or Suricata to detect compromised hosting clients. And possibly something to measure unusual traffic volume, or traffic to suspicious regions or hosts.
  - dmix 12y ago
    Also HIDS, such as OSSEC could detect tampering of the operating system. Such as a rootkit installing a module or modifying a bin.
- mobiplayer 12y ago
  They were probably spamming someone else and choking some network uplink at their provider, which is why the provider itself told them about the possible compromise. Either that or killing other resources and raising very simple alerts (disk, CPU, RAM, Ethernet ports, etc).
glibgil 12y ago
Where's your type system now, Haskell?
- jimmcslim 12y ago
  It has been... coerced.
- nilved 12y ago
  Less type safety, more internet safety!
  - wk_end 12y ago
    Type safety is one of the ways we can get more internet safety.
- slashnull 12y ago
  Mh, this is intriguing, anybody knows what's the stack under that repo?
- Buttons840 12y ago
  Is this deb package repository built on Haskell technologies? I don't think Haskell's type system is involved here.
  - jacquesm 12y ago
    That's the whole point of the GPs comment. You can have all the security related features you want baked into your language if you distribute it in a way that can be compromised that's all for naught.
_wmd 12y ago
Such a waste.. probably ran some crappy spam/DoS drone on it, when binaries from that host are probably pulled and executed inside more or less every major financial institution on the planet.
- intortus 12y ago
  That's what they'd like you to think.
  - voltagex_ 12y ago
    Excel runs the world, right?
    - dippyskoodlez 12y ago
      Can confirm. It runs Sprint.
- carterschonwald 12y ago
  I"m pretty sure most banks are using RHEL or the like, rather than Debian. At least for production environments, something about about enterprise support contracts and the like.
- mpeg 12y ago
  I once found this compromised system that was being used as a spam bot / ddos machine. It also happened to host the firmware packages for a very large device maker, which were not signed.
- dons 12y ago
  ... wouldn't take random binary packages from random sites...
tdicola 12y ago
I've been looking at hosting some apt packages in a custom repo and am truly impressed/horrified by the layers of complexity necessary & suggested to create one. It boggles my mind why an apt host has to be more than a static list of files that are hosted on S3, raw github, etc. Running and maintaining a full blown server just to host some files seems crazy.
- eikenberry 12y ago
  You can host a dep/apt repo as a static list of files in S3. Did that myself before using reprepro to generate it, then pushed it up to S3. Nothing fancy, but it worked fine. Not sure where you got the idea is was more complicated. I've also done it using apt-ftparchive using a simple script. Very bare bones, but easy enough. Was looking at trying aptly next time I needed one. It looked interesting. https://packages.debian.org/jessie/reprepro https://packages.debian.org/jessie/reprepro https://packages.debian.org/jessie/apt-utils https://packages.debian.org/jessie/apt-utils https://packages.debian.org/jessie/aptly https://packages.debian.org/jessie/aptly [edit to include all links]
  - tdicola 12y ago
    Yeah the problem is getting to that point. Check out the docs on setting up an apt repo: https://wiki.debian.org/HowToSetupADebianRepository https://wiki.debian.org/HowToSetupADebianRepository Good luck figuring out the easy way of using reprepro and S3 from that page.
    - eikenberry 12y ago
      Point taken.
    - geofft 12y ago
      The sad part is it's even easier than that, if your goal is hosting like two packages. Stick a bunch of debs in a directory, make sure your default GPG key is reasonable, then run `apt-ftparchive packages . > Packages`, `apt-ftparchive release . > Release`, and `gpg -ab < Release > Release.gpg`. Then toss the whole thing on a web server that supports plain HTTP. Honestly I bet GitHub Pages is good enough. The syntax for sources.list is then "deb http://path/to/your/directory http://path/to/your/directory ./". Users can pick up your key by piping it into "sudo apt-key add -", or they can "sudo apt-key adv --keyserver... --recv-key..." with the usual GPG options (and check the fingerprint the same way). The problem is that most serious users will quickly want tooling to do things like keep track of which versions of which packages are in the archive, not have two versions of the same package, support multiple distros/releases, support separate "test" and "production" areas, etc. And that's where the complexity shows up. (Also why you never see "./" sources.list lines in real life.) But for "Hey, I built these three packages", it's kind of perfect.
hoodoof 12y ago
Why? Why do systems like this get hacked?
- tedunangst 12y ago
  Because they can be.
- ChuckMcM 12y ago
  any vector into a target is a good vector ...
- andrewchambers 12y ago
  Well, this distributes binaries that package managers install on peoples systems, good way to spread a virus.
- kryptiskt 12y ago
  It can be because it was targeted or it can be because it's connected to the internet. Any vulnerable server will do for a spammer or botnet herder (they use hijacked servers to control the bots).
kfreds 12y ago
The most important information is missing - did the attackers get the package signing key? The comment "the window for package compromise was very very small" is a bit ambiguous. That the security of package integrity is so fragile should greatly concern all of us. Almost all big Linux distributions rely on downloading compiled packages signed by a single key. If the build system is compromised your Linux distribution of choice will happily give you trojaned packages that you install as root. If the signing key is compromised a man-in-the-middle is required, but the difficulty to mount an attack is decreased due to the fact that most repositories don't transport over TLS. The situation can be improved in many ways. By making the build process deterministic many parties can compile the same package and compare the result, alerting the community if one of the build systems report a different checksum. Package managers like apt and yum should be extended with the ability to rely on signatures from multiple parties. The Tor Project, Debian and Fedora has started working on the first problem, but I don't know of any efforts to support multi-sig. https://blog.torproject.org/blog/deterministic-builds-part-one-cyberwar-and-global-compromise https://blog.torproject.org/blog/deterministic-builds-part-o... https://blog.torproject.org/blog/deterministic-builds-part-two-technical-details https://blog.torproject.org/blog/deterministic-builds-part-t... https://wiki.debian.org/ReproducibleBuilds https://wiki.debian.org/ReproducibleBuilds https://securityblog.redhat.com/2013/09/18/reproducible-builds-for-fedora/ https://securityblog.redhat.com/2013/09/18/reproducible-buil...
- vezzy-fnord 12y ago
  Nix already solved the first problem a while ago, though its adoption is hindered by the fact that it would essentially deprecate the entire infrastructure that actually makes a distribution a distribution, and is therefore unaligned with best interests. This is in spite of widespread claims of wanting standardization.
  - drostie 12y ago
    Nix is just simply amazing, and there IS a distribution (NixOS[1]) which ships it. The big problem is of course keeping packages current. Nix and SmartOS[2] are both really important in my book. Nobody's talking about them as distributions but they both have intense potential to completely change not just server-administration (which is what they solve right now) but having a nice desktop experience. It would look like the Windows desktop experience, where they try to hide C: from you and just give you a set of shared-across-your-applications folders: Documents, Desktop Icons, Pictures. When every application occupies its own universe, then the OS itself behaves just the way that the "window" metaphor suggests; moreover you can start to seamlessly include VMs and get Windows applications living next to OSX ones. The only cost is hard-drive space, but hard drive space can be shared if we can package the software and share packages with the same checksums. Eventually, I hope that people will just assume that an application "comes with" its operating environment. 1. http://nixos.org/ http://nixos.org/ 2. https://smartos.org/ https://smartos.org/
sclv 12y ago
For the record, the hosted files were daily builds of HEAD. Despite the domain name, these packages did not have "official status" anywhere and were not distributed outside of that site to our knowledge. In fact there are very few users that we know of -- to the point that when the site went down, there were no reports filed or complaints made. That said, when the service gets up and running again, the key will indeed need to be replaced.

114 points by corndoge 12y ago

49i4 12y ago

Curious what monitoring system they have to detect things like that.

meowface 12y ago
Looks like the hosting provider picked up on it. Likely an IDS such as Snort or Suricata to detect compromised hosting clients. And possibly something to measure unusual traffic volume, or traffic to suspicious regions or hosts.
- dmix 12y ago
  Also HIDS, such as OSSEC could detect tampering of the operating system. Such as a rootkit installing a module or modifying a bin.
mobiplayer 12y ago
They were probably spamming someone else and choking some network uplink at their provider, which is why the provider itself told them about the possible compromise. Either that or killing other resources and raising very simple alerts (disk, CPU, RAM, Ethernet ports, etc).

glibgil 12y ago

Where's your type system now, Haskell?

jimmcslim 12y ago
It has been... coerced.
nilved 12y ago
Less type safety, more internet safety!
- wk_end 12y ago
  Type safety is one of the ways we can get more internet safety.
slashnull 12y ago
Mh, this is intriguing, anybody knows what's the stack under that repo?
Buttons840 12y ago
Is this deb package repository built on Haskell technologies? I don't think Haskell's type system is involved here.
- jacquesm 12y ago
  That's the whole point of the GPs comment. You can have all the security related features you want baked into your language if you distribute it in a way that can be compromised that's all for naught.

_wmd 12y ago

Such a waste.. probably ran some crappy spam/DoS drone on it, when binaries from that host are probably pulled and executed inside more or less every major financial institution on the planet.

intortus 12y ago
That's what they'd like you to think.
- voltagex_ 12y ago
  Excel runs the world, right?
  - dippyskoodlez 12y ago
    Can confirm. It runs Sprint.
carterschonwald 12y ago
I"m pretty sure most banks are using RHEL or the like, rather than Debian. At least for production environments, something about about enterprise support contracts and the like.
mpeg 12y ago
I once found this compromised system that was being used as a spam bot / ddos machine. It also happened to host the firmware packages for a very large device maker, which were not signed.
dons 12y ago
... wouldn't take random binary packages from random sites...

tdicola 12y ago

I've been looking at hosting some apt packages in a custom repo and am truly impressed/horrified by the layers of complexity necessary & suggested to create one. It boggles my mind why an apt host has to be more than a static list of files that are hosted on S3, raw github, etc. Running and maintaining a full blown server just to host some files seems crazy.

eikenberry 12y ago
You can host a dep/apt repo as a static list of files in S3. Did that myself before using reprepro to generate it, then pushed it up to S3. Nothing fancy, but it worked fine. Not sure where you got the idea is was more complicated. I've also done it using apt-ftparchive using a simple script. Very bare bones, but easy enough. Was looking at trying aptly next time I needed one. It looked interesting. https://packages.debian.org/jessie/reprepro https://packages.debian.org/jessie/reprepro https://packages.debian.org/jessie/apt-utils https://packages.debian.org/jessie/apt-utils https://packages.debian.org/jessie/aptly https://packages.debian.org/jessie/aptly [edit to include all links]
- tdicola 12y ago
  Yeah the problem is getting to that point. Check out the docs on setting up an apt repo: https://wiki.debian.org/HowToSetupADebianRepository https://wiki.debian.org/HowToSetupADebianRepository Good luck figuring out the easy way of using reprepro and S3 from that page.
  - eikenberry 12y ago
    Point taken.
  - geofft 12y ago
    The sad part is it's even easier than that, if your goal is hosting like two packages. Stick a bunch of debs in a directory, make sure your default GPG key is reasonable, then run `apt-ftparchive packages . > Packages`, `apt-ftparchive release . > Release`, and `gpg -ab < Release > Release.gpg`. Then toss the whole thing on a web server that supports plain HTTP. Honestly I bet GitHub Pages is good enough. The syntax for sources.list is then "deb http://path/to/your/directory http://path/to/your/directory ./". Users can pick up your key by piping it into "sudo apt-key add -", or they can "sudo apt-key adv --keyserver... --recv-key..." with the usual GPG options (and check the fingerprint the same way). The problem is that most serious users will quickly want tooling to do things like keep track of which versions of which packages are in the archive, not have two versions of the same package, support multiple distros/releases, support separate "test" and "production" areas, etc. And that's where the complexity shows up. (Also why you never see "./" sources.list lines in real life.) But for "Hey, I built these three packages", it's kind of perfect.

hoodoof 12y ago

Why? Why do systems like this get hacked?

tedunangst 12y ago
Because they can be.
ChuckMcM 12y ago
any vector into a target is a good vector ...
andrewchambers 12y ago
Well, this distributes binaries that package managers install on peoples systems, good way to spread a virus.
kryptiskt 12y ago
It can be because it was targeted or it can be because it's connected to the internet. Any vulnerable server will do for a spammer or botnet herder (they use hijacked servers to control the bots).

kfreds 12y ago

The most important information is missing - did the attackers get the package signing key? The comment "the window for package compromise was very very small" is a bit ambiguous. That the security of package integrity is so fragile should greatly concern all of us. Almost all big Linux distributions rely on downloading compiled packages signed by a single key. If the build system is compromised your Linux distribution of choice will happily give you trojaned packages that you install as root. If the signing key is compromised a man-in-the-middle is required, but the difficulty to mount an attack is decreased due to the fact that most repositories don't transport over TLS. The situation can be improved in many ways. By making the build process deterministic many parties can compile the same package and compare the result, alerting the community if one of the build systems report a different checksum. Package managers like apt and yum should be extended with the ability to rely on signatures from multiple parties. The Tor Project, Debian and Fedora has started working on the first problem, but I don't know of any efforts to support multi-sig. https://blog.torproject.org/blog/deterministic-builds-part-one-cyberwar-and-global-compromise https://blog.torproject.org/blog/deterministic-builds-part-o... https://blog.torproject.org/blog/deterministic-builds-part-two-technical-details https://blog.torproject.org/blog/deterministic-builds-part-t... https://wiki.debian.org/ReproducibleBuilds https://wiki.debian.org/ReproducibleBuilds https://securityblog.redhat.com/2013/09/18/reproducible-builds-for-fedora/ https://securityblog.redhat.com/2013/09/18/reproducible-buil...

vezzy-fnord 12y ago
Nix already solved the first problem a while ago, though its adoption is hindered by the fact that it would essentially deprecate the entire infrastructure that actually makes a distribution a distribution, and is therefore unaligned with best interests. This is in spite of widespread claims of wanting standardization.
- drostie 12y ago
  Nix is just simply amazing, and there IS a distribution (NixOS[1]) which ships it. The big problem is of course keeping packages current. Nix and SmartOS[2] are both really important in my book. Nobody's talking about them as distributions but they both have intense potential to completely change not just server-administration (which is what they solve right now) but having a nice desktop experience. It would look like the Windows desktop experience, where they try to hide C: from you and just give you a set of shared-across-your-applications folders: Documents, Desktop Icons, Pictures. When every application occupies its own universe, then the OS itself behaves just the way that the "window" metaphor suggests; moreover you can start to seamlessly include VMs and get Windows applications living next to OSX ones. The only cost is hard-drive space, but hard drive space can be shared if we can package the software and share packages with the same checksums. Eventually, I hope that people will just assume that an application "comes with" its operating environment. 1. http://nixos.org/ http://nixos.org/ 2. https://smartos.org/ https://smartos.org/

sclv 12y ago

For the record, the hosted files were daily builds of HEAD. Despite the domain name, these packages did not have "official status" anywhere and were not distributed outside of that site to our knowledge. In fact there are very few users that we know of -- to the point that when the site went down, there were no reports filed or complaints made. That said, when the service gets up and running again, the key will indeed need to be replaced.