6 ms·

Can someone please enlighten my uninformed self why NixOS needs to store this mindboggling amount of data and why it would be damaging to the research community

Can someone please enlighten my uninformed self why NixOS needs to store this mindboggling amount of data and why it would be damaging to the research community specifically if it were to be lost? 9000$ a month is a LOT.

asdfaoeu 3y ago
In comparison it seems Debian is using 1/100th of it https://www.debian.org/mirror/size https://www.debian.org/mirror/size .
- znpy 3y ago
  I guess that reflects the fact that debian is a distro from another time, when you had to be mindful of the resources you consumed. But i too wonder if they really have to keep everything around.
  - kaba0 3y ago
    One has hard dependencies with Nix, so a modification down the line will cause a cascading recompilation. As mentioned, you doesn’t have to, but it has great value to be able to checkout any version.
    - znpy 3y ago
      sure, but at 425TB i'm guessing they're storing either everything in full or everything in full and uncompressed. and compression can do a lot for text files like sources etc. for example (last time i did that) the linux kernel source code was about 1.2GB uncompressed, but ~60-70MB in tar+gz format. surely there must be a way to reduce that size.
      - aidenn0 3y ago
        It's stored compressed with xzip
  - c0l0 3y ago
    > I guess that reflects the fact that debian is a distro from another time, when you had to be mindful of the resources you consumed. I am inclined to argue that the fact NixOS is having this problem and discussion goes to show that you *still* have to be mindful of the resources you consume :) While public cloud resources may seem infinite to some, at least the depth of the pockets of those shouldering the resulting costs should not.
microtonal 3y ago
Nix works quite differently compared to other distributions. If you have eg. the hello package in Debian it specifies that it is dependent on the glibc C library with some version range. You can install another version of glibc and the package manager will consider hello's glibc dependency satisfied as long as it is within the version range. In Nix on the other hand, a particular hello package/binary will be dependent on the glibc that it is built with. That doesn't mean just the same glibc version, even if you'd change something like glibc's build flags it'd be considered a different version [1]. One of the benefits is reproducibility. If you build hello from the same nixpkgs revision as I do (barring any impurities), you'll have exactly the same hello with the same transitive dependencies. Now, with hello this may not matter as much, because it's a simple program and glibc is fairly stable. But now imagine that you are doing some machine learning project and you want to provide a reproducible machine learning experiment. Building on a typical packaging ecosystem (eg. pip) is like building on quicksand. There are all kinds of subtle behaviors that can change and influence your experiment (eg. you machine learning library rewrites some compute kernels, and the small numerical differences pile up after a large number of layers). Though usually it ends at: my system has Python 3.11 and this experiment requires an old PyTorch version that was only built for Python 3.5 and earlier. The Nix approach gives you the benefit of pinning almost the whole world - your experiment and all its transitive dependencies. Suppose you have a package repository like nixpkgs or Debian that contains tens of thousands of packages that are continuously updated. In Debian, if glibc is updated, it gets rebuilt and you are done. glibc has a stable ABI, so packages that were built against the previous glibc version don't need to be rebuilt. In nixpkgs if glibc is updated, every package that uses glibc needs to be rebuilt due to Nix' use of exact dependencies. And this is true for almost all non-leaf packages --- a package update requires all packages that are (transitively) dependent on that package to be rebuilt. And you need to store all these builds, so that users can download them, otherwise their machines need to rebuild the packages on every update. This is how you end up with a gigantic binary cache. [1] This is a bit different now that Nix supports content-addressing for non-fixed output derivations.
- tehlike 3y ago
  What's the compressibility of this? As a whole it looks highly compressible but with something like s3 and everything being its own file, it may not be getting utilized.

asdfaoeu 3y ago

In comparison it seems Debian is using 1/100th of it https://www.debian.org/mirror/size https://www.debian.org/mirror/size .

znpy 3y ago
I guess that reflects the fact that debian is a distro from another time, when you had to be mindful of the resources you consumed. But i too wonder if they really have to keep everything around.
- kaba0 3y ago
  One has hard dependencies with Nix, so a modification down the line will cause a cascading recompilation. As mentioned, you doesn’t have to, but it has great value to be able to checkout any version.
  - znpy 3y ago
    sure, but at 425TB i'm guessing they're storing either everything in full or everything in full and uncompressed. and compression can do a lot for text files like sources etc. for example (last time i did that) the linux kernel source code was about 1.2GB uncompressed, but ~60-70MB in tar+gz format. surely there must be a way to reduce that size.
    - aidenn0 3y ago
      It's stored compressed with xzip
- c0l0 3y ago
  > I guess that reflects the fact that debian is a distro from another time, when you had to be mindful of the resources you consumed. I am inclined to argue that the fact NixOS is having this problem and discussion goes to show that you *still* have to be mindful of the resources you consume :) While public cloud resources may seem infinite to some, at least the depth of the pockets of those shouldering the resulting costs should not.

microtonal 3y ago

Nix works quite differently compared to other distributions. If you have eg. the hello package in Debian it specifies that it is dependent on the glibc C library with some version range. You can install another version of glibc and the package manager will consider hello's glibc dependency satisfied as long as it is within the version range. In Nix on the other hand, a particular hello package/binary will be dependent on the glibc that it is built with. That doesn't mean just the same glibc version, even if you'd change something like glibc's build flags it'd be considered a different version [1]. One of the benefits is reproducibility. If you build hello from the same nixpkgs revision as I do (barring any impurities), you'll have exactly the same hello with the same transitive dependencies. Now, with hello this may not matter as much, because it's a simple program and glibc is fairly stable. But now imagine that you are doing some machine learning project and you want to provide a reproducible machine learning experiment. Building on a typical packaging ecosystem (eg. pip) is like building on quicksand. There are all kinds of subtle behaviors that can change and influence your experiment (eg. you machine learning library rewrites some compute kernels, and the small numerical differences pile up after a large number of layers). Though usually it ends at: my system has Python 3.11 and this experiment requires an old PyTorch version that was only built for Python 3.5 and earlier. The Nix approach gives you the benefit of pinning almost the whole world - your experiment and all its transitive dependencies. Suppose you have a package repository like nixpkgs or Debian that contains tens of thousands of packages that are continuously updated. In Debian, if glibc is updated, it gets rebuilt and you are done. glibc has a stable ABI, so packages that were built against the previous glibc version don't need to be rebuilt. In nixpkgs if glibc is updated, every package that uses glibc needs to be rebuilt due to Nix' use of exact dependencies. And this is true for almost all non-leaf packages --- a package update requires all packages that are (transitively) dependent on that package to be rebuilt. And you need to store all these builds, so that users can download them, otherwise their machines need to rebuild the packages on every update. This is how you end up with a gigantic binary cache. [1] This is a bit different now that Nix supports content-addressing for non-fixed output derivations.

tehlike 3y ago
What's the compressibility of this? As a whole it looks highly compressible but with something like s3 and everything being its own file, it may not be getting utilized.