4 ms·

The nix cache compresses store artifacts in NAR archives, which I think defeats de-duplication. Its possible to serve a nix cache directly out of a nix store,

The nix cache compresses store artifacts in NAR archives, which I think defeats de-duplication. Its possible to serve a nix cache directly out of a nix store, creating the NAR archives on-the-fly. But that would require a) a big filesystem and b) a lot more compute. Which are not insurmountable challenges, but they are challenges.

mikepurvis 3y ago
My org does the live-store-on-zfs model with a thin proxy cache on the front end to take the load off the CPU constantly regenerating the top-5% most popular NARs. It seems to work well for us, but I am curious how that would look at 425TB scale— would filesystem deduplication pay for the losses in compression, and if so, by how much? How does that change if you start sharding the store across multiple ZFS hosts?
- gigatexal 3y ago
  Dedup can be used with compression I thought.
  - hamandcheese 3y ago
    I think you are confusing deduplication and compression provided by the filesystem vs. deduplication and compression provided by nix. If you were just storing a static set of pregenerated NAR archives, you will not see any benefit from filesystem-provided compression or deduplication. If you host a live nix store (i.e. uncompressed files under /nix/store), then you could benefit from filesystem-provided compression and deduplication. Also, nix itself can replace duplicates with hard links. But the downside is then that you have to generate the NAR archives on the fly when a client requests a derivation. That might be worth it, especially since they get great hit rates from Fastly. But on the other hand, that means a lot more moving parts to maintain and monitor vs. simple blob storage.
    - mikepurvis 3y ago
      Definitely. It's not hard to imagine that even if they end up rolling their own for this, it might make sense for it to be a bunch of off-the-shelf Synology NASes stuffed with 22TB drives in donated rackspace around the world, running MinIO. If the "hot" data is all still in S3/CDN/cache, then you're really just keeping the rest of it around on in-case basis, and for that, the simpler the better.
      - gigatexal 3y ago
        Yeah that's what I was thinking. Say 3 nodes in 3 different data centers all sync'd running enough drives of spinning rust to meet their needs and then some plus the overhead that ZFS requires (i.e the rule of thumb of not using more than 80%? of the usable storage to prevent fragmentation I think) and then exposing that via an S3 compatible API via minio + a CDN of their choice.
- vamega 3y ago
  Is this done with a off the shelf tool that I could run? Or did you build this in house?
  - hamandcheese 3y ago
    nix-serve is off the shelf: https://github.com/edolstra/nix-serve https://github.com/edolstra/nix-serve Then put whatever cache you want in front.
    - mikepurvis 3y ago
      We started there, briefly tried nix-serve-ng [1], and then landed on Harmonia [2], which has worked well. [1]: Ran into https://github.com/aristanetworks/nix-serve-ng/issues/19 https://github.com/aristanetworks/nix-serve-ng/issues/19 with it [2]: https://github.com/nix-community/harmonia https://github.com/nix-community/harmonia

mikepurvis 3y ago

My org does the live-store-on-zfs model with a thin proxy cache on the front end to take the load off the CPU constantly regenerating the top-5% most popular NARs. It seems to work well for us, but I am curious how that would look at 425TB scale— would filesystem deduplication pay for the losses in compression, and if so, by how much? How does that change if you start sharding the store across multiple ZFS hosts?

gigatexal 3y ago
Dedup can be used with compression I thought.
- hamandcheese 3y ago
  I think you are confusing deduplication and compression provided by the filesystem vs. deduplication and compression provided by nix. If you were just storing a static set of pregenerated NAR archives, you will not see any benefit from filesystem-provided compression or deduplication. If you host a live nix store (i.e. uncompressed files under /nix/store), then you could benefit from filesystem-provided compression and deduplication. Also, nix itself can replace duplicates with hard links. But the downside is then that you have to generate the NAR archives on the fly when a client requests a derivation. That might be worth it, especially since they get great hit rates from Fastly. But on the other hand, that means a lot more moving parts to maintain and monitor vs. simple blob storage.
  - mikepurvis 3y ago
    Definitely. It's not hard to imagine that even if they end up rolling their own for this, it might make sense for it to be a bunch of off-the-shelf Synology NASes stuffed with 22TB drives in donated rackspace around the world, running MinIO. If the "hot" data is all still in S3/CDN/cache, then you're really just keeping the rest of it around on in-case basis, and for that, the simpler the better.
    - gigatexal 3y ago
      Yeah that's what I was thinking. Say 3 nodes in 3 different data centers all sync'd running enough drives of spinning rust to meet their needs and then some plus the overhead that ZFS requires (i.e the rule of thumb of not using more than 80%? of the usable storage to prevent fragmentation I think) and then exposing that via an S3 compatible API via minio + a CDN of their choice.
vamega 3y ago
Is this done with a off the shelf tool that I could run? Or did you build this in house?
- hamandcheese 3y ago
  nix-serve is off the shelf: https://github.com/edolstra/nix-serve https://github.com/edolstra/nix-serve Then put whatever cache you want in front.
  - mikepurvis 3y ago
    We started there, briefly tried nix-serve-ng [1], and then landed on Harmonia [2], which has worked well. [1]: Ran into https://github.com/aristanetworks/nix-serve-ng/issues/19 https://github.com/aristanetworks/nix-serve-ng/issues/19 with it [2]: https://github.com/nix-community/harmonia https://github.com/nix-community/harmonia