5 ms·

NixOS stores its build artifacts in S3, but it's now too expensive, so they're exploring other options, including Storj, Cloudflare R2, and self-hosting.

anonymousiam 3y ago
Amazon literally hasn't reduced prices for S3 since 2016: https://www.petercai.com/storage-prices-have-sort-of-stopped-dropping/ https://www.petercai.com/storage-prices-have-sort-of-stopped... Hard drive cost $/B continue to fall: https://www.backblaze.com/blog/hard-drive-cost-per-gigabyte/ https://www.backblaze.com/blog/hard-drive-cost-per-gigabyte/ Self-hosting is looking more and more attractive.
- orf 3y ago
  Intelligent tiering was released after 2016 and is fantastic at reducing prices?
- electroly 3y ago
  AWS has repeatedly released new, lower-cost tiers. Indeed, Nix is already using the Infrequent Access tier. That tier already existed when that blog post was made, but it hurts the author's point so they didn't mention it. The cheapest tier, Glacier Deep Archive, is $0.00099/GB. There are still lots of reasons to consider alternative hosting providers (the biggest one: egress) but that blog post--frequently reposted and annoyingly misleading--is not a good summary of the situation here.
  - nine_k 3y ago
    Glacier Deep Archive can take hours to access abucket, likely because the right tape should be shipped from a warehouse.
    - electroly 3y ago
      We have... a lot of terabytes in GDA. I'm well aware :) I'm trying to show the wide range of tiering options available to fit various needs. OP's article pretends that only Standard exists. You'd be a fool to raw dog 425TB of cold data on the Standard tier, and indeed Nix is not doing that.
  - AceJohnny2 3y ago
    > The cheapest tier, Glacier Deep Archive, is $0.00099/GB. Bad example. "Latency", if it can even be called that, is 12-48 hours. It's not cloud storage, it's archiving.
  - anonymousiam 3y ago
    Yeah, $0.00099/GB (roughly $1/TB per month) sounds cheap, but retrieving that data will cost you almost two orders of magnitude more than the storage, so not really such a good deal. Also, obviously the latency of the "glacial" tier is so awful that it is useless for anything other than archival of long-term data without any need for speedy restoration.
lrvick 3y ago
They also store them without any distributed software supply chain integrity. One compromised person and the whole system fails. I wish they were less concerned with where they artifacts are stored and more about how nixpkg can prove they are authentic and who authored them. Nix has so many innovations over other other distributions but the massive step backwards in supply chain integrity still makes it too dangerous to use for any system you would not trust strangers with ssh access to.
- predictabl3 3y ago
  I don't follow. The NARs are signed by a key that (presumably?) only Hydra has access to. (The signature is then recorded in the narinfo file). The integrity is the nixpkgs history, and the signature verification that happens when you pull the narinfo+NAR from a cache that hydra pushed to.
  - lrvick 3y ago
    How can Hydra know the nixpkgs repo was not tampered with? Maintainers impersonated? How can anyone know the Hydra signing key was not tampered with? These are problems other linux distros have solved for decades by just requiring maintainers press a blinking yubikey or similar to sign their contributions.
- anderskaseorg 3y ago
  NixOS binary caches are secured by public-key signatures, as well as HTTPS. Hashes for all source artifacts are checked by builders, and this is strongly enforced by a build sandbox that disables network access for derivations without a known output hash. NixOS is leading the effort toward reproducible builds (https://reproducible.nixos.org/ https://reproducible.nixos.org/), with many sources of nonreproducibility automatically ruled out by design. Supply chain security is of course a massively multifaceted technical and social problem, but I’m curious what other distributions you think are doing it better in practice?
  - lrvick 3y ago
    Nix packages are blindly and automatically signed by a single private key last I reviewed it. Bribe or compromise one person and the entire model fails. There is good reason to do this too given that many major blockchain projects use it for builds. Hundreds of millions to be made by one that impersonates the right maintainer before the right software release. Also no signed commits or signed authorship means someone with Github access can just fake history and inject whatever they want after code reviews are completed, which will then be blindly and automatically signed. Some of the people with write access to the nixpgs repo even have SMS recovery enabled on their github recovery email accounts. One sim swap to compromise all nix users. I will not call them out, but go try to do a email password reset on recent committers for yourself. A malicious github employee could also of course do whatever they want to an unsigned repo. Or a well placed BGP attack. Lots of options. It is hard to prevent such things, but author commit signing would mitigate the risk and can be enforced. I made my case for this to the nix team but in the end it was concluded people would stop maintaining packages if they had to do the bare minimum like commit signing or hardware 2FA. https://github.com/NixOS/rfcs/pull/34 https://github.com/NixOS/rfcs/pull/34 All this is fine, but it means effectively a decision was made for NixOS to be a hobby distro not suitable for any targeted applications or individuals. It really sucks, because I love everything else about nix design. Instead I am forced to bootstrap high security applications using arch and debian toolchains which are worse than nix in every way but supply chain integrity given that all authors directly sign package sources with their personal well verified keys. They have a ton of other security and even their own supply chain problems but they at least can survive phishing, a malicious mirror, or a sim swap. It is a low bar nix sadly does not meet.

anonymousiam 3y ago

Amazon literally hasn't reduced prices for S3 since 2016: https://www.petercai.com/storage-prices-have-sort-of-stopped-dropping/ https://www.petercai.com/storage-prices-have-sort-of-stopped... Hard drive cost $/B continue to fall: https://www.backblaze.com/blog/hard-drive-cost-per-gigabyte/ https://www.backblaze.com/blog/hard-drive-cost-per-gigabyte/ Self-hosting is looking more and more attractive.

orf 3y ago
Intelligent tiering was released after 2016 and is fantastic at reducing prices?
electroly 3y ago
AWS has repeatedly released new, lower-cost tiers. Indeed, Nix is already using the Infrequent Access tier. That tier already existed when that blog post was made, but it hurts the author's point so they didn't mention it. The cheapest tier, Glacier Deep Archive, is $0.00099/GB. There are still lots of reasons to consider alternative hosting providers (the biggest one: egress) but that blog post--frequently reposted and annoyingly misleading--is not a good summary of the situation here.
- nine_k 3y ago
  Glacier Deep Archive can take hours to access abucket, likely because the right tape should be shipped from a warehouse.
  - electroly 3y ago
    We have... a lot of terabytes in GDA. I'm well aware :) I'm trying to show the wide range of tiering options available to fit various needs. OP's article pretends that only Standard exists. You'd be a fool to raw dog 425TB of cold data on the Standard tier, and indeed Nix is not doing that.
- AceJohnny2 3y ago
  > The cheapest tier, Glacier Deep Archive, is $0.00099/GB. Bad example. "Latency", if it can even be called that, is 12-48 hours. It's not cloud storage, it's archiving.
- anonymousiam 3y ago
  Yeah, $0.00099/GB (roughly $1/TB per month) sounds cheap, but retrieving that data will cost you almost two orders of magnitude more than the storage, so not really such a good deal. Also, obviously the latency of the "glacial" tier is so awful that it is useless for anything other than archival of long-term data without any need for speedy restoration.

lrvick 3y ago

They also store them without any distributed software supply chain integrity. One compromised person and the whole system fails. I wish they were less concerned with where they artifacts are stored and more about how nixpkg can prove they are authentic and who authored them. Nix has so many innovations over other other distributions but the massive step backwards in supply chain integrity still makes it too dangerous to use for any system you would not trust strangers with ssh access to.

predictabl3 3y ago
I don't follow. The NARs are signed by a key that (presumably?) only Hydra has access to. (The signature is then recorded in the narinfo file). The integrity is the nixpkgs history, and the signature verification that happens when you pull the narinfo+NAR from a cache that hydra pushed to.
- lrvick 3y ago
  How can Hydra know the nixpkgs repo was not tampered with? Maintainers impersonated? How can anyone know the Hydra signing key was not tampered with? These are problems other linux distros have solved for decades by just requiring maintainers press a blinking yubikey or similar to sign their contributions.
anderskaseorg 3y ago
NixOS binary caches are secured by public-key signatures, as well as HTTPS. Hashes for all source artifacts are checked by builders, and this is strongly enforced by a build sandbox that disables network access for derivations without a known output hash. NixOS is leading the effort toward reproducible builds (https://reproducible.nixos.org/ https://reproducible.nixos.org/), with many sources of nonreproducibility automatically ruled out by design. Supply chain security is of course a massively multifaceted technical and social problem, but I’m curious what other distributions you think are doing it better in practice?
- lrvick 3y ago
  Nix packages are blindly and automatically signed by a single private key last I reviewed it. Bribe or compromise one person and the entire model fails. There is good reason to do this too given that many major blockchain projects use it for builds. Hundreds of millions to be made by one that impersonates the right maintainer before the right software release. Also no signed commits or signed authorship means someone with Github access can just fake history and inject whatever they want after code reviews are completed, which will then be blindly and automatically signed. Some of the people with write access to the nixpgs repo even have SMS recovery enabled on their github recovery email accounts. One sim swap to compromise all nix users. I will not call them out, but go try to do a email password reset on recent committers for yourself. A malicious github employee could also of course do whatever they want to an unsigned repo. Or a well placed BGP attack. Lots of options. It is hard to prevent such things, but author commit signing would mitigate the risk and can be enforced. I made my case for this to the nix team but in the end it was concluded people would stop maintaining packages if they had to do the bare minimum like commit signing or hardware 2FA. https://github.com/NixOS/rfcs/pull/34 https://github.com/NixOS/rfcs/pull/34 All this is fine, but it means effectively a decision was made for NixOS to be a hobby distro not suitable for any targeted applications or individuals. It really sucks, because I love everything else about nix design. Instead I am forced to bootstrap high security applications using arch and debian toolchains which are worse than nix in every way but supply chain integrity given that all authors directly sign package sources with their personal well verified keys. They have a ton of other security and even their own supply chain problems but they at least can survive phishing, a malicious mirror, or a sim swap. It is a low bar nix sadly does not meet.