17 ms·

The NixOS Foundation’s Call to Action: S3 Costs Require Community Support

304 points by ghuntley 3y ago

deleted 3y ago
[deleted]
LanternLight83 3y ago
I get that it's out of scope right now, but it'd be so cool to have a decentralized solution for regional caching
- candiddevmike 3y ago
  That's basically what every other distro does, seems like nix's architecture doesn't support this?
  - anon291 3y ago
    Nix supports mirroring. It's pretty easy to set up your own mirror.
    - candiddevmike 3y ago
      Why are they not considering a decentralized mirrorlist?
      - microtonal 3y ago
        The storage requirements are too large for most potential mirrors. Nix is not like any regular Linux distribution.
  - 1MachineElf 3y ago
    But AWS S3 is decentralized - all objects are replicated across a region's availability zones. /s
  - deleted 3y ago
    [deleted]
  - slabity 3y ago
    It does. Nix has an extremely powerful (and flexible) caching system. Not to mention popular caching systems like cachix that are easy to integrate into your system. Unlike most other packaging systems though, Nix packages are rebuilt when there's changes to any of their dependencies. This results in a huge amount of storage data used in their caches compared to other systems. Not to mention the compute power required to rebuild those packages. For example, let's say something like `glibc` updated with a new patch. Every single package that has `glibc` as a dependency needs to be rebuilt, along with every package that has any of those packages as a dependency. This could easily cause the rebuild of tens of thousands of packages, even if they are only dynamically linking to it.
    - pxc 3y ago
      Maybe we should be using something like Guix grafts to save not just rebuild time, but space (and thus money).
      - dlahoda 3y ago
        nix has security patch tech.
      - slabity 3y ago
        Guix's graft system is a hack specifically made to get security updates out ASAP. They should really only be seen as a last resort because it circumvents the primary benefit of using Guix as a functional build system (pure, reproducible builds). I'm not sure if Nix has a similar system. IMHO I'd prefer to see compile-time caches like ccache or sccache be better integrated into Nix while still providing the guarantees that a functional build system provides. That alone would save massive amounts of compute power when rebuilding packages, as you'd essentially only need to recompile the files that change in a package. Not only on continuous build systems like Nix's Hydra, but also when just rebuilding packages locally. This won't solve the storage cost issue, but would definitely alleviate compute costs.
        xyzzy_plugh 3y ago
- LanternLight83 3y ago
  Just to clarify, the missing component in my mind is in mirror advertisement. The need for manual compilation and review is a hindrance to casual participants, we need a config option that you just toggle (maybe a configuration struct with eg. bandwidth limits) to opt-in to IPFS or GNUnet distribution of store items present on your system.
- simonjgreen 3y ago
  Anycast BGP and Colo is the cheapest (and imo simplest) way
justahuman74 3y ago
The nix store seems on the face of it like a perfect use case for something like IPFS
- rkeene2 3y ago
  My AppFS project [0] is basically this, a content-addressable remote package-oriented system. [0] https://AppFS.net/ https://AppFS.net/
- pxc 3y ago
  There has been work ongoing to support this in Nix for a long time, actually. Maybe this will help push that effort along, or direct more resources towards it. See: https://github.com/NixOS/rfcs/pull/133 https://github.com/NixOS/rfcs/pull/133
- nlitened 3y ago
  Or torrents
- shp0ngle 3y ago
  IPFS in not magic you can just sprinkle on things, say “p2p! decentralised!” and be done. you still need either pay someone to “pin it”, which is more expensive than dumping it on s3. Pinning services (that’s a thing) are all sketchy (as the big ones are all somehow connected to NFTs) and if you do it yourself it’s actually harder than just storing it
  - mindslight 3y ago
    As a NixOS user (who doesn't even use the binaries), I would happily pin NixOS packages! The point of IPFS is to decouple the names of resources from what's serving them. It's exactly what is needed for distributed libre software projects where many are willing to donate some resources, but few are able to donate all the resources.
    - joepie91_ 3y ago
      But the problem here isn't distribution, it's durable storage. Something along the lines of Tahoe-LAFS would be far more suitable.
toomuchtodo 3y ago
Can folks who have AWS credits to burn use them to cover egress costs?
- cyberax 3y ago
  Yes, but not easily. It's possible to set the bucket to "requester pays" model, but this will require touching all the cached objects (which also will cost some money).
justahuman74 3y ago
Are any large companies using NixOS? I presume they'd be candidates for sponsorship
- solarkraft 3y ago
  repl.it comes to mind, but I think recently a few others have popped up.
- pxc 3y ago
  The most Nix-centric companies are all consultancies and startups, but off the top of my head: - Target - Shopify - Pinterest all used Nix fairly heavily at some point, and I believe they still do. (This is actually about more than NixOS— the binary caches affect all Nix users, including those on macOS as well as any Linux distro.)
- groodt 3y ago
  Shopify and Target have been known to use it.
heavyset_go 3y ago
S3's costs are insane for this use case, can't imagine using it to for a Linux distribution's repositories. edit: Oh man, there's potentially a $32k migration fee to pay for S3 egress, as well.
- bombcar 3y ago
  At least use one of those s3 competitors with sane pricing. GitHub? backblaze? Cloud flare?
  - mkl 3y ago
    > same pricing I think you meant "sane"? Otherwise there's not much point in switching.
    - bombcar 3y ago
      Yeah, that was an unfortunate typo.
      - sitkack 3y ago
        Some things can only be said as a joke or by accident or both.
- scottlamb 3y ago
  "We will cover egress and migration fees for customers migrating over 10 TB of data from US, Canada, and Europe regions, and storing it with us for at least 12 months." - https://www.backblaze.com/b2/c2c-migration.html https://www.backblaze.com/b2/c2c-migration.html Also free egress from Backblaze to CDNs (Cloudflare or Fastly).
- pxc 3y ago
  > Oh man, there's potentially a $32k migration fee to pay for S3 egress, as well. I've known about this for a while, but never bothered to ask. Why is that kind of fee even legal? Is there anything pro-social or valuable about that kind of business practice?
johnklos 3y ago
Half a petabyte in storage and 3 TB transfer a day? Shit. That's nothing, unless you're "saving money by using the cloud". While I feel for them, they put themselves in this position. Even when someone gives you something apparently for free, you have to consider what might happen when it disappears, and it seems that nobody thought about it, or if they did, they didn't care. People often learn how expensive "the cloud" is the hard way.
- paulddraper 3y ago
  > they put themselves in this position You say this like they're hopeless.
  - simonjgreen 3y ago
    I think the implication is either naive or wilfully ignorant, rather than hopeless
- n2d4 3y ago
  I'm curious. What would you do, host on-prem? Even if I had the hardware, as a pure software person, I have no clue how I'd even start setting up a physical location to serve data on that scale. And even if the people on the NixOS team know, it sounds like their time would be better spent working on NixOS than building & maintaining a data center (even if small). Bare metal storage services seem to be on a similar price level as lower-cost cloud services like Cloudflare or Backblaze, but I'm very interested if you happen to know a cheaper alternative! Plus, even if you deal with the storage itself, how do you deal with serving the traffic globally? Optimally you'd want a distributed CDN, but that doesn't really sound feasible without the cloud. Would you still use a cloud CDN, or just serve directly from your own server?
  - jtolds 3y ago
    (piping up from another thread, but) > I'm very interested if you happen to know a cheaper alternative! and > Optimally you'd want a distributed CDN https://www.storj.io/ https://www.storj.io/ is this thing!
siddiqi123 3y ago
[flagged]
- Rychard 3y ago
  This is a direct copy of a single sentence of a comment[0] posted on TFA. It lacks any of the context of the original comment, and appears to be shameless plagiarism. 0: https://discourse.nixos.org/t/the-nixos-foundations-call-to-action-s3-costs-require-community-support/28672/4 https://discourse.nixos.org/t/the-nixos-foundations-call-to-...
  - sagarm 3y ago
    Spot checking a few other comments, they are similarly copied verbatim from elsewhere.
deleted 3y ago
[deleted]
botanical 3y ago
I've never heard of LogicBlox but they were sponsoring $9000 a month? That's crazy, normally even big sponsors will give a once-off $5000. Well, well done to them
- pxc 3y ago
  The author and lead maintainer of Nix used to work there for some years, as well. Maybe he pushed for it, or maybe they saw both things as similar kinds of investments in their stack.
poxrud 3y ago
“That would incur a fixed cost of around $32k for the migration.” They should consider Backblaze b2 if they want s3 api compatibility. Backblaze currently offers to cover the migration costs assuming some spend commitment.
andrewstuart 3y ago
Cloudflare R2 does not charge egress.
- adobrawy 3y ago
  Cloudflare R2 does apply charge egress: https://pricing.ceru.dev/r2 https://pricing.ceru.dev/r2 They can apply for OSS sponsorship, but they still have to pay the AWS bill - either $9k/month for keeping AWS services, or $32k/month in one-off migration costs.
  - Dylan16807 3y ago
    What are you trying to show with that page? It says at the top R2 has no bandwidth costs. And if you drag the slider for read percentage around, you can see how it doesn't affect the R2 price at all.
    - adobrawy 3y ago
      I wanted to show that there is a charge associated with the increase in load (API operations), but you are right that the bandwidth is free.
      - Dylan16807 3y ago
        Well, there is a cost for read requests, but if you're going to point that out you need to give a sense of scale. A terabyte on S3 is $90. If we go with 5MB files, that's 200k requests. For 200k requests, S3 costs 8 cents and R2 costs 7.2 cents. The chart in the post suggests that NixOS is dealing with about 10 requests per second on average. Which is about $10 per month in fees on either service.
  - vlovich123 3y ago
    Not sure where you get your info. No r2 does not charge egress. Source: former TL for R2, work at cloudflare. You should probably use the official pricing calculator: https://r2-calculator.cloudflare.com/ https://r2-calculator.cloudflare.com/ You would have to pay your S3 egress costs to migrate but there’s a migration tool to make that easy and we’re not in charge of your AWS bill. I imagine Matthew would be happy to have Amazon join the Bandwidth Alliance. You can also just drop Cache Reserve in front which transparently stores into R2 the assets as you access them depending on what you need from this. That being said, it looks like only $900 is for egress with the rest (~4k if I’m reading it right) being storage. I haven’t done the math but I don’t know how much cheaper we’ll be on that right now.
colordrops 3y ago
I wonder if some plugin could be created to use torrents and/or ipfs for caching.
- dlahoda 3y ago
  there was attempt which stopped. yeah, having ipfs. i would donate in crypto for that. and sure share my local storage. filesystem engine of nix considers remote and local location as same, and does delta copy, gc, and other operations afaik of things. so it would require rewrite of some super robust core parts to work with ipfs.
  - edude03 3y ago
    It’s still going, just slowly. I checked in on the RFC and it was updated 3 days ago
andrewstuart 3y ago
2 days ago someone said they had thousands of dollars of AWS credits to use. https://news.ycombinator.com/item?id=36149966 https://news.ycombinator.com/item?id=36149966
- supriyo-biswas 3y ago
  The sponsor could use this to enable a one-time migration out of S3, into something like Cloudflare R2. 1. The sponsor generates an IAM user and the corresponding access key to access objects in the NixOS buckets. 2. The NixOS operator grants access to said IAM user and (this is important) enables "requester pays" on the buckets so that the sponsor is billed. 3. The sponsor's keys are added to Cloudflare R2, which pulls objects from S3 and stores them.
  - dizhn 3y ago
    I hope there's no place there for error. It would suck if the gifter ends up with unused credits + a huge bill. :)
BonoboIO 3y ago
Deleted
- dlahoda 3y ago
  would be awesome i could donate in crypto, as if crypto would be real thing.
- viraptor 3y ago
  Just post the link or the relevant bits. This is spam : (
  - BonoboIO 3y ago
    I wanted to copy paste a part of it ... iPhone copied the whole page ... why?!
user6723 3y ago
One of my colocated servers has 60TB of space to spare and has a 10gbe unmetered uplink on the provider's IP-transit blend. It is easy to configure traffic shaping on the box so that any unused upload bandwidth can be used without any impact to existing applications. I would be happy to seed NixOS torrents. There is an extension to torrents where files can be added to a torrent after a torrent is released that might make sense to look at.
- toastal 3y ago
  Makes you wonder why the whole protocol isn't Bittorrent with users seeding their builds to others.
  - kortilla 3y ago
    Because the cloud movement in all of its hype stifled any investment in this area.
    - toastal 3y ago
      I believe it. I foresee a lot of holes being dug for projects buying into “the cloud” and “the edge”.
  - ritchiey 3y ago
    The reason usually cited for why not BitTorrent for software packages is because it makes public what package versions you have and haven’t downloaded. This information could be useful to an attacker.
    - goodpoint 3y ago
      Hardly. Plus I'd rather have a random bunch of people get a small fraction of such information rather than a single company see everything.
    - __MatrixMan__ 3y ago
      Only if you're only seeding what you're using. It would be easy to add an obfuscation layer that seeds more.
jtolds 3y ago
Cloudflyer (https://www.cloudflyer.io/ https://www.cloudflyer.io/) offers free migrations from S3 to Storj. Storj is $4/TB/mo, $7/TB egress. It's decentralized storage, so the base functionality has CDN-like performance (no Cloudfront needed), and you don't need to pay extra for multiregion redundancy. I jumped on the Nix Foundation Matrix to try and help them directly, but for anyone reading this thread, Storj might be able to cut a zero off your storage costs. Check us out! Full disclosure - I work for Storj. https://www.storj.io/ https://www.storj.io/
- labawi 3y ago
  I really like the storj concept - distributed storage, local encryption. Comes from tahoe-lafs I believe, except you don't need to find 10 buddies. There is one main reason I don't use it besides web share: Despite being OSS, uplink-cli is not available in linux distribution packages, which - makes me uneasy about stability/maturity of the platform (e.g. do you not provide packages because you don't have a stable API, and if so, should I really rely on you to keep my data?) - it increases friction - either build from source or manually copy binaries; and you have need to do this on all your machines and servers; and it means no automatic updates - binaries/packages without trusted update channels may be against corporate/personal policies, and you usually want your storage accessible on all/most of your computers - if available in non-distribution package repositories (didn't see in quick-start) - it would still create trust issues Have you considered getting your cli packages in distributions? If I could just `apt install uplink-cli` or similar, I would already be using it.
- optymizer 3y ago
  On an unrelated note, the Storj landing page spells the word 'compatibility' wrong ('compatability').
  - jtolds 3y ago
    ooh, that's embarrassing. thanks for pointing that out. fixed now
    - 3y ago
rglullis 3y ago
That seems like the perfect use case for IPFS. I can even bet that if the people at the NixOS foundation played their cards right, they could get the Filecoin bag holders to pay for the development of this new feature.
- __MatrixMan__ 3y ago
  It's in the works: https://discourse.nixos.org/t/obsidian-systems-is-excited-to-bring-ipfs-support-to-nix/7375/47?page=2 https://discourse.nixos.org/t/obsidian-systems-is-excited-to...
fnordpiglet 3y ago
I’m surprised aws doesn’t get in front of this and offer the s3 storage as a donation. It would be tax deductible at retail price, and generate good will. In fact I always thought they should open up storage for popular distributions of Linux and stuff like brew etc.
- cperciva 3y ago
  AWS does provide sponsorship for some open source usage of their infrastructure. They host some FreeBSD mirrors, for example. I've reached out to a couple of my contacts to point them at this.
- adobrawy 3y ago
  Here is info on how to apply for OSS sponsorship by AWS: https://aws.amazon.com/blogs/opensource/aws-promotional-credits-open-source-projects/ https://aws.amazon.com/blogs/opensource/aws-promotional-cred...
garysomerhalder 3y ago
What about looking at Wasabi? It’s $5.99 USD + (any applicable taxes) per TB per month for S3 compatible storage on the pay as you go plan https://wasabi.com https://wasabi.com
temptemptemp111 3y ago
[dead]
exabrial 3y ago
You know, we used to solve these problems through organizations like the Apache Software Foundation and Eclipse Foundation....
getcrunk 3y ago
Shoutout to bunny cdn. It’s dirt cheap in comparison. $5/tb egress. Storage is similarly priced to s3 tho.
predictabl3 3y ago
Personally, I'll shovel money to the TVL folks for tvix-store before I give a single penny that might end up in Amazon's pocket for S3 hosting...
- tazjin 3y ago
  As one of the TVL folks, we aim for cache compatibility (i.e. output path equality), so the NixOS cache itself is highly relevant to us, too. Would we choose to host it on S3? Probably not, but that's where it is right now and we have to be realistic about whether or not that will change. Personally I don't think so, though even on S3 there are still some optimisations to be done.
charcircuit 3y ago
I wonder if it would be cheaper to just rebuild some of these objects from the derivation. Isn't that the point of reproducable builds?
aidenn0 3y ago
The only thing I wonder is how much space it would use if the nar files were stored without .xz and in a store that does smart deduplication. Because of the way Nix works, there's going to be a lot of e.g. ELF files that are nearly identical except for the rpath.
- eliaspro 3y ago
  See this post in the original discussion regarding flokli's work on tvix-store, which provides a completely different storage model, while still being able to provide NAR compatibility by producing them on demand: https://discourse.nixos.org/t/the-nixos-foundations-call-to-action-s3-costs-require-community-support/28672/12 https://discourse.nixos.org/t/the-nixos-foundations-call-to-... The tvix project itself is quite fascinating and worth a look. It reminds me a bit of the Gentoo times when Paludis came around and blew Portage's performance out of the water - but tvix seems to care more about compatibility with the existing ecosystem instead of being a competitor.
  - aidenn0 3y ago
    tvix-store is neat and I look forward to what it might bring in the future (along with the rest of tvix) However there are many deduplicating storage systems that exist right now. Switching to one of them could potentially be an intermediate solution, though compression would have to be solved; perhaps a low -memory version of zstd for compressing in transit would make sense.
    - eliaspro 3y ago
      It might be worth a try to run Caddy, which supports zstd out of the box (the only webserver which does so to my knowledge), in front of an uncompressed store to see whether on-the-fly encoding/compression is feasible. https://caddyserver.com/docs/caddyfile/directives/encode#encode https://caddyserver.com/docs/caddyfile/directives/encode#enc...
janosdebugs 3y ago
Can someone please enlighten my uninformed self why NixOS needs to store this mindboggling amount of data and why it would be damaging to the research community specifically if it were to be lost? 9000$ a month is a LOT.
- asdfaoeu 3y ago
  In comparison it seems Debian is using 1/100th of it https://www.debian.org/mirror/size https://www.debian.org/mirror/size .
  - znpy 3y ago
    I guess that reflects the fact that debian is a distro from another time, when you had to be mindful of the resources you consumed. But i too wonder if they really have to keep everything around.
    - kaba0 3y ago
      One has hard dependencies with Nix, so a modification down the line will cause a cascading recompilation. As mentioned, you doesn’t have to, but it has great value to be able to checkout any version.
      - znpy 3y ago
        sure, but at 425TB i'm guessing they're storing either everything in full or everything in full and uncompressed. and compression can do a lot for text files like sources etc. for example (last time i did that) the linux kernel source code was about 1.2GB uncompressed, but ~60-70MB in tar+gz format. surely there must be a way to reduce that size.
        aidenn0 3y ago
        It's stored compressed with xzip
    - c0l0 3y ago
deleted 3y ago
[deleted]
hamandcheese 3y ago
> Migrating the buckets to Cloudflare R2 52, and using the generous OSS sponsorship offer they recently announced 89. That would incur a fixed cost of around $32k for the migration. I assume this is do to egress from S3. I wonder how that cost compares to re-generating the historic caches. Presumably that is possible with nix.
- 8organicbits 3y ago
  It may even be worth leaving historic data in S3, assuming it gets low enough traffic, then expire it once it's too old. Definitely look for ways to avoid egressing it all from S3.
- microtonal 3y ago
  I assume this is do to egress from S3. I wonder how that cost compares to re-generating the historic caches. Presumably that is possible with nix. One issue that you'll run into is that many upstream source archives are not available anymore at the original URL. Though I think it should be possible to collect all fixed-output derivations and copy their outputs over to a new cache and then try to build. Once these are in the cache, Nix will not try to download the sources from the original location but use the binary cache instead.
gigatexal 3y ago
Any cloud provider or CDN that agreed to just sponsor such an important project would gain so much community love that I think they should just do it. Cloudflare should just do an agreement special with Nix and store everything for free. The cost would be not negligible but the reputation benefit and goodwill will be immense.
- kvz 3y ago
  Like LogicBlox received so much community love? I’m a Nix user and their paying $9k/mo for this never amounted to me learning their name. Could say a lot about me but could also say a lot about how appealing it will be to sponsor this. Could be changed tho, Nix leadership should brainstorm how to make this an attractive thing to sponsor. Feature name prominently on site, conferences, social media, even CLI(?).
  - gigatexal 3y ago
    A sponsor like CF giving them free hosting and r2 storage etc would get a lot of love. Or it might not. Honestly I think Hetzner should just host them it could be a ton cheaper than cloud though it'd not be so geographically optimal I guess.
testemailfordg2 3y ago
Maybe explore Wasabi, costs $5 per TB/month with no egress costs https://wasabi.com/cloud-storage-pricing/#three-info https://wasabi.com/cloud-storage-pricing/#three-info Can get cheaper as well if you go for reservered capacity model.
- xgbi 3y ago
  It is only a good fit if the egress amount is less than total storage, as specified in their FAQ: https://wasabi.com/paygo-pricing-faq/#free-egress-policy https://wasabi.com/paygo-pricing-faq/#free-egress-policy > For example, if you store 100 TB with Wasabi and download (egress) 100 TB or less within a monthly billing cycle, then your storage use case is a good fit for our policy. If your monthly downloads exceed 100 TB, then your use case is not a good fit.
crazypython 3y ago
S3 is a highly reliable storage media. You should use a CDN that transparently caches S3 artifacts to save on reads, such as AWS CloudFront or Cloudflare R2 are good choices.
10000truths 3y ago
Google and some napkin math tell me that this kind of workload should not be on the cloud. A 4U or 8U rack server with 1 PB of storage and plenty of cores: ~$40K Colo facilities in the Bay Area: $400-500/mo Compared against the status quo of $9K/mo, the ROI ends up being 5 months per box, for twice the storage capacity they need. Set up multiple of those boxes around the world to the extent that you care about redundancy/distribution, and they still end up far ahead in a few years time.
- marginalia_nu 3y ago
  Sounds about right. You might possibly be able to bring the server cost down to like $20K if you really turn every stone looking for good offers (for a storage server like this you can probably go with a refurbished machine; the demand on the hardware is really nothing to write home about), but you should be paying hundreds per month, absolutely nowhere near $10k/month.
- georgyo 3y ago
  I am no fan of the cloud, but with that many disks, failure rate goes up. So you also need someone to go to the DC. All the disks will bought around the same time, which means failures start to cluster up as well. The machine also needs to be replaced every $interval. After 4 or 5 years, keep it up is more hassle that replacing it. A single machine patches and will need to reboot. What should happen to people trying to access the store? Keep replicas help, ensuring a petabyte or even half a petabyte of small files is in sync across multiple sites with various link speeds is not as trivial as rsync.
  - bsagdiyev 3y ago
    If the NixOS team cannot handle this then I question their capabilities. This is cake work.
    - quickthrower2 3y ago
      Handle yes, but is this “cheaper”
znpy 3y ago
425 TB? Really? I wonder how much of that could be thrown away.
- microtonal 3y ago
  Not much, if you want historical nixpkgs revisions to be usable (without rebuilding the world locally, which would probably fail because of now-dead upstream download sites). Yanking historical output paths would do serious damage to some of Nix' claimed benefits (eg. reproducibility).
  - simonjgreen 3y ago
    Which is an artifact of designing without consideration to cost
    - jabradoodle 3y ago
      Have any examples of poor design choices?
    - _piif 3y ago
      Which I'm glad has been the case here since it results in the most stable and trouble-free Linux distro and package manager I have ever used. Complete with full source reproducibility for everyone to top it off. It'd be hard to convince me to switch back to a traditional distro at this point.
      - znpy 3y ago
        > Complete with full source reproducibility if you have full source reproducibility, why are artifacts needed then?
        _piif 3y ago
        To not force every user to build (potentially all) derivations which is very time consuming, especially considering the way Nix derivations work which requires a rebuild each time a dependency changes in any way meaning something like a glibc update results in a recompile / redownload of the entire userspace. So it's not a necessity per se but constantly rebuilding a significant amount of packages would easily turn the majority of users away. I'm not sure how many Nixpkgs revisions the Hydra build server keeps hold of or if they even clean up older revisions at all - perhaps it might be time to start doing so at this point. That would be an issue for ancient Nixpkgs revisions which point to broken urls though as the archival of sources is out of the scope for the NixOS infrastructure. Nix is only concerned about the state of a given source using checksums, not the availability itself. So in that sense "full source reproducibility" is only a guarantee that the input doesn't differ (and optimally should result in the same output bit-for-bit which is a completely different beast and something that is being tackled not only by NixOS but by several other distros). It's not a guarantee for "I will be able to grab this Nix derivation in 30 years and be able to get an output out of it" unless you have a mirror of the original sources.
1letterunixname 3y ago
When you reach scale and need a CDN, DIY on bare metal on colo and roundrobin using AnyCast. And then find out why you're consuming so much bandwidth and try to use less.
woopwoop24 3y ago
tl:dr operating yourself around 6-10k€/year fixed price for hosting the 500TB. If cloudflare offers to store that and operate it for free this is a no brainer for cloudfare R2, if the 32k migration is really a one time only cost. i personally would invest in a solution which decreases or at least flattens the cost, when the storage needs increase, instead of the current linear curve. Biggest problem in "cloud" and biggest win in hosting yourself. if you want to do it yourself i would probably think about something like this: 500TB with 20tb hdd's are around 28 disks needed. This will cost between 15-20.000€ per server(4U with sata/sas controller and 36-40 disk chassis) with no raid and high availability you will need 2 locations and 2 servers in a colocated setup. i would say colo will cost you 500-1000€ a month Then you would use the cloudflare cdn with their OSS offering and be done with it. initial cost 40k operational cost + colo = around 6-12k/year i would assume, if you have people nearby to do maintenance.
- tpetry 3y ago
  You can‘t use the CloudFlare CDN to offload the egress for file downloads. That‘s against their terms of service.
lsca94 3y ago
I would suggest them a service like wasabi.com. There they wouldn‘t pay the egress traffic only the used storage. But yeah probably the best solutions would be cloudflare if they manage to get the OSS sponsorship.
londons_explore 3y ago
> That would incur a fixed cost of around $32k for the migration The EU needs to get onto banning/reducing lock in like this by making migration expensive. All the cloud providers clearly have pricing models to try to lock customers from moving elsewhere.
- londons_explore 3y ago
  I wonder what would happen if the EU mandated that all cloud providers supported 'cloud porting', where all cloud assets (VM's, IP addresses, S3 buckets) must be portable, for free, to another provider. Parts that cannot be ported (eg. Redirection proxy services) must be run for free. Phone number portability was done via a similar mandate, and despite industry claiming at first that it wasn't technically possible, it has worked out pretty well.
tutfbhuf 3y ago
I can recommend Hetzner Storage boxes https://www.hetzner.com/storage/storage-box?country=us https://www.hetzner.com/storage/storage-box?country=us its quite cheap (€40.60 for 20 TB). Just order 25x BX41 Boxes with 500 TB for 1015€. They do not support S3 protocol, but have plenty of other protocol options (FTP, SFTP, HTTPS, rsync, mount as network drive, ...).
- foepys 3y ago
  These storage boxes have very narrow concurrent connection limits and are not suited for the problem in the OP without additional hardware/services.
  - anotherhue 3y ago
    Yeah, those boxes are meant for archive only and it shows. High latency, slow transfer, frequent maintenance downtime. Not the SLA you're looking for
chillbill 3y ago
I’m disappointed to learn they use S3 to be honest.
ris 3y ago
Reminder that S3 supports bittorrent natively.
- liftm 3y ago
  Just for my understanding: That wouldn't help, right? - It doesn't change storage costs - It doesn't decrease the cost of a full exit - They already have a CDN in front of it which reduces the traffic by a factor of 60.
- greyface- 3y ago
  No longer true: https://news.ycombinator.com/item?id=27524549 https://news.ycombinator.com/item?id=27524549 > As of April 29, 2021 Amazon S3 is discontinuing the S3 BitTorrent feature and it will no longer be available to enable. AWS will support customers currently using the S3 BitTorrent feature for 12 months. After April 29, 2022, BitTorrent clients will no longer connect to Amazon S3.
Havoc 3y ago
I don't get why these things aren't designed around bittorrent, at least as first line of defence. Also means the most popular packages have the most seeders I bet it is easier to get people to seed a bit than to get them to pull out a credit card
JonChesterfield 3y ago
This seems to be about a cache of compiled stuff where it'll cost $32k to move the data out of S3. So... rebuild the cache somewhere else and drop the original data?
- abathur 3y ago
  I think this is technically an option, but IIUC it'll have plenty of its own hurdles. The cache is populated by the results of builds run by NixOS's hydra build farm over a period of many years, and hydra stays busy enough that it's hard to imagine this approach being viable without standing up a separate cluster of build machines dedicated to the task. To complete it in a meaningful timeframe, the cluster might also need to be quite a bit bigger. (NixOS hydra obviously has some spare capacity or it'd never catch up, but I would be surprised if it has enough idle capacity to rebuild everything over any meaningful timeframe without also significantly affecting velocity on everything else.) Calling on the community to dedicate temporary build resources might work--but the systems need to be trusted, so it can't quite be Folding@home... I guess if there's a bit of Pareto in the distribution of build output sizes, there might be meaningful savings in rebuilding the packages that take up the most storage and paying to move the small stuff. It might also be possible to triage the sources in the cache--re-fetch all that are still available from the original source or a mirror, and then just export the ones that either couldn't be fetched or could be fetched but the contents no longer match the hash.
  - JonChesterfield 3y ago
    Probably does take a lot of compute to recreate the cache, but doing it in order of requests works pretty well as a heuristic for most important first, gradually backfilling when idle. Adding machines to the build infra seems like less of a loss than paying egress fees too. Overall seems plausible that this data store isn't just a cache of things that can be recreated at will. The sibling comment about source being unavailable (or still available but at a different location) for the earlier builds is interesting in that respect.
    - RaitoBezarius 3y ago
nixer6872 3y ago
There has been a controversy around the server cache.nixos.org. The signing ed25519 cache key is from the year 2015 and there is no plan to rotate the key. It is apparent that a key cannot hold for all these years. I have been made aware of the matter when somebody on the matrix claimed that there is no plan to rotate the cache key. He/She said reasons behind that are explained by Eelco D. in https://www.youtube.com/watch?v=yXPQdZh8o7c https://www.youtube.com/watch?v=yXPQdZh8o7c. Unfortunately, I couldn't confirm this. That person afterwards claimed that the video is cut and that part was removed intentionally. If that's the case, it might be the reason why ppl stop paying for the cache server costs. Furthermore, if the foundation attempted to hide the facts by cutting the video and re-uploading, for me, this is a matter of dishonesty and somebody should take the responsibility for it. I believe this person is Ron E., the current director of the Nixos Foundation. But in any case, this claim should be backed by finding the original stream of the Nix 20th Community Panel recording. For the rest of us, until some cryptogeek fixes nix to support certs and key lifecycles, the solution is to build a local cache with hydra and flakes. There are many ppl keen on the subject, you can look for one on the matrix.
jensenbox 3y ago
Why is Backblaze with their B2 solution not on their list? From what I recall they'll cover your initial egress costs.
Tonyjosh 3y ago
[flagged]
unixhero 3y ago
July 1st is not plenty of time as a heads up.
unixhero 3y ago
Just get sponsored by an educational institution or a unicorn with plenty of bandwidth and deep pockets.
ronef 3y ago
We will be holding the community call on Wednesday June 7th at 5pm (GMT+2) Planned Agenda * Brief budget and timeline review * Review/discuss all potential options * Brainstorm/Q&A Info to join is in the original discourse post. Thank you again for jumping all in on this with us!

304 points by ghuntley 3y ago

deleted 3y ago

[deleted]

LanternLight83 3y ago

I get that it's out of scope right now, but it'd be so cool to have a decentralized solution for regional caching

candiddevmike 3y ago
That's basically what every other distro does, seems like nix's architecture doesn't support this?
- anon291 3y ago
  Nix supports mirroring. It's pretty easy to set up your own mirror.
  - candiddevmike 3y ago
    Why are they not considering a decentralized mirrorlist?
    - microtonal 3y ago
      The storage requirements are too large for most potential mirrors. Nix is not like any regular Linux distribution.
- 1MachineElf 3y ago
  But AWS S3 is decentralized - all objects are replicated across a region's availability zones. /s
- deleted 3y ago
  [deleted]
- slabity 3y ago
  It does. Nix has an extremely powerful (and flexible) caching system. Not to mention popular caching systems like cachix that are easy to integrate into your system. Unlike most other packaging systems though, Nix packages are rebuilt when there's changes to any of their dependencies. This results in a huge amount of storage data used in their caches compared to other systems. Not to mention the compute power required to rebuild those packages. For example, let's say something like `glibc` updated with a new patch. Every single package that has `glibc` as a dependency needs to be rebuilt, along with every package that has any of those packages as a dependency. This could easily cause the rebuild of tens of thousands of packages, even if they are only dynamically linking to it.
  - pxc 3y ago
    Maybe we should be using something like Guix grafts to save not just rebuild time, but space (and thus money).
    - dlahoda 3y ago
      nix has security patch tech.
    - slabity 3y ago
      Guix's graft system is a hack specifically made to get security updates out ASAP. They should really only be seen as a last resort because it circumvents the primary benefit of using Guix as a functional build system (pure, reproducible builds). I'm not sure if Nix has a similar system. IMHO I'd prefer to see compile-time caches like ccache or sccache be better integrated into Nix while still providing the guarantees that a functional build system provides. That alone would save massive amounts of compute power when rebuilding packages, as you'd essentially only need to recompile the files that change in a package. Not only on continuous build systems like Nix's Hydra, but also when just rebuilding packages locally. This won't solve the storage cost issue, but would definitely alleviate compute costs.
      - xyzzy_plugh 3y ago
LanternLight83 3y ago
Just to clarify, the missing component in my mind is in mirror advertisement. The need for manual compilation and review is a hindrance to casual participants, we need a config option that you just toggle (maybe a configuration struct with eg. bandwidth limits) to opt-in to IPFS or GNUnet distribution of store items present on your system.
simonjgreen 3y ago
Anycast BGP and Colo is the cheapest (and imo simplest) way

justahuman74 3y ago

The nix store seems on the face of it like a perfect use case for something like IPFS

rkeene2 3y ago
My AppFS project [0] is basically this, a content-addressable remote package-oriented system. [0] https://AppFS.net/ https://AppFS.net/
pxc 3y ago
There has been work ongoing to support this in Nix for a long time, actually. Maybe this will help push that effort along, or direct more resources towards it. See: https://github.com/NixOS/rfcs/pull/133 https://github.com/NixOS/rfcs/pull/133
nlitened 3y ago
Or torrents
shp0ngle 3y ago
IPFS in not magic you can just sprinkle on things, say “p2p! decentralised!” and be done. you still need either pay someone to “pin it”, which is more expensive than dumping it on s3. Pinning services (that’s a thing) are all sketchy (as the big ones are all somehow connected to NFTs) and if you do it yourself it’s actually harder than just storing it
- mindslight 3y ago
  As a NixOS user (who doesn't even use the binaries), I would happily pin NixOS packages! The point of IPFS is to decouple the names of resources from what's serving them. It's exactly what is needed for distributed libre software projects where many are willing to donate some resources, but few are able to donate all the resources.
  - joepie91_ 3y ago
    But the problem here isn't distribution, it's durable storage. Something along the lines of Tahoe-LAFS would be far more suitable.

toomuchtodo 3y ago

Can folks who have AWS credits to burn use them to cover egress costs?

cyberax 3y ago
Yes, but not easily. It's possible to set the bucket to "requester pays" model, but this will require touching all the cached objects (which also will cost some money).

justahuman74 3y ago

Are any large companies using NixOS? I presume they'd be candidates for sponsorship

solarkraft 3y ago
repl.it comes to mind, but I think recently a few others have popped up.
pxc 3y ago
The most Nix-centric companies are all consultancies and startups, but off the top of my head: - Target - Shopify - Pinterest all used Nix fairly heavily at some point, and I believe they still do. (This is actually about more than NixOS— the binary caches affect all Nix users, including those on macOS as well as any Linux distro.)
groodt 3y ago
Shopify and Target have been known to use it.

heavyset_go 3y ago

S3's costs are insane for this use case, can't imagine using it to for a Linux distribution's repositories. edit: Oh man, there's potentially a $32k migration fee to pay for S3 egress, as well.

bombcar 3y ago
At least use one of those s3 competitors with sane pricing. GitHub? backblaze? Cloud flare?
- mkl 3y ago
  > same pricing I think you meant "sane"? Otherwise there's not much point in switching.
  - bombcar 3y ago
    Yeah, that was an unfortunate typo.
    - sitkack 3y ago
      Some things can only be said as a joke or by accident or both.
scottlamb 3y ago
"We will cover egress and migration fees for customers migrating over 10 TB of data from US, Canada, and Europe regions, and storing it with us for at least 12 months." - https://www.backblaze.com/b2/c2c-migration.html https://www.backblaze.com/b2/c2c-migration.html Also free egress from Backblaze to CDNs (Cloudflare or Fastly).
pxc 3y ago
> Oh man, there's potentially a $32k migration fee to pay for S3 egress, as well. I've known about this for a while, but never bothered to ask. Why is that kind of fee even legal? Is there anything pro-social or valuable about that kind of business practice?

johnklos 3y ago

Half a petabyte in storage and 3 TB transfer a day? Shit. That's nothing, unless you're "saving money by using the cloud". While I feel for them, they put themselves in this position. Even when someone gives you something apparently for free, you have to consider what might happen when it disappears, and it seems that nobody thought about it, or if they did, they didn't care. People often learn how expensive "the cloud" is the hard way.

paulddraper 3y ago
> they put themselves in this position You say this like they're hopeless.
- simonjgreen 3y ago
  I think the implication is either naive or wilfully ignorant, rather than hopeless
n2d4 3y ago
I'm curious. What would you do, host on-prem? Even if I had the hardware, as a pure software person, I have no clue how I'd even start setting up a physical location to serve data on that scale. And even if the people on the NixOS team know, it sounds like their time would be better spent working on NixOS than building & maintaining a data center (even if small). Bare metal storage services seem to be on a similar price level as lower-cost cloud services like Cloudflare or Backblaze, but I'm very interested if you happen to know a cheaper alternative! Plus, even if you deal with the storage itself, how do you deal with serving the traffic globally? Optimally you'd want a distributed CDN, but that doesn't really sound feasible without the cloud. Would you still use a cloud CDN, or just serve directly from your own server?
- jtolds 3y ago
  (piping up from another thread, but) > I'm very interested if you happen to know a cheaper alternative! and > Optimally you'd want a distributed CDN https://www.storj.io/ https://www.storj.io/ is this thing!

siddiqi123 3y ago

[flagged]

Rychard 3y ago
This is a direct copy of a single sentence of a comment[0] posted on TFA. It lacks any of the context of the original comment, and appears to be shameless plagiarism. 0: https://discourse.nixos.org/t/the-nixos-foundations-call-to-action-s3-costs-require-community-support/28672/4 https://discourse.nixos.org/t/the-nixos-foundations-call-to-...
- sagarm 3y ago
  Spot checking a few other comments, they are similarly copied verbatim from elsewhere.

deleted 3y ago

[deleted]

botanical 3y ago

I've never heard of LogicBlox but they were sponsoring $9000 a month? That's crazy, normally even big sponsors will give a once-off $5000. Well, well done to them

pxc 3y ago
The author and lead maintainer of Nix used to work there for some years, as well. Maybe he pushed for it, or maybe they saw both things as similar kinds of investments in their stack.

poxrud 3y ago

“That would incur a fixed cost of around $32k for the migration.” They should consider Backblaze b2 if they want s3 api compatibility. Backblaze currently offers to cover the migration costs assuming some spend commitment.

andrewstuart 3y ago

Cloudflare R2 does not charge egress.

adobrawy 3y ago
Cloudflare R2 does apply charge egress: https://pricing.ceru.dev/r2 https://pricing.ceru.dev/r2 They can apply for OSS sponsorship, but they still have to pay the AWS bill - either $9k/month for keeping AWS services, or $32k/month in one-off migration costs.
- Dylan16807 3y ago
  What are you trying to show with that page? It says at the top R2 has no bandwidth costs. And if you drag the slider for read percentage around, you can see how it doesn't affect the R2 price at all.
  - adobrawy 3y ago
    I wanted to show that there is a charge associated with the increase in load (API operations), but you are right that the bandwidth is free.
    - Dylan16807 3y ago
      Well, there is a cost for read requests, but if you're going to point that out you need to give a sense of scale. A terabyte on S3 is $90. If we go with 5MB files, that's 200k requests. For 200k requests, S3 costs 8 cents and R2 costs 7.2 cents. The chart in the post suggests that NixOS is dealing with about 10 requests per second on average. Which is about $10 per month in fees on either service.
- vlovich123 3y ago
  Not sure where you get your info. No r2 does not charge egress. Source: former TL for R2, work at cloudflare. You should probably use the official pricing calculator: https://r2-calculator.cloudflare.com/ https://r2-calculator.cloudflare.com/ You would have to pay your S3 egress costs to migrate but there’s a migration tool to make that easy and we’re not in charge of your AWS bill. I imagine Matthew would be happy to have Amazon join the Bandwidth Alliance. You can also just drop Cache Reserve in front which transparently stores into R2 the assets as you access them depending on what you need from this. That being said, it looks like only $900 is for egress with the rest (~4k if I’m reading it right) being storage. I haven’t done the math but I don’t know how much cheaper we’ll be on that right now.

colordrops 3y ago

I wonder if some plugin could be created to use torrents and/or ipfs for caching.

dlahoda 3y ago
there was attempt which stopped. yeah, having ipfs. i would donate in crypto for that. and sure share my local storage. filesystem engine of nix considers remote and local location as same, and does delta copy, gc, and other operations afaik of things. so it would require rewrite of some super robust core parts to work with ipfs.
- edude03 3y ago
  It’s still going, just slowly. I checked in on the RFC and it was updated 3 days ago

andrewstuart 3y ago

2 days ago someone said they had thousands of dollars of AWS credits to use. https://news.ycombinator.com/item?id=36149966 https://news.ycombinator.com/item?id=36149966

supriyo-biswas 3y ago
The sponsor could use this to enable a one-time migration out of S3, into something like Cloudflare R2. 1. The sponsor generates an IAM user and the corresponding access key to access objects in the NixOS buckets. 2. The NixOS operator grants access to said IAM user and (this is important) enables "requester pays" on the buckets so that the sponsor is billed. 3. The sponsor's keys are added to Cloudflare R2, which pulls objects from S3 and stores them.
- dizhn 3y ago
  I hope there's no place there for error. It would suck if the gifter ends up with unused credits + a huge bill. :)

BonoboIO 3y ago

Deleted

dlahoda 3y ago
would be awesome i could donate in crypto, as if crypto would be real thing.
viraptor 3y ago
Just post the link or the relevant bits. This is spam : (
- BonoboIO 3y ago
  I wanted to copy paste a part of it ... iPhone copied the whole page ... why?!

user6723 3y ago

One of my colocated servers has 60TB of space to spare and has a 10gbe unmetered uplink on the provider's IP-transit blend. It is easy to configure traffic shaping on the box so that any unused upload bandwidth can be used without any impact to existing applications. I would be happy to seed NixOS torrents. There is an extension to torrents where files can be added to a torrent after a torrent is released that might make sense to look at.

toastal 3y ago
Makes you wonder why the whole protocol isn't Bittorrent with users seeding their builds to others.
- kortilla 3y ago
  Because the cloud movement in all of its hype stifled any investment in this area.
  - toastal 3y ago
    I believe it. I foresee a lot of holes being dug for projects buying into “the cloud” and “the edge”.
- ritchiey 3y ago
  The reason usually cited for why not BitTorrent for software packages is because it makes public what package versions you have and haven’t downloaded. This information could be useful to an attacker.
  - goodpoint 3y ago
    Hardly. Plus I'd rather have a random bunch of people get a small fraction of such information rather than a single company see everything.
  - __MatrixMan__ 3y ago
    Only if you're only seeding what you're using. It would be easy to add an obfuscation layer that seeds more.

jtolds 3y ago

Cloudflyer (https://www.cloudflyer.io/ https://www.cloudflyer.io/) offers free migrations from S3 to Storj. Storj is $4/TB/mo, $7/TB egress. It's decentralized storage, so the base functionality has CDN-like performance (no Cloudfront needed), and you don't need to pay extra for multiregion redundancy. I jumped on the Nix Foundation Matrix to try and help them directly, but for anyone reading this thread, Storj might be able to cut a zero off your storage costs. Check us out! Full disclosure - I work for Storj. https://www.storj.io/ https://www.storj.io/

labawi 3y ago
I really like the storj concept - distributed storage, local encryption. Comes from tahoe-lafs I believe, except you don't need to find 10 buddies. There is one main reason I don't use it besides web share: Despite being OSS, uplink-cli is not available in linux distribution packages, which - makes me uneasy about stability/maturity of the platform (e.g. do you not provide packages because you don't have a stable API, and if so, should I really rely on you to keep my data?) - it increases friction - either build from source or manually copy binaries; and you have need to do this on all your machines and servers; and it means no automatic updates - binaries/packages without trusted update channels may be against corporate/personal policies, and you usually want your storage accessible on all/most of your computers - if available in non-distribution package repositories (didn't see in quick-start) - it would still create trust issues Have you considered getting your cli packages in distributions? If I could just `apt install uplink-cli` or similar, I would already be using it.
optymizer 3y ago
On an unrelated note, the Storj landing page spells the word 'compatibility' wrong ('compatability').
- jtolds 3y ago
  ooh, that's embarrassing. thanks for pointing that out. fixed now
  - 3y ago

rglullis 3y ago

That seems like the perfect use case for IPFS. I can even bet that if the people at the NixOS foundation played their cards right, they could get the Filecoin bag holders to pay for the development of this new feature.

__MatrixMan__ 3y ago
It's in the works: https://discourse.nixos.org/t/obsidian-systems-is-excited-to-bring-ipfs-support-to-nix/7375/47?page=2 https://discourse.nixos.org/t/obsidian-systems-is-excited-to...

fnordpiglet 3y ago

I’m surprised aws doesn’t get in front of this and offer the s3 storage as a donation. It would be tax deductible at retail price, and generate good will. In fact I always thought they should open up storage for popular distributions of Linux and stuff like brew etc.

cperciva 3y ago
AWS does provide sponsorship for some open source usage of their infrastructure. They host some FreeBSD mirrors, for example. I've reached out to a couple of my contacts to point them at this.
adobrawy 3y ago
Here is info on how to apply for OSS sponsorship by AWS: https://aws.amazon.com/blogs/opensource/aws-promotional-credits-open-source-projects/ https://aws.amazon.com/blogs/opensource/aws-promotional-cred...

garysomerhalder 3y ago

What about looking at Wasabi? It’s $5.99 USD + (any applicable taxes) per TB per month for S3 compatible storage on the pay as you go plan https://wasabi.com https://wasabi.com

temptemptemp111 3y ago

[dead]

exabrial 3y ago

You know, we used to solve these problems through organizations like the Apache Software Foundation and Eclipse Foundation....

getcrunk 3y ago

Shoutout to bunny cdn. It’s dirt cheap in comparison. $5/tb egress. Storage is similarly priced to s3 tho.

predictabl3 3y ago

Personally, I'll shovel money to the TVL folks for tvix-store before I give a single penny that might end up in Amazon's pocket for S3 hosting...

tazjin 3y ago
As one of the TVL folks, we aim for cache compatibility (i.e. output path equality), so the NixOS cache itself is highly relevant to us, too. Would we choose to host it on S3? Probably not, but that's where it is right now and we have to be realistic about whether or not that will change. Personally I don't think so, though even on S3 there are still some optimisations to be done.

charcircuit 3y ago

I wonder if it would be cheaper to just rebuild some of these objects from the derivation. Isn't that the point of reproducable builds?

aidenn0 3y ago

The only thing I wonder is how much space it would use if the nar files were stored without .xz and in a store that does smart deduplication. Because of the way Nix works, there's going to be a lot of e.g. ELF files that are nearly identical except for the rpath.

eliaspro 3y ago
See this post in the original discussion regarding flokli's work on tvix-store, which provides a completely different storage model, while still being able to provide NAR compatibility by producing them on demand: https://discourse.nixos.org/t/the-nixos-foundations-call-to-action-s3-costs-require-community-support/28672/12 https://discourse.nixos.org/t/the-nixos-foundations-call-to-... The tvix project itself is quite fascinating and worth a look. It reminds me a bit of the Gentoo times when Paludis came around and blew Portage's performance out of the water - but tvix seems to care more about compatibility with the existing ecosystem instead of being a competitor.
- aidenn0 3y ago
  tvix-store is neat and I look forward to what it might bring in the future (along with the rest of tvix) However there are many deduplicating storage systems that exist right now. Switching to one of them could potentially be an intermediate solution, though compression would have to be solved; perhaps a low -memory version of zstd for compressing in transit would make sense.
  - eliaspro 3y ago
    It might be worth a try to run Caddy, which supports zstd out of the box (the only webserver which does so to my knowledge), in front of an uncompressed store to see whether on-the-fly encoding/compression is feasible. https://caddyserver.com/docs/caddyfile/directives/encode#encode https://caddyserver.com/docs/caddyfile/directives/encode#enc...

janosdebugs 3y ago

Can someone please enlighten my uninformed self why NixOS needs to store this mindboggling amount of data and why it would be damaging to the research community specifically if it were to be lost? 9000$ a month is a LOT.

asdfaoeu 3y ago
In comparison it seems Debian is using 1/100th of it https://www.debian.org/mirror/size https://www.debian.org/mirror/size .
- znpy 3y ago
  I guess that reflects the fact that debian is a distro from another time, when you had to be mindful of the resources you consumed. But i too wonder if they really have to keep everything around.
  - kaba0 3y ago
    One has hard dependencies with Nix, so a modification down the line will cause a cascading recompilation. As mentioned, you doesn’t have to, but it has great value to be able to checkout any version.
    - znpy 3y ago
      sure, but at 425TB i'm guessing they're storing either everything in full or everything in full and uncompressed. and compression can do a lot for text files like sources etc. for example (last time i did that) the linux kernel source code was about 1.2GB uncompressed, but ~60-70MB in tar+gz format. surely there must be a way to reduce that size.
      - aidenn0 3y ago
        It's stored compressed with xzip
  - c0l0 3y ago

deleted 3y ago

[deleted]

hamandcheese 3y ago

> Migrating the buckets to Cloudflare R2 52, and using the generous OSS sponsorship offer they recently announced 89. That would incur a fixed cost of around $32k for the migration. I assume this is do to egress from S3. I wonder how that cost compares to re-generating the historic caches. Presumably that is possible with nix.

8organicbits 3y ago
It may even be worth leaving historic data in S3, assuming it gets low enough traffic, then expire it once it's too old. Definitely look for ways to avoid egressing it all from S3.
microtonal 3y ago
I assume this is do to egress from S3. I wonder how that cost compares to re-generating the historic caches. Presumably that is possible with nix. One issue that you'll run into is that many upstream source archives are not available anymore at the original URL. Though I think it should be possible to collect all fixed-output derivations and copy their outputs over to a new cache and then try to build. Once these are in the cache, Nix will not try to download the sources from the original location but use the binary cache instead.

gigatexal 3y ago

Any cloud provider or CDN that agreed to just sponsor such an important project would gain so much community love that I think they should just do it. Cloudflare should just do an agreement special with Nix and store everything for free. The cost would be not negligible but the reputation benefit and goodwill will be immense.

kvz 3y ago
Like LogicBlox received so much community love? I’m a Nix user and their paying $9k/mo for this never amounted to me learning their name. Could say a lot about me but could also say a lot about how appealing it will be to sponsor this. Could be changed tho, Nix leadership should brainstorm how to make this an attractive thing to sponsor. Feature name prominently on site, conferences, social media, even CLI(?).
- gigatexal 3y ago
  A sponsor like CF giving them free hosting and r2 storage etc would get a lot of love. Or it might not. Honestly I think Hetzner should just host them it could be a ton cheaper than cloud though it'd not be so geographically optimal I guess.

testemailfordg2 3y ago

Maybe explore Wasabi, costs $5 per TB/month with no egress costs https://wasabi.com/cloud-storage-pricing/#three-info https://wasabi.com/cloud-storage-pricing/#three-info Can get cheaper as well if you go for reservered capacity model.

xgbi 3y ago
It is only a good fit if the egress amount is less than total storage, as specified in their FAQ: https://wasabi.com/paygo-pricing-faq/#free-egress-policy https://wasabi.com/paygo-pricing-faq/#free-egress-policy > For example, if you store 100 TB with Wasabi and download (egress) 100 TB or less within a monthly billing cycle, then your storage use case is a good fit for our policy. If your monthly downloads exceed 100 TB, then your use case is not a good fit.

crazypython 3y ago

S3 is a highly reliable storage media. You should use a CDN that transparently caches S3 artifacts to save on reads, such as AWS CloudFront or Cloudflare R2 are good choices.

10000truths 3y ago

Google and some napkin math tell me that this kind of workload should not be on the cloud. A 4U or 8U rack server with 1 PB of storage and plenty of cores: ~$40K Colo facilities in the Bay Area: $400-500/mo Compared against the status quo of $9K/mo, the ROI ends up being 5 months per box, for twice the storage capacity they need. Set up multiple of those boxes around the world to the extent that you care about redundancy/distribution, and they still end up far ahead in a few years time.

marginalia_nu 3y ago
Sounds about right. You might possibly be able to bring the server cost down to like $20K if you really turn every stone looking for good offers (for a storage server like this you can probably go with a refurbished machine; the demand on the hardware is really nothing to write home about), but you should be paying hundreds per month, absolutely nowhere near $10k/month.
georgyo 3y ago
I am no fan of the cloud, but with that many disks, failure rate goes up. So you also need someone to go to the DC. All the disks will bought around the same time, which means failures start to cluster up as well. The machine also needs to be replaced every $interval. After 4 or 5 years, keep it up is more hassle that replacing it. A single machine patches and will need to reboot. What should happen to people trying to access the store? Keep replicas help, ensuring a petabyte or even half a petabyte of small files is in sync across multiple sites with various link speeds is not as trivial as rsync.
- bsagdiyev 3y ago
  If the NixOS team cannot handle this then I question their capabilities. This is cake work.
  - quickthrower2 3y ago
    Handle yes, but is this “cheaper”

znpy 3y ago

425 TB? Really? I wonder how much of that could be thrown away.

microtonal 3y ago
Not much, if you want historical nixpkgs revisions to be usable (without rebuilding the world locally, which would probably fail because of now-dead upstream download sites). Yanking historical output paths would do serious damage to some of Nix' claimed benefits (eg. reproducibility).
- simonjgreen 3y ago
  Which is an artifact of designing without consideration to cost
  - jabradoodle 3y ago
    Have any examples of poor design choices?
  - _piif 3y ago
    Which I'm glad has been the case here since it results in the most stable and trouble-free Linux distro and package manager I have ever used. Complete with full source reproducibility for everyone to top it off. It'd be hard to convince me to switch back to a traditional distro at this point.
    - znpy 3y ago
      > Complete with full source reproducibility if you have full source reproducibility, why are artifacts needed then?
      - _piif 3y ago
        To not force every user to build (potentially all) derivations which is very time consuming, especially considering the way Nix derivations work which requires a rebuild each time a dependency changes in any way meaning something like a glibc update results in a recompile / redownload of the entire userspace. So it's not a necessity per se but constantly rebuilding a significant amount of packages would easily turn the majority of users away. I'm not sure how many Nixpkgs revisions the Hydra build server keeps hold of or if they even clean up older revisions at all - perhaps it might be time to start doing so at this point. That would be an issue for ancient Nixpkgs revisions which point to broken urls though as the archival of sources is out of the scope for the NixOS infrastructure. Nix is only concerned about the state of a given source using checksums, not the availability itself. So in that sense "full source reproducibility" is only a guarantee that the input doesn't differ (and optimally should result in the same output bit-for-bit which is a completely different beast and something that is being tackled not only by NixOS but by several other distros). It's not a guarantee for "I will be able to grab this Nix derivation in 30 years and be able to get an output out of it" unless you have a mirror of the original sources.

1letterunixname 3y ago

When you reach scale and need a CDN, DIY on bare metal on colo and roundrobin using AnyCast. And then find out why you're consuming so much bandwidth and try to use less.

woopwoop24 3y ago

tl:dr operating yourself around 6-10k€/year fixed price for hosting the 500TB. If cloudflare offers to store that and operate it for free this is a no brainer for cloudfare R2, if the 32k migration is really a one time only cost. i personally would invest in a solution which decreases or at least flattens the cost, when the storage needs increase, instead of the current linear curve. Biggest problem in "cloud" and biggest win in hosting yourself. if you want to do it yourself i would probably think about something like this: 500TB with 20tb hdd's are around 28 disks needed. This will cost between 15-20.000€ per server(4U with sata/sas controller and 36-40 disk chassis) with no raid and high availability you will need 2 locations and 2 servers in a colocated setup. i would say colo will cost you 500-1000€ a month Then you would use the cloudflare cdn with their OSS offering and be done with it. initial cost 40k operational cost + colo = around 6-12k/year i would assume, if you have people nearby to do maintenance.

tpetry 3y ago
You can‘t use the CloudFlare CDN to offload the egress for file downloads. That‘s against their terms of service.

lsca94 3y ago

I would suggest them a service like wasabi.com. There they wouldn‘t pay the egress traffic only the used storage. But yeah probably the best solutions would be cloudflare if they manage to get the OSS sponsorship.

londons_explore 3y ago

> That would incur a fixed cost of around $32k for the migration The EU needs to get onto banning/reducing lock in like this by making migration expensive. All the cloud providers clearly have pricing models to try to lock customers from moving elsewhere.

londons_explore 3y ago
I wonder what would happen if the EU mandated that all cloud providers supported 'cloud porting', where all cloud assets (VM's, IP addresses, S3 buckets) must be portable, for free, to another provider. Parts that cannot be ported (eg. Redirection proxy services) must be run for free. Phone number portability was done via a similar mandate, and despite industry claiming at first that it wasn't technically possible, it has worked out pretty well.

tutfbhuf 3y ago

I can recommend Hetzner Storage boxes https://www.hetzner.com/storage/storage-box?country=us https://www.hetzner.com/storage/storage-box?country=us its quite cheap (€40.60 for 20 TB). Just order 25x BX41 Boxes with 500 TB for 1015€. They do not support S3 protocol, but have plenty of other protocol options (FTP, SFTP, HTTPS, rsync, mount as network drive, ...).

foepys 3y ago
These storage boxes have very narrow concurrent connection limits and are not suited for the problem in the OP without additional hardware/services.
- anotherhue 3y ago
  Yeah, those boxes are meant for archive only and it shows. High latency, slow transfer, frequent maintenance downtime. Not the SLA you're looking for

chillbill 3y ago

I’m disappointed to learn they use S3 to be honest.

ris 3y ago

Reminder that S3 supports bittorrent natively.

liftm 3y ago
Just for my understanding: That wouldn't help, right? - It doesn't change storage costs - It doesn't decrease the cost of a full exit - They already have a CDN in front of it which reduces the traffic by a factor of 60.
greyface- 3y ago
No longer true: https://news.ycombinator.com/item?id=27524549 https://news.ycombinator.com/item?id=27524549 > As of April 29, 2021 Amazon S3 is discontinuing the S3 BitTorrent feature and it will no longer be available to enable. AWS will support customers currently using the S3 BitTorrent feature for 12 months. After April 29, 2022, BitTorrent clients will no longer connect to Amazon S3.

Havoc 3y ago

I don't get why these things aren't designed around bittorrent, at least as first line of defence. Also means the most popular packages have the most seeders I bet it is easier to get people to seed a bit than to get them to pull out a credit card

JonChesterfield 3y ago

This seems to be about a cache of compiled stuff where it'll cost $32k to move the data out of S3. So... rebuild the cache somewhere else and drop the original data?

abathur 3y ago
I think this is technically an option, but IIUC it'll have plenty of its own hurdles. The cache is populated by the results of builds run by NixOS's hydra build farm over a period of many years, and hydra stays busy enough that it's hard to imagine this approach being viable without standing up a separate cluster of build machines dedicated to the task. To complete it in a meaningful timeframe, the cluster might also need to be quite a bit bigger. (NixOS hydra obviously has some spare capacity or it'd never catch up, but I would be surprised if it has enough idle capacity to rebuild everything over any meaningful timeframe without also significantly affecting velocity on everything else.) Calling on the community to dedicate temporary build resources might work--but the systems need to be trusted, so it can't quite be Folding@home... I guess if there's a bit of Pareto in the distribution of build output sizes, there might be meaningful savings in rebuilding the packages that take up the most storage and paying to move the small stuff. It might also be possible to triage the sources in the cache--re-fetch all that are still available from the original source or a mirror, and then just export the ones that either couldn't be fetched or could be fetched but the contents no longer match the hash.
- JonChesterfield 3y ago
  Probably does take a lot of compute to recreate the cache, but doing it in order of requests works pretty well as a heuristic for most important first, gradually backfilling when idle. Adding machines to the build infra seems like less of a loss than paying egress fees too. Overall seems plausible that this data store isn't just a cache of things that can be recreated at will. The sibling comment about source being unavailable (or still available but at a different location) for the earlier builds is interesting in that respect.
  - RaitoBezarius 3y ago

nixer6872 3y ago

There has been a controversy around the server cache.nixos.org. The signing ed25519 cache key is from the year 2015 and there is no plan to rotate the key. It is apparent that a key cannot hold for all these years. I have been made aware of the matter when somebody on the matrix claimed that there is no plan to rotate the cache key. He/She said reasons behind that are explained by Eelco D. in https://www.youtube.com/watch?v=yXPQdZh8o7c https://www.youtube.com/watch?v=yXPQdZh8o7c. Unfortunately, I couldn't confirm this. That person afterwards claimed that the video is cut and that part was removed intentionally. If that's the case, it might be the reason why ppl stop paying for the cache server costs. Furthermore, if the foundation attempted to hide the facts by cutting the video and re-uploading, for me, this is a matter of dishonesty and somebody should take the responsibility for it. I believe this person is Ron E., the current director of the Nixos Foundation. But in any case, this claim should be backed by finding the original stream of the Nix 20th Community Panel recording. For the rest of us, until some cryptogeek fixes nix to support certs and key lifecycles, the solution is to build a local cache with hydra and flakes. There are many ppl keen on the subject, you can look for one on the matrix.

jensenbox 3y ago

Why is Backblaze with their B2 solution not on their list? From what I recall they'll cover your initial egress costs.

Tonyjosh 3y ago

[flagged]

unixhero 3y ago

July 1st is not plenty of time as a heads up.

unixhero 3y ago

Just get sponsored by an educational institution or a unicorn with plenty of bandwidth and deep pockets.

ronef 3y ago

We will be holding the community call on Wednesday June 7th at 5pm (GMT+2) Planned Agenda * Brief budget and timeline review * Review/discuss all potential options * Brainstorm/Q&A Info to join is in the original discourse post. Thank you again for jumping all in on this with us!