9 ms·

Super Colliding Nix Stores: Nix Flakes for Millions of Developers

112 points by osener 3y ago

peterldowns 3y ago
I think the best part about Nix is that it makes it super easy to get software into your development environment. This layered store is a pretty cool setup to make that efficient for Replit. Not sure how if Nix/Replit has the program you’re looking for? Or what package name to use to actually get the program you want? Check out my cli search tool: https://github.com/peterldowns/nix-search-cli https://github.com/peterldowns/nix-search-cli Edit: I can’t help myself from ranting, it’s insane that I had to write this tool. Speaks to the lack of user focus in the Nix community. Even the blog post here doesn’t mention how to actually find packages to install! What good is a package manager that makes finding packages difficult? Who should be expected to tolerate such a thing?
- ghuntley 3y ago
  or https://search.nixos.org https://search.nixos.org if folks want a easy (and good way) to get up and running in the nix ecosystem w/services ala replace docker compose then https://devenv.sh https://devenv.sh is the shizz
  - soraminazuki 3y ago
    Also, package search is also prominently featured in the NixOS homepage for folks who don't want to remember the URL.
- otabdeveloper4 3y ago
  > Speaks to the lack of user focus in the Nix community. Not really. Nix is not a tool, it's a toolkit for making your own tools. Very much in the vein of Unix, shell, etc. Many people don't use Nix for "installing" stuff. I personally use it for deploying software to servers without the (congnitive and performance) overhead of containers. I find search.nixos.org immensely useful, but don't really need it from the CLI.
  - sterlind 3y ago
    > Not really. Nix is not a tool, it's a toolkit for making your own tools. Very much in the vein of Unix, shell, etc even tools for tools should be ergonomic to use. the Nix language server should have autocomplete, searches and interacting with the language should be easy, etc. and nearly everyone needs to hunt for packages, even if they're not "installing" them - they might be needed for the toolchain. Nix is like early Git. all the raw power is there, but little of the porcelain. it's so much better than anything before it that we accept the pain of dealing with exposed plumbing, but we shouldn't get used to it.
    - ghuntley 3y ago
      The last two years have seen an increased focus on usability, education and polish now that after circa 15 years of establishing solid foundations are in place. All good things in time. There’s no venture backed cash here paying for the polish that folks typically see in devtools launched in 2023. That polish is expensive to do and isn’t “fun or engaging” for the majority of developers thus classic open source problem of over indexing on serving the needs/existing people who do not need the polish. There’s some notable call-outs here of folks putting in serious effort such as https://nix.dev https://nix.dev ala https://cachix.org https://cachix.org out of love for nix to succeed. Yeah nix wants to grow, and it is, largest bazaar GitHub community/repo out there right now. starting to see venture backed companies forming who are building the polish as porcelain over the top and giving back. see https://floxdev.com/ https://floxdev.com/ at this stage nix is a damn safe bet because of what it can do, the problems it solves and the size of community.
      - pxc 3y ago
        > at this stage nix is a damn safe bet because of what it can do, the problems it solves and the size of community. In the last few years, Nix has gone from a place where I was sometimes uncomfortable recommending it to people to one where I feel like it would be unstrategic not to use it where it fits. If you know Nix today, you should be pushing it at your company wherever you can see a good use case.
- sterlind 3y ago
  why does it query search.nixos.org, rather than looking through your local nixpkgs? just because of how tricky it'd be to handle the nix language or hacking on the nix toolchain proper? also I agree that it's ridiculous nix doesn't have a search command. it baffles me.
  - ghuntley 3y ago
    search.nixos.org also includes flakes that people PR into the index via GitHub at https://github.com/NixOS/nixos-search/issues https://github.com/NixOS/nixos-search/issues ps: $ nix search exists via experimental flags https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-search.html https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3...
  - lilyball 3y ago
    It does though. `nix search`.
  - peter_l_downs 3y ago
    This exists and searches `search.nixos.org` for a few reasons: - you may want to search without downloading and indexing nixpkgs and the hydra results yourself - you may want to search by the name of the binaries/programs that will be installed, which is impossible to do by looking through a local package store alone - you may want the search to be fast - you may want to search from the command line `nix search` is a cruel joke which doesn't allow searching by the name/program that would be installed, only package name, and requires a flake name every time. Absolutely terrible interface. My tool is a single-install binary that performs fast and accurate search to help you find the right package name to install a given binary. I don't understand how after years of using other package managers anyone could want a search tool that does anything other than this by default. For more details on why this exists, check out https://github.com/peterldowns/nix-search-cli#motivation https://github.com/peterldowns/nix-search-cli#motivation
  - aidenn0 3y ago
    Nix has several search commands, it's just none of them do quite what people want (e.g. "nix search" with flakes and "nix-query" without flakes search by name and description, "nix-locate" can search by contents but the interface is a bit obtuse) For those who want a local search for executables, and don't mind running nix-index: #!/bin/sh results="$(nix-locate -1 --regex --top-level "^/bin/${1}\$")" for item in $results; do nix search "nixpkgs#${item%.*}" done will search for a specific executable and give you (hopefully) reasonable output. If you don't have flakes enabled, you can just print the result of the command-substitution directly, but you won't get the short description.
- jljljl 3y ago
  Finding a specific version of a package is also really difficult. It requires you to find the commit hash in the Nixpkgs repo that last contained it, and there isn't a central search engine that lets you map version to commit. We ended up building package version search into the Devbox CLI for this reason (https://www.jetpack.io/devbox/docs/guides/pinning_packages/#searching-for-available-packages https://www.jetpack.io/devbox/docs/guides/pinning_packages/#...). There's also a 3rd party site that lets you search for Nix package versions (https://lazamar.co.uk/nix-versions/ https://lazamar.co.uk/nix-versions/)
  - peter_l_downs 3y ago
    I'll have to check out devbox, thanks for the link. I don't understand why Nix doesn't have some way for derivations to claim that they install certain artifacts, which would make searching so much easier. The version thing is also really annoying, thanks for the helpful link, I hadn't seen that. Not sure how this will work in a Flakes world. Seems weird to me that sometimes packages are important enough to have multiple derivations each installing a different version (like the pythonFull packages) but there is no standard/good/default solution for it.
    - jljljl 3y ago
      Versioning is a bit easier with Flakes because they're generally published in their own Git repository, meaning you can use tags or branch refs to publish/install specific versions of a package. You can usually do something like `github:org/repo/ref#package` to get what you need.
      - peter_l_downs 3y ago
        Yes, but how will this work for the `nixpkgs` repository? Still very unclear! I'm 100% onboard with flakes in general, and use flakes to describe devshells + publish binaries when applicable -- like with the tool I linked in my earlier post.
  - NathanOsullivan 3y ago
    Normally I would eye-roll at these kind of self promotions but it looks like you are trying to help with a real pain point of Nix. I put a good amount of time getting to grips with "raw" nix with the I imagine common yo-yo-ing between "I don't get it" and "oh I see, this is great" but when I realised how the intersection of nixpkgs and package versioning actually worked.. I was done. For a tool that from the outside seems is so heavily focused on immutability to just continually throw away old versions in nixpkgs head is a head scratcher, and as a monorepo isn't a great fit for utilising different revisions for different packages either. I can only guess that due to single repo containing every package it wasn't seen as practical to just continually append versions to, but when the diff log is just full of 'delete version X URL and it's hash, add X+1 and new hash' from the outside at least, it felt like a real missed opportunity.
- adarsh-krishna 3y ago
  Ended up building a tool which indexes every version of every package into a searchable tool. Was hacked together in a weekend, so the UI needs some polishing, but it's functional :) https://nixdex.dev https://nixdex.dev
- deleted 3y ago
  [deleted]
- jcpsimmons 3y ago
  What is confusing about finding Nix packages? Just go to search.nixos.org There has been a trend in software dev towards solving problems that don’t need solved. I’ve never been confused about how to find packages for nix.
  - soraminazuki 3y ago
    It's genuinely nice to have a CLI frontend for search.nixos.org, so I wouldn't call it a problem that doesn't need solving. But the rant about not being able to search packages is simply wrong IMHO.
  - tadfisher 3y ago
    Or `nix search nixpkgs <query>`
- toastal 3y ago
  That's a use case, but for from the best. Nix lets you create immutable, reproducible packages/OSs/deploys--and can be built on arbitrary fleets of machines with shared caches. Just bringing in tools to the developer environment means the distribution of its tools is immutable but everything you plan to do with it inside that environment is mututable. The forest is missed for the trees if folks think dev shells are the killer feature.
  - pxc 3y ago
    devShells are nice because they have a really low investment to benefit ratio. They're nice because they can often be a clear win for many teams even if they have little prior Nix experience. But you're right! Deeper investment, starting with actually performing your builds with Nix, yields greater benefits, and teams that never take that next step miss out on that.
    - toastal 3y ago
      I wouldn’t disagree, but it’s like stepping into the kiddy pool without bothering to learn to swim to unlock all the other opportunities to swim (and possibly save others too).
- masfrost 3y ago
  If you try to run a binary from replit's shell it will find a package to run it with and will prompt you if there are multiple options
smasher164 3y ago
I love Nix as a package manager, and flakes are awesome for setting up environments. I wish it had more affordances for being a proper build system. If only there were a Buck2 and Nix hybrid.
- ghuntley 3y ago
  would love it if the compilation closure was more incremental rather than at a derivation level. specifically caching sub modules. many folks working on improving this space. last week benchmarks from tvnix (a rust rewrite of the nix cli) was published which saw a THIRTEEN second evaluation speed up on a basic derivation evaluation. https://twitter.com/matthewcroughan/status/1660613835655348228?s=46 https://twitter.com/matthewcroughan/status/16606138356553482...
  - pxc 3y ago
    I saw that tweet and thought it was pretty exciting! How much of Nixpkgs can Tvix build these days?
    - RaitoBezarius 3y ago
      Those benchmarks are wrong and misleading. Tvix contributor (but mostly reviewer I would say) here. We still don't have finished the integration with our own store, using the Nix store forces us to be slower than Nix, therefore the claims are bogus alas. Tvix can *evaluate* a serious (?) chunk of Nixpkgs, but we have strictness bugs in some areas. Tvix has no builder yet.
      - pxc 3y ago
        Well, congrats to y'all on good progress with the evaluator, then! That's definitely still worth celebrating. :)
- chills 3y ago
  > If only there were a Buck2 and Nix hybrid You'll be interested in https://github.com/thoughtpolice/buck2-nix https://github.com/thoughtpolice/buck2-nix
Stammon 3y ago
So do they share their implementation of overlaying stores somewhere? It would be a huge benefit for the community to have it open source and available for being used in countless other use cases.
- bb010g 3y ago
  > In the coming weeks, we will be developing the Layered Store features, releasing a Nix Community RFC, and working to upstream it into Nix. Doesn't look like specs or implementations are public yet.
woleium 3y ago
But oh my, the security implications. Nix is great, but there's no chain of trust on where these packages come from. Did we learn nothing from 'left_pad'?
- tomberek 3y ago
  Other than the very explicit tracking of the packages, their dependencies, the full build instructions, the public history, cryptographic signatures for the pre-built binaries, and the trivial ability for anyone to re-build from source and audit the entire chain all the way from bootstrap if they wished? I’m not sure what golden standard we are comparing this to. It is not perfect, but I’d say this is a far more solid bedrock upon which to build software than anything else I’ve encountered.
  - RaitoBezarius 3y ago
    Did you mention sandboxed builds? Actively preventing pip to execute code arbitrarily at install time? Same for Node.js? I cannot live without that anymore, that's the new golden standard.
    - tomberek 3y ago
      It’s funny; I forgot to mention sandboxed builds because at this point I’ve forgotten that they are not ubiquitous. Oops. Thanks!
  - woleium 3y ago
    Heh, til. I'll have to take a closer look at the platform. Thank you!
  - pxc 3y ago
    > I’m not sure what golden standard we are comparing this to. There isn't any, of course. But one high standard of excellence that we should celebrate and look up to (or just build upon!) is the Guix full-source bootstrap! They've achieved something pretty incredible in terms of supply chain auditability: https://guix.gnu.org/blog/2023/the-full-source-bootstrap-building-from-source-all-the-way-down/ https://guix.gnu.org/blog/2023/the-full-source-bootstrap-bui... Maybe we could also do more with Trustix, and/or build in something like `guix challenge` to the new Nix CLI, to make verifying package provenance and contents easier for users. > I’d say this is a far more solid bedrock upon which to build software than anything else I’ve encountered. With the exception (in some areas) of our little sister, Guix, absolutely. It's a real relief to have such a predictable, inspectable system as you get on NixOS. The drive for standards like software bills of materials comes from the chaos of the old world, the clumsy ways of building and distributing software, that proper functional package management like Nix obsoletes. (Unfortunately we do still cope with and suffer from those old ways in Nixpkgs where the upstream build systems inflict it on us. The JVM and .NET come to mind.)
tormeh 3y ago
Nix Flakes are deemed experimental and explicitly not recommended in prod by the Nix project, afaict. Despite these warnings it’s widely used, but breakage is very much a possibility because it hasn’t gone through the Nix RFC process.
- breakds 3y ago
  Have been using flakes in production for more than 2 years - I think it is stable enough for daily use.
- tomberek 3y ago
  Stabilization of flakes and certain other experimental features is a top priority for the Nix team at the moment (disclaimer: myself included). Please take a look at RFC136 for an outline of our plan for the way ahead…… and please comment.
  - JamesSwift 3y ago
    From the github issue [1]: First, the non-Flakes CLI wll be stabilized, in phases. Afterwards, Flakes itself and its CLI components can be stabilized. The final design of Flakes will also require another RFC. That seems like Flakes are still quite a ways away. [1] - https://github.com/NixOS/rfcs/pull/136 https://github.com/NixOS/rfcs/pull/136
- whateveracct 3y ago
  Where did anyone say "not recommended in prod"? The experimental tag is mostly saying they may change the API when they upgrade Nix itself. Also, "in prod" is such an odd thing to say about Nix. It's a built-time thing. Nix doesn't run in prod. Even NixOS isn't running Nix code. It's running systemd units.
  - rgoulter 3y ago
    > Also, "in prod" is such an odd thing to say about Nix. It's a built-time thing. Nix doesn't run in prod. What people mean by "use X in production" is that X is a key dependency as part of the production process. If you describe a NixOS configuration for the system where your services are running, you're "using" Nix, even though the Nix-specific stuff is already done. This relates to risk. -- If it turned out that using Nix to build packages or to configure a system then caused problems when running services later, that would be a problem. Whereas, "using in prod" suggests confidence in no problems, or in being able to fix any issue in a timely manner.
deleted 3y ago
[deleted]
nwmcsween 3y ago
A less painful alternative to dancing around the issue is just to use the filesystem itself as a global database.
hbrundage 3y ago
Dang this is cool! I get why replit went so heavy on nix but I also feel like it must have a cost for them — nix is hard to learn, especially for folks new to development which I know makes up a lot of replits customer base. We built a solution to the same problem with a similar approach[1], but that just snapshots any old files instead of doing nix derivations. Nix couples the build process to the content-addressability of the output, which works great if you want to put all the effort in to deterministic builds. We just read files like git does which works great for non-deterministic processes like npm install (tragically). I like the idea of the Big Disk style of attaching a content addressable cache, but in our experiments we still found the network latency to the attached disk too high when reading file by file, like when booting a node app, so we’re caching a much smaller amount on a local SSD for each prod server. Maybe replit isn’t as sensitive to read perf from the cache layer, or they have fancy local per-node read through caching within the overlay setup? Regardless, cool!! [1]: https://github.com/gadget-inc/dateilager https://github.com/gadget-inc/dateilager
firstlink 3y ago
Nix is solving the wrong problems, and I'm dismayed that it is being adopted more and more to solve those wrong problems. The very concept of a "development environment" is indicative of broken build systems. Ecosystems like python are trying desperately to tame the insanity using venvs, while ecosystems like rust were built correctly from the start with Cargo. Neither of these benefits in the least way from Nix or the most popular alternative, Docker (except that both end up falling victim to the insanity of other ecosystems when FFI comes into play). What Nix could have been was a global un-fscked dependency manager, a la Portage but with Nix's special sauce for local environments -- thereby working around fundamentally and permanently broken ecosystems like C, which would be useful even in the FFI case. But somehow it never got around to actually managing dependencies. By failing to manage dependencies (by pretending that versioning doesn't exist, and supporting only exact software sets), Nix only re-entrenches the same bad behavior and broken systems which it should have been replacing: i.e., encouraging "works-on-my-system" build environments, by making them part of mandatory collaboration workflows. So, soon: Encounter a bug because the build depends on the environment in some inappropriate, insane way? Then it's your own fault for not adhering to the approved Nix build instructions which would set up a system which doesn't exhibit that bug.
- colordrops 3y ago
  Is this a real problem? I've been using Nix for a year and a half constantly updating on 3 systems and haven't seen the issues you describe.
- kaba0 3y ago
  There is a lot to unpack here — first of all, Nix and Docker are not at all competitors. Docker doesn’t even try to be reproducible, most builds will just apt install the newest available version that could easily change between invocations. Next, Cargo is not a solution: it can only do a tiny part of the problem, namely managing Rust dependencies for a Rust project. What if you are building a tool that needs a shared lib to communicate with.. basically anything? Rust has an enthusiastic community, but it is still a tiny language — most problems require more than what can be found in any one language’s berks. Also, what even is a version? 3.4 of package X doesn’t mean anything — if it has like 10 flags than it can be built 2^10 different ways, and then we didn’t even count its dependencies that are also not just a number. Nix does the correct thing, not the naive one.
npub1zenn0 3y ago
I don't get how people find nix useful. Setting up a python environment, for example, is a total nightmare. Nix lang itself seems completely unnecessary. Guix is using scheme, which I assume people would be more incentivized to learn just because it's a general programming language. So why is nix winning?
- peterbecich 3y ago
  I cannot comment on Guix versus Nix. I do agree the Nix language is unusual. For instance, this statement: buildInputs = with pkgs;[ python34 python34Packages.docopt ]; In any other language, the first semi-colon in that position would be the end of the statement. But in Nix, it is not. The writer of the Nix expression does have to pay a significant cost, in my opinion. If done correctly, the users of the Nix expression can benefit enormously. Using a Nix project can be simple. These commands: nix flake show nix build ... nix run ... nix flake check cover a lot of ground, and don't necessitate understanding the Nix language. It is the best way I know to synchronize a team of developers and provide them with identical working environments, on their own operating systems. A very complex application written in multiple languages with multiple services can be set up with a Nix expression. The application developer then needs only the commands above to build, run, and test the application. Furthermore, Nix can provide the compiler and IDE support (such as an LSP server) to the developer.

112 points by osener 3y ago

I think the best part about Nix is that it makes it super easy to get software into your development environment. This layered store is a pretty cool setup to make that efficient for Replit. Not sure how if Nix/Replit has the program you’re looking for? Or what package name to use to actually get the program you want? Check out my cli search tool: https://github.com/peterldowns/nix-search-cli https://github.com/peterldowns/nix-search-cli Edit: I can’t help myself from ranting, it’s insane that I had to write this tool. Speaks to the lack of user focus in the Nix community. Even the blog post here doesn’t mention how to actually find packages to install! What good is a package manager that makes finding packages difficult? Who should be expected to tolerate such a thing?

ghuntley 3y ago
or https://search.nixos.org https://search.nixos.org if folks want a easy (and good way) to get up and running in the nix ecosystem w/services ala replace docker compose then https://devenv.sh https://devenv.sh is the shizz
- soraminazuki 3y ago
  Also, package search is also prominently featured in the NixOS homepage for folks who don't want to remember the URL.
otabdeveloper4 3y ago
> Speaks to the lack of user focus in the Nix community. Not really. Nix is not a tool, it's a toolkit for making your own tools. Very much in the vein of Unix, shell, etc. Many people don't use Nix for "installing" stuff. I personally use it for deploying software to servers without the (congnitive and performance) overhead of containers. I find search.nixos.org immensely useful, but don't really need it from the CLI.
- sterlind 3y ago
  > Not really. Nix is not a tool, it's a toolkit for making your own tools. Very much in the vein of Unix, shell, etc even tools for tools should be ergonomic to use. the Nix language server should have autocomplete, searches and interacting with the language should be easy, etc. and nearly everyone needs to hunt for packages, even if they're not "installing" them - they might be needed for the toolchain. Nix is like early Git. all the raw power is there, but little of the porcelain. it's so much better than anything before it that we accept the pain of dealing with exposed plumbing, but we shouldn't get used to it.
  - ghuntley 3y ago
    The last two years have seen an increased focus on usability, education and polish now that after circa 15 years of establishing solid foundations are in place. All good things in time. There’s no venture backed cash here paying for the polish that folks typically see in devtools launched in 2023. That polish is expensive to do and isn’t “fun or engaging” for the majority of developers thus classic open source problem of over indexing on serving the needs/existing people who do not need the polish. There’s some notable call-outs here of folks putting in serious effort such as https://nix.dev https://nix.dev ala https://cachix.org https://cachix.org out of love for nix to succeed. Yeah nix wants to grow, and it is, largest bazaar GitHub community/repo out there right now. starting to see venture backed companies forming who are building the polish as porcelain over the top and giving back. see https://floxdev.com/ https://floxdev.com/ at this stage nix is a damn safe bet because of what it can do, the problems it solves and the size of community.
    - pxc 3y ago
      > at this stage nix is a damn safe bet because of what it can do, the problems it solves and the size of community. In the last few years, Nix has gone from a place where I was sometimes uncomfortable recommending it to people to one where I feel like it would be unstrategic not to use it where it fits. If you know Nix today, you should be pushing it at your company wherever you can see a good use case.
sterlind 3y ago
why does it query search.nixos.org, rather than looking through your local nixpkgs? just because of how tricky it'd be to handle the nix language or hacking on the nix toolchain proper? also I agree that it's ridiculous nix doesn't have a search command. it baffles me.
- ghuntley 3y ago
  search.nixos.org also includes flakes that people PR into the index via GitHub at https://github.com/NixOS/nixos-search/issues https://github.com/NixOS/nixos-search/issues ps: $ nix search exists via experimental flags https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-search.html https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3...
- lilyball 3y ago
  It does though. `nix search`.
- peter_l_downs 3y ago
  This exists and searches `search.nixos.org` for a few reasons: - you may want to search without downloading and indexing nixpkgs and the hydra results yourself - you may want to search by the name of the binaries/programs that will be installed, which is impossible to do by looking through a local package store alone - you may want the search to be fast - you may want to search from the command line `nix search` is a cruel joke which doesn't allow searching by the name/program that would be installed, only package name, and requires a flake name every time. Absolutely terrible interface. My tool is a single-install binary that performs fast and accurate search to help you find the right package name to install a given binary. I don't understand how after years of using other package managers anyone could want a search tool that does anything other than this by default. For more details on why this exists, check out https://github.com/peterldowns/nix-search-cli#motivation https://github.com/peterldowns/nix-search-cli#motivation
- aidenn0 3y ago
  Nix has several search commands, it's just none of them do quite what people want (e.g. "nix search" with flakes and "nix-query" without flakes search by name and description, "nix-locate" can search by contents but the interface is a bit obtuse) For those who want a local search for executables, and don't mind running nix-index: #!/bin/sh results="$(nix-locate -1 --regex --top-level "^/bin/${1}\$")" for item in $results; do nix search "nixpkgs#${item%.*}" done will search for a specific executable and give you (hopefully) reasonable output. If you don't have flakes enabled, you can just print the result of the command-substitution directly, but you won't get the short description.
jljljl 3y ago
Finding a specific version of a package is also really difficult. It requires you to find the commit hash in the Nixpkgs repo that last contained it, and there isn't a central search engine that lets you map version to commit. We ended up building package version search into the Devbox CLI for this reason (https://www.jetpack.io/devbox/docs/guides/pinning_packages/#searching-for-available-packages https://www.jetpack.io/devbox/docs/guides/pinning_packages/#...). There's also a 3rd party site that lets you search for Nix package versions (https://lazamar.co.uk/nix-versions/ https://lazamar.co.uk/nix-versions/)
- peter_l_downs 3y ago
  I'll have to check out devbox, thanks for the link. I don't understand why Nix doesn't have some way for derivations to claim that they install certain artifacts, which would make searching so much easier. The version thing is also really annoying, thanks for the helpful link, I hadn't seen that. Not sure how this will work in a Flakes world. Seems weird to me that sometimes packages are important enough to have multiple derivations each installing a different version (like the pythonFull packages) but there is no standard/good/default solution for it.
  - jljljl 3y ago
    Versioning is a bit easier with Flakes because they're generally published in their own Git repository, meaning you can use tags or branch refs to publish/install specific versions of a package. You can usually do something like `github:org/repo/ref#package` to get what you need.
    - peter_l_downs 3y ago
      Yes, but how will this work for the `nixpkgs` repository? Still very unclear! I'm 100% onboard with flakes in general, and use flakes to describe devshells + publish binaries when applicable -- like with the tool I linked in my earlier post.
- NathanOsullivan 3y ago
  Normally I would eye-roll at these kind of self promotions but it looks like you are trying to help with a real pain point of Nix. I put a good amount of time getting to grips with "raw" nix with the I imagine common yo-yo-ing between "I don't get it" and "oh I see, this is great" but when I realised how the intersection of nixpkgs and package versioning actually worked.. I was done. For a tool that from the outside seems is so heavily focused on immutability to just continually throw away old versions in nixpkgs head is a head scratcher, and as a monorepo isn't a great fit for utilising different revisions for different packages either. I can only guess that due to single repo containing every package it wasn't seen as practical to just continually append versions to, but when the diff log is just full of 'delete version X URL and it's hash, add X+1 and new hash' from the outside at least, it felt like a real missed opportunity.
adarsh-krishna 3y ago
Ended up building a tool which indexes every version of every package into a searchable tool. Was hacked together in a weekend, so the UI needs some polishing, but it's functional :) https://nixdex.dev https://nixdex.dev
deleted 3y ago
[deleted]
jcpsimmons 3y ago
What is confusing about finding Nix packages? Just go to search.nixos.org There has been a trend in software dev towards solving problems that don’t need solved. I’ve never been confused about how to find packages for nix.
- soraminazuki 3y ago
  It's genuinely nice to have a CLI frontend for search.nixos.org, so I wouldn't call it a problem that doesn't need solving. But the rant about not being able to search packages is simply wrong IMHO.
- tadfisher 3y ago
  Or `nix search nixpkgs <query>`
toastal 3y ago
That's a use case, but for from the best. Nix lets you create immutable, reproducible packages/OSs/deploys--and can be built on arbitrary fleets of machines with shared caches. Just bringing in tools to the developer environment means the distribution of its tools is immutable but everything you plan to do with it inside that environment is mututable. The forest is missed for the trees if folks think dev shells are the killer feature.
- pxc 3y ago
  devShells are nice because they have a really low investment to benefit ratio. They're nice because they can often be a clear win for many teams even if they have little prior Nix experience. But you're right! Deeper investment, starting with actually performing your builds with Nix, yields greater benefits, and teams that never take that next step miss out on that.
  - toastal 3y ago
    I wouldn’t disagree, but it’s like stepping into the kiddy pool without bothering to learn to swim to unlock all the other opportunities to swim (and possibly save others too).
masfrost 3y ago
If you try to run a binary from replit's shell it will find a package to run it with and will prompt you if there are multiple options

smasher164 3y ago

I love Nix as a package manager, and flakes are awesome for setting up environments. I wish it had more affordances for being a proper build system. If only there were a Buck2 and Nix hybrid.

ghuntley 3y ago
would love it if the compilation closure was more incremental rather than at a derivation level. specifically caching sub modules. many folks working on improving this space. last week benchmarks from tvnix (a rust rewrite of the nix cli) was published which saw a THIRTEEN second evaluation speed up on a basic derivation evaluation. https://twitter.com/matthewcroughan/status/1660613835655348228?s=46 https://twitter.com/matthewcroughan/status/16606138356553482...
- pxc 3y ago
  I saw that tweet and thought it was pretty exciting! How much of Nixpkgs can Tvix build these days?
  - RaitoBezarius 3y ago
    Those benchmarks are wrong and misleading. Tvix contributor (but mostly reviewer I would say) here. We still don't have finished the integration with our own store, using the Nix store forces us to be slower than Nix, therefore the claims are bogus alas. Tvix can *evaluate* a serious (?) chunk of Nixpkgs, but we have strictness bugs in some areas. Tvix has no builder yet.
    - pxc 3y ago
      Well, congrats to y'all on good progress with the evaluator, then! That's definitely still worth celebrating. :)
chills 3y ago
> If only there were a Buck2 and Nix hybrid You'll be interested in https://github.com/thoughtpolice/buck2-nix https://github.com/thoughtpolice/buck2-nix

Stammon 3y ago

So do they share their implementation of overlaying stores somewhere? It would be a huge benefit for the community to have it open source and available for being used in countless other use cases.

bb010g 3y ago
> In the coming weeks, we will be developing the Layered Store features, releasing a Nix Community RFC, and working to upstream it into Nix. Doesn't look like specs or implementations are public yet.

woleium 3y ago

But oh my, the security implications. Nix is great, but there's no chain of trust on where these packages come from. Did we learn nothing from 'left_pad'?

tomberek 3y ago
Other than the very explicit tracking of the packages, their dependencies, the full build instructions, the public history, cryptographic signatures for the pre-built binaries, and the trivial ability for anyone to re-build from source and audit the entire chain all the way from bootstrap if they wished? I’m not sure what golden standard we are comparing this to. It is not perfect, but I’d say this is a far more solid bedrock upon which to build software than anything else I’ve encountered.
- RaitoBezarius 3y ago
  Did you mention sandboxed builds? Actively preventing pip to execute code arbitrarily at install time? Same for Node.js? I cannot live without that anymore, that's the new golden standard.
  - tomberek 3y ago
    It’s funny; I forgot to mention sandboxed builds because at this point I’ve forgotten that they are not ubiquitous. Oops. Thanks!
- woleium 3y ago
  Heh, til. I'll have to take a closer look at the platform. Thank you!
- pxc 3y ago
  > I’m not sure what golden standard we are comparing this to. There isn't any, of course. But one high standard of excellence that we should celebrate and look up to (or just build upon!) is the Guix full-source bootstrap! They've achieved something pretty incredible in terms of supply chain auditability: https://guix.gnu.org/blog/2023/the-full-source-bootstrap-building-from-source-all-the-way-down/ https://guix.gnu.org/blog/2023/the-full-source-bootstrap-bui... Maybe we could also do more with Trustix, and/or build in something like `guix challenge` to the new Nix CLI, to make verifying package provenance and contents easier for users. > I’d say this is a far more solid bedrock upon which to build software than anything else I’ve encountered. With the exception (in some areas) of our little sister, Guix, absolutely. It's a real relief to have such a predictable, inspectable system as you get on NixOS. The drive for standards like software bills of materials comes from the chaos of the old world, the clumsy ways of building and distributing software, that proper functional package management like Nix obsoletes. (Unfortunately we do still cope with and suffer from those old ways in Nixpkgs where the upstream build systems inflict it on us. The JVM and .NET come to mind.)

tormeh 3y ago

Nix Flakes are deemed experimental and explicitly not recommended in prod by the Nix project, afaict. Despite these warnings it’s widely used, but breakage is very much a possibility because it hasn’t gone through the Nix RFC process.

breakds 3y ago
Have been using flakes in production for more than 2 years - I think it is stable enough for daily use.
tomberek 3y ago
Stabilization of flakes and certain other experimental features is a top priority for the Nix team at the moment (disclaimer: myself included). Please take a look at RFC136 for an outline of our plan for the way ahead…… and please comment.
- JamesSwift 3y ago
  From the github issue [1]: First, the non-Flakes CLI wll be stabilized, in phases. Afterwards, Flakes itself and its CLI components can be stabilized. The final design of Flakes will also require another RFC. That seems like Flakes are still quite a ways away. [1] - https://github.com/NixOS/rfcs/pull/136 https://github.com/NixOS/rfcs/pull/136
whateveracct 3y ago
Where did anyone say "not recommended in prod"? The experimental tag is mostly saying they may change the API when they upgrade Nix itself. Also, "in prod" is such an odd thing to say about Nix. It's a built-time thing. Nix doesn't run in prod. Even NixOS isn't running Nix code. It's running systemd units.
- rgoulter 3y ago
  > Also, "in prod" is such an odd thing to say about Nix. It's a built-time thing. Nix doesn't run in prod. What people mean by "use X in production" is that X is a key dependency as part of the production process. If you describe a NixOS configuration for the system where your services are running, you're "using" Nix, even though the Nix-specific stuff is already done. This relates to risk. -- If it turned out that using Nix to build packages or to configure a system then caused problems when running services later, that would be a problem. Whereas, "using in prod" suggests confidence in no problems, or in being able to fix any issue in a timely manner.

deleted 3y ago

[deleted]

nwmcsween 3y ago

A less painful alternative to dancing around the issue is just to use the filesystem itself as a global database.

hbrundage 3y ago

Dang this is cool! I get why replit went so heavy on nix but I also feel like it must have a cost for them — nix is hard to learn, especially for folks new to development which I know makes up a lot of replits customer base. We built a solution to the same problem with a similar approach[1], but that just snapshots any old files instead of doing nix derivations. Nix couples the build process to the content-addressability of the output, which works great if you want to put all the effort in to deterministic builds. We just read files like git does which works great for non-deterministic processes like npm install (tragically). I like the idea of the Big Disk style of attaching a content addressable cache, but in our experiments we still found the network latency to the attached disk too high when reading file by file, like when booting a node app, so we’re caching a much smaller amount on a local SSD for each prod server. Maybe replit isn’t as sensitive to read perf from the cache layer, or they have fancy local per-node read through caching within the overlay setup? Regardless, cool!! [1]: https://github.com/gadget-inc/dateilager https://github.com/gadget-inc/dateilager

firstlink 3y ago

Nix is solving the wrong problems, and I'm dismayed that it is being adopted more and more to solve those wrong problems. The very concept of a "development environment" is indicative of broken build systems. Ecosystems like python are trying desperately to tame the insanity using venvs, while ecosystems like rust were built correctly from the start with Cargo. Neither of these benefits in the least way from Nix or the most popular alternative, Docker (except that both end up falling victim to the insanity of other ecosystems when FFI comes into play). What Nix could have been was a global un-fscked dependency manager, a la Portage but with Nix's special sauce for local environments -- thereby working around fundamentally and permanently broken ecosystems like C, which would be useful even in the FFI case. But somehow it never got around to actually managing dependencies. By failing to manage dependencies (by pretending that versioning doesn't exist, and supporting only exact software sets), Nix only re-entrenches the same bad behavior and broken systems which it should have been replacing: i.e., encouraging "works-on-my-system" build environments, by making them part of mandatory collaboration workflows. So, soon: Encounter a bug because the build depends on the environment in some inappropriate, insane way? Then it's your own fault for not adhering to the approved Nix build instructions which would set up a system which doesn't exhibit that bug.

colordrops 3y ago
Is this a real problem? I've been using Nix for a year and a half constantly updating on 3 systems and haven't seen the issues you describe.
kaba0 3y ago
There is a lot to unpack here — first of all, Nix and Docker are not at all competitors. Docker doesn’t even try to be reproducible, most builds will just apt install the newest available version that could easily change between invocations. Next, Cargo is not a solution: it can only do a tiny part of the problem, namely managing Rust dependencies for a Rust project. What if you are building a tool that needs a shared lib to communicate with.. basically anything? Rust has an enthusiastic community, but it is still a tiny language — most problems require more than what can be found in any one language’s berks. Also, what even is a version? 3.4 of package X doesn’t mean anything — if it has like 10 flags than it can be built 2^10 different ways, and then we didn’t even count its dependencies that are also not just a number. Nix does the correct thing, not the naive one.

npub1zenn0 3y ago

I don't get how people find nix useful. Setting up a python environment, for example, is a total nightmare. Nix lang itself seems completely unnecessary. Guix is using scheme, which I assume people would be more incentivized to learn just because it's a general programming language. So why is nix winning?

peterbecich 3y ago
I cannot comment on Guix versus Nix. I do agree the Nix language is unusual. For instance, this statement: buildInputs = with pkgs;[ python34 python34Packages.docopt ]; In any other language, the first semi-colon in that position would be the end of the statement. But in Nix, it is not. The writer of the Nix expression does have to pay a significant cost, in my opinion. If done correctly, the users of the Nix expression can benefit enormously. Using a Nix project can be simple. These commands: nix flake show nix build ... nix run ... nix flake check cover a lot of ground, and don't necessitate understanding the Nix language. It is the best way I know to synchronize a team of developers and provide them with identical working environments, on their own operating systems. A very complex application written in multiple languages with multiple services can be set up with a Nix expression. The application developer then needs only the commands above to build, run, and test the application. Furthermore, Nix can provide the compiler and IDE support (such as an LSP server) to the developer.