8 ms·

Tvix: We Are Rewriting Nix

144 points by robto 5y ago

thumbellina 5y ago
I don't know enough about Nix internals to know how much of a problem this is, or the right way to improve it. However, it's good to see an ecosystem forming around the derivation data type, with the possibility of reuse across projects that use derivations. Good stuff! I'd be interested to hear a perspective from the Nix and Guix authors.
- pxc 5y ago
  > it's good to see an ecosystem forming around the derivation data type, with the possibility of reuse across projects that use derivations. Guix used to use the same derivation format as Nix, but they've since diverged
  - johnisgood 5y ago
    Does Guix have any performance issues?
    - mbakke 5y ago
      Sure, just different ones. Compiling Guix and its package modules (aka 'guix pull') takes minutes on powerful hardware, and is nigh impossible on low-end systems. Computing derivations that import a lot of Guile/Guix modules can take some time for the same reason. These computations are cached of course, but still change frequently enough that you'll notice if you have to build them locally (i.e. custom packages not in Guix proper). OTOH, Guix operates on in-memory representations of profiles, packages and derivations, and delays using the store until an actual build needs to be performed; whereas I believe Nix keeps making RPC's to the store during evaluation (please correct me if this is wrong). I don't think adding Guix support to Tvix is as easy as plugging in a Guile evaluator, as the Nix tooling would not be able to understand the higher-level abstractions in Guix. You really need Guix and all its supporting modules to evaluate Guix packages.
      - johnisgood 5y ago
        Thank you! I am wondering, is there a ticket open to follow for optimizing Guix?
  - thumbellina 5y ago
    TIL. Do you have any links that go into more detail about these differences?
    - southerntofu 5y ago
      I don't have a link with low-level details, but this talk highlighted the main differences: https://xana.lepiller.eu/guix-days-2020/guix-days-2020-andrew-tropin-nix-and-guix.mp4 https://xana.lepiller.eu/guix-days-2020/guix-days-2020-andre...
d3nj4l 5y ago
OT for nix, but that's an interesting approach to building an organisation on the internet. They seem to have a good sense of cohesion despite being just a group of people with common interests who hang out on an IRC and are distributed across the world?
- southerntofu 5y ago
  They hang around on hackint, the hacking/hackerspace IRC network. So it's very likely they have some AFK connections as well, and also some common political motivations driving them.
darksaints 5y ago
Is tvix/nix pretty tightly coupled to the Linux kernel? I know there is some effort to use the nix package manager on OSX, but I'm not sure if that is a hack or an actual full port to a different kernel. Could it be used on, say, OpenBSD? SeL4?
- tazjin 5y ago
  For the core parts of Tvix we won't have any OS-specific dependencies, but each platform will need a working builder and sandbox implementation. I suspect that OpenBSD has at least one OCI-compatible sandboxing/containerisation method so it should end up running fine on there.
- akavel 5y ago
  Nix itself is in first approximation "just a package manager", so not particularly tied to a kernel AFAIU. As to NixOS, this one's a Linux distro, so it currently mostly assumes Linux kernel AFAIK. I tried exploring plugging in L4Linux in that place, but failed due to not being able to understand related areas of NixOS/Nixpkgs well enough. Guix OS on the other hand has some (experimental?) support for GNU Mach kernel I believe.
  - pxc 5y ago
    Guix System (the full OS distribution) supports both Linux and GNU Hurd as kernels
    - akavel 5y ago
      Ouch, right, I borked the name of the GNU kernel - sorry and thanks!
  - chriswarbo 5y ago
    > Nix itself is in first approximation "just a package manager" I'd go even further: Nix is just a build tool, like Make. In fact, it assumes far less about the system than Make; e.g. Make does all sorts of shell-related things, whilst Nix simply executes a binary (with a list of arguments and a set of env vars). (Nixpkgs definitions almost always use a bash binary as their executable, so it feels more like Make; but Nix doesn't care)
Valodim 5y ago
So this sounds like it is mostly concerned with the derivation part. Is it going to be based on Nickel then, or a new custom parser? Would be interested in the rationale there
- tazjin 5y ago
  No, we are writing an implementation of the Nix language. Nixpkgs is written in Nix and we believe that's the most important part of the ecosystem, so Tvix will support it.
sidkshatriya 5y ago
There are noble goals to this project to rewriting Nix to improve performance. The reasons feel valid. However I wonder about certain issues: - Nix itself has problems of compatibility with Nix! Let me explain...recently NixOS 21.11 released. It still comes packaged with Nix 2.3.16 instead of Nix 2.4 because of incompatibility issues. When the Nix project itself is not able to deliver backwards compatibility I feel uncertain that Tvix might be able to deliver their promise of full compatibility with all the deep architectural changes it plans. Nix does not have a formal specification: Nix is what its C/C++ implementation says. Nix also has deep architectural changes planned going ahead. So Tvix will need to re-architect Nix keeping in mind (and keeping up) with the future changes that are coming. It's going to be tough to build this "alternative" implementation. Users are going to run into weird compatibility issues, not have the patience to resolve them and then just go back to the original Nix implementation. - A rewrite to a popular project will find it difficult to succeed. There is simply too much momentum in the parent Nix implementation (I'm not talking about the package definitions here) that very few people are likely to adopt this rewrite. This is because most people might not really need the speedup. However, it's possible that the pain around slow evaluation runs so deep that other companies might adopt this too. I don't know enough here. - The rewrite needs to offer more reason to shift. Right now this rewrite seems to be mainly about performance. There is also a proposal to be able to integrate with GNU Guile. To me this is just avoiding one untyped language (Nix) to replace with another slightly better untyped language (Guile Scheme) (GNU Guix uses Guile also AFAIK). What about introducing typing? https://github.com/tweag/nickel https://github.com/tweag/nickel is an interesting project and it would be great to have a system like Nix with optional typing to make writing the derivations more robust. What about rewriting in a safe programming language (e.g. Rust or a GC-ed language)? These things would be likely to excite would-be-contributors and would make your rewrite viable. In other words, the rewrite needs to offer more than just the promise of performance. Perhaps I have not understood the project goals fully. Maybe the project is all about just being able to leverage the Nix package definitions but have a totally different implementation to instantiate them? But the issue here is that nix-the-language is very intimately tied up with the nix-the-platform and the rewritten implementation will still need to really be extremely compatible to capture all the implicit assumptions in nixpkgs. Once again, improving nix through a rewrite is a noble goal. It's a tough problem and it would be awesome if the project succeeds! I have obviously not thought about this as much as you guys and would love to learn how you wish to tackle some of these problems. Maybe some of these issues are not as big as them seem??
throwaway984393 5y ago
Every distribution has the exact same problems that are impossible to prevent. It's all about the inherent limitations of how software is developed and run in operating systems today. All this stuff about 'solving dependency hell' and dependency paths tied up with merkle trees doesn't change the fact that you can't use two completely different libraries in one application at the same time, nor shell out to two incompatible versions of a binary from the same application. It is only possible if the software itself is designed to do that, using the features of a dynamic linker that allow linking in multiple versions of the same function and specifying which version you're calling from what source function. We don't actually need a new package manager or distribution at all, because they can't solve one of the most fundamental problems - only developing software differently can do that. Sorry, but Nix is a total waste of time.
- Ericson2314 5y ago
  You should spend more time in the Nix community, because actually people are very aware of these limitations. For C packages, yes the best we can easily do is ensure each binary has coherent library choices, but allow mixing libraries in different binaries. That is still useful, though! That said, you underestimate the power of being able to hack on any dependency -- for real purpose of Nix is to defeat Conway's law, allowing any person to work on any codebase. It would be quite easy to fork the linker/loader to try out some experiment, or even just do some per-package symbol mangling, for example. I wouldn't want to try that experiment with any other distro, because it would be too hard to make my modification and then put everything together again!
  - throwaway984393 5y ago
    Do you find you need to make experimental changes to the linker often? Back when I was a Gentoo user, I too bought the story that being able to re-emerge my whole system was some kind of performance or configuration boon. Eventually I realized that I just wanted to use my computer to get through life, that literally any operating system would work, and that any experimental hacks could simply be done when they were needed rather than trying to manage an over-complicated, poorly-supported hobby OS. I use Ubuntu now. It sucks. But I spend my time on doing real world things now, rather than constantly tweaking my computer to eventually do a real world thing. For work I use containers with any distribution at all. They all have the same result in a container.
ncmncm 5y ago
What would seem most useful would be a compile-time-typed language (perhaps with some inference propagation, for convenience) that existing Nix code could be translated into, maybe even while emulating execution to discover the actual types to annotate with. You have the advantage that all existing production Nix code is known to terminate in short order. Then, you could generate a new version of the entire 30,000-strong package collection automatically, with the new language swapped in in place of the Nix code, which could then be abandoned.
- choeger 5y ago
  > that existing Nix code could be translated into, It is highly unlikely that this is feasible. I don't know nix in detail but I infer from your post that it is an untyped language. Untyped languages can, in general, not be translated to (statically decidable) typed languages. > which could then be abandoned. It could not, unless you convince a majority of the contributors to the package repo to switch to your new language.
  - southerntofu 5y ago
    > It is highly unlikely that this is feasible Some more complex nix features would not be easy to transpile automatically, but many (i don't have stats sorry) packages are very simple declarations which could be ported automatically. For example, all language-specific libraries imported from rust/python/etc could very easily be automatically ported, and in my experience, writing declarations for those many dependencies is what takes most time in making a small package (although admittedly i only tried once).
  - funcDropShadow 5y ago
    Nix, is a dynamically typed language, i.e. the language gives every value at runtime a type, but there a no compile-time (i.e. static) checks whether any function or operator calls will match at runtime to their parameters. Programs can be translated from dynamically typed-languages to statically typed languages. Either by infering types at translation time or by exhaustively adding lots of cases to every function or operator call depending on the runtime. The first approach can in principle only translate a subset of all programs, the latter does not give you any additional guarantees, because you have encoded an interpreter for the dynamically-type language into the statically typed language. Whether the first approach could be used on a large codebase like nixpkgs is highly questionable, the second approach would not yield any advantages here.
di4na 5y ago
Hey Tazjin, i see you are around that one. Quick question. Can you expand on IFD and integrations? That is one thing i would love to know more about your plans. Also i hope it works, the nox code base is imho the biggest impediment to the nix model being more used. Small advise: please consider making it really easy to contribute to through wll the tooling. This is one of the reason Nix codebase rotted
- southerntofu 5y ago
  > please consider making it really easy to contribute Yes, that's one area where Guix shines in comparison to nix. They use an established language with its standard library (Guile lisp) making it very easy for people to contribute without learning arcane incantations. I would love to see nix/Guix implemented in a statically-typed, high-level language like Rust.
  - di4na 5y ago
    I mean nix is in Cpp so uh... technically...
kripke 5y ago
I don't know much about Nix, although it has been on my radar for some time. The claim that the language performance is bad pops up time and again, and I'm sure many people would be happy to have a faster alternative --- so props to them for trying to address this clearly important problem. That being said, in perusing related issues on GitHub such as [1], I have found no serious attempts to even profile the damn thing. The typical attempt seems to be throwing a random idea at the evaluator and crossing fingers. The example of a "performance experiment" [2] from the article is self-described as follows: > Add an alternative impl of the now-abstract Bindings base class that is backed by a std::vector, somewhat similar but stylistically a little superior to the array-backed implementation in upstream nix. I fail to see how that is a performance experiment when it does not mention performance at all. The post also mentions issues with the general Nix design and a desire to modernize it, which I am unqualified to comment on, but seems a worthwhile endeavor in and of itself. I do hope that they won't end up writing a new evaluator for the Nix language without a proper understanding of what makes the current one slow and a clear plan not to repeat its mistakes, or the new one risks being just as slow as the old one. [1] : https://github.com/NixOS/nix/issues/2652 https://github.com/NixOS/nix/issues/2652 [2] : https://cl.tvl.fyi/c/depot/+/1123/ https://cl.tvl.fyi/c/depot/+/1123/
- tazjin 5y ago
  The background on our side is that we did do quite a lot of profiling and investigation of the current evaluator - last year, informally, over video chats and IRC. We have a bunch of logs and various perf measurements spread out over some machines but right now the knowledge lives in our heads, mostly because at the time we did not expect this to spawn a serious project. Once we get further along with the evaluator we will be incorporating some of that material into more specific blog posts, but it seemed a little too deep for the initial announcement. Regarding that CL linked from the post: It was part of laying the groundwork for trying to have the same Nix language type backed by different implementations; the reason for this is that Nix's attribute sets (similar to dictionaries or k/v maps in other languages) have a wide variety of use-cases with wildly conflicting performance characteristics. For example, one very common use-case is the data structure representing the top-level of nixpkgs: An enormous attribute set with many thousands of keys that is frequently modified & copied. An even more common one is related to the builtin function `listToAttrs` which deals with two-key attribute sets (name, value pairs). In current Nix, these are some of the biggest evaluator hotspots (apart from higher-level issues like cache-unfriendliness due to the tree-walking evaluation style) and optimising for one case vastly diminishes performance of the other. The problem we ran into with all experiments of this form is that there is no clear interface separation in `libexpr` (the part of the Nix codebase that implements the evaluator): Everything is modifying and reading internal state of everything else, including in other parts of Nix that depend on libexpr. It made it basically impossible to really try anything without refactoring huge amounts of (memory-unsafe) code. Now it's important to mention that the reason for this is understandable: Nix is an old project (18 years now I believe?) and I don't think anyone expected it to get to where it is now, so obviously different decisions were made in its implementation than would be made if it were started today knowing everything we know.
MadeThisToReply 5y ago
How do you pronounce "Tvix"?
- tazjin 5y ago
  Твикс - does that help?) It seems that North Americans struggle with the word, but most Europeans (including Brits) don't. Don't know enough about linguistics to guess why! Try saying "Twix" the way you'd imagine a German or Scandinavian accent to sound.
  - ogogmad 5y ago
    > Твикс - does that help? Yes, definitely!! I think it wasn't clear whether you were supposed to say "tee-vix" or /tvɪks/. For non-Russians, try and figure out which of the above two is right. No one help them, because you'll be taking away a learning opportunity. https://www.youtube.com/watch?v=9hNphbAm_Hs https://www.youtube.com/watch?v=9hNphbAm_Hs
kungito 5y ago
I really wish there was a statically typed alternative to nix. Nix docs are horrible and incomplete and the language is called the same as the rest of tye ecosystem so it's hard to google. I don't think there can be a good declarative functional language without static types because how the hell do you debug? I'ts been really hard whenever I have to do something in a bigger nix environment. I get the same feeling like when I jump into a bigger Python project which isn't documented and I have to read around to see what is what, only goto definition doesn't work, the language is weird and everything is some specific function which I don't know is built in or where it's from and googling gives no answers except maybe link to source code
- southerntofu 5y ago
  I strongly agree with your sentiment. Both nix and guix have terrible error messages and searching the web for answers if doomed to failure. Although i must point out both have very friendly communities and asking for help in chat is likely to yield constructive answers. As guix has shown, nix principles can be applied as a DSL/library within your favorite language of choice. I would be curious to see what a "rustix" would look like!
- aseipp 5y ago
  For error messages I think you're ultimately a bit cornered. Even typed languages have serious problems with straightforward and reliable errors, everything else aside. The ones that do tend to have to make careful tradeoffs in the system to achieve that (HKDs probably just straight up make error messages worse, for instance.) That said I think having types would be useful for helping navigate code, but I think you'd need a fairly advanced type system to capture all the "open ended" parts of Nixpkgs, and that's where all the value really is, the tons of little ways people have built up to use it over time. So it's a mixed bag. I sympathize though; I've spent a lot of time on spurious Nix bullshit, even if I still love the tools. As a contributor I actually think that the success of projects like TypeScript have shown that there should be a way to build a type checker for something like Nix, with incremental migration. But it probably isn't going to be the ideal solution we could all imagine and it will be very very difficult nonetheless. But it could work with the code we have now, and that's a very important bar to clear (I don't take some of the other suggestions like "auto translate 30,000 packages to some other better language that doesn't exist & isn't the one people are using now" all too seriously, I'm afraid.) I do think having a store implementation (even a "virtual one") decoupled from the evaluator and builder tooling is a step in the right direction for experimentation and hope that it works out.

144 points by robto 5y ago

I don't know enough about Nix internals to know how much of a problem this is, or the right way to improve it. However, it's good to see an ecosystem forming around the derivation data type, with the possibility of reuse across projects that use derivations. Good stuff! I'd be interested to hear a perspective from the Nix and Guix authors.

pxc 5y ago
> it's good to see an ecosystem forming around the derivation data type, with the possibility of reuse across projects that use derivations. Guix used to use the same derivation format as Nix, but they've since diverged
- johnisgood 5y ago
  Does Guix have any performance issues?
  - mbakke 5y ago
    Sure, just different ones. Compiling Guix and its package modules (aka 'guix pull') takes minutes on powerful hardware, and is nigh impossible on low-end systems. Computing derivations that import a lot of Guile/Guix modules can take some time for the same reason. These computations are cached of course, but still change frequently enough that you'll notice if you have to build them locally (i.e. custom packages not in Guix proper). OTOH, Guix operates on in-memory representations of profiles, packages and derivations, and delays using the store until an actual build needs to be performed; whereas I believe Nix keeps making RPC's to the store during evaluation (please correct me if this is wrong). I don't think adding Guix support to Tvix is as easy as plugging in a Guile evaluator, as the Nix tooling would not be able to understand the higher-level abstractions in Guix. You really need Guix and all its supporting modules to evaluate Guix packages.
    - johnisgood 5y ago
      Thank you! I am wondering, is there a ticket open to follow for optimizing Guix?
- thumbellina 5y ago
  TIL. Do you have any links that go into more detail about these differences?
  - southerntofu 5y ago
    I don't have a link with low-level details, but this talk highlighted the main differences: https://xana.lepiller.eu/guix-days-2020/guix-days-2020-andrew-tropin-nix-and-guix.mp4 https://xana.lepiller.eu/guix-days-2020/guix-days-2020-andre...

d3nj4l 5y ago

OT for nix, but that's an interesting approach to building an organisation on the internet. They seem to have a good sense of cohesion despite being just a group of people with common interests who hang out on an IRC and are distributed across the world?

southerntofu 5y ago
They hang around on hackint, the hacking/hackerspace IRC network. So it's very likely they have some AFK connections as well, and also some common political motivations driving them.

darksaints 5y ago

Is tvix/nix pretty tightly coupled to the Linux kernel? I know there is some effort to use the nix package manager on OSX, but I'm not sure if that is a hack or an actual full port to a different kernel. Could it be used on, say, OpenBSD? SeL4?

tazjin 5y ago
For the core parts of Tvix we won't have any OS-specific dependencies, but each platform will need a working builder and sandbox implementation. I suspect that OpenBSD has at least one OCI-compatible sandboxing/containerisation method so it should end up running fine on there.
akavel 5y ago
Nix itself is in first approximation "just a package manager", so not particularly tied to a kernel AFAIU. As to NixOS, this one's a Linux distro, so it currently mostly assumes Linux kernel AFAIK. I tried exploring plugging in L4Linux in that place, but failed due to not being able to understand related areas of NixOS/Nixpkgs well enough. Guix OS on the other hand has some (experimental?) support for GNU Mach kernel I believe.
- pxc 5y ago
  Guix System (the full OS distribution) supports both Linux and GNU Hurd as kernels
  - akavel 5y ago
    Ouch, right, I borked the name of the GNU kernel - sorry and thanks!
- chriswarbo 5y ago
  > Nix itself is in first approximation "just a package manager" I'd go even further: Nix is just a build tool, like Make. In fact, it assumes far less about the system than Make; e.g. Make does all sorts of shell-related things, whilst Nix simply executes a binary (with a list of arguments and a set of env vars). (Nixpkgs definitions almost always use a bash binary as their executable, so it feels more like Make; but Nix doesn't care)

Valodim 5y ago

So this sounds like it is mostly concerned with the derivation part. Is it going to be based on Nickel then, or a new custom parser? Would be interested in the rationale there

tazjin 5y ago
No, we are writing an implementation of the Nix language. Nixpkgs is written in Nix and we believe that's the most important part of the ecosystem, so Tvix will support it.

sidkshatriya 5y ago

There are noble goals to this project to rewriting Nix to improve performance. The reasons feel valid. However I wonder about certain issues: - Nix itself has problems of compatibility with Nix! Let me explain...recently NixOS 21.11 released. It still comes packaged with Nix 2.3.16 instead of Nix 2.4 because of incompatibility issues. When the Nix project itself is not able to deliver backwards compatibility I feel uncertain that Tvix might be able to deliver their promise of full compatibility with all the deep architectural changes it plans. Nix does not have a formal specification: Nix is what its C/C++ implementation says. Nix also has deep architectural changes planned going ahead. So Tvix will need to re-architect Nix keeping in mind (and keeping up) with the future changes that are coming. It's going to be tough to build this "alternative" implementation. Users are going to run into weird compatibility issues, not have the patience to resolve them and then just go back to the original Nix implementation. - A rewrite to a popular project will find it difficult to succeed. There is simply too much momentum in the parent Nix implementation (I'm not talking about the package definitions here) that very few people are likely to adopt this rewrite. This is because most people might not really need the speedup. However, it's possible that the pain around slow evaluation runs so deep that other companies might adopt this too. I don't know enough here. - The rewrite needs to offer more reason to shift. Right now this rewrite seems to be mainly about performance. There is also a proposal to be able to integrate with GNU Guile. To me this is just avoiding one untyped language (Nix) to replace with another slightly better untyped language (Guile Scheme) (GNU Guix uses Guile also AFAIK). What about introducing typing? https://github.com/tweag/nickel https://github.com/tweag/nickel is an interesting project and it would be great to have a system like Nix with optional typing to make writing the derivations more robust. What about rewriting in a safe programming language (e.g. Rust or a GC-ed language)? These things would be likely to excite would-be-contributors and would make your rewrite viable. In other words, the rewrite needs to offer more than just the promise of performance. Perhaps I have not understood the project goals fully. Maybe the project is all about just being able to leverage the Nix package definitions but have a totally different implementation to instantiate them? But the issue here is that nix-the-language is very intimately tied up with the nix-the-platform and the rewritten implementation will still need to really be extremely compatible to capture all the implicit assumptions in nixpkgs. Once again, improving nix through a rewrite is a noble goal. It's a tough problem and it would be awesome if the project succeeds! I have obviously not thought about this as much as you guys and would love to learn how you wish to tackle some of these problems. Maybe some of these issues are not as big as them seem??

throwaway984393 5y ago

Every distribution has the exact same problems that are impossible to prevent. It's all about the inherent limitations of how software is developed and run in operating systems today. All this stuff about 'solving dependency hell' and dependency paths tied up with merkle trees doesn't change the fact that you can't use two completely different libraries in one application at the same time, nor shell out to two incompatible versions of a binary from the same application. It is only possible if the software itself is designed to do that, using the features of a dynamic linker that allow linking in multiple versions of the same function and specifying which version you're calling from what source function. We don't actually need a new package manager or distribution at all, because they can't solve one of the most fundamental problems - only developing software differently can do that. Sorry, but Nix is a total waste of time.

Ericson2314 5y ago
You should spend more time in the Nix community, because actually people are very aware of these limitations. For C packages, yes the best we can easily do is ensure each binary has coherent library choices, but allow mixing libraries in different binaries. That is still useful, though! That said, you underestimate the power of being able to hack on any dependency -- for real purpose of Nix is to defeat Conway's law, allowing any person to work on any codebase. It would be quite easy to fork the linker/loader to try out some experiment, or even just do some per-package symbol mangling, for example. I wouldn't want to try that experiment with any other distro, because it would be too hard to make my modification and then put everything together again!
- throwaway984393 5y ago
  Do you find you need to make experimental changes to the linker often? Back when I was a Gentoo user, I too bought the story that being able to re-emerge my whole system was some kind of performance or configuration boon. Eventually I realized that I just wanted to use my computer to get through life, that literally any operating system would work, and that any experimental hacks could simply be done when they were needed rather than trying to manage an over-complicated, poorly-supported hobby OS. I use Ubuntu now. It sucks. But I spend my time on doing real world things now, rather than constantly tweaking my computer to eventually do a real world thing. For work I use containers with any distribution at all. They all have the same result in a container.

ncmncm 5y ago

What would seem most useful would be a compile-time-typed language (perhaps with some inference propagation, for convenience) that existing Nix code could be translated into, maybe even while emulating execution to discover the actual types to annotate with. You have the advantage that all existing production Nix code is known to terminate in short order. Then, you could generate a new version of the entire 30,000-strong package collection automatically, with the new language swapped in in place of the Nix code, which could then be abandoned.

choeger 5y ago
> that existing Nix code could be translated into, It is highly unlikely that this is feasible. I don't know nix in detail but I infer from your post that it is an untyped language. Untyped languages can, in general, not be translated to (statically decidable) typed languages. > which could then be abandoned. It could not, unless you convince a majority of the contributors to the package repo to switch to your new language.
- southerntofu 5y ago
  > It is highly unlikely that this is feasible Some more complex nix features would not be easy to transpile automatically, but many (i don't have stats sorry) packages are very simple declarations which could be ported automatically. For example, all language-specific libraries imported from rust/python/etc could very easily be automatically ported, and in my experience, writing declarations for those many dependencies is what takes most time in making a small package (although admittedly i only tried once).
- funcDropShadow 5y ago
  Nix, is a dynamically typed language, i.e. the language gives every value at runtime a type, but there a no compile-time (i.e. static) checks whether any function or operator calls will match at runtime to their parameters. Programs can be translated from dynamically typed-languages to statically typed languages. Either by infering types at translation time or by exhaustively adding lots of cases to every function or operator call depending on the runtime. The first approach can in principle only translate a subset of all programs, the latter does not give you any additional guarantees, because you have encoded an interpreter for the dynamically-type language into the statically typed language. Whether the first approach could be used on a large codebase like nixpkgs is highly questionable, the second approach would not yield any advantages here.

di4na 5y ago

Hey Tazjin, i see you are around that one. Quick question. Can you expand on IFD and integrations? That is one thing i would love to know more about your plans. Also i hope it works, the nox code base is imho the biggest impediment to the nix model being more used. Small advise: please consider making it really easy to contribute to through wll the tooling. This is one of the reason Nix codebase rotted

southerntofu 5y ago
> please consider making it really easy to contribute Yes, that's one area where Guix shines in comparison to nix. They use an established language with its standard library (Guile lisp) making it very easy for people to contribute without learning arcane incantations. I would love to see nix/Guix implemented in a statically-typed, high-level language like Rust.
- di4na 5y ago
  I mean nix is in Cpp so uh... technically...

kripke 5y ago

I don't know much about Nix, although it has been on my radar for some time. The claim that the language performance is bad pops up time and again, and I'm sure many people would be happy to have a faster alternative --- so props to them for trying to address this clearly important problem. That being said, in perusing related issues on GitHub such as [1], I have found no serious attempts to even profile the damn thing. The typical attempt seems to be throwing a random idea at the evaluator and crossing fingers. The example of a "performance experiment" [2] from the article is self-described as follows: > Add an alternative impl of the now-abstract Bindings base class that is backed by a std::vector, somewhat similar but stylistically a little superior to the array-backed implementation in upstream nix. I fail to see how that is a performance experiment when it does not mention performance at all. The post also mentions issues with the general Nix design and a desire to modernize it, which I am unqualified to comment on, but seems a worthwhile endeavor in and of itself. I do hope that they won't end up writing a new evaluator for the Nix language without a proper understanding of what makes the current one slow and a clear plan not to repeat its mistakes, or the new one risks being just as slow as the old one. [1] : https://github.com/NixOS/nix/issues/2652 https://github.com/NixOS/nix/issues/2652 [2] : https://cl.tvl.fyi/c/depot/+/1123/ https://cl.tvl.fyi/c/depot/+/1123/

tazjin 5y ago
The background on our side is that we did do quite a lot of profiling and investigation of the current evaluator - last year, informally, over video chats and IRC. We have a bunch of logs and various perf measurements spread out over some machines but right now the knowledge lives in our heads, mostly because at the time we did not expect this to spawn a serious project. Once we get further along with the evaluator we will be incorporating some of that material into more specific blog posts, but it seemed a little too deep for the initial announcement. Regarding that CL linked from the post: It was part of laying the groundwork for trying to have the same Nix language type backed by different implementations; the reason for this is that Nix's attribute sets (similar to dictionaries or k/v maps in other languages) have a wide variety of use-cases with wildly conflicting performance characteristics. For example, one very common use-case is the data structure representing the top-level of nixpkgs: An enormous attribute set with many thousands of keys that is frequently modified & copied. An even more common one is related to the builtin function `listToAttrs` which deals with two-key attribute sets (name, value pairs). In current Nix, these are some of the biggest evaluator hotspots (apart from higher-level issues like cache-unfriendliness due to the tree-walking evaluation style) and optimising for one case vastly diminishes performance of the other. The problem we ran into with all experiments of this form is that there is no clear interface separation in `libexpr` (the part of the Nix codebase that implements the evaluator): Everything is modifying and reading internal state of everything else, including in other parts of Nix that depend on libexpr. It made it basically impossible to really try anything without refactoring huge amounts of (memory-unsafe) code. Now it's important to mention that the reason for this is understandable: Nix is an old project (18 years now I believe?) and I don't think anyone expected it to get to where it is now, so obviously different decisions were made in its implementation than would be made if it were started today knowing everything we know.

MadeThisToReply 5y ago

How do you pronounce "Tvix"?

tazjin 5y ago
Твикс - does that help?) It seems that North Americans struggle with the word, but most Europeans (including Brits) don't. Don't know enough about linguistics to guess why! Try saying "Twix" the way you'd imagine a German or Scandinavian accent to sound.
- ogogmad 5y ago
  > Твикс - does that help? Yes, definitely!! I think it wasn't clear whether you were supposed to say "tee-vix" or /tvɪks/. For non-Russians, try and figure out which of the above two is right. No one help them, because you'll be taking away a learning opportunity. https://www.youtube.com/watch?v=9hNphbAm_Hs https://www.youtube.com/watch?v=9hNphbAm_Hs

kungito 5y ago

I really wish there was a statically typed alternative to nix. Nix docs are horrible and incomplete and the language is called the same as the rest of tye ecosystem so it's hard to google. I don't think there can be a good declarative functional language without static types because how the hell do you debug? I'ts been really hard whenever I have to do something in a bigger nix environment. I get the same feeling like when I jump into a bigger Python project which isn't documented and I have to read around to see what is what, only goto definition doesn't work, the language is weird and everything is some specific function which I don't know is built in or where it's from and googling gives no answers except maybe link to source code

southerntofu 5y ago
I strongly agree with your sentiment. Both nix and guix have terrible error messages and searching the web for answers if doomed to failure. Although i must point out both have very friendly communities and asking for help in chat is likely to yield constructive answers. As guix has shown, nix principles can be applied as a DSL/library within your favorite language of choice. I would be curious to see what a "rustix" would look like!
aseipp 5y ago
For error messages I think you're ultimately a bit cornered. Even typed languages have serious problems with straightforward and reliable errors, everything else aside. The ones that do tend to have to make careful tradeoffs in the system to achieve that (HKDs probably just straight up make error messages worse, for instance.) That said I think having types would be useful for helping navigate code, but I think you'd need a fairly advanced type system to capture all the "open ended" parts of Nixpkgs, and that's where all the value really is, the tons of little ways people have built up to use it over time. So it's a mixed bag. I sympathize though; I've spent a lot of time on spurious Nix bullshit, even if I still love the tools. As a contributor I actually think that the success of projects like TypeScript have shown that there should be a way to build a type checker for something like Nix, with incremental migration. But it probably isn't going to be the ideal solution we could all imagine and it will be very very difficult nonetheless. But it could work with the code we have now, and that's a very important bar to clear (I don't take some of the other suggestions like "auto translate 30,000 packages to some other better language that doesn't exist & isn't the one people are using now" all too seriously, I'm afraid.) I do think having a store implementation (even a "virtual one") decoupled from the evaluator and builder tooling is a step in the right direction for experimentation and hope that it works out.