7 ms·

I've had the (dis)pleasure of working with several projects that have been built by developers that have religion around Nix. These projects were contract work

I've had the (dis)pleasure of working with several projects that have been built by developers that have religion around Nix. These projects were contract work where the client paid a significant amount of money, and the final product was really poor quality. One of them is a financial application that has strict security requirements, so having a reproducible build system and some of the other qualities of Nix sounds great, however, the auditor, which is one of the top auditors in this industry had this to say in their report: > Overall, we found the dashboard code to be difficult to review and build. In particular, the manner in which the frontend JavaScript is compiled made it difficult to comprehensively evaluate. Given that this includes the driver code for Ledger devices and handling of user input, we believe this to be an area of concern and poses a significant risk. Building the dashboard took hours on an AWS EC2 VM (with a four-core CPU and 32 GB RAM) and while much of that build time is attributable to Nix's way of building dependencies, the code complexity adds unnecessary risk. We encourage a rework of this area of the project to follow more idiomatic web development practices and patterns. This would make the code more comprehensible and conducive to security evaluations, therefore reducing the risk of vulnerabilities that go unnoticed.

WatchDog 6y ago
I haven't used Nix, but I would have thought that builds would be fast, due to how cache-able the dependencies should be.
- throwaway894345 6y ago
  Sounds like they were building everything from scratch, for some reason.
  - maytc 6y ago
    That’s how nix guarantees reproducibility. The compiled artifacts can be cached but that didnt quite work out of the box
    - throwaway894345 6y ago
      Nix guarantees reproducibility, which means anything can be rebuilt from scratch, but that's a very abnormal use case. If it didn't work out of the box, it's a problem with the package scripts (the "derivations"). That said, all of our software tends to bottom out in a bunch of shitty C libraries that are all delicately cobbled together with autotools and cmake, so anything that aspires to reproduce these things is going to have issues. This tends to make Nix difficult to use, because it doesn't have nearly the same investment/manpower (yet) as other package ecosystems that is necessary to wrangle these dependencies into a stable foundation that doesn't leak its underlying havoc to higher levels of the stack.
      - microphone987 6y ago
        This comment is really, really confusing. If it builds once Nix, it will build again with Nix... If you can find a derivation that builds on one machine and not on the other there's usually a fundamental difference - either in CPU arch, your nixpkgs config differs (overlays), etc. Or I guess you could write a build script that non-deterministic ally fails, but that has nothing to do with Nix's maturity. Nix ensures the tooling is called the same. I can understand having a non-autotools project that is more difficult to write a derivation for, but "Nix can't wrangle messy libraries" makes absolutely no sense, either from a "make compiling reliable" or anything at use-time. Do you have a specific example in mind that is less hand-wavy?
        throwaway894345 6y ago
        > If it builds once Nix, it will build again with Nix... That's the aspiration, but it doesn't always pan out. The rate at which you run into problems depends a lot on the packages you use, how much attention is given to them, and how hard it is to reproducibly package them. I've noticed in particular that the Python ecosystem is really fragile. > Do you have a specific example in mind that is less hand-wavy? A specific example that comes to mind was the psycopg2 Python package, which would build on some developers' machines but not on others. This sort of thing happens all the time, usually on macOS, and usually with C packages (sadly, so much of the Python ecosystem is built on C and its shoddy build tooling). I've also found quite a few packages in nixpkgs that simply don't build on macOS, but which presumably build on Linux; however, I forget which ones specifically.
- mzaccari 6y ago
  We've been using Nix for deploying a Rails app for an enterprise customer for quite a few years now. One area where it shines for us is the ability to build it on relatively recent version of Ubuntu and deploy to a (almost EOL) RHEL6 box. Bundling, assets and various other tasks take just a few minutes. We also have ~20 Go services that are also deployed via Nix, and building takes seconds. However, it can be quite cumbersome to get a Nix expression to the point where it builds reliably for something that takes multiple steps like a Rails app, especially if you're building on macOS and deploying to Linux. It's come a _long_ way in recent years, but with Enterprise customers now embracing containerization we migrated everything to that and haven't looked back.
  - smichel17 6y ago
    I'm having a hard time parsing -- you migrated everything from nix to containers (containers removed the need for nix) or to nix+containers (containers solved the "having to build for multiple platforms" issue)?
    - mzaccari 6y ago
      Ah my apologies - We use Bazel to build our services, and the output artifacts were then pulled into Nix and deployed as Nix packages. Bazel has _excellent_ support taking the same application code and creating Docker images from them (https://github.com/bazelbuild/rules_docker#language-rules https://github.com/bazelbuild/rules_docker#language-rules), and the tools available for deploying containers is orders of magnitude more feature-full and higher quality than what you get with Nix today. So we no longer have Nix anywhere in our pipeline, and all of our artifacts are now deployed inside containers.
- takeda 6y ago
  It is, perhaps they didn't use cache to store previously built packages and build everything from scratch. With no cache Nix will start with building compilers and glibc until it has everything to build the actual application.
- illumin8 6y ago
  While this is true if you are building new versions of dependencies on an existing system, in many cases in high security environments you want to start with a clean build environment every time. Cached dependencies might be considered to be a security risk. Bootstrapping an entire Nix environment can take a lot of time.
  - ryukafalz 6y ago
    Sure, but in most systems you wouldn’t have much of a choice; if you’re using Debian or Ubuntu, you’re probably using binary packages built by someone else. It takes a lot more work to build everything you want from source in a traditional distro. It seems like a pretty straightforward tradeoff: do you want to rely on prebuilt packages for speed, or do you want to rebuild it all (and accept the extra time that takes)? Nix and Guix at least make the latter pretty straightforward to do.
yawaramin 6y ago
Sounds like their main concern wasn't Nix but the complexity of the code itself.
- addicted 6y ago
  The hours long build system appears to be attributed to Nix. Is Nix really that slow?
  - mason55 6y ago
    Dependencies can take awhile if you can’t used cached versions of the binaries. Building some Haskell tooling from scratch took ~3 hours on my MBP.
    - gmfawcett 6y ago
      There are few compilers that are slower than GHC. I suspect this issue has nothing to do with Nix, and the tools would have taken ages to build from scratch without Nix.
  - geofft 6y ago
    If you're applying idiomatic Nix to modern JS, I wouldn't be surprised - modern JS tends to involve installing thousands of packages by just combining them into a directory, but Nix doesn't want you to edit existing directories. So your dependency graph turns into a build-dependency graph, with each JS package requiring a full build of everything it depends on, and the Nix build system is presumably not optimized for fast turnaround times on five-line JS modules that don't even have a compilation step. (Personally, I wouldn't try to apply idiomatic Nix to modern JS - I'd apply it to the major components like "my web server" and "my database library" and have "all the JS I depend on" as one big Nix package. That's not really a claim about Nix, I wouldn't try to apply idiomatic, say, Debian packaging to modern JS either. In both cases I'd still get about 90% of the benefit of using Nix/Debian as a delivery mechanism.) The other thing that could be slow is if you start compiling all your dependencies, including gcc and node.js, from scratch. While there's some security benefit in doing so, the reality is that just about nobody actually does that. You'd want to set things up to use the precompiled packages, or at least set up your own cache server and have compiled binaries you trust but only do it once.
woah 6y ago
Ah yes.. cryptocurrency, where bizarre programming practices secure large sums of money
- CapriciousCptl 6y ago
  LOL. Reading that audit report made me immediately wonder who in the heck is blowing money on bringing new tech to their critical finance systems. And then I saw your post and, yes, that answers that.
  - danharaj 6y ago
    I am aware of a few trading firms that are phasing in nix to handle their dependency clusterfucks.
Kinrany 6y ago
So the auditors said that the system was hard to audit?
- mehrdadn 6y ago
  Putting it this way it misses the point of that statement, but yes, that does appear to be what they said.
- karatestomp 6y ago
  Considering that “difficult to audit” overlaps with: difficult to understand, difficult to deploy, difficult to onboard, and a bunch of other things that are also very important to non-auditors, that seems like an entirely reasonable and informative complaint. If you hire a financial auditor and their report was basically an nicer version of “‘books‘ were written in pencil on napkins, many food stained, some illegible. Attempting to make sense of them took forever. Fix this shit to meet minimum standards for business accounting, it’s entirely unsuitable for its purpose” that’s be great info, if you didn’t already know it.
  - yakshaving_jgt 6y ago
    "books written in pencil on napkins" is analogous to how most web application projects are organised today. Nix would be the formalisation of this.
    - atoav 6y ago
      Never used nix, but their stated goal is to make implicit dependecies explicit (and reproducible). Isn't it only natural for this to sometimes bring a dependency hell to light, that otherwise could have been swept under the rug? Or phrased differently: maybe the problem here wasn't nix, but the way developers chose their dependecies
bulldoa 6y ago
Woah, I didn't know there is an entire ecosystem of contract job and auditors for big projects. How do companies usually hire contract jobs (outsourced HR, upwork, Accenture)? And how do they hire auditors?
- all_blue_chucks 6y ago
  This sounds like a security audit, and auditors found the build system difficult to work with when attempting to audit code.
- kitplummer 6y ago
  This used to be called "quality assurance". Web and .com blew that apart. We ain't got time for that.
m_mueller 6y ago
So I understand correctly from this that nix has no binary packages? If so, why not? To me, compiling from source is not strictly necessary to get reproduceability guarantees. It might actually be harmful if you're not carefully checksumming the build products (e.g. cosmic rays messing with complex builds).
- yakshaving_jgt 6y ago
  You do not understand correctly. Nix users use binary caches[0] instead of building everything from source. You can use the binary cache provided by NixOS, or Cachix[1], or something you set up and host yourself, or all of the above. [0]: https://nixos.wiki/wiki/Binary_Cache https://nixos.wiki/wiki/Binary_Cache [1]: https://cachix.org/ https://cachix.org/
  - m_mueller 6y ago
    Then why would a build take so long as GP describes?
    - yakshaving_jgt 6y ago
      I can't imagine why some unknown people encountered difficulty building some unknown project. We use Nix to build and deploy our big Haskell monolith along with the operating system itself, and all system dependencies, e.g., PostgreSQL, Redis, Grafana, collectd, influxdb, openssh, ejabberd, Elm applications, etc, and it works fabulously.
    - illumin8 6y ago
      GP here. My guess is that they intentionally disabled binary caching because this is a financial application and they wanted to make sure no untrusted binaries got into the build pipeline.
      - m_mueller 6y ago
        Thanks! That makes some sense. Though, I work for a financial service provider as well and have never found compile-from source to be solution to that problem. If you do have dependencies, whether you trust the sources or the binaries shouldn't really matter, as long as you trust the repositories where they come from. And if you don't trust it I wanna see the army of developers actually checking all the sources down to the system level.
sadfklsjlkjwt 6y ago
But the build time dependencies would still be there. Just implicit and undocumented.
floriol 6y ago
It sounds like "this is the first time I've seen this build system and don't really see how it works, would be better to change it to a more often used one". Which is fair for a security auditor, but it stems from the newness of the project and nothing inherently bad with it - like I doubt they know what actually happens inside whatever other build system other's use, but since they empirically know it poses no threat they are okay with it. If you ask me, this is absolutely no reason to not use Nix - well maybe not for a bank (though on long term they would definitely win with it)

WatchDog 6y ago

I haven't used Nix, but I would have thought that builds would be fast, due to how cache-able the dependencies should be.

throwaway894345 6y ago
Sounds like they were building everything from scratch, for some reason.
- maytc 6y ago
  That’s how nix guarantees reproducibility. The compiled artifacts can be cached but that didnt quite work out of the box
  - throwaway894345 6y ago
    Nix guarantees reproducibility, which means anything can be rebuilt from scratch, but that's a very abnormal use case. If it didn't work out of the box, it's a problem with the package scripts (the "derivations"). That said, all of our software tends to bottom out in a bunch of shitty C libraries that are all delicately cobbled together with autotools and cmake, so anything that aspires to reproduce these things is going to have issues. This tends to make Nix difficult to use, because it doesn't have nearly the same investment/manpower (yet) as other package ecosystems that is necessary to wrangle these dependencies into a stable foundation that doesn't leak its underlying havoc to higher levels of the stack.
    - microphone987 6y ago
      This comment is really, really confusing. If it builds once Nix, it will build again with Nix... If you can find a derivation that builds on one machine and not on the other there's usually a fundamental difference - either in CPU arch, your nixpkgs config differs (overlays), etc. Or I guess you could write a build script that non-deterministic ally fails, but that has nothing to do with Nix's maturity. Nix ensures the tooling is called the same. I can understand having a non-autotools project that is more difficult to write a derivation for, but "Nix can't wrangle messy libraries" makes absolutely no sense, either from a "make compiling reliable" or anything at use-time. Do you have a specific example in mind that is less hand-wavy?
      - throwaway894345 6y ago
        > If it builds once Nix, it will build again with Nix... That's the aspiration, but it doesn't always pan out. The rate at which you run into problems depends a lot on the packages you use, how much attention is given to them, and how hard it is to reproducibly package them. I've noticed in particular that the Python ecosystem is really fragile. > Do you have a specific example in mind that is less hand-wavy? A specific example that comes to mind was the psycopg2 Python package, which would build on some developers' machines but not on others. This sort of thing happens all the time, usually on macOS, and usually with C packages (sadly, so much of the Python ecosystem is built on C and its shoddy build tooling). I've also found quite a few packages in nixpkgs that simply don't build on macOS, but which presumably build on Linux; however, I forget which ones specifically.
mzaccari 6y ago
We've been using Nix for deploying a Rails app for an enterprise customer for quite a few years now. One area where it shines for us is the ability to build it on relatively recent version of Ubuntu and deploy to a (almost EOL) RHEL6 box. Bundling, assets and various other tasks take just a few minutes. We also have ~20 Go services that are also deployed via Nix, and building takes seconds. However, it can be quite cumbersome to get a Nix expression to the point where it builds reliably for something that takes multiple steps like a Rails app, especially if you're building on macOS and deploying to Linux. It's come a _long_ way in recent years, but with Enterprise customers now embracing containerization we migrated everything to that and haven't looked back.
- smichel17 6y ago
  I'm having a hard time parsing -- you migrated everything from nix to containers (containers removed the need for nix) or to nix+containers (containers solved the "having to build for multiple platforms" issue)?
  - mzaccari 6y ago
    Ah my apologies - We use Bazel to build our services, and the output artifacts were then pulled into Nix and deployed as Nix packages. Bazel has _excellent_ support taking the same application code and creating Docker images from them (https://github.com/bazelbuild/rules_docker#language-rules https://github.com/bazelbuild/rules_docker#language-rules), and the tools available for deploying containers is orders of magnitude more feature-full and higher quality than what you get with Nix today. So we no longer have Nix anywhere in our pipeline, and all of our artifacts are now deployed inside containers.
takeda 6y ago
It is, perhaps they didn't use cache to store previously built packages and build everything from scratch. With no cache Nix will start with building compilers and glibc until it has everything to build the actual application.
illumin8 6y ago
While this is true if you are building new versions of dependencies on an existing system, in many cases in high security environments you want to start with a clean build environment every time. Cached dependencies might be considered to be a security risk. Bootstrapping an entire Nix environment can take a lot of time.
- ryukafalz 6y ago
  Sure, but in most systems you wouldn’t have much of a choice; if you’re using Debian or Ubuntu, you’re probably using binary packages built by someone else. It takes a lot more work to build everything you want from source in a traditional distro. It seems like a pretty straightforward tradeoff: do you want to rely on prebuilt packages for speed, or do you want to rebuild it all (and accept the extra time that takes)? Nix and Guix at least make the latter pretty straightforward to do.

yawaramin 6y ago

Sounds like their main concern wasn't Nix but the complexity of the code itself.

addicted 6y ago
The hours long build system appears to be attributed to Nix. Is Nix really that slow?
- mason55 6y ago
  Dependencies can take awhile if you can’t used cached versions of the binaries. Building some Haskell tooling from scratch took ~3 hours on my MBP.
  - gmfawcett 6y ago
    There are few compilers that are slower than GHC. I suspect this issue has nothing to do with Nix, and the tools would have taken ages to build from scratch without Nix.
- geofft 6y ago
  If you're applying idiomatic Nix to modern JS, I wouldn't be surprised - modern JS tends to involve installing thousands of packages by just combining them into a directory, but Nix doesn't want you to edit existing directories. So your dependency graph turns into a build-dependency graph, with each JS package requiring a full build of everything it depends on, and the Nix build system is presumably not optimized for fast turnaround times on five-line JS modules that don't even have a compilation step. (Personally, I wouldn't try to apply idiomatic Nix to modern JS - I'd apply it to the major components like "my web server" and "my database library" and have "all the JS I depend on" as one big Nix package. That's not really a claim about Nix, I wouldn't try to apply idiomatic, say, Debian packaging to modern JS either. In both cases I'd still get about 90% of the benefit of using Nix/Debian as a delivery mechanism.) The other thing that could be slow is if you start compiling all your dependencies, including gcc and node.js, from scratch. While there's some security benefit in doing so, the reality is that just about nobody actually does that. You'd want to set things up to use the precompiled packages, or at least set up your own cache server and have compiled binaries you trust but only do it once.

woah 6y ago

Ah yes.. cryptocurrency, where bizarre programming practices secure large sums of money

CapriciousCptl 6y ago
LOL. Reading that audit report made me immediately wonder who in the heck is blowing money on bringing new tech to their critical finance systems. And then I saw your post and, yes, that answers that.
- danharaj 6y ago
  I am aware of a few trading firms that are phasing in nix to handle their dependency clusterfucks.

Kinrany 6y ago

So the auditors said that the system was hard to audit?

mehrdadn 6y ago
Putting it this way it misses the point of that statement, but yes, that does appear to be what they said.
karatestomp 6y ago
Considering that “difficult to audit” overlaps with: difficult to understand, difficult to deploy, difficult to onboard, and a bunch of other things that are also very important to non-auditors, that seems like an entirely reasonable and informative complaint. If you hire a financial auditor and their report was basically an nicer version of “‘books‘ were written in pencil on napkins, many food stained, some illegible. Attempting to make sense of them took forever. Fix this shit to meet minimum standards for business accounting, it’s entirely unsuitable for its purpose” that’s be great info, if you didn’t already know it.
- yakshaving_jgt 6y ago
  "books written in pencil on napkins" is analogous to how most web application projects are organised today. Nix would be the formalisation of this.
  - atoav 6y ago
    Never used nix, but their stated goal is to make implicit dependecies explicit (and reproducible). Isn't it only natural for this to sometimes bring a dependency hell to light, that otherwise could have been swept under the rug? Or phrased differently: maybe the problem here wasn't nix, but the way developers chose their dependecies

bulldoa 6y ago

Woah, I didn't know there is an entire ecosystem of contract job and auditors for big projects. How do companies usually hire contract jobs (outsourced HR, upwork, Accenture)? And how do they hire auditors?

all_blue_chucks 6y ago
This sounds like a security audit, and auditors found the build system difficult to work with when attempting to audit code.
kitplummer 6y ago
This used to be called "quality assurance". Web and .com blew that apart. We ain't got time for that.

m_mueller 6y ago

So I understand correctly from this that nix has no binary packages? If so, why not? To me, compiling from source is not strictly necessary to get reproduceability guarantees. It might actually be harmful if you're not carefully checksumming the build products (e.g. cosmic rays messing with complex builds).

yakshaving_jgt 6y ago
You do not understand correctly. Nix users use binary caches[0] instead of building everything from source. You can use the binary cache provided by NixOS, or Cachix[1], or something you set up and host yourself, or all of the above. [0]: https://nixos.wiki/wiki/Binary_Cache https://nixos.wiki/wiki/Binary_Cache [1]: https://cachix.org/ https://cachix.org/
- m_mueller 6y ago
  Then why would a build take so long as GP describes?
  - yakshaving_jgt 6y ago
    I can't imagine why some unknown people encountered difficulty building some unknown project. We use Nix to build and deploy our big Haskell monolith along with the operating system itself, and all system dependencies, e.g., PostgreSQL, Redis, Grafana, collectd, influxdb, openssh, ejabberd, Elm applications, etc, and it works fabulously.
  - illumin8 6y ago
    GP here. My guess is that they intentionally disabled binary caching because this is a financial application and they wanted to make sure no untrusted binaries got into the build pipeline.
    - m_mueller 6y ago
      Thanks! That makes some sense. Though, I work for a financial service provider as well and have never found compile-from source to be solution to that problem. If you do have dependencies, whether you trust the sources or the binaries shouldn't really matter, as long as you trust the repositories where they come from. And if you don't trust it I wanna see the army of developers actually checking all the sources down to the system level.

sadfklsjlkjwt 6y ago

But the build time dependencies would still be there. Just implicit and undocumented.

floriol 6y ago

It sounds like "this is the first time I've seen this build system and don't really see how it works, would be better to change it to a more often used one". Which is fair for a security auditor, but it stems from the newness of the project and nothing inherently bad with it - like I doubt they know what actually happens inside whatever other build system other's use, but since they empirically know it poses no threat they are okay with it. If you ask me, this is absolutely no reason to not use Nix - well maybe not for a bank (though on long term they would definitely win with it)