4 ms·

Regarding disappearing sources: Nix offers a content-addressed mirror for sources downloaded by the Hydra CI system. As a random example, here is the latest Chr

by ymse 9y ago

Regarding disappearing sources: Nix offers a content-addressed mirror for sources downloaded by the Hydra CI system. As a random example, here is the latest Chromium source tarball: http://tarballs.nixos.org/sha256/3dfa02e873ff51a11ee02b9ca391807f0c8ea0529a4924afa645fbf97163f9d4 http://tarballs.nixos.org/sha256/3dfa02e873ff51a11ee02b9ca39... So disappearing sources is not a huge problem in my experience. Obviously if you have package declarations outside of Nixpkgs proper things are different. This problem is also something the Software Heritage project[0] aims to solve, but I don't think they have a good API yet. [0] https://www.softwareheritage.org/ https://www.softwareheritage.org/

vishvananda 9y ago
Yeah I'm aware of the build mirror, but crashcart builds things under a different prefix so it must build things from source each time. I realize this doesn't affect most users (which is why it takes a while for disappearing sources to be found and fixed). A content addressed mirror for sources as well would solve the problem nicely.
- ymse 9y ago
  I'm sorry, the mirror is actually for sources, not build artifacts. I've updated my comment to clarify.
  - Filligree 9y ago
    When was that added? Well fetchurl automatically poke it if it can't find the sources otherwise?
    - ymse 9y ago
      Yes, fetchurl will try it: https://github.com/NixOS/nixpkgs/blob/733151b7825ee8840f743705076683c88e8d098e/pkgs/build-support/fetchurl/mirrors.nix#L4 https://github.com/NixOS/nixpkgs/blob/733151b7825ee8840f7437... Edit: It was apparently added in 2007: https://nixos.org/releases/nixpkgs/nixpkgs-0.11/release-notes/ https://nixos.org/releases/nixpkgs/nixpkgs-0.11/release-note...
  - vishvananda 9y ago
    interesting I wasn't aware of that. I wonder why my builds are not using it.
    - catern 9y ago
      It might be related to using a different Nix prefix for your builds, which is a little poorly supported. Just curious: Why are you using a different prefix?
      - vishvananda 9y ago
        The point of crashcart is to be able to side load a filesystem with utilities into a running container. It is very important that the location we pick doesn't conflict with any path already in the container. If we used /nix as the mount path it would conflict with any container that uses nix. In order to prevent this (probably rare) conflict, we build our utilities in /dev/crashcart/ instead.
        catern 9y ago
        Hmmmm, interesting... That does seem like a pretty good reason, though maybe you could bind mount over /nix? That should be fine from a Nix perspective, not sure how well it works with container technologies.
        vishvananda 9y ago
        we could bind mount over /nix or /nix/store, but that means any existing nix packages from the container would not be available. The whole point is to have the whole container file system available along with the utilities. We could alternatively find each package that we need for our debugging utilities and bind mount each directory from the store individually. This would work due to unique paths in the store, but that means potentially hundreds of bind mounts and is an orchestration nightmare.
        catern 9y ago

by ymse 9y ago

vishvananda 9y ago

Yeah I'm aware of the build mirror, but crashcart builds things under a different prefix so it must build things from source each time. I realize this doesn't affect most users (which is why it takes a while for disappearing sources to be found and fixed). A content addressed mirror for sources as well would solve the problem nicely.

ymse 9y ago
I'm sorry, the mirror is actually for sources, not build artifacts. I've updated my comment to clarify.
- Filligree 9y ago
  When was that added? Well fetchurl automatically poke it if it can't find the sources otherwise?
  - ymse 9y ago
    Yes, fetchurl will try it: https://github.com/NixOS/nixpkgs/blob/733151b7825ee8840f743705076683c88e8d098e/pkgs/build-support/fetchurl/mirrors.nix#L4 https://github.com/NixOS/nixpkgs/blob/733151b7825ee8840f7437... Edit: It was apparently added in 2007: https://nixos.org/releases/nixpkgs/nixpkgs-0.11/release-notes/ https://nixos.org/releases/nixpkgs/nixpkgs-0.11/release-note...
- vishvananda 9y ago
  interesting I wasn't aware of that. I wonder why my builds are not using it.
  - catern 9y ago
    It might be related to using a different Nix prefix for your builds, which is a little poorly supported. Just curious: Why are you using a different prefix?
    - vishvananda 9y ago
      The point of crashcart is to be able to side load a filesystem with utilities into a running container. It is very important that the location we pick doesn't conflict with any path already in the container. If we used /nix as the mount path it would conflict with any container that uses nix. In order to prevent this (probably rare) conflict, we build our utilities in /dev/crashcart/ instead.
      - catern 9y ago
        Hmmmm, interesting... That does seem like a pretty good reason, though maybe you could bind mount over /nix? That should be fine from a Nix perspective, not sure how well it works with container technologies.
        vishvananda 9y ago
        we could bind mount over /nix or /nix/store, but that means any existing nix packages from the container would not be available. The whole point is to have the whole container file system available along with the utilities. We could alternatively find each package that we need for our debugging utilities and bind mount each directory from the store individually. This would work due to unique paths in the store, but that means potentially hundreds of bind mounts and is an orchestration nightmare.
        catern 9y ago