14 ms·

Fedora change aims for 99% package reproducibility

436 points by voxadam 2y ago

ajross 2y ago
Linux folks continue with running away with package security paradigms while NPM, PyPI, cargo, et. al. (like that VSCode extension registry that was on the front page last week) think they can still get away with just shipping what some rando pushes.
- hedora 2y ago
  Shipping what randos push works great for iOS and Android too. System perl is actually good. It's too bad the Linux vendors don't bother with system versions of newer languages.
  - heinrich5991 2y ago
    System rustc is also good on Arch Linux. I think system rustc-web is also fine on Debian.
    - WD-42 2y ago
      I recently re-installed and instead of installing rustup I thought I'd give the Arch rust package a shot. Supposedly it's not the "correct" way to do it but so far it's working great, none of my projects require nightly. Updates have come in maybe a day after upstream. One less thing to think about, I like it.
  - g-b-r 2y ago
    Sure, there's never malware on the stores...
  - ajross 2y ago
    > Shipping what randos push works great for iOS and Android too. App store software is excruciatingly vetted, though. Apple and Google spend far, far, FAR more on validating the software they ship to customers than Fedora or Canonical, and it's not remotely close. It only looks like "randos" because the armies of auditors and datacenters of validation software are hidden behind the paywall.
    - IshKebab 2y ago
      It's really not. At least on Android they have some automated vetting but that's about it. Any human vetting is mostly for business reasons. Also Windows and Mac have existed for decades and there's zero vetting there. Yeah malware exists but its easy to avoid and easily worth the benefit of actually being able to get up-to-date software from anywhere.
      - skydhash 2y ago
        > Also Windows and Mac have existed for decades and there's zero vetting there. Isn't that only for applications? All the system software are provided and vetted by the OS developer.
        IshKebab 2y ago
        Yeah but distro package repositories mostly consist of applications and non-system libraries.
        skydhash 2y ago
        Most users only have an handful of applications installed. They may not be the same one, but flatpak is easy to setup if you want the latest version. And you're not tied to the system package manager if you want the latest for CLI software (nix, brew, toolbox,...) The nice thing about Debian is that you can have 2 full years of routine maintenance while getting reading for the next big updates. The main issue is upstream developer having bug fixes and features update on the same patch.
      - ndiddy 2y ago
        The vetting on Windows is that basically any software that isn't signed by an EV certificate will show a scary SmartScreen warning on users' computers. Even if your users aren't deterred by the warning, you also have a significant chance of your executable getting mistakenly flagged by Windows Defender, and then you have to file a request for Microsoft to whitelist you. The vetting on Mac is that any unsigned software will show a scary warning and make your users have to dig into the security options in Settings to get the software to open. This isn't really proactive, but it means that if you ship malware, Microsoft/Apple can revoke your certificate. If you're interested in something similar to this distribution model on Linux, I would check out Flatpak. It's similar to how distribution works on Windows/Mac with the added benefit that updates are handled centrally (so you don't need to write auto-update functionality into each program) and that all programs are manually vetted both before they go up on Flathub and when they change any permissions. It also doesn't cost any money to list software, unlike the "no scary warnings" distribution options for both Windows and Mac.
- anotherhue 2y ago
  I have observed a sharp disconnect in the philosophies of 'improving developer experience' and 'running a tight ship'. I think the last twenty years of quasi-marketing/sales/recruiting DevRel roles have pushed a narrative of frictionless development, while on the flip side security and correctness have mostly taken a back seat (special industries aside). I think it's a result of the massive market growth, but I so welcome the pendulum swinging back a little bit. Typo squatting packages being a concern at the same time as speculative execution exploits shows mind bending immaturity.
  - psadauskas 2y ago
    "Security" and "Convenience" is always a tradeoff, you can never have both.
    - bluGill 2y ago
      Ture. Then the Convenience folks don't understand why the rest of us don't want the things they think are so great. There are good middle grounds, but most package managers don't even acknowledge other concerns as valid.
    - benterix 2y ago
      This is obvious, the question here is why everybody traded security for convenience and what else has to happen for people to start taking security seriously.
      - ziddoap 2y ago
        >the question here is why everybody traded security for convenience I don't think security was traded away for convenience. Everything started with convenience, and security has been trying to gain ground ever since. >happen for people to start taking security seriously Law with enforced and non-trivial consequences are the only thing that will force people to take security seriously. And even then, most probably still wont.
      - vladms 2y ago
- esafak 2y ago
  The future is not evenly distributed.
  - deleted 2y ago
    [deleted]
- Palomides 2y ago
  distros get unbelievable amounts of hate for not immediately integrating upstream changes, there's really no winning
  - IshKebab 2y ago
    Rightly so. The idea that all software should be packaged for all distros, and that you shouldn't want to use the latest version of software is clearly ludicrous. It only seems vaguely reasonable because it's what's always been done. If Linux had evolved a more sensible system and someone came along and suggested "no actually I think each distro should have its own package format and they should all be responsible for packaging all software in the world, and they should use old versions too for stability" they would rightly be laughed out of the room.
    - bluGill 2y ago
      There are too many distributions / formats. However distribution package are much better than snap/flatpack/docker for most uses, the only hard part is there are so many that no program can put "and is integrated into the package manager" in their release steps. You can ship a docker container in your program release - it is often done but rarely what should be done.
      - oivey 2y ago
        For some cases maybe that makes sense, but in a very large percentage it does not. As example, what if I want to build and use/deploy a Python app that needs the latest NumPy, and the system package manager doesn’t have it. It would be hard to justify for me to figure out and build a distro specific package for this rather than just using the Python package on PyPI.
        bluGill 2y ago
        The point is the distro should provide the numpy you need.
- tsimionescu 2y ago
  I think the opposite is mostly true. Linux packaging folks are carefully sculpting their toys, while everyone else is mostly using upstream packages and docker containers to work around the beautiful systems. For half the software I care about on my Debian system, I have a version installed either directly from the web (curl | bash style), from the developer's own APT repo, or most likely from a separate package manager (be it MELPA, pypi, Go cache, Maven, etc).
  - justinrubek 2y ago
    That sounds like an incredibly annoying way to manage software. I don't have a single thing installed that way.
  - kombine 2y ago
    I use nix package manager on 3 of the systems I'm working daily (one of them HPC cluster) and none of them run NixOS. It's possible to carefully sculpt one's tools and use latest and greatest.
  - deleted 2y ago
    [deleted]
kachapopopow 2y ago
[flagged]
sheepscreek 2y ago
YES! I want more tools to be deterministic. My wish-list has Proxmox config at the very top.
- TheDong 2y ago
  Want to give this a try and see if it works? https://github.com/SaumonNet/proxmox-nixos?tab=readme-ov-file#-declarative-vms https://github.com/SaumonNet/proxmox-nixos?tab=readme-ov-fil...
knowitnone 2y ago
99%? Debbie Downer says it only takes 1 package to screw the pooch
- ethersteeds 2y ago
  I would still much prefer playing 100:1 Russian roulette than 1:1, if those are my options.
- nwah1 2y ago
  There's a long tail of obscure packages that are rarely used, and almost certainly a power law in terms of which packages are common. Reproducibility often requires coordination between both the packagers and the developers, and achieving that for each and every package is optimistic. If they just started quarantining the long tail of obscure packages, then people would get upset. And failing to be 100% reproducible will make a subset of users upset. Lose-lose proposition there, given that intelligent users could just consciously avoid packages that aren't passing reproducibility tests. 100% reproducibility is a good goal, but as long as the ubiquitous packages are reproducible then that is probably going to cover most. Would be interesting to provide an easy way to disallow non-reproducible packages. I'm sure one day they will be able to make it a requirement for inclusion into the official repos.
  - yjftsjthsd-h 2y ago
    There's an interesting thought - in addition to aiming for 99% of all packages, perhaps it would be a good idea to target 100% of the packages that, say, land in the official install media? (I wouldn't even be surprised if they already meet that goal TBH, but making it explicit and documenting it has value)
- EasyMark 2y ago
  "All I see is 1% of complete failure" --Bad Dads everywhere
patrakov 2y ago
This goal feels like a marketing OKR to me. A proper technical goal would be "all packages, except the ones that have a valid reason, such as signatures, not to be reproducible".
- RegnisGnaw 2y ago
  As someone who dabbles a bit in the RHEL world, IIRC all packages in Fedora are signed. In additional the DNF/Yum meta-data is also signed. IIRC I don't think Debian packages themselves are signed themselves but the apt meta-data is signed.
  - westurner 2y ago
    I learned this from an ansible molecule test env setup script for use in containers and VMs years ago; because `which` isn't necessarily installed in containers for example: type -p apt && (set -x; apt install -y debsums; debsums | grep -v 'OK$') || \ type -p rpm && rpm -Va # --verify --all dnf reads .repo files from /etc/yum.repos.d/ [1] which have various gpg options; here's an /etc/yum.repos.d/fedora-updates.repo: [updates] name=Fedora $releasever - $basearch - Updates #baseurl=http://download.example/pub/fedora/linux/updates/$releasever/Everything/$basearch/ metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-released-f$releasever&arch=$basearch enabled=1 countme=1 repo_gpgcheck=0 type=rpm gpgcheck=1 metadata_expire=6h gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch skip_if_unavailable=False From the dnf conf docs [1], there are actually even more per-repo gpg options: gpgkey gpgkey_dns_verification repo_gpgcheck localpkg_gpgcheck gpgcheck 1. https://dnf.readthedocs.io/en/latest/conf_ref.html#repo-options https://dnf.readthedocs.io/en/latest/conf_ref.html#repo-opti... 2. https://docs.ansible.com/ansible/latest/collections/ansible/builtin/yum_repository_module.html#parameter-gpgcakey https://docs.ansible.com/ansible/latest/collections/ansible/... lists a gpgcakey parameter for the ansible.builtin.yum_repository module For Debian, Ubuntu, Raspberry Pi OS and other dpkg .deb and apt distros: man sources.list man sources.list | grep -i keyring -C 10 # trusted: # signed-by: # /etc/apt/ trusted.gpg.d/ man apt-secure man apt-key apt-key help less "$(type -p apt-key)" signing-apt-repo-faq: https://github.com/crystall1nedev/signing-apt-repo-faq https://github.com/crystall1nedev/signing-apt-repo-faq From "New requirements for APT repository signing in 24.04" (2024) https://discourse.ubuntu.com/t/new-requirements-for-apt-repository-signing-in-24-04/42854 https://discourse.ubuntu.com/t/new-requirements-for-apt-repo... : > In Ubuntu 24.04, APT will require repositories to be signed using one of the following public key algorithms: [ RSA with at least 2048-bit keys, Ed25519, Ed448 ] > This has been made possible thanks to recent work in GnuPG 2.4 82 by Werner Koch to allow us to specify a “public key algorithm assertion” in APT when calling the gpgv tool for verifying repositories.
binarymax 2y ago
I often see initiatives and articles like this but no mention of Nix. Is it just not well known enough for comparison? Because to me that’s the standard.
- esseph 2y ago
  It's an article about Fedora, specifically.
  - binarymax 2y ago
    Yes, I know that. But when talking about reproducible packages, we can and should learn from existing techniques. Is Fedoras system capable of this goal? Is it built the right way? Should it adopt an alternate package manager to achieve this with less headache?
    - dogleash 2y ago
      "What about Nix tho?" is the "Rewrite it in Rust" of reproducible builds.
    - deleted 2y ago
      [deleted]
    - johnny22 2y ago
      No other currently distro whether it be debian, fedora, or arch, or opensuse seems to want to switch up their design to match nix's approach. They have too many people familiar with the current approaches.
      - nonameiguess 2y ago
        That's not even really true. Hacker News readers seem to fixate heavily on the most popular desktop versions of Linux distros, but that is far from all they make. SUSE with SLE Micro and Edge Image Builder is a declarative build system for creating your own installer images. If you don't want SLE Micro, they even give you the Elemental Toolkit, which can be used to build your own custom distro in a declarative manner. I don't know much about Debian and Red Hat's approaches to this, but I know they have similar tooling. The "problem," if you want to call it that, is developers, if I may be uncharitable for a brief moment here, are amazingly solipsistic. The major Linux vendors are targeting enterprise edge computing use cases, whereas developers only ever think about the endpoint PCs they themselves use for development. Plenty of distro tooling will give you declaratively built, transactionally updated, immutable server images, but if they don't give you the same for your personal laptop, those efforts are invisible to the modal Hacker News commenter. For what it's worth, there's also Guix, which is literally a clone of Nix but part of the GNU project, so it only uses free software and opts for Guile instead of a custom DSL for configuration. It wasn't a pre-existing distro that changed, of course.
froh 2y ago
nice to see they're in this too. https://news.opensuse.org/2025/02/18/rbos-project-hits-milestone/ https://news.opensuse.org/2025/02/18/rbos-project-hits-miles...
charcircuit 2y ago
This is a waste of time compared to investing in sandboxing which will actually protect users as opposed to stopping theoretical attacks. Fedora's sandbox capabilities for apps is so far behind other operating systems like Android that it is much more important of an area to address.
- johnny22 2y ago
  I think you have to do both sandboxing and this.
  - charcircuit 2y ago
    Both are good for security, but prioritization is important. The areas that are weakest in terms of security should get the most attention.
    - jcgl 2y ago
      That assumes that there is a fixed amount of effort and resources to be split between the two things—that there is always opportunity cost. When it comes to community efforts, it’s rarely the case that all things have opportunity cost—people who contribute effort for X would not have necessarily done so for Y. The Fedora Project is of course not a purely community effort, so I don’t know exactly how that applies here. But just wanted to point out that prioritization and opportunity cost don’t always work like you suggested.
- AshamedCaptain 2y ago
  I am yet to see a form of sandboxing for the desktop that is not: a) effectively useless or b) makes me want to throw my computer through the window and replace it with a 1990's device (still more useful than your average Android).
- fsflover 2y ago
  If you want security through compartmentalization, you should consider Qubes OS, my daily driver, https://qubes-os.org https://qubes-os.org.
nimish 2y ago
As a user of fedora what does this actually get me? I mean I understand it for hermetic builds but why?
- deleted 2y ago
  [deleted]
- jacobgkau 2y ago
  My impression is that reproducible builds improve your security by helping make it more obvious that packages haven't been tampered with in late stages of the build system. * Edit, it's quoted in the linked article: > Jędrzejewski-Szmek said that one of the benefits of reproducible builds was to help detect and mitigate any kind of supply-chain attack on Fedora's builders and allow others to perform independent verification that the package sources match the binaries that are delivered by Fedora.
  - Zamicol 2y ago
    Bingo.
  - deleted 2y ago
    [deleted]
  - kazinator 2y ago
    The supply chain attacks you have to most worry about are not someone breaking into Fedora build machines. It's the attacks on the upstream packages themselves. Reproducible builds would absolutely not catch a situation like the XZ package being compromised a year ago, due to the project merging a contribution from a malicious actor. A downstream package system or OS distro will just take that malicious update and spin it into a beautifully reproducing build.
    - yjftsjthsd-h 2y ago
      Don't let the perfect be the enemy of the good; this doesn't prevent upstream problems but it removes one place for compromises to happen.
Dwedit 2y ago
Reproducibility is at odds with Profile-Guided-Optimization. Especially on anything that involves networking and other IO that isn't consistent.
- michaelt 2y ago
  Why should it be? Does the profiler not output a hprof file or whatever, which is the input to the compiler making the release binary? Why not just store that?
- gnulinux 2y ago
  It's not at odds at all but it'll be "Monadic" in the sense that the output of system A will be part of the input to system A+1 which is complicated to organize in a systems setting, especially if you don't have access to a language that can verify. But it's absolutely achievable if you do have such a tool, e.g. you can do this in nix.
- zbobet2012 2y ago
  That's only the case if you did PGO with "live" data instead of replays from captured runs, which is best practice afaik.
- nrvn 2y ago
  from Go documentation[0]: > Committing profiles directly in the source repository is recommended as profiles are an input to the build important for reproducible (and performant!) builds. Storing alongside the source simplifies the build experience as there are no additional steps to get the profile beyond fetching the source. I very much hope other languages/frameworks can do the same. [0]: https://go.dev/doc/pgo#building https://go.dev/doc/pgo#building
  - nyrikki 2y ago
    The Performant claim there is counter to research I have heard. Plus as the PGO profile data is non-deterministic in most cases, even when compiled on the same hardware as the end machine "Committing profiles directly in the source repository" is the reason why they are deleted or at least excluded from the comparison. A quote from the paper that I remember on the subject[1] as these profiles are just about as machine dependent as you can get. > Unfortunately, most code improvements are not machine independent, and the few that truly are machine independent interact with those that are machine dependent causing phase-ordering problems. Hence, effectively there are no machine-independent code improvements. There were some differences between various Xeon chip's implementations of the same or neighboring generations that I personally ran into when we tried to copy profiles to avoid the cost of the profile runs that may make me a bit more sensitive to this, but I personally saw huge drops in performance well into the double digits that threw off our regression testing. IMHO this is exactly why your link suggested the following: > Your production environment is the best source of representative profiles for your application, as described in Collecting profiles. That is very different from Fedora using some random or generic profile for x86_64, which may or may not match the end users specific profile. [1] https://dl.acm.org/doi/10.5555/184716.184723 https://dl.acm.org/doi/10.5555/184716.184723
barotalomey 2y ago
The real treasure was the friend I found along the way https://github.com/keszybz/add-determinism https://github.com/keszybz/add-determinism
- m463 2y ago
  I kind of wonder if this or something similar could somehow nullify timestamps so you could compare two logfiles... further would be the ability to compare logfiles with pointer addresses or something
  - didericis 2y ago
    A different but more powerful method of ensuring reproducibility is more rigorous compilation using formally verifiable proofs. That’s what https://pi2.network/ https://pi2.network/ does. It uses K-Framework, which is imo very underrated/deserves more attention as a long term way of solving this kind of problem.
  - yjftsjthsd-h 2y ago
    I'm not confident that I understand what you're asking for, but couldn't you just sed off the timestamp from every line? Or for a more extreme example, I have occasionally used ... tr, I think? to completely remove all numbers from logs in order to aggregate error messages without worrying about the fact that they kept including irrelevant changing numbers (something like tail -5000 logfile | tr -d [0-9] | sort | uniq -c | sort -n or so).
    - winwang 2y ago
      how would you do it if your logs were printed on paper with a printer, each line printed with stochastic timing (due to a bug), with an ink containing a chemical tracer with halflife `h` (after being put to paper), but the ink is randomly sampled from several (`m`) inks of different halflives `h1`, h2`,... `hn`? assume `p` different printers scattered across the 10 most populous US cities. you may use standard unix utilities.
unit149 2y ago
[dead]
frainfreeze 2y ago
Amazing to see this progress! Cudos to everyone who put in the effort. Related news from March https://news.ycombinator.com/item?id=43484520 https://news.ycombinator.com/item?id=43484520 (Debian bookworm live images now fully reproducible)
kccqzy 2y ago
> For example, Haskell packages are not currently reproducible when compiled by more than one thread Doesn't seem like a big issue to me. The gcc compiler doesn't even support multithreaded compiling. In the C world, parallelism comes from compiling multiple translation units in parallel, not any one with multiple threads.
apatheticonion 2y ago
Another thing I'd love to see is more statically linked binaries. Something like Python, for instance, is a nightmare to install and work with
- theteapot 2y ago
  I think general consensus is against you. Fedora packaging policy [1]: > Packages including libraries should exclude static libs as far as possible (eg by configuring with --disable-static). Static libraries should only be included in exceptional circumstances. Applications linking against libraries should as far as possible link against shared libraries not static versions. [1]: https://docs.fedoraproject.org/en-US/packaging-guidelines/ https://docs.fedoraproject.org/en-US/packaging-guidelines/
  - Brian_K_White 2y ago
    I'd far rather a static binary than a bundled vm for a single app which produces all the same bad points of a static binary plus 900 new bad points on top. Packaging guidelines from a distros docs like this are not any kind of counter argument to that comment. This is the current orthodoxy, so obviously all docs say it. We all know the standard argument for the current standard. Their comment was explicitly "I'd like to see a change from the current orthodoxy". They are saying that maybe that argument is not all it promised to be back in the 90's when we started using dynamic libs.
    - booi 2y ago
      So instead of 1000 programs and 1000 libraries, you’d rather 1000 programs and 1,000,000 libraries?
      - Brian_K_White 2y ago
        Baffled how you got there, but not interested.
        2y ago
trod1234 2y ago
Can someone provide a brief clarification about build reproducibility in general? The stated aim is that when you compile the same source, environment, and instructions the end result is bit identical. There is, however; hardware specific optimizations that will naturally negate this stated aim, and I don't see how there's any way to avoid throwing out the baby with the bathwater. I understand why having a reproducible build is needed on a lot of fronts, but the stated requirements don't seem to be in line with the realities. At its most basic, there is hardware, where the hardware may advertise features it doesn't have, or doesn't perform the same instructions in the same way, and other nuances that break determinism as a property, and that naturally taints the entire stack since computers rely heavily on emergent design. This is often hidden in layers of abstraction and/or may be separated into pieces that are architecture dependent vs independent (freestanding), but it remains there. Most if not all of the beneficial properties of reproducible builds rely on the environment being limited to a deterministic scope, and the reality is manufacturers ensure these things remain in a stochastic scope.
- amarshall 2y ago
  Well the point is that if N of M machines produce the same output, it provides the opportunity to question why it is different on the others. If the build is not reproducible then one just throws up their arms. It’s not clear if you’re also talking about compiler optimizations—a reproducible build must have a fixed target for that.
  - trod1234 2y ago
    I was thinking more from a reference point along the lines of LLVM type performance optimizations when I was speaking about optimizations, if that sufficiently clarifies.
- Crestwave 2y ago
  > hardware specific optimizations that will naturally negate this stated aim Distro packages are compiled on their build server and distributed to users with all kinds of systems; therefore, by nature, it should not use optimizations specific to the builder's hardware. On source-based distros like Gentoo, yes, users adding optimization flags would get a different output. But there is still value in having the same hardware/compilation flags result in the same output.
curtisszmania 2y ago
[dead]
koreanguy 2y ago
[dead]
kaelit 2y ago
Interesting take. I'm building something related to zk systems — will share once it's up.
nigel_doug 2y ago
"trust in the contents of both source and binary packages is low." - I wonder will this convince organisations to adopt proper artifact management processes? If supply chain attacks are on the rise, than surely its more imperative than ever for businesses to adopt secure artifact scanning with tools like Cloudsmith or jFrog?

436 points by voxadam 2y ago

Linux folks continue with running away with package security paradigms while NPM, PyPI, cargo, et. al. (like that VSCode extension registry that was on the front page last week) think they can still get away with just shipping what some rando pushes.

hedora 2y ago
Shipping what randos push works great for iOS and Android too. System perl is actually good. It's too bad the Linux vendors don't bother with system versions of newer languages.
- heinrich5991 2y ago
  System rustc is also good on Arch Linux. I think system rustc-web is also fine on Debian.
  - WD-42 2y ago
    I recently re-installed and instead of installing rustup I thought I'd give the Arch rust package a shot. Supposedly it's not the "correct" way to do it but so far it's working great, none of my projects require nightly. Updates have come in maybe a day after upstream. One less thing to think about, I like it.
- g-b-r 2y ago
  Sure, there's never malware on the stores...
- ajross 2y ago
  > Shipping what randos push works great for iOS and Android too. App store software is excruciatingly vetted, though. Apple and Google spend far, far, FAR more on validating the software they ship to customers than Fedora or Canonical, and it's not remotely close. It only looks like "randos" because the armies of auditors and datacenters of validation software are hidden behind the paywall.
  - IshKebab 2y ago
    It's really not. At least on Android they have some automated vetting but that's about it. Any human vetting is mostly for business reasons. Also Windows and Mac have existed for decades and there's zero vetting there. Yeah malware exists but its easy to avoid and easily worth the benefit of actually being able to get up-to-date software from anywhere.
    - skydhash 2y ago
      > Also Windows and Mac have existed for decades and there's zero vetting there. Isn't that only for applications? All the system software are provided and vetted by the OS developer.
      - IshKebab 2y ago
        Yeah but distro package repositories mostly consist of applications and non-system libraries.
        skydhash 2y ago
        Most users only have an handful of applications installed. They may not be the same one, but flatpak is easy to setup if you want the latest version. And you're not tied to the system package manager if you want the latest for CLI software (nix, brew, toolbox,...) The nice thing about Debian is that you can have 2 full years of routine maintenance while getting reading for the next big updates. The main issue is upstream developer having bug fixes and features update on the same patch.
    - ndiddy 2y ago
      The vetting on Windows is that basically any software that isn't signed by an EV certificate will show a scary SmartScreen warning on users' computers. Even if your users aren't deterred by the warning, you also have a significant chance of your executable getting mistakenly flagged by Windows Defender, and then you have to file a request for Microsoft to whitelist you. The vetting on Mac is that any unsigned software will show a scary warning and make your users have to dig into the security options in Settings to get the software to open. This isn't really proactive, but it means that if you ship malware, Microsoft/Apple can revoke your certificate. If you're interested in something similar to this distribution model on Linux, I would check out Flatpak. It's similar to how distribution works on Windows/Mac with the added benefit that updates are handled centrally (so you don't need to write auto-update functionality into each program) and that all programs are manually vetted both before they go up on Flathub and when they change any permissions. It also doesn't cost any money to list software, unlike the "no scary warnings" distribution options for both Windows and Mac.
anotherhue 2y ago
I have observed a sharp disconnect in the philosophies of 'improving developer experience' and 'running a tight ship'. I think the last twenty years of quasi-marketing/sales/recruiting DevRel roles have pushed a narrative of frictionless development, while on the flip side security and correctness have mostly taken a back seat (special industries aside). I think it's a result of the massive market growth, but I so welcome the pendulum swinging back a little bit. Typo squatting packages being a concern at the same time as speculative execution exploits shows mind bending immaturity.
- psadauskas 2y ago
  "Security" and "Convenience" is always a tradeoff, you can never have both.
  - bluGill 2y ago
    Ture. Then the Convenience folks don't understand why the rest of us don't want the things they think are so great. There are good middle grounds, but most package managers don't even acknowledge other concerns as valid.
  - benterix 2y ago
    This is obvious, the question here is why everybody traded security for convenience and what else has to happen for people to start taking security seriously.
    - ziddoap 2y ago
      >the question here is why everybody traded security for convenience I don't think security was traded away for convenience. Everything started with convenience, and security has been trying to gain ground ever since. >happen for people to start taking security seriously Law with enforced and non-trivial consequences are the only thing that will force people to take security seriously. And even then, most probably still wont.
    - vladms 2y ago
esafak 2y ago
The future is not evenly distributed.
- deleted 2y ago
  [deleted]
Palomides 2y ago
distros get unbelievable amounts of hate for not immediately integrating upstream changes, there's really no winning
- IshKebab 2y ago
  Rightly so. The idea that all software should be packaged for all distros, and that you shouldn't want to use the latest version of software is clearly ludicrous. It only seems vaguely reasonable because it's what's always been done. If Linux had evolved a more sensible system and someone came along and suggested "no actually I think each distro should have its own package format and they should all be responsible for packaging all software in the world, and they should use old versions too for stability" they would rightly be laughed out of the room.
  - bluGill 2y ago
    There are too many distributions / formats. However distribution package are much better than snap/flatpack/docker for most uses, the only hard part is there are so many that no program can put "and is integrated into the package manager" in their release steps. You can ship a docker container in your program release - it is often done but rarely what should be done.
    - oivey 2y ago
      For some cases maybe that makes sense, but in a very large percentage it does not. As example, what if I want to build and use/deploy a Python app that needs the latest NumPy, and the system package manager doesn’t have it. It would be hard to justify for me to figure out and build a distro specific package for this rather than just using the Python package on PyPI.
      - bluGill 2y ago
        The point is the distro should provide the numpy you need.
tsimionescu 2y ago
I think the opposite is mostly true. Linux packaging folks are carefully sculpting their toys, while everyone else is mostly using upstream packages and docker containers to work around the beautiful systems. For half the software I care about on my Debian system, I have a version installed either directly from the web (curl | bash style), from the developer's own APT repo, or most likely from a separate package manager (be it MELPA, pypi, Go cache, Maven, etc).
- justinrubek 2y ago
  That sounds like an incredibly annoying way to manage software. I don't have a single thing installed that way.
- kombine 2y ago
  I use nix package manager on 3 of the systems I'm working daily (one of them HPC cluster) and none of them run NixOS. It's possible to carefully sculpt one's tools and use latest and greatest.
- deleted 2y ago
  [deleted]

kachapopopow 2y ago

[flagged]

sheepscreek 2y ago

YES! I want more tools to be deterministic. My wish-list has Proxmox config at the very top.

TheDong 2y ago
Want to give this a try and see if it works? https://github.com/SaumonNet/proxmox-nixos?tab=readme-ov-file#-declarative-vms https://github.com/SaumonNet/proxmox-nixos?tab=readme-ov-fil...

knowitnone 2y ago

99%? Debbie Downer says it only takes 1 package to screw the pooch

ethersteeds 2y ago
I would still much prefer playing 100:1 Russian roulette than 1:1, if those are my options.
nwah1 2y ago
There's a long tail of obscure packages that are rarely used, and almost certainly a power law in terms of which packages are common. Reproducibility often requires coordination between both the packagers and the developers, and achieving that for each and every package is optimistic. If they just started quarantining the long tail of obscure packages, then people would get upset. And failing to be 100% reproducible will make a subset of users upset. Lose-lose proposition there, given that intelligent users could just consciously avoid packages that aren't passing reproducibility tests. 100% reproducibility is a good goal, but as long as the ubiquitous packages are reproducible then that is probably going to cover most. Would be interesting to provide an easy way to disallow non-reproducible packages. I'm sure one day they will be able to make it a requirement for inclusion into the official repos.
- yjftsjthsd-h 2y ago
  There's an interesting thought - in addition to aiming for 99% of all packages, perhaps it would be a good idea to target 100% of the packages that, say, land in the official install media? (I wouldn't even be surprised if they already meet that goal TBH, but making it explicit and documenting it has value)
EasyMark 2y ago
"All I see is 1% of complete failure" --Bad Dads everywhere

patrakov 2y ago

This goal feels like a marketing OKR to me. A proper technical goal would be "all packages, except the ones that have a valid reason, such as signatures, not to be reproducible".

RegnisGnaw 2y ago
As someone who dabbles a bit in the RHEL world, IIRC all packages in Fedora are signed. In additional the DNF/Yum meta-data is also signed. IIRC I don't think Debian packages themselves are signed themselves but the apt meta-data is signed.
- westurner 2y ago
  I learned this from an ansible molecule test env setup script for use in containers and VMs years ago; because `which` isn't necessarily installed in containers for example: type -p apt && (set -x; apt install -y debsums; debsums | grep -v 'OK$') || \ type -p rpm && rpm -Va # --verify --all dnf reads .repo files from /etc/yum.repos.d/ [1] which have various gpg options; here's an /etc/yum.repos.d/fedora-updates.repo: [updates] name=Fedora $releasever - $basearch - Updates #baseurl=http://download.example/pub/fedora/linux/updates/$releasever/Everything/$basearch/ metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-released-f$releasever&arch=$basearch enabled=1 countme=1 repo_gpgcheck=0 type=rpm gpgcheck=1 metadata_expire=6h gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch skip_if_unavailable=False From the dnf conf docs [1], there are actually even more per-repo gpg options: gpgkey gpgkey_dns_verification repo_gpgcheck localpkg_gpgcheck gpgcheck 1. https://dnf.readthedocs.io/en/latest/conf_ref.html#repo-options https://dnf.readthedocs.io/en/latest/conf_ref.html#repo-opti... 2. https://docs.ansible.com/ansible/latest/collections/ansible/builtin/yum_repository_module.html#parameter-gpgcakey https://docs.ansible.com/ansible/latest/collections/ansible/... lists a gpgcakey parameter for the ansible.builtin.yum_repository module For Debian, Ubuntu, Raspberry Pi OS and other dpkg .deb and apt distros: man sources.list man sources.list | grep -i keyring -C 10 # trusted: # signed-by: # /etc/apt/ trusted.gpg.d/ man apt-secure man apt-key apt-key help less "$(type -p apt-key)" signing-apt-repo-faq: https://github.com/crystall1nedev/signing-apt-repo-faq https://github.com/crystall1nedev/signing-apt-repo-faq From "New requirements for APT repository signing in 24.04" (2024) https://discourse.ubuntu.com/t/new-requirements-for-apt-repository-signing-in-24-04/42854 https://discourse.ubuntu.com/t/new-requirements-for-apt-repo... : > In Ubuntu 24.04, APT will require repositories to be signed using one of the following public key algorithms: [ RSA with at least 2048-bit keys, Ed25519, Ed448 ] > This has been made possible thanks to recent work in GnuPG 2.4 82 by Werner Koch to allow us to specify a “public key algorithm assertion” in APT when calling the gpgv tool for verifying repositories.

binarymax 2y ago

I often see initiatives and articles like this but no mention of Nix. Is it just not well known enough for comparison? Because to me that’s the standard.

esseph 2y ago
It's an article about Fedora, specifically.
- binarymax 2y ago
  Yes, I know that. But when talking about reproducible packages, we can and should learn from existing techniques. Is Fedoras system capable of this goal? Is it built the right way? Should it adopt an alternate package manager to achieve this with less headache?
  - dogleash 2y ago
    "What about Nix tho?" is the "Rewrite it in Rust" of reproducible builds.
  - deleted 2y ago
    [deleted]
  - johnny22 2y ago
    No other currently distro whether it be debian, fedora, or arch, or opensuse seems to want to switch up their design to match nix's approach. They have too many people familiar with the current approaches.
    - nonameiguess 2y ago
      That's not even really true. Hacker News readers seem to fixate heavily on the most popular desktop versions of Linux distros, but that is far from all they make. SUSE with SLE Micro and Edge Image Builder is a declarative build system for creating your own installer images. If you don't want SLE Micro, they even give you the Elemental Toolkit, which can be used to build your own custom distro in a declarative manner. I don't know much about Debian and Red Hat's approaches to this, but I know they have similar tooling. The "problem," if you want to call it that, is developers, if I may be uncharitable for a brief moment here, are amazingly solipsistic. The major Linux vendors are targeting enterprise edge computing use cases, whereas developers only ever think about the endpoint PCs they themselves use for development. Plenty of distro tooling will give you declaratively built, transactionally updated, immutable server images, but if they don't give you the same for your personal laptop, those efforts are invisible to the modal Hacker News commenter. For what it's worth, there's also Guix, which is literally a clone of Nix but part of the GNU project, so it only uses free software and opts for Guile instead of a custom DSL for configuration. It wasn't a pre-existing distro that changed, of course.

froh 2y ago

nice to see they're in this too. https://news.opensuse.org/2025/02/18/rbos-project-hits-milestone/ https://news.opensuse.org/2025/02/18/rbos-project-hits-miles...

charcircuit 2y ago

This is a waste of time compared to investing in sandboxing which will actually protect users as opposed to stopping theoretical attacks. Fedora's sandbox capabilities for apps is so far behind other operating systems like Android that it is much more important of an area to address.

johnny22 2y ago
I think you have to do both sandboxing and this.
- charcircuit 2y ago
  Both are good for security, but prioritization is important. The areas that are weakest in terms of security should get the most attention.
  - jcgl 2y ago
    That assumes that there is a fixed amount of effort and resources to be split between the two things—that there is always opportunity cost. When it comes to community efforts, it’s rarely the case that all things have opportunity cost—people who contribute effort for X would not have necessarily done so for Y. The Fedora Project is of course not a purely community effort, so I don’t know exactly how that applies here. But just wanted to point out that prioritization and opportunity cost don’t always work like you suggested.
AshamedCaptain 2y ago
I am yet to see a form of sandboxing for the desktop that is not: a) effectively useless or b) makes me want to throw my computer through the window and replace it with a 1990's device (still more useful than your average Android).
fsflover 2y ago
If you want security through compartmentalization, you should consider Qubes OS, my daily driver, https://qubes-os.org https://qubes-os.org.

nimish 2y ago

As a user of fedora what does this actually get me? I mean I understand it for hermetic builds but why?

deleted 2y ago
[deleted]
jacobgkau 2y ago
My impression is that reproducible builds improve your security by helping make it more obvious that packages haven't been tampered with in late stages of the build system. * Edit, it's quoted in the linked article: > Jędrzejewski-Szmek said that one of the benefits of reproducible builds was to help detect and mitigate any kind of supply-chain attack on Fedora's builders and allow others to perform independent verification that the package sources match the binaries that are delivered by Fedora.
- Zamicol 2y ago
  Bingo.
- deleted 2y ago
  [deleted]
- kazinator 2y ago
  The supply chain attacks you have to most worry about are not someone breaking into Fedora build machines. It's the attacks on the upstream packages themselves. Reproducible builds would absolutely not catch a situation like the XZ package being compromised a year ago, due to the project merging a contribution from a malicious actor. A downstream package system or OS distro will just take that malicious update and spin it into a beautifully reproducing build.
  - yjftsjthsd-h 2y ago
    Don't let the perfect be the enemy of the good; this doesn't prevent upstream problems but it removes one place for compromises to happen.

Dwedit 2y ago

Reproducibility is at odds with Profile-Guided-Optimization. Especially on anything that involves networking and other IO that isn't consistent.

michaelt 2y ago
Why should it be? Does the profiler not output a hprof file or whatever, which is the input to the compiler making the release binary? Why not just store that?
gnulinux 2y ago
It's not at odds at all but it'll be "Monadic" in the sense that the output of system A will be part of the input to system A+1 which is complicated to organize in a systems setting, especially if you don't have access to a language that can verify. But it's absolutely achievable if you do have such a tool, e.g. you can do this in nix.
zbobet2012 2y ago
That's only the case if you did PGO with "live" data instead of replays from captured runs, which is best practice afaik.
nrvn 2y ago
from Go documentation[0]: > Committing profiles directly in the source repository is recommended as profiles are an input to the build important for reproducible (and performant!) builds. Storing alongside the source simplifies the build experience as there are no additional steps to get the profile beyond fetching the source. I very much hope other languages/frameworks can do the same. [0]: https://go.dev/doc/pgo#building https://go.dev/doc/pgo#building
- nyrikki 2y ago
  The Performant claim there is counter to research I have heard. Plus as the PGO profile data is non-deterministic in most cases, even when compiled on the same hardware as the end machine "Committing profiles directly in the source repository" is the reason why they are deleted or at least excluded from the comparison. A quote from the paper that I remember on the subject[1] as these profiles are just about as machine dependent as you can get. > Unfortunately, most code improvements are not machine independent, and the few that truly are machine independent interact with those that are machine dependent causing phase-ordering problems. Hence, effectively there are no machine-independent code improvements. There were some differences between various Xeon chip's implementations of the same or neighboring generations that I personally ran into when we tried to copy profiles to avoid the cost of the profile runs that may make me a bit more sensitive to this, but I personally saw huge drops in performance well into the double digits that threw off our regression testing. IMHO this is exactly why your link suggested the following: > Your production environment is the best source of representative profiles for your application, as described in Collecting profiles. That is very different from Fedora using some random or generic profile for x86_64, which may or may not match the end users specific profile. [1] https://dl.acm.org/doi/10.5555/184716.184723 https://dl.acm.org/doi/10.5555/184716.184723

barotalomey 2y ago

The real treasure was the friend I found along the way https://github.com/keszybz/add-determinism https://github.com/keszybz/add-determinism

m463 2y ago
I kind of wonder if this or something similar could somehow nullify timestamps so you could compare two logfiles... further would be the ability to compare logfiles with pointer addresses or something
- didericis 2y ago
  A different but more powerful method of ensuring reproducibility is more rigorous compilation using formally verifiable proofs. That’s what https://pi2.network/ https://pi2.network/ does. It uses K-Framework, which is imo very underrated/deserves more attention as a long term way of solving this kind of problem.
- yjftsjthsd-h 2y ago
  I'm not confident that I understand what you're asking for, but couldn't you just sed off the timestamp from every line? Or for a more extreme example, I have occasionally used ... tr, I think? to completely remove all numbers from logs in order to aggregate error messages without worrying about the fact that they kept including irrelevant changing numbers (something like tail -5000 logfile | tr -d [0-9] | sort | uniq -c | sort -n or so).
  - winwang 2y ago
    how would you do it if your logs were printed on paper with a printer, each line printed with stochastic timing (due to a bug), with an ink containing a chemical tracer with halflife `h` (after being put to paper), but the ink is randomly sampled from several (`m`) inks of different halflives `h1`, h2`,... `hn`? assume `p` different printers scattered across the 10 most populous US cities. you may use standard unix utilities.

unit149 2y ago

[dead]

frainfreeze 2y ago

Amazing to see this progress! Cudos to everyone who put in the effort. Related news from March https://news.ycombinator.com/item?id=43484520 https://news.ycombinator.com/item?id=43484520 (Debian bookworm live images now fully reproducible)

kccqzy 2y ago

> For example, Haskell packages are not currently reproducible when compiled by more than one thread Doesn't seem like a big issue to me. The gcc compiler doesn't even support multithreaded compiling. In the C world, parallelism comes from compiling multiple translation units in parallel, not any one with multiple threads.

apatheticonion 2y ago

Another thing I'd love to see is more statically linked binaries. Something like Python, for instance, is a nightmare to install and work with

theteapot 2y ago
I think general consensus is against you. Fedora packaging policy [1]: > Packages including libraries should exclude static libs as far as possible (eg by configuring with --disable-static). Static libraries should only be included in exceptional circumstances. Applications linking against libraries should as far as possible link against shared libraries not static versions. [1]: https://docs.fedoraproject.org/en-US/packaging-guidelines/ https://docs.fedoraproject.org/en-US/packaging-guidelines/
- Brian_K_White 2y ago
  I'd far rather a static binary than a bundled vm for a single app which produces all the same bad points of a static binary plus 900 new bad points on top. Packaging guidelines from a distros docs like this are not any kind of counter argument to that comment. This is the current orthodoxy, so obviously all docs say it. We all know the standard argument for the current standard. Their comment was explicitly "I'd like to see a change from the current orthodoxy". They are saying that maybe that argument is not all it promised to be back in the 90's when we started using dynamic libs.
  - booi 2y ago
    So instead of 1000 programs and 1000 libraries, you’d rather 1000 programs and 1,000,000 libraries?
    - Brian_K_White 2y ago
      Baffled how you got there, but not interested.
      - 2y ago

trod1234 2y ago

Can someone provide a brief clarification about build reproducibility in general? The stated aim is that when you compile the same source, environment, and instructions the end result is bit identical. There is, however; hardware specific optimizations that will naturally negate this stated aim, and I don't see how there's any way to avoid throwing out the baby with the bathwater. I understand why having a reproducible build is needed on a lot of fronts, but the stated requirements don't seem to be in line with the realities. At its most basic, there is hardware, where the hardware may advertise features it doesn't have, or doesn't perform the same instructions in the same way, and other nuances that break determinism as a property, and that naturally taints the entire stack since computers rely heavily on emergent design. This is often hidden in layers of abstraction and/or may be separated into pieces that are architecture dependent vs independent (freestanding), but it remains there. Most if not all of the beneficial properties of reproducible builds rely on the environment being limited to a deterministic scope, and the reality is manufacturers ensure these things remain in a stochastic scope.

amarshall 2y ago
Well the point is that if N of M machines produce the same output, it provides the opportunity to question why it is different on the others. If the build is not reproducible then one just throws up their arms. It’s not clear if you’re also talking about compiler optimizations—a reproducible build must have a fixed target for that.
- trod1234 2y ago
  I was thinking more from a reference point along the lines of LLVM type performance optimizations when I was speaking about optimizations, if that sufficiently clarifies.
Crestwave 2y ago
> hardware specific optimizations that will naturally negate this stated aim Distro packages are compiled on their build server and distributed to users with all kinds of systems; therefore, by nature, it should not use optimizations specific to the builder's hardware. On source-based distros like Gentoo, yes, users adding optimization flags would get a different output. But there is still value in having the same hardware/compilation flags result in the same output.

curtisszmania 2y ago

[dead]

koreanguy 2y ago

[dead]

kaelit 2y ago

Interesting take. I'm building something related to zk systems — will share once it's up.

nigel_doug 2y ago

"trust in the contents of both source and binary packages is low." - I wonder will this convince organisations to adopt proper artifact management processes? If supply chain attacks are on the rise, than surely its more imperative than ever for businesses to adopt secure artifact scanning with tools like Cloudsmith or jFrog?