9 ms·

I noticed that the two bugs they linked [1] [2] are both due to the distros introducing bugs by applying patches -- one for hurd support (!) -- that were not sh

by evmar 5y ago

I noticed that the two bugs they linked [1] [2] are both due to the distros introducing bugs by applying patches -- one for hurd support (!) -- that were not shipped by the upstream projects that the distros were repackaging. As we have discussed earlier [3] I think the way distros inject themselves into this software development process produces these bad outcomes due to getting the incentives wrong. [1] https://bugzilla.mozilla.org/show_bug.cgi?id=1679430 https://bugzilla.mozilla.org/show_bug.cgi?id=1679430 [2] https://bugzilla.mozilla.org/show_bug.cgi?id=1633467 https://bugzilla.mozilla.org/show_bug.cgi?id=1633467 [3] https://news.ycombinator.com/item?id=26216917 https://news.ycombinator.com/item?id=26216917

khuey 5y ago
This is exactly why historically Mozilla has required distributors to distribute an unmodified version of Mozilla software to use Mozilla trademarks such as the name Firefox, which (unlike the code) are not freely available. Unfortunately the specific bugs here are in library packages that Firefox uses and those have no such restrictions.
- IshKebab 5y ago
  If I were Mozilla I would strongly consider bundling more dependencies.
  - vetinari 5y ago
    Ironically, that was one of the reasons why distributions wanted their own builds.
  - trissylegs 5y ago
    They do have a bunch if you get it directly from them: /opt/firefox-nightly % ls *.so libfreeblpriv3.so liblgpllibs.so libmozavutil.so libmozsandbox.so libmozwayland.so libnss3.so libnssutil3.so libplc4.so libsmime3.so libssl3.so libgraphitewasm.so libmozavcodec.so libmozgtk.so libmozsqlite3.so libnspr4.so libnssckbi.so liboggwasm.so libplds4.so libsoftokn3.so libxul.so It did give me some grief a few weeks ago when their libmozwayland broke and I had to LD_PRELOAD libwayland to get it working.
- sneak 5y ago
  Yeah, I'm pretty thankful for distributions patching out or otherwise disabling the nonconsensual telemetry in browsers, Firefox included.
jcastro 5y ago
The blog post could be titled "Why the Linux distribution model is totally broken for end user software". As a linux user I appreciate the work, but it's tough to read this blog post and not think about the wasted effort that could be used to fix the bugs in the upstream software itself.
- ohthehugemanate 5y ago
  Really? Because I just think about how impossible this would be on Windows, where shared or external libs are inscrutable, as are OS build chains, version differences, etc etc.
  - gsvelto 5y ago
    We do gather symbol information on Windows too, but it's a lot less detailed than what we get from Linux. It also requires jumping through a lot of hoops. For example we get minimal information from Windows graphics drivers and we have to semi-manually scrape them ourselves.
  - kevingadd 5y ago
    To get symbols for basically any MS library I just add the official MS symbol server to my symbol search list. Chrome has one too, there's no real reason other vendors couldn't do it. You can also just ship a .pdb file next to your binaries, and I think the only good reasons for not doing that are keeping download sizes small (use a symbol server...) or paranoid secrecy. For example, NVIDIA's windows drivers are like 600mb at this point yet they don't even include function name symbols. It's inexcusable.
  - phendrenad2 5y ago
    Has Windows ever broken Firefox? Do you think it's more or less likely to happen in Windows (or Mac) or Linux?
vngzs 5y ago
Arch Linux does not do this, instead preferring to upstream patches [0]: > Arch Linux defines simplicity as without unnecessary additions or modifications. It ships software as released by the original developers (upstream) with minimal distribution-specific (downstream) changes: patches not accepted by upstream are avoided, and Arch's downstream patches consist almost entirely of backported bug fixes that are obsoleted by the project's next release. It's one of the core things that has kept me on Arch as a daily driver, even long after I've lost the urge to endlessly tweak my system configuration. I can trust that the software I use is simply vanilla upstream software with little or no modifications, and that's a great advantage when it comes to filing upstream bug reports and working on patches. In addition, it means the Arch Wiki is fairly general in its applicability, and effort spent documenting software for Arch can apply equally well to, for example, Void Linux (which also has this "vanilla software" philosophy). [0]: https://wiki.archlinux.org/title/Arch_Linux https://wiki.archlinux.org/title/Arch_Linux
- evmar 5y ago
  As a former Debian enthusiast (I even helped staff a Debian booth at a conference once!) who also tends to be conservative with new technology and who is consequently also skeptical of all these random Linux distros, I eventually tried out Arch and found it ... super awesome, I really love it! I highly recommend it to anyone else like me, who is generally cranky about new things. They did a really great job with Arch. This policy you mention is a great example of what I like about it.
  - rjzzleep 5y ago
    I've been using arch for a few years as daily driver, and I mostly like it, but I would be lying if I said it doesn't break at random intervals during system updates. Nothing unrecoverable, but it's definitely a thing that comes with frequent upgrades.
    - nemetroid 5y ago
      What part of it breaks for you? It's not that I don't believe you, and I've seen this sentiment before, but I've used Arch as my daily driver for about five years, and can't recall a single time where it broke from a system update.
intrepidhero 5y ago
I'm not sure I agree. To quote one of the cases: >For example, at some point, Debian updated the fontconfig package by backporting an upstream fix for a memory leak. Unfortunately, the fix contained a bug that would crash Firefox and possibly other software too. We spotted the new crash only six days after the change landed in Debian sources and only a couple of weeks afterwards the issue had been fixed both upstream and in Debian. We sent reports and fixes to other projects too including Mesa, GTK, glib, PCSC, SQLite and more. That sounds a lot like Linus' "many eyes make all bugs shallow" idea working as intended.
- gsvelto 5y ago
  Yes, that's one of the benefits. Firefox has a very large user-base compared to other FOSS projects so we will often spot bugs that others haven't noticed simply thanks to the sheer volume of crash reports we get.
- eikenberry 5y ago
  Even more to the point is that these bugs were found in Debian's unstable/testing distribution. Reiterating that the release process is happening as it should and the bugs are being found by people who volunteered to help test the software.
JoshTriplett 5y ago
I think it's a good thing that there's enough information to easily correlate things like crash reports from the latest bleeding-edge packages with uploads of dependency packages: > the libfontconfig1 package was updated on Debian on the 21st of April and the first crash we have on record was sent on the 22nd. Here's the changelog: I think that's a good argument for this kind of crash-telemetry, working in conjunction with Linux distributions. And it helped, in this case, that the dependencies could be updated independently from Firefox. On the other hand, the libdrm hurd patch breaking things on Linux seems like an excellent example of how portability to obscure platforms has a non-zero cost, and it isn't as simple as "just accept portability patches".
AnIdiotOnTheNet 5y ago
Agree wholeheartedly. As far as I am concerned the whole repo/package manager model is the root cause of most of the reasons the Linux Desktop experience is as awful as it is.
- hulitu 5y ago
  No, the reason is SW development process - developers which introduce big changes between library versions ( why do i need fontconfig 4.3.2_pl6 for example). When the library is a moving target any package which depend on it and also the security of the system become a moving target. Unfortunately Firefox does the same ( they discontinued ESR because it is boring fixing bugs).
  - jfk13 5y ago
    > they discontinued ESR Did they? Looks like it's still there: https://www.mozilla.org/en-US/firefox/all/#product-desktop-esr https://www.mozilla.org/en-US/firefox/all/#product-desktop-e...
- matheusmoreira 5y ago
  It's actually pretty great. Linux distributions have actual human maintainers that everybody trusts and they generally do the right thing. The fact random developers don't have direct access to people's installs is a major feature. If you want to release a Linux package, you should have to work with distribution maintainers in order to actually integrate your package with their ecosystem. It's not like PyPI/rubygems/npm/whatever where any random person can make an account and start pushing packages. The maintainers have to actually trust you.
  - AnIdiotOnTheNet 5y ago
    > It's actually pretty great. Subjective, but needless to say I disagree. I disagree with all of your reasoning too. Inserting a middleman into the software distribution process is needless friction and adds another point of failure. Even Linus has spoken on how much of a pain in the ass it has made it to distribute software: https://www.youtube.com/watch?v=5PmHRSeA2c8&t=340s https://www.youtube.com/watch?v=5PmHRSeA2c8&t=340s But hey, you guys keep doing what you're doing and ignoring what potential users want. I'm sure the Year of The Linux Desktop is just around the corner. P.S. Yes, I know, it's been your Year of the Linux Desktop since 2001 or whatever. We both know that's moving the goalposts. So is saying "but Android" since it replaced everything above the kernel with a completely new userspace stack and, oh yeah, it doesn't really run on the desktop anyway.
dan_quixote 5y ago
> the way distros inject themselves into this software development process produces these bad outcomes As a former maintainer of packages in an enterprise Linux ecosystem, I agree. But...the patches are often meant to solve a dependency mismatch and the maintainers can't easily pull related dependencies forward without breaking other applications. It's great that other distros like Arch just simply upstream the patches - it's the right thing to do after all - but that takes time that the enterprise distributions don't always have.
miohtama 5y ago
Switching to static binaries would solve some of these issues. Earlier discussion: https://news.ycombinator.com/item?id=27009044 https://news.ycombinator.com/item?id=27009044
- tadfisher 5y ago
  Are you saying switching to static binaries would help because... they are harder to patch? If I'm not mistaken, Debian and most Linux distros build Firefox from source. In this instance, if Firefox was only available as a statically-compiled binary, it would still be vulnerable to the issue Debian was patching!
bscphil 5y ago
> I noticed that the two bugs they linked [1] [2] This is misleading and in fact not even strictly true. They link three bugs, not two: https://bugzilla.mozilla.org/show_bug.cgi?id=1633459 https://bugzilla.mozilla.org/show_bug.cgi?id=1633459 This third bug was not due to Debian patches. And in fact the two Debian patch cases in the article were included to make the point that they can now catch issues caused by distribution patches, not to claim that most of the issues they find are caused in this way. Assuming on the basis of an article written to make a wholly different point that because 2/3 were the result of distribution patches that this must be a huge problem in general for Linux software development just shows your bias on this issue. I like and use Arch Linux on my desktops. But I'd defend the choice by Debian to patch. Most of Debian's changes are intended to improve support for certain setups or introduce bug or security fixes that upstream hasn't backported. These are both good things. Furthermore, problems that are created tend to be caught while they're still in unstable or sid: if I'm not mistaken that's exactly what happened in these cases, which is the process working as intended. (Mozilla is certainly free to disregard bug reports from Debian users if they feel that Debian's patching process is simply causing too many problems.)
- evmar 5y ago
  You are right, sorry for missing the third bug.

by evmar 5y ago

khuey 5y ago

This is exactly why historically Mozilla has required distributors to distribute an unmodified version of Mozilla software to use Mozilla trademarks such as the name Firefox, which (unlike the code) are not freely available. Unfortunately the specific bugs here are in library packages that Firefox uses and those have no such restrictions.

IshKebab 5y ago
If I were Mozilla I would strongly consider bundling more dependencies.
- vetinari 5y ago
  Ironically, that was one of the reasons why distributions wanted their own builds.
- trissylegs 5y ago
  They do have a bunch if you get it directly from them: /opt/firefox-nightly % ls *.so libfreeblpriv3.so liblgpllibs.so libmozavutil.so libmozsandbox.so libmozwayland.so libnss3.so libnssutil3.so libplc4.so libsmime3.so libssl3.so libgraphitewasm.so libmozavcodec.so libmozgtk.so libmozsqlite3.so libnspr4.so libnssckbi.so liboggwasm.so libplds4.so libsoftokn3.so libxul.so It did give me some grief a few weeks ago when their libmozwayland broke and I had to LD_PRELOAD libwayland to get it working.
sneak 5y ago
Yeah, I'm pretty thankful for distributions patching out or otherwise disabling the nonconsensual telemetry in browsers, Firefox included.

jcastro 5y ago

The blog post could be titled "Why the Linux distribution model is totally broken for end user software". As a linux user I appreciate the work, but it's tough to read this blog post and not think about the wasted effort that could be used to fix the bugs in the upstream software itself.

ohthehugemanate 5y ago
Really? Because I just think about how impossible this would be on Windows, where shared or external libs are inscrutable, as are OS build chains, version differences, etc etc.
- gsvelto 5y ago
  We do gather symbol information on Windows too, but it's a lot less detailed than what we get from Linux. It also requires jumping through a lot of hoops. For example we get minimal information from Windows graphics drivers and we have to semi-manually scrape them ourselves.
- kevingadd 5y ago
  To get symbols for basically any MS library I just add the official MS symbol server to my symbol search list. Chrome has one too, there's no real reason other vendors couldn't do it. You can also just ship a .pdb file next to your binaries, and I think the only good reasons for not doing that are keeping download sizes small (use a symbol server...) or paranoid secrecy. For example, NVIDIA's windows drivers are like 600mb at this point yet they don't even include function name symbols. It's inexcusable.
- phendrenad2 5y ago
  Has Windows ever broken Firefox? Do you think it's more or less likely to happen in Windows (or Mac) or Linux?

vngzs 5y ago

Arch Linux does not do this, instead preferring to upstream patches [0]: > Arch Linux defines simplicity as without unnecessary additions or modifications. It ships software as released by the original developers (upstream) with minimal distribution-specific (downstream) changes: patches not accepted by upstream are avoided, and Arch's downstream patches consist almost entirely of backported bug fixes that are obsoleted by the project's next release. It's one of the core things that has kept me on Arch as a daily driver, even long after I've lost the urge to endlessly tweak my system configuration. I can trust that the software I use is simply vanilla upstream software with little or no modifications, and that's a great advantage when it comes to filing upstream bug reports and working on patches. In addition, it means the Arch Wiki is fairly general in its applicability, and effort spent documenting software for Arch can apply equally well to, for example, Void Linux (which also has this "vanilla software" philosophy). [0]: https://wiki.archlinux.org/title/Arch_Linux https://wiki.archlinux.org/title/Arch_Linux

evmar 5y ago
As a former Debian enthusiast (I even helped staff a Debian booth at a conference once!) who also tends to be conservative with new technology and who is consequently also skeptical of all these random Linux distros, I eventually tried out Arch and found it ... super awesome, I really love it! I highly recommend it to anyone else like me, who is generally cranky about new things. They did a really great job with Arch. This policy you mention is a great example of what I like about it.
- rjzzleep 5y ago
  I've been using arch for a few years as daily driver, and I mostly like it, but I would be lying if I said it doesn't break at random intervals during system updates. Nothing unrecoverable, but it's definitely a thing that comes with frequent upgrades.
  - nemetroid 5y ago
    What part of it breaks for you? It's not that I don't believe you, and I've seen this sentiment before, but I've used Arch as my daily driver for about five years, and can't recall a single time where it broke from a system update.

intrepidhero 5y ago

I'm not sure I agree. To quote one of the cases: >For example, at some point, Debian updated the fontconfig package by backporting an upstream fix for a memory leak. Unfortunately, the fix contained a bug that would crash Firefox and possibly other software too. We spotted the new crash only six days after the change landed in Debian sources and only a couple of weeks afterwards the issue had been fixed both upstream and in Debian. We sent reports and fixes to other projects too including Mesa, GTK, glib, PCSC, SQLite and more. That sounds a lot like Linus' "many eyes make all bugs shallow" idea working as intended.

gsvelto 5y ago
Yes, that's one of the benefits. Firefox has a very large user-base compared to other FOSS projects so we will often spot bugs that others haven't noticed simply thanks to the sheer volume of crash reports we get.
eikenberry 5y ago
Even more to the point is that these bugs were found in Debian's unstable/testing distribution. Reiterating that the release process is happening as it should and the bugs are being found by people who volunteered to help test the software.

JoshTriplett 5y ago

I think it's a good thing that there's enough information to easily correlate things like crash reports from the latest bleeding-edge packages with uploads of dependency packages: > the libfontconfig1 package was updated on Debian on the 21st of April and the first crash we have on record was sent on the 22nd. Here's the changelog: I think that's a good argument for this kind of crash-telemetry, working in conjunction with Linux distributions. And it helped, in this case, that the dependencies could be updated independently from Firefox. On the other hand, the libdrm hurd patch breaking things on Linux seems like an excellent example of how portability to obscure platforms has a non-zero cost, and it isn't as simple as "just accept portability patches".

AnIdiotOnTheNet 5y ago

Agree wholeheartedly. As far as I am concerned the whole repo/package manager model is the root cause of most of the reasons the Linux Desktop experience is as awful as it is.

hulitu 5y ago
No, the reason is SW development process - developers which introduce big changes between library versions ( why do i need fontconfig 4.3.2_pl6 for example). When the library is a moving target any package which depend on it and also the security of the system become a moving target. Unfortunately Firefox does the same ( they discontinued ESR because it is boring fixing bugs).
- jfk13 5y ago
  > they discontinued ESR Did they? Looks like it's still there: https://www.mozilla.org/en-US/firefox/all/#product-desktop-esr https://www.mozilla.org/en-US/firefox/all/#product-desktop-e...
matheusmoreira 5y ago
It's actually pretty great. Linux distributions have actual human maintainers that everybody trusts and they generally do the right thing. The fact random developers don't have direct access to people's installs is a major feature. If you want to release a Linux package, you should have to work with distribution maintainers in order to actually integrate your package with their ecosystem. It's not like PyPI/rubygems/npm/whatever where any random person can make an account and start pushing packages. The maintainers have to actually trust you.
- AnIdiotOnTheNet 5y ago
  > It's actually pretty great. Subjective, but needless to say I disagree. I disagree with all of your reasoning too. Inserting a middleman into the software distribution process is needless friction and adds another point of failure. Even Linus has spoken on how much of a pain in the ass it has made it to distribute software: https://www.youtube.com/watch?v=5PmHRSeA2c8&t=340s https://www.youtube.com/watch?v=5PmHRSeA2c8&t=340s But hey, you guys keep doing what you're doing and ignoring what potential users want. I'm sure the Year of The Linux Desktop is just around the corner. P.S. Yes, I know, it's been your Year of the Linux Desktop since 2001 or whatever. We both know that's moving the goalposts. So is saying "but Android" since it replaced everything above the kernel with a completely new userspace stack and, oh yeah, it doesn't really run on the desktop anyway.

dan_quixote 5y ago

> the way distros inject themselves into this software development process produces these bad outcomes As a former maintainer of packages in an enterprise Linux ecosystem, I agree. But...the patches are often meant to solve a dependency mismatch and the maintainers can't easily pull related dependencies forward without breaking other applications. It's great that other distros like Arch just simply upstream the patches - it's the right thing to do after all - but that takes time that the enterprise distributions don't always have.

miohtama 5y ago

Switching to static binaries would solve some of these issues. Earlier discussion: https://news.ycombinator.com/item?id=27009044 https://news.ycombinator.com/item?id=27009044

tadfisher 5y ago
Are you saying switching to static binaries would help because... they are harder to patch? If I'm not mistaken, Debian and most Linux distros build Firefox from source. In this instance, if Firefox was only available as a statically-compiled binary, it would still be vulnerable to the issue Debian was patching!

bscphil 5y ago

> I noticed that the two bugs they linked [1] [2] This is misleading and in fact not even strictly true. They link three bugs, not two: https://bugzilla.mozilla.org/show_bug.cgi?id=1633459 https://bugzilla.mozilla.org/show_bug.cgi?id=1633459 This third bug was not due to Debian patches. And in fact the two Debian patch cases in the article were included to make the point that they can now catch issues caused by distribution patches, not to claim that most of the issues they find are caused in this way. Assuming on the basis of an article written to make a wholly different point that because 2/3 were the result of distribution patches that this must be a huge problem in general for Linux software development just shows your bias on this issue. I like and use Arch Linux on my desktops. But I'd defend the choice by Debian to patch. Most of Debian's changes are intended to improve support for certain setups or introduce bug or security fixes that upstream hasn't backported. These are both good things. Furthermore, problems that are created tend to be caught while they're still in unstable or sid: if I'm not mistaken that's exactly what happened in these cases, which is the process working as intended. (Mozilla is certainly free to disregard bug reports from Debian users if they feel that Debian's patching process is simply causing too many problems.)

evmar 5y ago
You are right, sorry for missing the third bug.