3 ms·

Working for a vendor that ships Linux code, we find regressions all the time. Because we actually test things. We do try to be good citizens, and Linux is the r

Working for a vendor that ships Linux code, we find regressions all the time. Because we actually test things. We do try to be good citizens, and Linux is the right answer, but it isn't a false assertion on the part of the author at all. In a previous job, 20 years ago, we found that the Linux USB stack was so bad that it was full of deadlocks and unprotected shared memory on 2.6. Just atrocious. And subsequent releases broke it again and again. About 5 years after that, working for a different company, I had a well-known kernel developer tell me point blank that as long as we release our code to the kernel, they will "keep it working." When I probed what that meant, he genuinely saw no difference between "compiling" and "working." There are various efforts to add testing/tests. Those are good. But we lost a lot when Solaris, Irix, etc. all died, and Linux has never quite gotten there, and I doubt it ever will.

ryao 2y ago
OpenSolaris still lives in illumos: https://illumos.org/ https://illumos.org/
- LeFantome 2y ago
  I am sure what he means that is we lost a lot when it become unprofitable for companies to offer solid UNIX offerings that were supported in the way he would like. Other than the fact that it evolves glacially, Illumos is in an even worse position than Linux when it comes to offering support. If we are being honest though, in the terms that this article is arguing, "Linux" as an ecosystem is the wrong label. When it comes to stability and support, we need to think of each Linux distro as an Operating System. In that sense, things like RHEL are certainly well supported--including on the desktop. If your app targets RHEL, you pretty much have none of the problems discussed in this article. You still have a smaller universe of applications than Windows of course which is to be expected. So, the problem boils down to "nobody is making sufficient money off desktop Linux to steward a desktop Linux distro". That is not careless devs or bad design decisions though--no matter how chicken-and-egg you try to paint it. It is simply the network effect. Decent desktop operating systems exist and they already have large application universes and large, monetizable user bases. They appeared into a vacuum that lacked such operating systems but that vacuum no longer exists. That makes it almost impossible to economically introduce a new desktop platform. Certainly none of the commercial attempts have succeeded in the last few decades. In fact, the system that has been the MOST successful is Linux. By that metric, saying that Linux is doing it all wrong does not ring very true. Valve is doing a lot to make Linux gaming a reality because, when it comes to the network effect, it is largely true that Steam ( not just Windows ) is the true platform. So, if you can bring Steam to Linux, you can move the users there too. It is still early days for that but it appears to be working. On the mobile side, the network effect is very much alive as well. This is why Windows, using of course all of Microsoft's tricks, strengths, and knowledge, completely failed while the Linux kernel ( in the form of Android ) has totally dominated. And that is not for any of the reasons cited in the article. Do you think an Android app from 2008 will run on Android today unmodified? An iOS app from 2008 on iOS today? I don't think so.
imiric 2y ago
> [...] it isn't a false assertion on the part of the author at all. I wasn't arguing that it was entirely false, but that the generalizations of "regressions are introduced all the time" and "Linux developers spend little to no time" can't be applied to the entire Linux ecosystem. I wouldn't discredit your experience, but speaking of Linux kernel development specifically, my understanding as a user is that it's held to a very high standard of scrutiny. Rants from Linus are famous for drilling into people that "WE DON'T BREAK USERSPACE". If you consider how many thousands of lines of code are changed on a regular basis, it's an engineering wonder that it works as well as it does. In many ways Linux is a victim of its success.
- foobiekr 2y ago
  Yes, they are very careful about not breaking userspace. That has nothing to do with holding the _user_ in high regard, though. The sweeping changes in Linux are amazing, but they also break a lot, frequently, and the testing approach leaves a lot to be desired. I am old and a former embedded coder, I understand how the 90s developers feel about unit testing, and I am sympathetic to the idea that test-driven-development and heavy UT in modern teams is often completely out of control, but Linux is distinctly lacking in real testing of independent units in isolation. I am aware of kunit, kselftest, etc.
  - tuna74 2y ago
    If you report a regression within an -RC series it will be fixed or the commit reverted. Of course, that requires actual testing.
    - foobiekr 2y ago
      I am very familiar with reporting bugs on the Linux kernel, as you may imagine from the previous comments. But you’re just reiterating that the kernel development itself lacks sufficient testing which necessitates this. Everyone is a volunteer; the result is as you’d expect. Look, there’s nothing wrong with this. In reality all the people who are electing to run Linux without it actually having sufficient quality for their purposes are the actual problem, who’d which there are hundreds of companies that just don’t care. We created this situation by undervaluing quality vs cost. But it’s also the reason it won’t be a success on the desktop. The only way that happens is if windows takes an even more substantial nosedive. Things are going to have to get pretty bad for that to happen.
devops99 2y ago
illumos is the living and thriving fork of OpenSolaris. I run some customers' private clouds on illumos. There are some contributions we may be able to upstream before 2025Q3. Looking back on how things panned out, going all in on illumos a bit over ten years ago is something I wish I had done. addendum: by "all in" I mean as an operating systems developer.

ryao 2y ago

OpenSolaris still lives in illumos: https://illumos.org/ https://illumos.org/

LeFantome 2y ago
I am sure what he means that is we lost a lot when it become unprofitable for companies to offer solid UNIX offerings that were supported in the way he would like. Other than the fact that it evolves glacially, Illumos is in an even worse position than Linux when it comes to offering support. If we are being honest though, in the terms that this article is arguing, "Linux" as an ecosystem is the wrong label. When it comes to stability and support, we need to think of each Linux distro as an Operating System. In that sense, things like RHEL are certainly well supported--including on the desktop. If your app targets RHEL, you pretty much have none of the problems discussed in this article. You still have a smaller universe of applications than Windows of course which is to be expected. So, the problem boils down to "nobody is making sufficient money off desktop Linux to steward a desktop Linux distro". That is not careless devs or bad design decisions though--no matter how chicken-and-egg you try to paint it. It is simply the network effect. Decent desktop operating systems exist and they already have large application universes and large, monetizable user bases. They appeared into a vacuum that lacked such operating systems but that vacuum no longer exists. That makes it almost impossible to economically introduce a new desktop platform. Certainly none of the commercial attempts have succeeded in the last few decades. In fact, the system that has been the MOST successful is Linux. By that metric, saying that Linux is doing it all wrong does not ring very true. Valve is doing a lot to make Linux gaming a reality because, when it comes to the network effect, it is largely true that Steam ( not just Windows ) is the true platform. So, if you can bring Steam to Linux, you can move the users there too. It is still early days for that but it appears to be working. On the mobile side, the network effect is very much alive as well. This is why Windows, using of course all of Microsoft's tricks, strengths, and knowledge, completely failed while the Linux kernel ( in the form of Android ) has totally dominated. And that is not for any of the reasons cited in the article. Do you think an Android app from 2008 will run on Android today unmodified? An iOS app from 2008 on iOS today? I don't think so.

imiric 2y ago

> [...] it isn't a false assertion on the part of the author at all. I wasn't arguing that it was entirely false, but that the generalizations of "regressions are introduced all the time" and "Linux developers spend little to no time" can't be applied to the entire Linux ecosystem. I wouldn't discredit your experience, but speaking of Linux kernel development specifically, my understanding as a user is that it's held to a very high standard of scrutiny. Rants from Linus are famous for drilling into people that "WE DON'T BREAK USERSPACE". If you consider how many thousands of lines of code are changed on a regular basis, it's an engineering wonder that it works as well as it does. In many ways Linux is a victim of its success.

foobiekr 2y ago
Yes, they are very careful about not breaking userspace. That has nothing to do with holding the _user_ in high regard, though. The sweeping changes in Linux are amazing, but they also break a lot, frequently, and the testing approach leaves a lot to be desired. I am old and a former embedded coder, I understand how the 90s developers feel about unit testing, and I am sympathetic to the idea that test-driven-development and heavy UT in modern teams is often completely out of control, but Linux is distinctly lacking in real testing of independent units in isolation. I am aware of kunit, kselftest, etc.
- tuna74 2y ago
  If you report a regression within an -RC series it will be fixed or the commit reverted. Of course, that requires actual testing.
  - foobiekr 2y ago
    I am very familiar with reporting bugs on the Linux kernel, as you may imagine from the previous comments. But you’re just reiterating that the kernel development itself lacks sufficient testing which necessitates this. Everyone is a volunteer; the result is as you’d expect. Look, there’s nothing wrong with this. In reality all the people who are electing to run Linux without it actually having sufficient quality for their purposes are the actual problem, who’d which there are hundreds of companies that just don’t care. We created this situation by undervaluing quality vs cost. But it’s also the reason it won’t be a success on the desktop. The only way that happens is if windows takes an even more substantial nosedive. Things are going to have to get pretty bad for that to happen.

devops99 2y ago

illumos is the living and thriving fork of OpenSolaris. I run some customers' private clouds on illumos. There are some contributions we may be able to upstream before 2025Q3. Looking back on how things panned out, going all in on illumos a bit over ten years ago is something I wish I had done. addendum: by "all in" I mean as an operating systems developer.