4 ms·

FLOSS developers are real heroes, but so are the people willing to spend time testing newer non-LTS versions of the code and report their issues. I have enough

by hnarn 3y ago

FLOSS developers are real heroes, but so are the people willing to spend time testing newer non-LTS versions of the code and report their issues. I have enough on my plate just dealing with the issues arising from using stable code, I think it’s admirable that people find the time raising their glance to future releases and helping us all enjoying a less panic-inducing experience.

georgyo 3y ago
In my experience, bleeding edge and stable are about the same amount of pain. Breakage isn't actually that common, and fixes come a lot faster. And even if you perfer stable, the latest will become stable eventually. Not trying your workload out on the next releases has pretty much the same risk profile of just running latest. Many problems can only be found by running your particular workload.
- ilyt 3y ago
  That seems to be mostly bathtub curve for most of the software for us when it comes to amount of work. Running on "latest commit from master" from many projects (not Linux) will just get you code nobody even tested and so a lot of bugs fixed quickly. Running on "latest stable" (whatever that means for project) means fixes from time to time when it updates, but in vast majority of cases not that much work. Anything behind that like LTS releases ? Extra work. Now any doc you find might be about never release or feature that changed. "Bugs" might not get fixed if they are not big enough to backport. Upgrade to new LTS version will also get you years of changes in app that you then have to apply to the system, vs having to do it "change by change" when keeping up to date. If you use configuration management that also often means multiple different configs to manage at the very least till previous LTS version gets finally upgraded
talhah 3y ago
Bleeding edge arch linux user here, I've barely come across any major bugs in the last couple of years. Whenever I find something I do report it and it usually gets fixed really quickly. In fact, many of these bugs were on stable releases too.
- awill 3y ago
  exactly. A RHEL kernel is likely a lot more stable than the kernel.org LTS kernel. Often bugfixes and security patches are backported to the LTS kernel, meaning both can be affected by similar bugs.
drewg123 3y ago
We run bleeding edge FreeBSD at Netflix and are never more than a few weeks behind the FreeBSD main branch. This has worked out quite well for us. We used to run -stable, and update every few years, like from FreeBSD 9.x to FreeBSD 10.x. We found that when we did that, we would often encounter some small subtle bug that was tickled in our environment, and which was incredibly hard to track down. That sort of bug was hard to track down because the diff between branches was enormous, and because there were thousands of commits to sift through, and because the person responsible for the bug may have committed it months or years ago, and has forgotten about it. We eventually decided to track the main branch, updating frequently. This means that while we find more bugs, but they are far easier to fix because they were introduced more recently, and there are a lot fewer commits to look through to find where they came from.
- hpb42 3y ago
  Is there a position open on your team? This sounds like the stuff I'm into!
- freedomben 3y ago
  Fascinating! I worked with a company that figured out a similar thing with ruby and rails versions. By staying close to master, it's much easier to figure out what broke when things happen.
- bombcar 3y ago
  This is why I prefer rolling distributions compared to stable ones, sure you only have to upgrade an LTS every few years but everything will break; whereas the small breaks you get with a rolling are easier to track down and diagnose.
- hnarn 3y ago
  It’s an interesting take that makes a lot of sense, but I still feel like the angle of “having less code to sift through to spot bugs” is very specific to your use case and high competence relative to other operations. For a lot of companies, inspecting source code and filing bugs directly is just not a capacity that exists, which is where LTS of a Linux distribution makes a bit more sense — and without throwing any shade on FreeBSD (I love it), maybe the smaller amount of users globally compared to Linux means that “stable” isn’t quite as stable, especially if you’re doing bleeding edge stuff anyway. I guess you could say the same thing is true for a company like Cloudflare considering their network related patches to the Linux kernel. Thanks for the perspective!

by hnarn 3y ago

georgyo 3y ago

In my experience, bleeding edge and stable are about the same amount of pain. Breakage isn't actually that common, and fixes come a lot faster. And even if you perfer stable, the latest will become stable eventually. Not trying your workload out on the next releases has pretty much the same risk profile of just running latest. Many problems can only be found by running your particular workload.

ilyt 3y ago
That seems to be mostly bathtub curve for most of the software for us when it comes to amount of work. Running on "latest commit from master" from many projects (not Linux) will just get you code nobody even tested and so a lot of bugs fixed quickly. Running on "latest stable" (whatever that means for project) means fixes from time to time when it updates, but in vast majority of cases not that much work. Anything behind that like LTS releases ? Extra work. Now any doc you find might be about never release or feature that changed. "Bugs" might not get fixed if they are not big enough to backport. Upgrade to new LTS version will also get you years of changes in app that you then have to apply to the system, vs having to do it "change by change" when keeping up to date. If you use configuration management that also often means multiple different configs to manage at the very least till previous LTS version gets finally upgraded

talhah 3y ago

Bleeding edge arch linux user here, I've barely come across any major bugs in the last couple of years. Whenever I find something I do report it and it usually gets fixed really quickly. In fact, many of these bugs were on stable releases too.

awill 3y ago
exactly. A RHEL kernel is likely a lot more stable than the kernel.org LTS kernel. Often bugfixes and security patches are backported to the LTS kernel, meaning both can be affected by similar bugs.

drewg123 3y ago

We run bleeding edge FreeBSD at Netflix and are never more than a few weeks behind the FreeBSD main branch. This has worked out quite well for us. We used to run -stable, and update every few years, like from FreeBSD 9.x to FreeBSD 10.x. We found that when we did that, we would often encounter some small subtle bug that was tickled in our environment, and which was incredibly hard to track down. That sort of bug was hard to track down because the diff between branches was enormous, and because there were thousands of commits to sift through, and because the person responsible for the bug may have committed it months or years ago, and has forgotten about it. We eventually decided to track the main branch, updating frequently. This means that while we find more bugs, but they are far easier to fix because they were introduced more recently, and there are a lot fewer commits to look through to find where they came from.

hpb42 3y ago
Is there a position open on your team? This sounds like the stuff I'm into!
freedomben 3y ago
Fascinating! I worked with a company that figured out a similar thing with ruby and rails versions. By staying close to master, it's much easier to figure out what broke when things happen.
bombcar 3y ago
This is why I prefer rolling distributions compared to stable ones, sure you only have to upgrade an LTS every few years but everything will break; whereas the small breaks you get with a rolling are easier to track down and diagnose.
hnarn 3y ago
It’s an interesting take that makes a lot of sense, but I still feel like the angle of “having less code to sift through to spot bugs” is very specific to your use case and high competence relative to other operations. For a lot of companies, inspecting source code and filing bugs directly is just not a capacity that exists, which is where LTS of a Linux distribution makes a bit more sense — and without throwing any shade on FreeBSD (I love it), maybe the smaller amount of users globally compared to Linux means that “stable” isn’t quite as stable, especially if you’re doing bleeding edge stuff anyway. I guess you could say the same thing is true for a company like Cloudflare considering their network related patches to the Linux kernel. Thanks for the perspective!