5 ms·

I am an upstream maintainer of quite a few projects as well as a downstream maintainer of quite a few packages, so I've seen this from both sides of the exchang

by cyphar 7y ago

I am an upstream maintainer of quite a few projects as well as a downstream maintainer of quite a few packages, so I've seen this from both sides of the exchange. Sometimes the user-facing issue isn't an actual patch, it's the default configuration (which might be distribution-specific) or any other host of non-code changes (default security policy and so on). These things should be reported to the distribution, but often get reported upstream which acts as spam. And note that most downstream patches are backported bugfixes, not feature patches -- in my experience those are the exception. > So please submit the patch upstream, discuss it and wait for the next release. What if you're being paid for support and a customer needs a fix which you need to backport (it's not a CVE but it's a serious issue)? This is the problem we find ourselves in very often, and is why we push patches upstream but also patch our downstream packages so that the issue is resolved for our users. Some projects have release life-cycles that are ~6 months apart -- how do you explain to users that they need to wait 6 months in order for a fix which is already written and merged to be shipped to them? If we only ship it to those customers then it's not fair to the rest of our users nor the rest of the community (this is why SUSE has a Factory-first policy where all SLE changes must land in openSUSE first). And finally (in rare cases) the upstream might not accept patches that are required in order for some distribution features to work because they fundamentally disagree with the feature. What are we supposed to do in those cases? Either way, someone will complain because the best solution (merge it upstream) is closed off. While it would be ideal if downstream patching was unnecessary, that's not the world we live in. Again, I do submit patches upstream religiously -- but it's not as cut-and-dry as you might think.

pg_bot 7y ago
Fork the project, prefix the command with your company name and tell them to use that command with the patch applied if they need it today. This solves all your problems.
- cyphar 7y ago
  I think you're over-estimating how many users are willing to change their workflow or scripts to accommodate things like this. They will switch away from your distribution to one which patches the actual package. I also don't agree this would result in fewer upstream bug reports -- "suse-foobar" will still result in bug reports for "foobar" (I've seen some cases of this happening). You'd need to rename the project entirely so that users don't know what the upstream GitHub repo is, and that's even more anti-community than any other suggestion.
  - wolfgke 7y ago
    > and that's even more anti-community than any other suggestion. Shipping a fork that is different from the version that is created by the original author is also very anti-community. At least make it clear that you ship something different from the program that the author maintains.
    - cyphar 7y ago
      The key phrase was "even more". I disagree that all patches are somehow ethically wrong (bugfixes and security patches are obvious counterexamples). Not to mention that if the author felt otherwise, they wouldn't have released the code under a license that allowed you to modify and redistribute your modifications. But making massive changes to a project that are incompatible with upstream is definitely not a good thing to do without reason.
      - wolfgke 7y ago
        > I disagree that all patches are somehow ethically wrong (bugfixes and security patches are obvious counterexamples). Not to mention that if the author felt otherwise, they wouldn't have released the code under a license that allowed you to modify and redistribute your modifications. The fact that the author does not forbid it does not mean that he/she wants changes or even encourages them. It just means that the author believes that the downsides of completely disallowing changes are even worse.
        cyphar 7y ago
        There are free software licenses that require you to rename the project if you modify it. But I digress. This whole discussion is about trade-offs -- if you cannot get the patch upstream but you need to ship it what is the next best thing. I would contend that patching it is better than patching and renaming because renaming doesn't help solve the problem (unless you are very radical and rename the project entirely) and makes things less convenient for users. And note that distribution users are part of the community of people using the software.
kbknapp 7y ago
How about using some of that revenue to pay the project maintainer to incorporate the patch upstream and make a release? I've seen this very often in my personal experience that a company says exactly like you're saying now, "We have paying customers who demand certain patches, but the upstream project may be unable or unwilling to patch and release...so we downstream only." Alternatively, some downstream consumers simply "throw code over the wall" in the form of an upstream patch. However, those patches are sometimes duct-tape solutions which may not fit into the overall architecture/vision of the project maintainer(s). It's not fair to say, "accept this or else..." where the 'or else' is a downstream deviation (which in turn sometimes forces the upstream's hand). The ethical way to do this work with upstream, whether it be direct compensation or more back-and-forth vice code-over-the-wall.
- cyphar 7y ago
  I think it's a incredibly unfair to assume that I'm not acting in good faith when I send patches upstream. I was contributing to the free software community long before I was getting paid to do so. I spend almost all of my working hours doing upstream maintainership work or writing patches. I have PRs that are several years old and I still am working on getting them merged upstream. I'm also just a developer, I'm not in a position to start spending money on projects on behalf of my employer. I do happen to donate to plenty of upstream projects and foundations, but that's money out of my own pocket. The issue is that there are some (rare) cases where upstream is completely unwilling to merge a patch for philosophical reasons. If you want me to be blunt -- the example I'm thinking of is Docker (which doesn't have a cash-flow problem), and has refused outright on many occasions to merge patches that allow for build-time secrets. On SLE we need this because in order to use a SLE image on a SLE machine you need to "register it" and the only way to sanely register it automatically is to bind-mount some files into all containers on-build as well as on-run (which you cannot do with upstream Docker today). Red Hat spent a long time trying to get a patch like this upstream, as did we.
  - kbknapp 7y ago
    To be clear, I'm not say you personally. I don't know you, or anything about your work so I'm speaking in purely a general sense about a position that you were representing. I don't think all upstream contributions are in bad faith. I'm just saying there tends to be competing priorities which leads to some instances of hostility. As for the Docker example, I don't know what the right answer is without digging deeper. My naive thought is to write a SLE/RHEL shim/plugin style component to allow functionality that's missing. This allows keeping the upstream vanilla, while not having to fork into something without the brand recognition. If that doesn't work, forking as `rhel-docker` or `sle-docker` doesn't seem that bad to me. Ubuntu does this with all the bcc-tools. This is of course predicated on having tried all previous solutions first (paying an upstream developer, work with upstream with a good back-and-forth to incorporate a patch, etc). In the end, if the project decides something against their philosophical viewpoint, they're perfectly entitled to not accept patches. It's at that point, I think it's not the best solution to fork, downstream patch, and release as if it's the vanilla upstream.
zapita 7y ago
All these problems are very easy to solve. Suse and Red Hat could stop trying to have it both ways: either you get the flexibility of a fork, or you get the brand recognition of the upstream project. But you want both - the upstream brand to get users, and the downstream flexibility so that you can “differentiate” and sell support. This problem is entirely created by distributions and their “packagers know best” philosophy. You say downstream patching is necessary. Go ahead and strip the upstream trademark from every package which you patch downstream, replace it with new Suse-specific trademarks, and let’s see what users prefer!
- ecnahc515 7y ago
  This isn't a SUSE or Red Hat caused issue, it's a users thing. Users want it both ways. Users want the flexibility to assume a project is the same as the upstream (but maybe feature frozen at a specific version), while also wanting all the security and bugs to not exist. Users are paying to have it both ways, and whether or not it's feasible, the market for it exists, and Suse/Red Hat are just trying to fill the gap. If it was entirely the vendors, then we would see significantly more adoption in fast moving OSes, but that's still not a very popular model for production servers, even in the age of the cloud, containers, etc.

by cyphar 7y ago

pg_bot 7y ago

Fork the project, prefix the command with your company name and tell them to use that command with the patch applied if they need it today. This solves all your problems.

cyphar 7y ago
I think you're over-estimating how many users are willing to change their workflow or scripts to accommodate things like this. They will switch away from your distribution to one which patches the actual package. I also don't agree this would result in fewer upstream bug reports -- "suse-foobar" will still result in bug reports for "foobar" (I've seen some cases of this happening). You'd need to rename the project entirely so that users don't know what the upstream GitHub repo is, and that's even more anti-community than any other suggestion.
- wolfgke 7y ago
  > and that's even more anti-community than any other suggestion. Shipping a fork that is different from the version that is created by the original author is also very anti-community. At least make it clear that you ship something different from the program that the author maintains.
  - cyphar 7y ago
    The key phrase was "even more". I disagree that all patches are somehow ethically wrong (bugfixes and security patches are obvious counterexamples). Not to mention that if the author felt otherwise, they wouldn't have released the code under a license that allowed you to modify and redistribute your modifications. But making massive changes to a project that are incompatible with upstream is definitely not a good thing to do without reason.
    - wolfgke 7y ago
      > I disagree that all patches are somehow ethically wrong (bugfixes and security patches are obvious counterexamples). Not to mention that if the author felt otherwise, they wouldn't have released the code under a license that allowed you to modify and redistribute your modifications. The fact that the author does not forbid it does not mean that he/she wants changes or even encourages them. It just means that the author believes that the downsides of completely disallowing changes are even worse.
      - cyphar 7y ago
        There are free software licenses that require you to rename the project if you modify it. But I digress. This whole discussion is about trade-offs -- if you cannot get the patch upstream but you need to ship it what is the next best thing. I would contend that patching it is better than patching and renaming because renaming doesn't help solve the problem (unless you are very radical and rename the project entirely) and makes things less convenient for users. And note that distribution users are part of the community of people using the software.

kbknapp 7y ago

How about using some of that revenue to pay the project maintainer to incorporate the patch upstream and make a release? I've seen this very often in my personal experience that a company says exactly like you're saying now, "We have paying customers who demand certain patches, but the upstream project may be unable or unwilling to patch and release...so we downstream only." Alternatively, some downstream consumers simply "throw code over the wall" in the form of an upstream patch. However, those patches are sometimes duct-tape solutions which may not fit into the overall architecture/vision of the project maintainer(s). It's not fair to say, "accept this or else..." where the 'or else' is a downstream deviation (which in turn sometimes forces the upstream's hand). The ethical way to do this work with upstream, whether it be direct compensation or more back-and-forth vice code-over-the-wall.

cyphar 7y ago
I think it's a incredibly unfair to assume that I'm not acting in good faith when I send patches upstream. I was contributing to the free software community long before I was getting paid to do so. I spend almost all of my working hours doing upstream maintainership work or writing patches. I have PRs that are several years old and I still am working on getting them merged upstream. I'm also just a developer, I'm not in a position to start spending money on projects on behalf of my employer. I do happen to donate to plenty of upstream projects and foundations, but that's money out of my own pocket. The issue is that there are some (rare) cases where upstream is completely unwilling to merge a patch for philosophical reasons. If you want me to be blunt -- the example I'm thinking of is Docker (which doesn't have a cash-flow problem), and has refused outright on many occasions to merge patches that allow for build-time secrets. On SLE we need this because in order to use a SLE image on a SLE machine you need to "register it" and the only way to sanely register it automatically is to bind-mount some files into all containers on-build as well as on-run (which you cannot do with upstream Docker today). Red Hat spent a long time trying to get a patch like this upstream, as did we.
- kbknapp 7y ago
  To be clear, I'm not say you personally. I don't know you, or anything about your work so I'm speaking in purely a general sense about a position that you were representing. I don't think all upstream contributions are in bad faith. I'm just saying there tends to be competing priorities which leads to some instances of hostility. As for the Docker example, I don't know what the right answer is without digging deeper. My naive thought is to write a SLE/RHEL shim/plugin style component to allow functionality that's missing. This allows keeping the upstream vanilla, while not having to fork into something without the brand recognition. If that doesn't work, forking as `rhel-docker` or `sle-docker` doesn't seem that bad to me. Ubuntu does this with all the bcc-tools. This is of course predicated on having tried all previous solutions first (paying an upstream developer, work with upstream with a good back-and-forth to incorporate a patch, etc). In the end, if the project decides something against their philosophical viewpoint, they're perfectly entitled to not accept patches. It's at that point, I think it's not the best solution to fork, downstream patch, and release as if it's the vanilla upstream.

zapita 7y ago

All these problems are very easy to solve. Suse and Red Hat could stop trying to have it both ways: either you get the flexibility of a fork, or you get the brand recognition of the upstream project. But you want both - the upstream brand to get users, and the downstream flexibility so that you can “differentiate” and sell support. This problem is entirely created by distributions and their “packagers know best” philosophy. You say downstream patching is necessary. Go ahead and strip the upstream trademark from every package which you patch downstream, replace it with new Suse-specific trademarks, and let’s see what users prefer!

ecnahc515 7y ago
This isn't a SUSE or Red Hat caused issue, it's a users thing. Users want it both ways. Users want the flexibility to assume a project is the same as the upstream (but maybe feature frozen at a specific version), while also wanting all the security and bugs to not exist. Users are paying to have it both ways, and whether or not it's feasible, the market for it exists, and Suse/Red Hat are just trying to fill the gap. If it was entirely the vendors, then we would see significantly more adoption in fast moving OSes, but that's still not a very popular model for production servers, even in the age of the cloud, containers, etc.