4 ms·

> This is not upstreamable in its current form What does "upstreamable" mean in this context? Is this good or bad? As a follow-up, what would need to be diffe

> This is not upstreamable in its current form What does "upstreamable" mean in this context? Is this good or bad? As a follow-up, what would need to be different so that it is upstreamable and would that be good or bad?

Veliladon 4y ago
Upstreamable means adding it to the official Linux source being distributed on linux.org. It just means you have to download drivers separately until Nvidia give the kernel guys a pull request for the drivers to be included in the kernel sources by default.
- eru 4y ago
  I think the more important thing than the separate download is that when someone works on the kernel, they usually make sure that the don't break other things that are in the official kernel. But there's no obligation not to break things outside the official kernel (apart from the interface to userland).
humanwhosits 4y ago
You want things to be 'upstreamable' as they can be included in the default upstream kernel. If it gets included in Linus' git tree then keeping it working on future version of the kernel is much less hassle
Tuna-Fish 4y ago
> What does "upstreamable" mean in this context? Is this good or bad? "Upstream" is the kernel source tree maintained by Linus (and friends). As Linux is open source, anyone can fork it for any reason, and a lot of people do. However, Linus still maintains the common base that people use to build on top of. "Upstreaming" is the process of getting your code included in these official releases. It's significant because there are no stable interfaces inside the kernel. That is, in any release the Linux developers can change any internal system of the kernel in any way they please, with the only caveat that to get that change included they have to fix up any breakage it causes, but only inside upstream. Or, if you are building your own kernels with your own patch sets, any release of the kernel can completely break your build. Because of this, if you want your code to be actually used by real people, upstreaming it is considered extremely good, to the point of being almost a necessity. However, it can be a significant hurdle to overcome. Partly because there are fairly strict code quality/style requirements (as any kernel dev is supposed to be able to jump into your code to fix issues they caused by modifications elsewhere), but mostly because unlike inside the kernel itself, all interfaces between kernel and userspace are stable. That is, if you start supporting some operation from userspace, you need to be able to support it in perpetuity. Also, the kernel devs are extremely reluctant to add any kind of interface to the kernel where the only thing that uses it is some specific non-free binary blob on the userspace side. Currently the main reason the code is not upstreamable as is is that it needs a very substantial userspace program to function, which currently only exists as nonfree, and nVidia doesn't think that the interface used to connect to it is well-enough decided they want to freeze it. So the main hurdle for upstreaming is developing that interface until they are happy with it, and then develop some free version of the userspace portion of their graphics stack that can plug into that same interface.
- MuffinFlavored 4y ago
  > upstreaming it is considered extremely good are we assuming some sort of automatic testing will be added? unit testing? integration testing? sounds super hard to do for a driver. but... high quality code just being merged into master can still have bugs introduced in the future on accident, right? it's tests that usually makes it safe?
  - rsaxvc 4y ago
    You can run these sorts of tests in a VM with PCIe passthrough.
    - djbusby 4y ago
      And hardware companies usually have large test-beds of hardware. So, Nvidia should be able to build a comprehensive test-suite.
      - MuffinFlavored 4y ago
        the same way they should have been able to build an upstreamable opensource driver all this time, right? lol
        djbusby 4y ago
        No. Cause one (hardware) is old-hat and the other (FOSS) is big and risky and scary.
  - loganberries 4y ago
    The kernel has bugs, all software has bugs. But there are massive massive test suites run by hundreds of companies on linux-next. If you email off a stack of patches that breaks something, chances are you'll get emails back telling you which patch had the breaking change in.
  - jsjohnst 4y ago
    The reason upstreaming it is considered extremely good really doesn’t have to do with unit testing or really exactly with testing at all. The issue, as stated eloquently by the parent of your post, is that without it being upstreamed, it’s Nvidia’s problem to fix any issues caused by one of their dependencies in the kernel changing, where as if it’s upstreamed, it’s only maybe partially their responsibility (main burden is on whoever changed the dependency).
nwallin 4y ago
'Upstreamable' means that they send Linus a pull request, and Linus merges it into the kernel. It would then become part of Linux. It's a negative that it's not upstreamable. I wouldn't go so far as to say it's bad, but it's less than good. That means you'll continue to maintain a separate kernel module that's rebuilt for each kernel. It would need to be modified to match the kernel's coding standards and whatnot. Part of what makes linux such a strong thing is that you're just not allowed to do certain things in the kernel. They won't include ZFS in the kernel because it's got its own entire vfs -- linux filesystems are only permitted to use linux's vfs, and if you need vfs changes to make it work, you modify the 'one true' vfs. They didn't include the original AMD Radeon driver because it had its own hardware abstraction layer; you're supposed to do that in user space or not at all. (the Radeon driver was changed to remove the abstraction layer, and it was later merged into the kernel) It's not immediately clear to me how much work it will take to get the driver into the kernel. Hopefully it won't be a lot of work, but it's possible that it would require basically rewriting it from scratch. It's at least written almost entirely in C, so that's a good start. Regardless, this is tremendously good news.
- prox 4y ago
  What would the practical effect be for me as a user, say I render in Blender, or I play a few intensive 3d games?
  - loganberries 4y ago
    Currently, you rely on Nvidia to keep the drivers up to date and working with the kernel you use in your OS. If the drivers aren't upstream, a future kernel can break them. So if you update your OS, it might no longer support your GPU without installing an alternative. Nvidia are responsible for getting this fixed in a timely manner. If the drivers are upstream, then people aren't allowed to break them when they make changes to the kernel. There should never be a kernel or OS upgrade that causes the drivers to no longer work, the people who made the breaking change are responsible for fixing it before it gets released to you.
    - 4y ago

Veliladon 4y ago

Upstreamable means adding it to the official Linux source being distributed on linux.org. It just means you have to download drivers separately until Nvidia give the kernel guys a pull request for the drivers to be included in the kernel sources by default.

eru 4y ago
I think the more important thing than the separate download is that when someone works on the kernel, they usually make sure that the don't break other things that are in the official kernel. But there's no obligation not to break things outside the official kernel (apart from the interface to userland).

humanwhosits 4y ago

You want things to be 'upstreamable' as they can be included in the default upstream kernel. If it gets included in Linus' git tree then keeping it working on future version of the kernel is much less hassle

Tuna-Fish 4y ago

> What does "upstreamable" mean in this context? Is this good or bad? "Upstream" is the kernel source tree maintained by Linus (and friends). As Linux is open source, anyone can fork it for any reason, and a lot of people do. However, Linus still maintains the common base that people use to build on top of. "Upstreaming" is the process of getting your code included in these official releases. It's significant because there are no stable interfaces inside the kernel. That is, in any release the Linux developers can change any internal system of the kernel in any way they please, with the only caveat that to get that change included they have to fix up any breakage it causes, but only inside upstream. Or, if you are building your own kernels with your own patch sets, any release of the kernel can completely break your build. Because of this, if you want your code to be actually used by real people, upstreaming it is considered extremely good, to the point of being almost a necessity. However, it can be a significant hurdle to overcome. Partly because there are fairly strict code quality/style requirements (as any kernel dev is supposed to be able to jump into your code to fix issues they caused by modifications elsewhere), but mostly because unlike inside the kernel itself, all interfaces between kernel and userspace are stable. That is, if you start supporting some operation from userspace, you need to be able to support it in perpetuity. Also, the kernel devs are extremely reluctant to add any kind of interface to the kernel where the only thing that uses it is some specific non-free binary blob on the userspace side. Currently the main reason the code is not upstreamable as is is that it needs a very substantial userspace program to function, which currently only exists as nonfree, and nVidia doesn't think that the interface used to connect to it is well-enough decided they want to freeze it. So the main hurdle for upstreaming is developing that interface until they are happy with it, and then develop some free version of the userspace portion of their graphics stack that can plug into that same interface.

MuffinFlavored 4y ago
> upstreaming it is considered extremely good are we assuming some sort of automatic testing will be added? unit testing? integration testing? sounds super hard to do for a driver. but... high quality code just being merged into master can still have bugs introduced in the future on accident, right? it's tests that usually makes it safe?
- rsaxvc 4y ago
  You can run these sorts of tests in a VM with PCIe passthrough.
  - djbusby 4y ago
    And hardware companies usually have large test-beds of hardware. So, Nvidia should be able to build a comprehensive test-suite.
    - MuffinFlavored 4y ago
      the same way they should have been able to build an upstreamable opensource driver all this time, right? lol
      - djbusby 4y ago
        No. Cause one (hardware) is old-hat and the other (FOSS) is big and risky and scary.
- loganberries 4y ago
  The kernel has bugs, all software has bugs. But there are massive massive test suites run by hundreds of companies on linux-next. If you email off a stack of patches that breaks something, chances are you'll get emails back telling you which patch had the breaking change in.
- jsjohnst 4y ago
  The reason upstreaming it is considered extremely good really doesn’t have to do with unit testing or really exactly with testing at all. The issue, as stated eloquently by the parent of your post, is that without it being upstreamed, it’s Nvidia’s problem to fix any issues caused by one of their dependencies in the kernel changing, where as if it’s upstreamed, it’s only maybe partially their responsibility (main burden is on whoever changed the dependency).

nwallin 4y ago

'Upstreamable' means that they send Linus a pull request, and Linus merges it into the kernel. It would then become part of Linux. It's a negative that it's not upstreamable. I wouldn't go so far as to say it's bad, but it's less than good. That means you'll continue to maintain a separate kernel module that's rebuilt for each kernel. It would need to be modified to match the kernel's coding standards and whatnot. Part of what makes linux such a strong thing is that you're just not allowed to do certain things in the kernel. They won't include ZFS in the kernel because it's got its own entire vfs -- linux filesystems are only permitted to use linux's vfs, and if you need vfs changes to make it work, you modify the 'one true' vfs. They didn't include the original AMD Radeon driver because it had its own hardware abstraction layer; you're supposed to do that in user space or not at all. (the Radeon driver was changed to remove the abstraction layer, and it was later merged into the kernel) It's not immediately clear to me how much work it will take to get the driver into the kernel. Hopefully it won't be a lot of work, but it's possible that it would require basically rewriting it from scratch. It's at least written almost entirely in C, so that's a good start. Regardless, this is tremendously good news.

prox 4y ago
What would the practical effect be for me as a user, say I render in Blender, or I play a few intensive 3d games?
- loganberries 4y ago
  Currently, you rely on Nvidia to keep the drivers up to date and working with the kernel you use in your OS. If the drivers aren't upstream, a future kernel can break them. So if you update your OS, it might no longer support your GPU without installing an alternative. Nvidia are responsible for getting this fixed in a timely manner. If the drivers are upstream, then people aren't allowed to break them when they make changes to the kernel. There should never be a kernel or OS upgrade that causes the drivers to no longer work, the people who made the breaking change are responsible for fixing it before it gets released to you.
  - 4y ago