6 ms·

Most of it is dumb automatic header generation as the article points out, but looking at the source[0], there seems to be a lot of code duplication, e.g. 5 diff

Most of it is dumb automatic header generation as the article points out, but looking at the source[0], there seems to be a lot of code duplication, e.g. 5 different vcn_*.c files that seem of significant size and largely identical at a glance. That frankly seems like horrible software design. [0] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/gpu/drm/amd/amdgpu?h=v5.9-rc8 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

pantalaimon 6y ago
> That frankly seems like horrible software design. Why? If it's auto-generated and the generated code is never touched by humans, that's not too bad. This information is sourced by some CAD/synthesis tool and just describes the hardware register layout. Manually maintaining that would be much worse and error prone.
- tylerhou 6y ago
  Compile times increase superlinearly with program size. How many cycles and developer hours are wasted if this duplication is not necessary?
  - mcguire 6y ago
    "Compile times increase superlinearly with program size." I've seen this said a couple of times lately. What is the source?
    - saagarjha 6y ago
      Internal compiler datastructures and the need to keep more things in memory?
      - jchw 6y ago
        This particular thought thread is about more translation units, not larger translation units. I can't see why those would not mostly scale linearly.
        tylerhou 6y ago
        Tail latency. See my other comment (https://news.ycombinator.com/item?id=24749000 https://news.ycombinator.com/item?id=24749000).
    - rational_indian 6y ago
      Are you saying C compilers are O(n)?
      - FartyMcFarter 6y ago
        They may well be, provided one stays away from bad patterns such as humongous complex functions?
      - formerly_proven 6y ago
        I very much expect O(n) compilation of stuff like array declarations, enums, CPP macro definitions and so forth. What else would it be?
      - barumi 6y ago
        > Are you saying C compilers are O(n)? OP asked for sources that substantiated a technical claim. No assertion was given. Unless you have a source that either supports or refutes the claim, trying to deflect the question does nothing to address the problem.
    - tylerhou 6y ago
      The other time that you saw it was also probably me. It's from this talk, which is about how a large amount of generated protocol buffer code at Google led to a quadratic increase in compile times: https://youtu.be/rHIkrotSwcc?t=720 https://youtu.be/rHIkrotSwcc?t=720. TL;DW: The reasoning is because if you use a distributed build system, then your compile time is gated by the file with the longest compile time (tail latency). The more files you have, the greater chance that one of them takes a long time. When you generate source files, you tend to produce more files than if you didn't. Most users don't use a distributed build system to compile the kernel, so on further thought, in that case compile times probably scale closer to linear with the number of translation units. But wasted cycles are still wasted cycles, and regardless of how exactly compile times scale, you should still consider the cost of longer compile times when you duplicate code. With regards to link time optimization: sophisticated analyses take superlinear complexity: https://cs.stackexchange.com/questions/22435/time-complexity-of-a-compiler https://cs.stackexchange.com/questions/22435/time-complexity.... Disclaimer: I work at Google.
      - deleted 6y ago
        [deleted]
    - jcranmer 6y ago
      Not knowing the context of the quote, I can guess a few causes: * Compiler optimizations mostly work on a per-function basis, so that the 'n' in O(n) or O(n²) is the size of a function and not the size of the entire codebase. Not all optimization algorithms are linear, and while good superlinear optimizations should have code threshold cutoffs, in practice these may be omitted. Function sizes follow something like a power law distribution, which means larger code bases contain larger functions that are more expensive to optimize. * For C/C++, you compile code by pasting all the header definitions into your translation unit, and feeding that to the actual lexer/parser/codegen. Larger projects are probably going to have correspondingly larger headers, and files will probably include more of them, so that the actual translation unit sizes are going to be larger even than the on-disk source files would indicate. * This is more speculative, but larger projects are also probably likelier to blow out the various caches on your computer, so that the operations you figure are O(1) (such as hashtable lookup!) are actually more like O(lg N) in practice. And as you get larger projects that are less likely to fit any cache, the effect of the non-constant time becomes more apparent.
  - alxlaz 6y ago
    Looking at things like e.g. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/gpu/drm/amd/amdgpu/navi10_sdma_pkt_open.h?h=v5.9-rc8 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... , this is very obviously specific to the underlying hardware's control model in all sorts of ways. If the hardware has a gazillion control registers, then the software will have to handle a gazillion registers and will therefore need several gazillion definitions in order to do it. The hardware won't get any simpler just because it takes a long time to compile the driver.
  - barumi 6y ago
    > Compile times increase superlinearly with program size. How many cycles and developer hours are wasted if this duplication is not necessary? Care to weight in the impact on developer hours and cycles of having to debug problems caused in how the autogenerated headers are created? If the code is not expected to change and was already subjected to verification and validation, ensuring that it stays the same is a good way to avoid wasting time tracking bugs on code paths that assuredly were already verified.
- atq2119 6y ago
  That part of the comment was specifically talking about the parts that are not auto-generated, but copy&pasted from one hardware generation to the next -- and then bugs need to be fixed and interface changes need to be applied in N different places forever.
deaddodo 6y ago
How do you figure? VCN 1.0[0] and VCN 2.0[1] differ within the first few lines and only do so further, the more you compare them. The files seem to correlate to each version of AMD's Video Core Next hardware[2]; and not anything to do with autogeneration. [0] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/gpu/drm/amd/amdgpu/vcn_v1_0.c?h=v5.9-rc8 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/gpu/drm/amd/amdgpu/vcn_v2_0.c?h=v5.9-rc8 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... [2] https://en.wikipedia.org/wiki/Video_Core_Next https://en.wikipedia.org/wiki/Video_Core_Next
- atq2119 6y ago
  Actually, there's a lot of commonality between those two files. Just look at all the "is_idle", "get_wptr", "set_wptr" type functions. They're either literally identical or the v2_0 version is a superset of functionality of the v1_0 version. This is a case where code is identical not by random chance but due to an underlying logical pattern (in this case: ring buffers for communication between CPU and GPU seem to continue working the same way). Hiding that via code duplication is going to make the code overall less maintainable, because the code lacks implicit knowledge of these patterns. P.S.: It's interesting how many people missed that my complaint was specifically not about the auto-generated part, but about the part that humans copy&paste.
  - fulafel 6y ago
    Code duplication this way (versions of code for different hw) is common in Linux though and makes a certain kind of sense.
  - flohofwoe 6y ago
    Code duplication sometimes make sense to better decouple different parts of the code base, however that often only becomes apparent long after the "code cleanup" to remove duplicates has been done.
afwe 6y ago
AMD drivers, at least on Windows, have a terrible reputation.
fizixer 6y ago
Yes. Despite being an avid code minimalism and refactoring advocate, I have zero problems with auto generated code size, as long as the size of "the source that generated the code" is reasonable. Here's how I count the size of the code: So if a codebase is 1 million lines, but 900k was autogenerated using, let's say, a 10k line 'metaprogram' (which is otherwise not included in the source), the real size of the code is 100k + 10k = 110k. Now, with the stupid vcn_* .c duplication, that I have serious problems with. But what you could do is take a diff of the two vcn_* .c files, keep those diffs as part of the source code, plus only one version of vcn_* .c file, but now add a small script that generates the remaining vcn_* .c files during the build process by using the diffs, e.g., using the patch command. Now the size of the diffs and the size of the patch script, plus the size of one vcn_* .c files, is a better measure of the size of the code. Of course this is a short-term workaround. There's still a need to refactor the code properly.
- IncRnd 6y ago
  That can be a useful way to calculate lines of code, however it is only nominally correct. In many cases it is more useful to calculate the lines of code according to what was checked into the build system, since those files can drift from the primary source. This is true, even if a million lines of checked-in source originated from 1k lines of meta-source.
  - fizixer 6y ago
    I'm not saying a line-of-code calculating program like cloc should change its behavior. I'm talking about the right way (IMO) of assessing the software complexity of the codebase. And I'm also talking about the fact that it is trivial to make cloc match my definition of source code size "if" ... the developers are willing to do a small amount of work by moving code-gen into the build process and not committing pre-generated code into the source, but the metaprogram scripts instead.
    - IncRnd 6y ago
      > I'm talking about the right way (IMO) of assessing the software complexity of the codebase. Yes, I get that, and that is what I meant, too. The reason is that complexity scales with the code that is checked into the build system, not just with the meta-code. I've written a lot of code generators for various purposes, and despite the efforts to only get meta-code checked into the build system, what has happened in almost every case is the generated code has been checked into the vcs. Of course, this depends on the particular teams (or consumers) of the generator. Eventually, the checked-in code gets changed. Maybe not right away, but it will most likely happen someday. That is why the complexity scales with the checked-in code and not with the original meta-code. There are also other force-multipliers, so to speak. For example, if there is a vulnerability in the generated code that was checked-in, the actual attack surface of the company can be multiplied by the number of instances of generated code not by the meta code. Fixing one instance doesn't fix any other instance. Complexity and risk are inseparably entwined and should not be looked at separately.
est31 6y ago
IDK it seems pretty common in the embedded world to add support for a new piece of hardware by copying the entire driver and then changing the parts that the new hardware supports instead of trying to come up with a driver that supports all generations of the hardware. Pretty sad if you ask me and very unmaintainable, but, at least if you ask the companies, people should just buy newer generations of their hardware if they want those bugfixes :).
- rictic 6y ago
  Wild speculation from someone who doesn't do hardware, but I'd guess that it's not so much writing the software as testing it that motivates that. It'd be hard to keep the testing matrix of apps X hardware X driver version from exploding out of control. Your customers will be hella pissed at you if their favorite game suddenly doesn't work on their system after a driver update. So you need to test a representative sample of previously-working software across all hardware that the updated driver applies to. That sounds like a nightmare, and a capital-intensive one because you need to keep machines up with all of these cards, both in continuous integration and for interactive debugging. I can understand the impulse to keep the list of hardware that new driver versions support small.
  - deleted 6y ago
    [deleted]
  - colonwqbang 6y ago
    This is the real reason in my experience, combined with the bringup/support cycle that comes with hardware development. If you fork the driver for bringup, you buy yourself a lot of freedom to change things. During support phase, when the hardware is already in the hands of the customer, changing things is more risky and so you have to be conservative. For example, maybe you had a chip bug in the previous hardware generation which caused the system to hang after several days of stress testing. You found a software workaround for the bug, but every time you touch the code you need to re-verify the workaround, which takes days. Of course, the downsides of forking the code are also very apparent...

pantalaimon 6y ago

> That frankly seems like horrible software design. Why? If it's auto-generated and the generated code is never touched by humans, that's not too bad. This information is sourced by some CAD/synthesis tool and just describes the hardware register layout. Manually maintaining that would be much worse and error prone.

tylerhou 6y ago
Compile times increase superlinearly with program size. How many cycles and developer hours are wasted if this duplication is not necessary?
- mcguire 6y ago
  "Compile times increase superlinearly with program size." I've seen this said a couple of times lately. What is the source?
  - saagarjha 6y ago
    Internal compiler datastructures and the need to keep more things in memory?
    - jchw 6y ago
      This particular thought thread is about more translation units, not larger translation units. I can't see why those would not mostly scale linearly.
      - tylerhou 6y ago
        Tail latency. See my other comment (https://news.ycombinator.com/item?id=24749000 https://news.ycombinator.com/item?id=24749000).
  - rational_indian 6y ago
    Are you saying C compilers are O(n)?
    - FartyMcFarter 6y ago
      They may well be, provided one stays away from bad patterns such as humongous complex functions?
    - formerly_proven 6y ago
      I very much expect O(n) compilation of stuff like array declarations, enums, CPP macro definitions and so forth. What else would it be?
    - barumi 6y ago
      > Are you saying C compilers are O(n)? OP asked for sources that substantiated a technical claim. No assertion was given. Unless you have a source that either supports or refutes the claim, trying to deflect the question does nothing to address the problem.
  - tylerhou 6y ago
    The other time that you saw it was also probably me. It's from this talk, which is about how a large amount of generated protocol buffer code at Google led to a quadratic increase in compile times: https://youtu.be/rHIkrotSwcc?t=720 https://youtu.be/rHIkrotSwcc?t=720. TL;DW: The reasoning is because if you use a distributed build system, then your compile time is gated by the file with the longest compile time (tail latency). The more files you have, the greater chance that one of them takes a long time. When you generate source files, you tend to produce more files than if you didn't. Most users don't use a distributed build system to compile the kernel, so on further thought, in that case compile times probably scale closer to linear with the number of translation units. But wasted cycles are still wasted cycles, and regardless of how exactly compile times scale, you should still consider the cost of longer compile times when you duplicate code. With regards to link time optimization: sophisticated analyses take superlinear complexity: https://cs.stackexchange.com/questions/22435/time-complexity-of-a-compiler https://cs.stackexchange.com/questions/22435/time-complexity.... Disclaimer: I work at Google.
    - deleted 6y ago
      [deleted]
  - jcranmer 6y ago
    Not knowing the context of the quote, I can guess a few causes: * Compiler optimizations mostly work on a per-function basis, so that the 'n' in O(n) or O(n²) is the size of a function and not the size of the entire codebase. Not all optimization algorithms are linear, and while good superlinear optimizations should have code threshold cutoffs, in practice these may be omitted. Function sizes follow something like a power law distribution, which means larger code bases contain larger functions that are more expensive to optimize. * For C/C++, you compile code by pasting all the header definitions into your translation unit, and feeding that to the actual lexer/parser/codegen. Larger projects are probably going to have correspondingly larger headers, and files will probably include more of them, so that the actual translation unit sizes are going to be larger even than the on-disk source files would indicate. * This is more speculative, but larger projects are also probably likelier to blow out the various caches on your computer, so that the operations you figure are O(1) (such as hashtable lookup!) are actually more like O(lg N) in practice. And as you get larger projects that are less likely to fit any cache, the effect of the non-constant time becomes more apparent.
- alxlaz 6y ago
  Looking at things like e.g. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/gpu/drm/amd/amdgpu/navi10_sdma_pkt_open.h?h=v5.9-rc8 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... , this is very obviously specific to the underlying hardware's control model in all sorts of ways. If the hardware has a gazillion control registers, then the software will have to handle a gazillion registers and will therefore need several gazillion definitions in order to do it. The hardware won't get any simpler just because it takes a long time to compile the driver.
- barumi 6y ago
  > Compile times increase superlinearly with program size. How many cycles and developer hours are wasted if this duplication is not necessary? Care to weight in the impact on developer hours and cycles of having to debug problems caused in how the autogenerated headers are created? If the code is not expected to change and was already subjected to verification and validation, ensuring that it stays the same is a good way to avoid wasting time tracking bugs on code paths that assuredly were already verified.
atq2119 6y ago
That part of the comment was specifically talking about the parts that are not auto-generated, but copy&pasted from one hardware generation to the next -- and then bugs need to be fixed and interface changes need to be applied in N different places forever.

deaddodo 6y ago

How do you figure? VCN 1.0[0] and VCN 2.0[1] differ within the first few lines and only do so further, the more you compare them. The files seem to correlate to each version of AMD's Video Core Next hardware[2]; and not anything to do with autogeneration. [0] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/gpu/drm/amd/amdgpu/vcn_v1_0.c?h=v5.9-rc8 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/gpu/drm/amd/amdgpu/vcn_v2_0.c?h=v5.9-rc8 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... [2] https://en.wikipedia.org/wiki/Video_Core_Next https://en.wikipedia.org/wiki/Video_Core_Next

atq2119 6y ago
Actually, there's a lot of commonality between those two files. Just look at all the "is_idle", "get_wptr", "set_wptr" type functions. They're either literally identical or the v2_0 version is a superset of functionality of the v1_0 version. This is a case where code is identical not by random chance but due to an underlying logical pattern (in this case: ring buffers for communication between CPU and GPU seem to continue working the same way). Hiding that via code duplication is going to make the code overall less maintainable, because the code lacks implicit knowledge of these patterns. P.S.: It's interesting how many people missed that my complaint was specifically not about the auto-generated part, but about the part that humans copy&paste.
- fulafel 6y ago
  Code duplication this way (versions of code for different hw) is common in Linux though and makes a certain kind of sense.
- flohofwoe 6y ago
  Code duplication sometimes make sense to better decouple different parts of the code base, however that often only becomes apparent long after the "code cleanup" to remove duplicates has been done.

afwe 6y ago

AMD drivers, at least on Windows, have a terrible reputation.

fizixer 6y ago

Yes. Despite being an avid code minimalism and refactoring advocate, I have zero problems with auto generated code size, as long as the size of "the source that generated the code" is reasonable. Here's how I count the size of the code: So if a codebase is 1 million lines, but 900k was autogenerated using, let's say, a 10k line 'metaprogram' (which is otherwise not included in the source), the real size of the code is 100k + 10k = 110k. Now, with the stupid vcn_* .c duplication, that I have serious problems with. But what you could do is take a diff of the two vcn_* .c files, keep those diffs as part of the source code, plus only one version of vcn_* .c file, but now add a small script that generates the remaining vcn_* .c files during the build process by using the diffs, e.g., using the patch command. Now the size of the diffs and the size of the patch script, plus the size of one vcn_* .c files, is a better measure of the size of the code. Of course this is a short-term workaround. There's still a need to refactor the code properly.

IncRnd 6y ago
That can be a useful way to calculate lines of code, however it is only nominally correct. In many cases it is more useful to calculate the lines of code according to what was checked into the build system, since those files can drift from the primary source. This is true, even if a million lines of checked-in source originated from 1k lines of meta-source.
- fizixer 6y ago
  I'm not saying a line-of-code calculating program like cloc should change its behavior. I'm talking about the right way (IMO) of assessing the software complexity of the codebase. And I'm also talking about the fact that it is trivial to make cloc match my definition of source code size "if" ... the developers are willing to do a small amount of work by moving code-gen into the build process and not committing pre-generated code into the source, but the metaprogram scripts instead.
  - IncRnd 6y ago
    > I'm talking about the right way (IMO) of assessing the software complexity of the codebase. Yes, I get that, and that is what I meant, too. The reason is that complexity scales with the code that is checked into the build system, not just with the meta-code. I've written a lot of code generators for various purposes, and despite the efforts to only get meta-code checked into the build system, what has happened in almost every case is the generated code has been checked into the vcs. Of course, this depends on the particular teams (or consumers) of the generator. Eventually, the checked-in code gets changed. Maybe not right away, but it will most likely happen someday. That is why the complexity scales with the checked-in code and not with the original meta-code. There are also other force-multipliers, so to speak. For example, if there is a vulnerability in the generated code that was checked-in, the actual attack surface of the company can be multiplied by the number of instances of generated code not by the meta code. Fixing one instance doesn't fix any other instance. Complexity and risk are inseparably entwined and should not be looked at separately.

est31 6y ago

IDK it seems pretty common in the embedded world to add support for a new piece of hardware by copying the entire driver and then changing the parts that the new hardware supports instead of trying to come up with a driver that supports all generations of the hardware. Pretty sad if you ask me and very unmaintainable, but, at least if you ask the companies, people should just buy newer generations of their hardware if they want those bugfixes :).

rictic 6y ago
Wild speculation from someone who doesn't do hardware, but I'd guess that it's not so much writing the software as testing it that motivates that. It'd be hard to keep the testing matrix of apps X hardware X driver version from exploding out of control. Your customers will be hella pissed at you if their favorite game suddenly doesn't work on their system after a driver update. So you need to test a representative sample of previously-working software across all hardware that the updated driver applies to. That sounds like a nightmare, and a capital-intensive one because you need to keep machines up with all of these cards, both in continuous integration and for interactive debugging. I can understand the impulse to keep the list of hardware that new driver versions support small.
- deleted 6y ago
  [deleted]
- colonwqbang 6y ago
  This is the real reason in my experience, combined with the bringup/support cycle that comes with hardware development. If you fork the driver for bringup, you buy yourself a lot of freedom to change things. During support phase, when the hardware is already in the hands of the customer, changing things is more risky and so you have to be conservative. For example, maybe you had a chip bug in the previous hardware generation which caused the system to hang after several days of stress testing. You found a software workaround for the bug, but every time you touch the code you need to re-verify the workaround, which takes days. Of course, the downsides of forking the code are also very apparent...