5 ms·

The Linux kernel is one of the world's largest collaborative projects, and it shows. The strength of Linux is that tens of thousands of people work on it, and i

The Linux kernel is one of the world's largest collaborative projects, and it shows. The strength of Linux is that tens of thousands of people work on it, and it can be expected to support virtually all technology you might want to apply it to. The disadvantage is that coordinating tens of thousands of people is nearly impossible, and while Linux has central leadership, they're mainly concerned with organizing development and less with providing a unified vision or architecture for the kernel. Each subsystem has a great deal of soverignty in their affairs. That being said, if a group took it upon themselves to be a general-purpose "refactoring" team, without a focus on a single subsystem, they would probably be able to address these problems, albiet only if they have a strong stomach for bikeshedding. But Linux is made of volunteers, so you, dear reader, need to step up if you want this done.

linkdd 6y ago
I would question the sanity of this refactoring team after 1 month of reading code written by so many people ^^ Joke aside, they would not even need to code IMHO, just be able to ask questions like: - why is it done this way? - we found common code in those subsystems, do you think we could generalize or do you have other concerns?
- DyslexicAtheist 6y ago
  yeah especially since the whole point of ASN.1 is the agreement of the format of data structures so this means they never get the value out of ASN.1 that they potentially could. A big problem IMO is that it's hard to find a good implementation of ASN.1 compilers that will work across (sub-)systems and that as a FOSS project gets properly maintained. If this can be solved it would open up incredible possibilities such as automated grammar based fuzzing[0]. [1] https://blog.doyensec.com/2020/05/14/asn1fuzz.html https://blog.doyensec.com/2020/05/14/asn1fuzz.html
  - chris_wot 6y ago
    Oh that link is awesome! Thank you!
stefan_ 6y ago
That assumes the correct number of ASN.1 parsers in the kernel is "1". I would strictly prefer my kernel to include "0" (zero, nil, none) ASN.1 parsers. It's a trash spec designed by committee that probably needs its own bytecode machine to be parsed safely (that certainly is the track record of ASN.1 parsers in the wild).
- ZeroGravitas 6y ago
  I think there was a recent kefuffle in the python community because they wanted to depend on a Rust library to parse ASN.1 and some more obscure platforms weren't ready for that change.
  - comfydragon 6y ago
    Not in core Python, but in a widely-used library, yes. https://github.com/pyca/cryptography/issues/5771 https://github.com/pyca/cryptography/issues/5771
- rwmj 6y ago
  How would you suggest that the kernel interoperates with systems that need ASN.1 such as X.509 encryption, CIFS filesystems, or BER encoding in the H.323 ITU standard?
  - tremon 6y ago
    For example, by having a userspace helper for it. Pretty similar to the same way that the kernel itself doesn't do firmware patching, but relies on userspace to handle the opaque blobs.
    - rwmj 6y ago
      Firmware patching happens rarely, but decoding packets in the firewall would be far slower and more fragile if it required calling out to a userspace process.
st_goliath 6y ago
> That being said, if a group took it upon themselves to be a general-purpose "refactoring" team, ... I pretty much agree with you, but I'm not sure if it really needs a special cleanup task force. Stuff is being refactored and cleaned up on subsystem level all the time already. If, for instance, you do a `git log fs/squashfs` (an example I picked because I'm somewhat familiar with it, and there is comparatively little feature development) and skip past the maintenance fixes, there are a bunch of refactoring patches where common filesystem or block layer code was changed. On the mailing lists I'm on I somewhat regularly see Christoph Hellwig submitting larger cleanup patch sets for block layer stuff (even when you don't count the big cleanup spree during the first 2020 lock down). In the MTD subsystem there was a multi-year effort (earliest discussion with Boris and Miguel about this that I remember right now was around 2016) to refactor the entire subsystem, pull out a generic NAND framework, etc... But those are just some of the instances I can think off of the top of my head, from subsystems I'm personally somewhat familiar with. Regarding the ASN.1 parsers, Linux does have a `lib/` directory with common algorithms and helper functions in it. In fact one of the parsers being linked to is from there (https://git.zx2c4.com/linux/tree/lib/asn1_decoder.c https://git.zx2c4.com/linux/tree/lib/asn1_decoder.c). Somebody(TM) would "simply"(TM) have to patch the code in $SUBSYSTEM to use that instead, delete the old one and submit a patch set to the appropriate mailing lists. Of course it's a lot easier and less risky to simply post a mocking tweet and walk away. EDIT: In fact, this has already been done for one of them: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=cc2d58634e0f489d28b5564c05abc69930b4d920 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
- JackC 6y ago
  > Of course it's a lot easier and less risky to simply post a mocking tweet and walk away. This tweet seems to be from Jason Donenfeld, who worked for years to refactor linux kernel crypto APIs as part of upstreaming wireguard, apparently with some success. (Random link for some context, though I haven't followed the whole arc of that story: https://www.phoronix.com/scan.php?page=news_item&px=Crypto-API-Doing-Some-Zinc https://www.phoronix.com/scan.php?page=news_item&px=Crypto-A... ) If you're aware of that context and think that Donenfeld is too critical of the kernel and hasn't done enough to fix it, then OK, but from an outside perspective it sure looks like he's put a ton of time into fixing things and earned some right to point out remaining issues.
swills 6y ago
> Linux is made of volunteers This doesn't seem to be true to me: https://lwn.net/Articles/780271/ https://lwn.net/Articles/780271/ The vast majority of contributors seem to be paid. Am I misunderstanding?
- Blikkentrekker 6y ago
  No, it is a common myth that Linux, or many other bazaar-based projects are made by a thousand small, unpaid, helping hands, — perhaps because it sounds romantic. Most of Linux is indeed the fruit of paid, professional labor.
- rvz 6y ago
  > The vast majority of contributors seem to be paid. Am I misunderstanding? Correct.
- cycloptic 6y ago
  In those cases it would be the company that is volunteering, the contribution is still voluntary, and the company is paying someone to volunteer on their behalf.
  - recursive 6y ago
    What does volunteer even mean? Does it just mean not-slavery?
  - kungito 6y ago
    The company is paying an employee to work on a product which their business probably uses in return for publicity, bigger voice in the direction or prioritization of bugs and features, and to ensure stability of the product they depend on

linkdd 6y ago

I would question the sanity of this refactoring team after 1 month of reading code written by so many people ^^ Joke aside, they would not even need to code IMHO, just be able to ask questions like: - why is it done this way? - we found common code in those subsystems, do you think we could generalize or do you have other concerns?

DyslexicAtheist 6y ago
yeah especially since the whole point of ASN.1 is the agreement of the format of data structures so this means they never get the value out of ASN.1 that they potentially could. A big problem IMO is that it's hard to find a good implementation of ASN.1 compilers that will work across (sub-)systems and that as a FOSS project gets properly maintained. If this can be solved it would open up incredible possibilities such as automated grammar based fuzzing[0]. [1] https://blog.doyensec.com/2020/05/14/asn1fuzz.html https://blog.doyensec.com/2020/05/14/asn1fuzz.html
- chris_wot 6y ago
  Oh that link is awesome! Thank you!

stefan_ 6y ago

That assumes the correct number of ASN.1 parsers in the kernel is "1". I would strictly prefer my kernel to include "0" (zero, nil, none) ASN.1 parsers. It's a trash spec designed by committee that probably needs its own bytecode machine to be parsed safely (that certainly is the track record of ASN.1 parsers in the wild).

ZeroGravitas 6y ago
I think there was a recent kefuffle in the python community because they wanted to depend on a Rust library to parse ASN.1 and some more obscure platforms weren't ready for that change.
- comfydragon 6y ago
  Not in core Python, but in a widely-used library, yes. https://github.com/pyca/cryptography/issues/5771 https://github.com/pyca/cryptography/issues/5771
rwmj 6y ago
How would you suggest that the kernel interoperates with systems that need ASN.1 such as X.509 encryption, CIFS filesystems, or BER encoding in the H.323 ITU standard?
- tremon 6y ago
  For example, by having a userspace helper for it. Pretty similar to the same way that the kernel itself doesn't do firmware patching, but relies on userspace to handle the opaque blobs.
  - rwmj 6y ago
    Firmware patching happens rarely, but decoding packets in the firewall would be far slower and more fragile if it required calling out to a userspace process.

st_goliath 6y ago

> That being said, if a group took it upon themselves to be a general-purpose "refactoring" team, ... I pretty much agree with you, but I'm not sure if it really needs a special cleanup task force. Stuff is being refactored and cleaned up on subsystem level all the time already. If, for instance, you do a `git log fs/squashfs` (an example I picked because I'm somewhat familiar with it, and there is comparatively little feature development) and skip past the maintenance fixes, there are a bunch of refactoring patches where common filesystem or block layer code was changed. On the mailing lists I'm on I somewhat regularly see Christoph Hellwig submitting larger cleanup patch sets for block layer stuff (even when you don't count the big cleanup spree during the first 2020 lock down). In the MTD subsystem there was a multi-year effort (earliest discussion with Boris and Miguel about this that I remember right now was around 2016) to refactor the entire subsystem, pull out a generic NAND framework, etc... But those are just some of the instances I can think off of the top of my head, from subsystems I'm personally somewhat familiar with. Regarding the ASN.1 parsers, Linux does have a `lib/` directory with common algorithms and helper functions in it. In fact one of the parsers being linked to is from there (https://git.zx2c4.com/linux/tree/lib/asn1_decoder.c https://git.zx2c4.com/linux/tree/lib/asn1_decoder.c). Somebody(TM) would "simply"(TM) have to patch the code in $SUBSYSTEM to use that instead, delete the old one and submit a patch set to the appropriate mailing lists. Of course it's a lot easier and less risky to simply post a mocking tweet and walk away. EDIT: In fact, this has already been done for one of them: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=cc2d58634e0f489d28b5564c05abc69930b4d920 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

JackC 6y ago
> Of course it's a lot easier and less risky to simply post a mocking tweet and walk away. This tweet seems to be from Jason Donenfeld, who worked for years to refactor linux kernel crypto APIs as part of upstreaming wireguard, apparently with some success. (Random link for some context, though I haven't followed the whole arc of that story: https://www.phoronix.com/scan.php?page=news_item&px=Crypto-API-Doing-Some-Zinc https://www.phoronix.com/scan.php?page=news_item&px=Crypto-A... ) If you're aware of that context and think that Donenfeld is too critical of the kernel and hasn't done enough to fix it, then OK, but from an outside perspective it sure looks like he's put a ton of time into fixing things and earned some right to point out remaining issues.

swills 6y ago

> Linux is made of volunteers This doesn't seem to be true to me: https://lwn.net/Articles/780271/ https://lwn.net/Articles/780271/ The vast majority of contributors seem to be paid. Am I misunderstanding?

Blikkentrekker 6y ago
No, it is a common myth that Linux, or many other bazaar-based projects are made by a thousand small, unpaid, helping hands, — perhaps because it sounds romantic. Most of Linux is indeed the fruit of paid, professional labor.
rvz 6y ago
> The vast majority of contributors seem to be paid. Am I misunderstanding? Correct.
cycloptic 6y ago
In those cases it would be the company that is volunteering, the contribution is still voluntary, and the company is paying someone to volunteer on their behalf.
- recursive 6y ago
  What does volunteer even mean? Does it just mean not-slavery?
- kungito 6y ago
  The company is paying an employee to work on a product which their business probably uses in return for publicity, bigger voice in the direction or prioritization of bugs and features, and to ensure stability of the product they depend on