5 ms·

I'm just curious, how would you suggest they debug and maintain support for that hardware, without even owning that particular device? It seems like anything th

by zxzax 5y ago

I'm just curious, how would you suggest they debug and maintain support for that hardware, without even owning that particular device? It seems like anything they do would just be a wild guess as to whether it actually works or not. In my experience with this specialized datacenter hardware, if people want any of this stuff fixed (in userspace or in the kernel) it usually falls on the hardware vendor to hire some developers to fix it and upstream it, or at the very least donate some hardware to the upstream project so they can test properly.

mjw1007 5y ago
If they felt they needed information from, say, the Linux scsi maintainers then they could have gone out and asked for it. If they thought that in order for udev to do a proper job with these devices they needed the sg3-utils maintainers to write something new, they could have gone and asked for that. As it was they just stuck a "needs-new-home" label on the bug and ignored it. I think useful action would have been more likely if udev had still been maintained by a team who believed they'd taken on the responsibility to make /dev/disk/by-path work, rather than maintained as a small part of a project whose maintainers are mostly interested in other things. (Also, IIRC from what I had to do to work around it, they could have fixed that bug by reverting the change that introduced it. It wasn't a case of existing udev code not working with new hardware; it was caused by someone making changes without understanding the consequences.)
- zxzax 5y ago
  That's what I mean though, all of that seems to be just wild guessing as to what will actually work and sending random emails out to random people who may or may not be able to fix it (Disclaimer: I don't know anything about this specific bug or the specific hardware). Wouldn't someone who actually owns the device and knows who to contact on the kernel side be a better person to lead the effort on that?
  - mjw1007 5y ago
    This was a bug in udev. The right people to lead the effort on fixing bugs in udev are the udev maintainers. If people did not wish to have the responsibility for leading the effort to fix bugs in udev, they had the option of not taking over the maintenance of udev. I agree with ohazi, above, that the world would be a better place if the systemd people had availed themselves of that option. (To be fair, it looks like this issue was fixed a few months later by someone who knew what they were doing. So the main lesson here may be something more like "@poettering should not be triaging udev bugs".)
    - zxzax 5y ago
      I see people saying this kind of thing often in open source and I think you are missing the point here: who is going to fix this bug? You can say systemd developers shouldn't triage it, and then you're left with nobody to triage the bug at all. So what will you do? The point is, they don't own the hardware, they can't fix the bug. You have to find somebody who actually does own the hardware who knows what is going on, which seems to be exactly what happened. But if you get unlucky and if zero of those people are contributing to udev or any of its forks, then the bug probably won't ever get fixed.
      - 3np 5y ago
        I think their point is: with smaller project scope, this bug wouldn’t have existed in the first place.
        zxzax 5y ago
        That is throwing the baby out with the bathwater though, people who want large scale projects will continue to make projects with large scopes.
        regularfry 5y ago
        And they will continue to be bitten by problems specific to large-scope projects. Wanting something, no matter how much, does not let them escape engineering reality.
setheron 5y ago
My biggest gripe is that the logs are in a binary format.
- silly-silly 5y ago
  You use a tool like cat, vim , emacs, or tail to see logs. You can't physically read the file off the disk, how is this different than using journalctl ? # journalctl -u ssh
  - rnhmjoj 5y ago
    I think the issue is not that it's binary, but that it's a custom half-assed format. It's actually pretty slow because there is not enough indexing, this is why `systemctl status` can takes seconds to show the last 10 log lines[1]. Even worse, systemd can detect corruption but can't repair the logs[2], in practice a power failure can mean months of logs thrown away. Thankfully it can skip over corruption, so logging continues to work. I would have no problem if systemd used a database like sqlite, lmdb or anything else with some valid tooling, indexing, caching and proper recovery solutions. [1]: https://github.com/systemd/systemd/issues/2460 https://github.com/systemd/systemd/issues/2460 [2]: https://unix.stackexchange.com/questions/86206 https://unix.stackexchange.com/questions/86206
    - zxzax 5y ago
      I looked into this a while ago -- a traditional transactional CRUD database is totally unnecessary here, these are not files that you want to be editable by random tools. The journald log files have the important property that they are append-only by a single-writer. I agree there are some outstanding issues but those issues should just be fixed, instead of throwing it all out and moving to an even more complicated database layer that brings with it all its own additional problems. I think it would be perfectly feasible for someone to improve the indexing to show the latest 10 lines, or to write a fsck-type tool to repair damaged logs.
      - throw0101a 5y ago

by zxzax 5y ago

mjw1007 5y ago

If they felt they needed information from, say, the Linux scsi maintainers then they could have gone out and asked for it. If they thought that in order for udev to do a proper job with these devices they needed the sg3-utils maintainers to write something new, they could have gone and asked for that. As it was they just stuck a "needs-new-home" label on the bug and ignored it. I think useful action would have been more likely if udev had still been maintained by a team who believed they'd taken on the responsibility to make /dev/disk/by-path work, rather than maintained as a small part of a project whose maintainers are mostly interested in other things. (Also, IIRC from what I had to do to work around it, they could have fixed that bug by reverting the change that introduced it. It wasn't a case of existing udev code not working with new hardware; it was caused by someone making changes without understanding the consequences.)

zxzax 5y ago
That's what I mean though, all of that seems to be just wild guessing as to what will actually work and sending random emails out to random people who may or may not be able to fix it (Disclaimer: I don't know anything about this specific bug or the specific hardware). Wouldn't someone who actually owns the device and knows who to contact on the kernel side be a better person to lead the effort on that?
- mjw1007 5y ago
  This was a bug in udev. The right people to lead the effort on fixing bugs in udev are the udev maintainers. If people did not wish to have the responsibility for leading the effort to fix bugs in udev, they had the option of not taking over the maintenance of udev. I agree with ohazi, above, that the world would be a better place if the systemd people had availed themselves of that option. (To be fair, it looks like this issue was fixed a few months later by someone who knew what they were doing. So the main lesson here may be something more like "@poettering should not be triaging udev bugs".)
  - zxzax 5y ago
    I see people saying this kind of thing often in open source and I think you are missing the point here: who is going to fix this bug? You can say systemd developers shouldn't triage it, and then you're left with nobody to triage the bug at all. So what will you do? The point is, they don't own the hardware, they can't fix the bug. You have to find somebody who actually does own the hardware who knows what is going on, which seems to be exactly what happened. But if you get unlucky and if zero of those people are contributing to udev or any of its forks, then the bug probably won't ever get fixed.
    - 3np 5y ago
      I think their point is: with smaller project scope, this bug wouldn’t have existed in the first place.
      - zxzax 5y ago
        That is throwing the baby out with the bathwater though, people who want large scale projects will continue to make projects with large scopes.
        regularfry 5y ago
        And they will continue to be bitten by problems specific to large-scope projects. Wanting something, no matter how much, does not let them escape engineering reality.

setheron 5y ago

My biggest gripe is that the logs are in a binary format.

silly-silly 5y ago
You use a tool like cat, vim , emacs, or tail to see logs. You can't physically read the file off the disk, how is this different than using journalctl ? # journalctl -u ssh
- rnhmjoj 5y ago
  I think the issue is not that it's binary, but that it's a custom half-assed format. It's actually pretty slow because there is not enough indexing, this is why `systemctl status` can takes seconds to show the last 10 log lines[1]. Even worse, systemd can detect corruption but can't repair the logs[2], in practice a power failure can mean months of logs thrown away. Thankfully it can skip over corruption, so logging continues to work. I would have no problem if systemd used a database like sqlite, lmdb or anything else with some valid tooling, indexing, caching and proper recovery solutions. [1]: https://github.com/systemd/systemd/issues/2460 https://github.com/systemd/systemd/issues/2460 [2]: https://unix.stackexchange.com/questions/86206 https://unix.stackexchange.com/questions/86206
  - zxzax 5y ago
    I looked into this a while ago -- a traditional transactional CRUD database is totally unnecessary here, these are not files that you want to be editable by random tools. The journald log files have the important property that they are append-only by a single-writer. I agree there are some outstanding issues but those issues should just be fixed, instead of throwing it all out and moving to an even more complicated database layer that brings with it all its own additional problems. I think it would be perfectly feasible for someone to improve the indexing to show the latest 10 lines, or to write a fsck-type tool to repair damaged logs.
    - throw0101a 5y ago