7 ms·

This is crazy, apt has been reading diffs a byte at a time? Think of the millions of hours that have been wasted due to this.

by orf 11y ago

grandinj 11y ago
No, they made the right choices. (1)Make it work. (2)Make it correct. (3)Make it fast. Can take a long time to get to step 3 with something as safety critical as an update tool.
- deleted 11y ago
  [deleted]
- k__ 11y ago
  > implying apt is a young tool and the maintainers didn't have many years to go from 1 to 3
  - baghira 11y ago
    According to the developer the bug was present only in the 1.1 series, which is just a few months old (it was pre-released at debconf 2015, and is not in Jessie if i'm not mistaken).
    - r0muald 11y ago
      Furthermore pdiffs were introduced in ~2006: https://www.debian-administration.org/article/439/Avoiding_slow_package_updates_with_package_diffs https://www.debian-administration.org/article/439/Avoiding_s...
- deleted 11y ago
  [deleted]
- rektide 11y ago
  Sometime between the start and the end, there's often a mandatory burn it all down step required to more fully satisfy either (2) or (3). Watching hundreds of MB of data getting written to my HD for an apt-get update makes me 90% certain that Debian indeed needs to burn it all down. To their credit, they started really early. And burning it all down seems to be an increasingly better option for purely extrinsic reasons; the late mover advantage keeps increasing, and better and better options keep emerging. A LMDB-based refactor perhaps? This is probably a pipe dream, alas: Debian developers must have a hundred different programs that all build and extend forward the original "make it work" technology, to make it survivable. And that's really my final counter to your cute linear notion of ongoing improvement: the mold is set, and there are dozens of things bound up in what has happened.
  - julian-klode 11y ago
    Where do you want LMDB? apt has a super fast binary cache, the repositories and dpkg store their package lists as text files. RPM systems have the advantage IIRC that they can basically just download a bunch of sqlite3 databases, which means they do not have to parse any data.
    - hyc_symas 11y ago
      AFAIK RPM uses BerkeleyDB, though it will probably switch to LMDB https://bugzilla.redhat.com/show_bug.cgi?id=1086784 https://bugzilla.redhat.com/show_bug.cgi?id=1086784
joosters 11y ago
That's unfair. Pick any big program and you'll find examples of really dumb inefficient code. We all write it, no matter how smart we think we are. Instead, think of all the millions of satisfied users of the software, none of whom bothered to fix up this problem until now.
- jlarocco 11y ago
  I wouldn't even call it dumb, it's just what happens when a big group of people are writing software together. Somebody writes a function, other people see it and start using it in ways the original author didn't intend. One byte at a time could've been a quick hack that was "good enough" in the original author's use case, because he wasn't reading a lot of data or only calling it once or something. Somebody else saw "readline" and decided it was exactly what he needed and decided to use it everywhere.
  - jberryman 11y ago
    OT, but that's one reason comments are important. Also in the alternative hypothetical where reading a byte at a time was required: if that's not documented someone might come along and thing that was merely incidental or even buggy behavior and change it. That's why I'm not sympathetic to programmers who think that they write "self-documenting" code or something. If you don't document (even briefly) what your code is supposed to do then bugs become diffuse things that no one is responsible for, and which are difficult to spot. </ramble>
    - molyss 11y ago
      I profoundly disagree with that. Comments always end up not following the actual implementation. In this example, "self documenting" the code would be to name the function "unbufferedRead" or something similar that shows that not having a buffer is part of the contract. You make it buffered, you're breaking the contract. Anything else is merely applying temporary (because they'll rot), hidden (because if someone is reading code that calls myVeryWellDocumentedFunction, they won't see the documentation, especially if the function is just incompletely named) and inaccurate (because the only accurate thing is the code) band aids. There's a reason it's said that naming stuff is one of the hardest things in programming.

This is crazy, apt has been reading diffs a byte at a time? Think of the millions of hours that have been wasted due to this.

by orf 11y ago

This is crazy, apt has been reading diffs a byte at a time? Think of the millions of hours that have been wasted due to this.

grandinj 11y ago

No, they made the right choices. (1)Make it work. (2)Make it correct. (3)Make it fast. Can take a long time to get to step 3 with something as safety critical as an update tool.

deleted 11y ago
[deleted]
k__ 11y ago
> implying apt is a young tool and the maintainers didn't have many years to go from 1 to 3
- baghira 11y ago
  According to the developer the bug was present only in the 1.1 series, which is just a few months old (it was pre-released at debconf 2015, and is not in Jessie if i'm not mistaken).
  - r0muald 11y ago
    Furthermore pdiffs were introduced in ~2006: https://www.debian-administration.org/article/439/Avoiding_slow_package_updates_with_package_diffs https://www.debian-administration.org/article/439/Avoiding_s...
deleted 11y ago
[deleted]
rektide 11y ago
Sometime between the start and the end, there's often a mandatory burn it all down step required to more fully satisfy either (2) or (3). Watching hundreds of MB of data getting written to my HD for an apt-get update makes me 90% certain that Debian indeed needs to burn it all down. To their credit, they started really early. And burning it all down seems to be an increasingly better option for purely extrinsic reasons; the late mover advantage keeps increasing, and better and better options keep emerging. A LMDB-based refactor perhaps? This is probably a pipe dream, alas: Debian developers must have a hundred different programs that all build and extend forward the original "make it work" technology, to make it survivable. And that's really my final counter to your cute linear notion of ongoing improvement: the mold is set, and there are dozens of things bound up in what has happened.
- julian-klode 11y ago
  Where do you want LMDB? apt has a super fast binary cache, the repositories and dpkg store their package lists as text files. RPM systems have the advantage IIRC that they can basically just download a bunch of sqlite3 databases, which means they do not have to parse any data.
  - hyc_symas 11y ago
    AFAIK RPM uses BerkeleyDB, though it will probably switch to LMDB https://bugzilla.redhat.com/show_bug.cgi?id=1086784 https://bugzilla.redhat.com/show_bug.cgi?id=1086784

joosters 11y ago

That's unfair. Pick any big program and you'll find examples of really dumb inefficient code. We all write it, no matter how smart we think we are. Instead, think of all the millions of satisfied users of the software, none of whom bothered to fix up this problem until now.

jlarocco 11y ago
I wouldn't even call it dumb, it's just what happens when a big group of people are writing software together. Somebody writes a function, other people see it and start using it in ways the original author didn't intend. One byte at a time could've been a quick hack that was "good enough" in the original author's use case, because he wasn't reading a lot of data or only calling it once or something. Somebody else saw "readline" and decided it was exactly what he needed and decided to use it everywhere.
- jberryman 11y ago
  OT, but that's one reason comments are important. Also in the alternative hypothetical where reading a byte at a time was required: if that's not documented someone might come along and thing that was merely incidental or even buggy behavior and change it. That's why I'm not sympathetic to programmers who think that they write "self-documenting" code or something. If you don't document (even briefly) what your code is supposed to do then bugs become diffuse things that no one is responsible for, and which are difficult to spot. </ramble>
  - molyss 11y ago
    I profoundly disagree with that. Comments always end up not following the actual implementation. In this example, "self documenting" the code would be to name the function "unbufferedRead" or something similar that shows that not having a buffer is part of the contract. You make it buffered, you're breaking the contract. Anything else is merely applying temporary (because they'll rot), hidden (because if someone is reading code that calls myVeryWellDocumentedFunction, they won't see the documentation, especially if the function is just incompletely named) and inaccurate (because the only accurate thing is the code) band aids. There's a reason it's said that naming stuff is one of the hardest things in programming.