15 ms·

Leaky Abstractions

480 points by jakub_g 5y ago

yurishimo 5y ago
These are the kinds of articles I love to see on HN! I think it's so interesting to see how different OS' have approached these long lived operations over the years. Arguably, *nix OS' have been able to move past these problems since it appears that package management is much more of a first party concern. "Oh, you want to upgrade this utility? Well, I rely on these other things so you should update them too." I'm a webdev so a lot of this is foreign to me w/ how it actually all functions but it's cool to read about these problems and speculate how/why they came to be and why they may only exist on certain platforms.
- Rerarom 5y ago
  This is the kind of stuff I expected to study in my CS degree
  - dhosek 5y ago
    The problem is that computer science ≠ software engineering, especially in academic contexts.
    - austhrow743 5y ago
      I've seen this a lot on here combined with cs degrees being frequently mentioned and software engineering degrees rarely mentioned. Are people widely choosing computer science over software engineering and then being surprised they aren't studying software engineering or is it rare for institutions to offer software engineering as an option?
      - korijn 5y ago
        I am not aware of anyone with a software engineering degree. Is that really a thing?
        austhrow743 5y ago
        Was where I went. Two separate degrees: https://www.newcastle.edu.au/degrees/bachelor-of-software-engineering-honours https://www.newcastle.edu.au/degrees/bachelor-of-software-en... https://www.newcastle.edu.au/degrees/bachelor-of-computer-science https://www.newcastle.edu.au/degrees/bachelor-of-computer-sc... Looks like there's even a "computer systems engineering" as well. https://www.newcastle.edu.au/degrees/bachelor-of-computer-systems-engineering-honours https://www.newcastle.edu.au/degrees/bachelor-of-computer-sy...
        deleted 5y ago
        [deleted]
        drran 5y ago
        Yep, I'm the software engineer with diploma. I have master degree in Computer Sciences (Software Engineering Department). I'm from Ukraine.
        MillenialMan 5y ago
        At least in the UK, "Software Engineering" degrees are just Computer Science degrees with a different name and a restricted subset of course options. From what I can tell, they're a marketing gimmick, but even respectable universities will offer them. The more questionable universities will also offer actual specialised courses under that moniker (in the same vein as "games design" degrees), but they aren't great. Some Unis also offer different tracks for Masters degrees - MA (Master of Arts) in Software Engineering, or MSc (Master of Sciences) in Computer Science. Those differences are a bit more meaningful. The MA will be softer, with less research focus, and is (ostensibly) explicitly designed as preparation for industry.
        Beldin 5y ago
        My uni offers both a one-year MSc SE, and a two-year MSc CS. For organisational reasons, the 2 yr program is basically SE+MSc Information Science. Students can pick their final subject; there is no difference in subjects between the two programmes, though we obviously expect some difference in result. After all, you should show some proficiency in both masters if you're doing the 2yr programme. I'm currently supervising a student (SE) on file recovery, to whom I'll definitely forward the link. It's right up his project's alley. Side remark: had he been enrolled in the 2yr programme, the subject would have been the same. I'd just expect him to extend his current work with a research question towards information science.
      - Siira 5y ago
        In the end, a good engineer rarely becomes a professor. This is both because the industry pays better, and the universities don’t value engineering expertise. The best SE courses in my university were offered by “guest” teachers from big software companies. (Which were very rare.)
    - MillenialMan 5y ago
      I don't think that's the problem. The problem is that Universities aren't punished for teaching inadequately. And when challenged, people justify it by saying, "ah, but this course that 90% of you took so that you could be prepared for a career as a developer wasn't really about being a developer."
      - unishark 5y ago
        > The problem is that Universities aren't punished for teaching inadequately. They are punished in a way. It hurts a school to get a bad reputation among local companies for poorly-prepared grads. R1 research schools, particularly if they are public, may feel it less. But even then, the ones I've dealt with have always chased all the fads in trying to improve learning and metrics. A tenured professor who's just there for the research may choose to ignore it all of course.
        highmastdon 5y ago
        Not if every university is like that. Punishment will only happen when there will be that one university that has adequate courses and stands out above the rest
        MillenialMan 5y ago
        This hasn't been my experience in practice. I went to a top-5 CS university in the UK and some of my classmates graduated almost unable to program (and they still, somehow, got jobs). Seeing that first-hand was extremely jarring. It didn't and still hasn't damaged the reputation of the university.
    - jjice 5y ago
      Unfortunately, at my University the SE program was mostly building UML and ER diagrams and memorizing agile terms. Software design was a minor part of it. I was CS, but had a lot of SE friends, and we all agree that the best SE course was the one that was mandatory for all computing majors. It covered version control and branching, testing, design process, abstraction design, etc. That class was probably one of the most daily applicable things I learned, I took it in my third semester. If my university offered more courses of that caliber, that didn't rehash and force memorization of buzzwords, then that program would've been incredible.
flameon 5y ago
Fauci emails reveal how the virus was created: https://twitter.com/Littleb29872980/status/1400153906089504777/photo/1 https://twitter.com/Littleb29872980/status/14001539060895047... Arrest Fauci Now!
bentcorner 5y ago
I wonder why it's implemented as a per-file copy+delete instead of a "copy all files" then "delete all files". I also have a gut feeling that doing similar operations to a connected android phone (e.g., moving photos from your phone to your PC over USB) is also slow, probably for similar reasons.
- genpfault 5y ago
  > doing similar operations to a connected android phone MTP is terrible[1]. [1]: https://en.wikipedia.org/wiki/Media_Transfer_Protocol#Performance https://en.wikipedia.org/wiki/Media_Transfer_Protocol#Perfor...
  - jakub_g 5y ago
    Oh boy. I was thinking USB 2.0 is main reason for Android<>Windows copying of photos to suck so much, but the rabbit hole is much deeper. It's sad that with cloud being the solution for everything those days, this will probably never be improved within next decade.
    - userbinator 5y ago
      It's ridiculously wasteful and slow to have to upload to a cloud server who-knows-where, then download it again to the computer several feet away. Yet with Android having removed USB mass storage, that's often the easiest way. It's against the interests of those who profit off selling cloud storage to make local transfers easy.
      - steerablesafe 5y ago
        If I need to move a lot of data I just use adb and USB debugging to access the files. That is actually fast, but it's ridiculous that I have to do it.
        mihi 5y ago
        Ha, I like that! I should try that as well.
vidarh 5y ago
This kind of thing doesn't surprise me at all - a surprising number of developers miss those types of things, and one of the things a lot of people miss seems to be checking for situations where you end up with excessive amount of calls doing little work, for some reason. E.g. at one point (long time ago), MySQL's C client library would call read() for 4 bytes to read a length indicator, and then read exactly the number of bytes indicated by the protocol, which led to a lot of time spent context-switching vs. reading into a larger user-space buffer and copying out of that. On Linux simple, plain "strace" remains one of my favourite first steps to find performance problems in part because these things are near immediately apparent if you strace a process because the really pathological cases often ends up dominating the output so much that you'll spot them right away. Another "favourite" issue that shows up often when you use strace like this is e.g. excessive include paths - running strace on MRI Ruby with rubygems enables and lots of gems pulled in is a good way of seeing that in action - like this zip problem it's an example that seems totally reasonable when the number of gems is small, and that first becomes apparent when you test with lots of gems, and look at what's actually happening under the hood. A similar example of testing with too small datasets and/or on a large enough machine to not spot what becomes immediately obvious if you trace execution was how a type 1 font reference library made available by Adobe (no idea if they wrote it or if it was from another source originally) which would exhibit another typical pathological behaviour and call malloc() hundreds of times on loading a font to allocate 4 bytes at the time instead of allocating larger buffers. (strace won't catch malloc(), but ltrace does)) To avoid messing too much with the code, we replaced the calls to malloc() with calls to a simple arena allocator and load speed shot through the roof and memory usage dropped massively. Too few people trace execution of their code. I know that because if most developers did, issues like the above would get caught much sooner, and would be rarer in published code.
- formerly_proven 5y ago
  > Another "favourite" issue that shows up often when you use strace like this is e.g. excessive include paths - running strace on MRI Ruby with rubygems enables and lots of gems pulled in is a good way of seeing that in action - like this zip problem it's an example that seems totally reasonable when the number of gems is small, and that first becomes apparent when you test with lots of gems, and look at what's actually happening under the hood. When you add e.g. Python's standard library as a ZIP to the Python search path, the thing will open/read/read/read/close that file approximately a gazillion times on startup. That's OK on Linux/Unix, where that is fairly cheap. Guess which OS doesn't like that pattern at all?
osipov 5y ago
The programmer responsible for this implementation successfully inverted a red-black tree during the job interview and explained a Big-Oh optimal solution for doing so.
deleted 5y ago
[deleted]
zbendefy 5y ago
Another similar and good article: https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/ https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a...
- an1sotropy 5y ago
  Another common leaky abstraction: floating-point numbers as an abstraction of the real numbers. It works nearly all the time, until you have to really know about numerical precision, or where a NaN came from. https://www.johndcook.com/blog/2009/04/06/numbers-are-a-leaky-abstraction/ https://www.johndcook.com/blog/2009/04/06/numbers-are-a-leak...
  - kortex 5y ago
    Relevant: the list of integers you can convert to float, invert twice, and don't get the same number back. https://oeis.org/A275419 https://oeis.org/A275419
    - an1sotropy 5y ago
      This is a treasure; thank you!
  - chii 5y ago
    i wouldn't call floating point numbers an abstraction on real numbers. It's a representation - like char[] is a representation of strings, which is itself an abstraction used to manipulate human readable text!
    - an1sotropy 5y ago
      Read the link above about FP as a leaky abstraction of the reals; it explains it better than I can. FP is not really a representation of reals, because you can't represent the entire number line with a finite number of bits. It is a representation of a cleverly chosen subset of the reals, so well-designed that many people forget that they're not dealing with real numbers. That intelligent people can write code with FP values, operating on them thinking they are real numbers, and not knowing how FP works, and get programs that often do nearly the right thing, speaks to the success of FP as an abstraction. Understanding how things can go wrong is the leaks (like, non-associativity). I don't think people manipulating char[] representations of strings are ever at risk of forgetting that it's just an array of chars (they program with full awareness of the implementation), so it doesn't feel like there's much abstraction happening there. Other higher-level data structures for text manipulation probably do implement some useful abstractions; I haven't worked with those.
Sol- 5y ago
Interesting, didn't even know you can cut files from a zip in that Windows zip file viewer. I would have thought it's some read-only filesystem like viewing a mounted CD or so. In that context I wonder if you could cut from rewritable CD-RWs as well back in the day (can't remember) - that seems like another abstraction that's similarly slow in reality.
- jakub_g 5y ago
  I never tried cutting from CD-RW, but AFAIR each burning would append a non-trivial header (like 20MBs or so) so that would be a pretty expensive thing to do :) AFAIR when you "copied" into CD-RW the files would show up semi-transparent (pending) and you'd have to click a button to process with burning. Probably same for cutting I guess.
  - bluGill 5y ago
    Been years, but the ISO file system says 16 sectors past the start of the last track is where the file system index starts. (the first 15 sectors are reserved for booting) Then however many sectors you need for the index - which depends on how many files you have, and how long the filenames are. Note that both CDR and CDRW don't allow you to write sectors in the middle of a track, so you need to know where every file will be on disk before you can write anything - you can't build the index on the fly. (I remember this part because I was writing backups and so I didn't know how big the file would be until I was done and thus couldn't write the index first - I eventually got around the rule by creating a new track with just the index)
  - TeMPOraL 5y ago
    Speaking of old Windows and CDs, Windows had some crazy trick to turn CD-R (not CD-RW) into rewriteable medium. I'm guessing they simulated a regular file system on top of an append-only representation. I never dug into the details back in the day, because I never used this feature. Unfortunately, IIRC, this trickery was enabled by default when copying files to CDs - which was a problem, because nothing else could read it. This caused me an unending stream of calls from friends and relatives, who all tried to burn some family/vacation photos onto a CD-R, in order to view them on their TVs, but the CD readers for TVs couldn't parse the format.
dsego 5y ago
Unless I'm mistaken, this seems to be the original programmer on youtube: Dave's Garage - Secret History of Windows ZIPFolders https://youtu.be/aQUtUQ_L8Yk https://youtu.be/aQUtUQ_L8Yk
- codetrotter 5y ago
  It is. I have seen a few of his videos before. They are generally interesting and worth checking out.
- ericlaw 5y ago
  As noted in the blog post, he's the author of the Shell Namespace code, but not the ZIP code beneath.
  - dsego 5y ago
    Oh, I can see now it's mentioned, I must've skipped it on first reading.
mrkramer 5y ago
"Unfortunately, the code hasn’t really been updated in a while. A long while. The timestamp in the module claims it was last updated on Valentine’s Day 1998" When I saw this timestamp sometime ago on my PC I thought it was a joke?! C'mon 1998 like WTF!
sachinjoseph 5y ago
A bug has been filed about this on the WinDev repo, might help resolve it faster if it gets some upvotes: https://github.com/microsoft/Windows-Dev-Performance/issues/91 https://github.com/microsoft/Windows-Dev-Performance/issues/...
amichal 5y ago
This article is about the details of zip file handling in the Shell Extension... Reading the first few paras I thought they would talk about the struggle it was to implement a shell namespace extension. I did a somewhat crappy one 20-something years ago and I still remember how confusing it was. At the time is spent all my days writing various COM things. I was very comfortable doing things like hand editing IDL headers to avoid changing GUIDs. NSEs were still the most baffling, brittle and confusing interfaces I've ever seen
ping_pong 5y ago
25 years without major bugs and still being extremely useful is a complete win in my book! This is definitely a bug, but I wonder if Microsoft will think it's worth it to even update it, it doesn't seem like it happens enough to cause widespread issues.
- foxbee 5y ago
  This was my exact thinking!
deleted 5y ago
[deleted]
monocasa 5y ago
Having written a shell namespace extension, I now use the concept as a great example of making the wrong abstraction interestingly enough. Shell namespaces are a odd win32 concept rather than a filesystem concept and cause a lot of headaches (read support calls) when people see that they work in explorer, but not from arbitrary applications depending on how they use the underlying APIs. If you're trying to abstract the filesystem with pluggable components... abstract _the filesystem_ with pluggable components. It'd be interesting to now what lead to the architectural decision of wear to cut that abstraction. It feels like a Microsoft team balkanization issue (the explorer team and POs being different from the NT kernel ones, I assume?), but it's near impossible to know without inside knowledge.
- bhawks 5y ago
  The traditional posix filesystem is yet another leaky abstraction. Inflexible ACLS and metadata, a horrible and confusing consistency story, an inability to reliably / cleanly rename files, interfaces that are painful to implement performantly without surprises, don't even get started on fsync... I just want to create, read, update and delete some resources :)
  - monocasa 5y ago
    But at least there I can expect programs run as the same user to see the same files. "Why can I see the files in explorer and not this program I run to use those files?" isn't an answer you have to burn support time on because it's not ultimately implementing an FS layer in special GUI components. It's not that I don't like their FS abstraction (there's a lot of abstractions I don't like; as an engineer that's not an interesting topic for discussion but instead basically expected). It's that they broke their own abstraction a decade later by implementing the multiplextion in a layer totally contrary to what the user expects. Like if FUSE was implemented entirely in Gnome components and anything that used open(2) broke as not seeing the veneer files but you could see them in every system application.
    - teddyh 5y ago
      > Like if FUSE was implemented entirely in Gnome components and anything that used open(2) broke as not seeing the veneer files but you could see them in every system application. Oh, you mean exactly like Gnome GVfs?
londons_explore 5y ago
I'd like to see API's like this implement deferred operations, with help from the kernel. For example, the zip folder could realise that deletion is an expensive task, and defer the job till later. It would tell the kernel that the raw bytes of the zip file are only available to other applications after this operation is complete (so that emailing someone the zip file can't represent the pre-deferred-operations state). The tree of deferred operations can then be optimized - for example, deleting multiple files in a zip file could be combined into one. Deleting stuff in a zip file, and then deleting the whole file can likewise be combined. Kinda like peephole optimisations for stuff your OS does.
redsparrow 5y ago
I don't know why but "your CAB archive is a folder" really took me back. It's nice to read something that makes me nostalgic for a time when Windows was a big part of my life. I don't know that I ever loved Windows, but we were intimate.
- 4gotunameagain 5y ago
  Stockholm syndrome ♡
  - creeble 5y ago
    Totally, I still have ‘fond’ memories of Hungarian notation variables starting with ‘h’ and ‘w’. What I can’t remember is what they stood for - ‘w’ was a word, and I think ‘h’ a half-word. But was a word 16 bits when I was writing that code? Thanks for the PTSD.
BitPirate 5y ago
Hussein Nasser just recently created a video on the topic of leaky abstractions in our everday tech stack. https://youtube.com/watch?v=4a3bI7AYsy4 https://youtube.com/watch?v=4a3bI7AYsy4
cryptica 5y ago
It's an interesting article but the term 'Leaky abstractions' is not suitable here. This abstraction is not leaky at all. It sounds like a great abstraction in fact. It's just the implementation that's slow and it's not because of the abstraction. The author himself admits "it’s relatively easy to think of ways to dramatically improve the performance of this scenario". That fact that the issue can be easily resolved without changing the abstraction (the plugin system) is proof that the abstraction is not leaky. A leaky abstraction would leak implementation details from the plugin to the caller and thus make it impossible to replace or modify the plugin without modifying the caller's logic. It doesn't appear to be the case here. It seems like the issue can be resolved just by changing the plugin's logic; the caller's logic would stay the same. This sounds like an excellent abstraction and could be the reason why the core logic hasn't had to be changed in 1998... It should be hailed as an achievement, not shunned as 'leaky'.
- ericlaw 5y ago
  The abstraction leaks because the performance characteristics are inherently impossible to mask from the higher layers. There are other places where the abstraction leaks as well (try to add a file with a Unicode name to a Compressed Folder). It's true that you could add more and more code to try to make the abstraction leak less.
  - cryptica 5y ago
    It's not impossible because the author of the article themselves admit that there are many ways to fix this. I can already think of one possible way they could speed up the copy-paste from the zip file; by waiting until all files have been copied to the file system before deleting files from the zip in one go. The fact that solutions do not require a change to the interface is a sign that this is not a problem with the abstraction. Is it the abstraction which is forcing the files to be copied 1 byte at a time? It doesn't seem like it. It's an implementation issue. The interface allowed the plugin to have been implemented in many other ways, the implementer of the plugin just happened to go about it the wrong way.
teknopaul 5y ago
leaky abstracttion is a... polite way of describing the Windows filesystem. :) Unix particularly Linux show how the filesystem abstraction can be sufficient for a huge range of problems, it's such a shame Pottering et al. fail to grok this. System services is a virus.
- teknopaul 5y ago
  Anyone interested in working on a library libportfs that allowed userland apps (in a cross platform manner) to create fs like /mnt/apache/connections and /mnt/apache/enabled
  - teknopaul 5y ago
    First job would be to expose the entire windows registry as a filesystem
JohnL4 5y ago
On a quick scan, I'm not seeing any mention of OS/2, but I feel like this extensible behavior of the shell first showed up there. Not sure if that was an IBM thing or a Microsoft thing (or both), but it was WAY WAY WAY different from what came before.
ivanche 5y ago
To this very day I wonder why people torture themselves with Windows Explorer instead of just using Total Commander and pressing Alt+F9.

480 points by jakub_g 5y ago

These are the kinds of articles I love to see on HN! I think it's so interesting to see how different OS' have approached these long lived operations over the years. Arguably, *nix OS' have been able to move past these problems since it appears that package management is much more of a first party concern. "Oh, you want to upgrade this utility? Well, I rely on these other things so you should update them too." I'm a webdev so a lot of this is foreign to me w/ how it actually all functions but it's cool to read about these problems and speculate how/why they came to be and why they may only exist on certain platforms.

Rerarom 5y ago
This is the kind of stuff I expected to study in my CS degree
- dhosek 5y ago
  The problem is that computer science ≠ software engineering, especially in academic contexts.
  - austhrow743 5y ago
    I've seen this a lot on here combined with cs degrees being frequently mentioned and software engineering degrees rarely mentioned. Are people widely choosing computer science over software engineering and then being surprised they aren't studying software engineering or is it rare for institutions to offer software engineering as an option?
    - korijn 5y ago
      I am not aware of anyone with a software engineering degree. Is that really a thing?
      - austhrow743 5y ago
        Was where I went. Two separate degrees: https://www.newcastle.edu.au/degrees/bachelor-of-software-engineering-honours https://www.newcastle.edu.au/degrees/bachelor-of-software-en... https://www.newcastle.edu.au/degrees/bachelor-of-computer-science https://www.newcastle.edu.au/degrees/bachelor-of-computer-sc... Looks like there's even a "computer systems engineering" as well. https://www.newcastle.edu.au/degrees/bachelor-of-computer-systems-engineering-honours https://www.newcastle.edu.au/degrees/bachelor-of-computer-sy...
      - deleted 5y ago
        [deleted]
      - drran 5y ago
        Yep, I'm the software engineer with diploma. I have master degree in Computer Sciences (Software Engineering Department). I'm from Ukraine.
      - MillenialMan 5y ago
        At least in the UK, "Software Engineering" degrees are just Computer Science degrees with a different name and a restricted subset of course options. From what I can tell, they're a marketing gimmick, but even respectable universities will offer them. The more questionable universities will also offer actual specialised courses under that moniker (in the same vein as "games design" degrees), but they aren't great. Some Unis also offer different tracks for Masters degrees - MA (Master of Arts) in Software Engineering, or MSc (Master of Sciences) in Computer Science. Those differences are a bit more meaningful. The MA will be softer, with less research focus, and is (ostensibly) explicitly designed as preparation for industry.
        Beldin 5y ago
        My uni offers both a one-year MSc SE, and a two-year MSc CS. For organisational reasons, the 2 yr program is basically SE+MSc Information Science. Students can pick their final subject; there is no difference in subjects between the two programmes, though we obviously expect some difference in result. After all, you should show some proficiency in both masters if you're doing the 2yr programme. I'm currently supervising a student (SE) on file recovery, to whom I'll definitely forward the link. It's right up his project's alley. Side remark: had he been enrolled in the 2yr programme, the subject would have been the same. I'd just expect him to extend his current work with a research question towards information science.
    - Siira 5y ago
      In the end, a good engineer rarely becomes a professor. This is both because the industry pays better, and the universities don’t value engineering expertise. The best SE courses in my university were offered by “guest” teachers from big software companies. (Which were very rare.)
  - MillenialMan 5y ago
    I don't think that's the problem. The problem is that Universities aren't punished for teaching inadequately. And when challenged, people justify it by saying, "ah, but this course that 90% of you took so that you could be prepared for a career as a developer wasn't really about being a developer."
    - unishark 5y ago
      > The problem is that Universities aren't punished for teaching inadequately. They are punished in a way. It hurts a school to get a bad reputation among local companies for poorly-prepared grads. R1 research schools, particularly if they are public, may feel it less. But even then, the ones I've dealt with have always chased all the fads in trying to improve learning and metrics. A tenured professor who's just there for the research may choose to ignore it all of course.
      - highmastdon 5y ago
        Not if every university is like that. Punishment will only happen when there will be that one university that has adequate courses and stands out above the rest
      - MillenialMan 5y ago
        This hasn't been my experience in practice. I went to a top-5 CS university in the UK and some of my classmates graduated almost unable to program (and they still, somehow, got jobs). Seeing that first-hand was extremely jarring. It didn't and still hasn't damaged the reputation of the university.
  - jjice 5y ago
    Unfortunately, at my University the SE program was mostly building UML and ER diagrams and memorizing agile terms. Software design was a minor part of it. I was CS, but had a lot of SE friends, and we all agree that the best SE course was the one that was mandatory for all computing majors. It covered version control and branching, testing, design process, abstraction design, etc. That class was probably one of the most daily applicable things I learned, I took it in my third semester. If my university offered more courses of that caliber, that didn't rehash and force memorization of buzzwords, then that program would've been incredible.

flameon 5y ago

Fauci emails reveal how the virus was created: https://twitter.com/Littleb29872980/status/1400153906089504777/photo/1 https://twitter.com/Littleb29872980/status/14001539060895047... Arrest Fauci Now!

bentcorner 5y ago

I wonder why it's implemented as a per-file copy+delete instead of a "copy all files" then "delete all files". I also have a gut feeling that doing similar operations to a connected android phone (e.g., moving photos from your phone to your PC over USB) is also slow, probably for similar reasons.

genpfault 5y ago
> doing similar operations to a connected android phone MTP is terrible[1]. [1]: https://en.wikipedia.org/wiki/Media_Transfer_Protocol#Performance https://en.wikipedia.org/wiki/Media_Transfer_Protocol#Perfor...
- jakub_g 5y ago
  Oh boy. I was thinking USB 2.0 is main reason for Android<>Windows copying of photos to suck so much, but the rabbit hole is much deeper. It's sad that with cloud being the solution for everything those days, this will probably never be improved within next decade.
  - userbinator 5y ago
    It's ridiculously wasteful and slow to have to upload to a cloud server who-knows-where, then download it again to the computer several feet away. Yet with Android having removed USB mass storage, that's often the easiest way. It's against the interests of those who profit off selling cloud storage to make local transfers easy.
    - steerablesafe 5y ago
      If I need to move a lot of data I just use adb and USB debugging to access the files. That is actually fast, but it's ridiculous that I have to do it.
      - mihi 5y ago
        Ha, I like that! I should try that as well.

vidarh 5y ago

This kind of thing doesn't surprise me at all - a surprising number of developers miss those types of things, and one of the things a lot of people miss seems to be checking for situations where you end up with excessive amount of calls doing little work, for some reason. E.g. at one point (long time ago), MySQL's C client library would call read() for 4 bytes to read a length indicator, and then read exactly the number of bytes indicated by the protocol, which led to a lot of time spent context-switching vs. reading into a larger user-space buffer and copying out of that. On Linux simple, plain "strace" remains one of my favourite first steps to find performance problems in part because these things are near immediately apparent if you strace a process because the really pathological cases often ends up dominating the output so much that you'll spot them right away. Another "favourite" issue that shows up often when you use strace like this is e.g. excessive include paths - running strace on MRI Ruby with rubygems enables and lots of gems pulled in is a good way of seeing that in action - like this zip problem it's an example that seems totally reasonable when the number of gems is small, and that first becomes apparent when you test with lots of gems, and look at what's actually happening under the hood. A similar example of testing with too small datasets and/or on a large enough machine to not spot what becomes immediately obvious if you trace execution was how a type 1 font reference library made available by Adobe (no idea if they wrote it or if it was from another source originally) which would exhibit another typical pathological behaviour and call malloc() hundreds of times on loading a font to allocate 4 bytes at the time instead of allocating larger buffers. (strace won't catch malloc(), but ltrace does)) To avoid messing too much with the code, we replaced the calls to malloc() with calls to a simple arena allocator and load speed shot through the roof and memory usage dropped massively. Too few people trace execution of their code. I know that because if most developers did, issues like the above would get caught much sooner, and would be rarer in published code.

formerly_proven 5y ago
> Another "favourite" issue that shows up often when you use strace like this is e.g. excessive include paths - running strace on MRI Ruby with rubygems enables and lots of gems pulled in is a good way of seeing that in action - like this zip problem it's an example that seems totally reasonable when the number of gems is small, and that first becomes apparent when you test with lots of gems, and look at what's actually happening under the hood. When you add e.g. Python's standard library as a ZIP to the Python search path, the thing will open/read/read/read/close that file approximately a gazillion times on startup. That's OK on Linux/Unix, where that is fairly cheap. Guess which OS doesn't like that pattern at all?

osipov 5y ago

The programmer responsible for this implementation successfully inverted a red-black tree during the job interview and explained a Big-Oh optimal solution for doing so.

deleted 5y ago

[deleted]

zbendefy 5y ago

Another similar and good article: https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/ https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a...

an1sotropy 5y ago
Another common leaky abstraction: floating-point numbers as an abstraction of the real numbers. It works nearly all the time, until you have to really know about numerical precision, or where a NaN came from. https://www.johndcook.com/blog/2009/04/06/numbers-are-a-leaky-abstraction/ https://www.johndcook.com/blog/2009/04/06/numbers-are-a-leak...
- kortex 5y ago
  Relevant: the list of integers you can convert to float, invert twice, and don't get the same number back. https://oeis.org/A275419 https://oeis.org/A275419
  - an1sotropy 5y ago
    This is a treasure; thank you!
- chii 5y ago
  i wouldn't call floating point numbers an abstraction on real numbers. It's a representation - like char[] is a representation of strings, which is itself an abstraction used to manipulate human readable text!
  - an1sotropy 5y ago
    Read the link above about FP as a leaky abstraction of the reals; it explains it better than I can. FP is not really a representation of reals, because you can't represent the entire number line with a finite number of bits. It is a representation of a cleverly chosen subset of the reals, so well-designed that many people forget that they're not dealing with real numbers. That intelligent people can write code with FP values, operating on them thinking they are real numbers, and not knowing how FP works, and get programs that often do nearly the right thing, speaks to the success of FP as an abstraction. Understanding how things can go wrong is the leaks (like, non-associativity). I don't think people manipulating char[] representations of strings are ever at risk of forgetting that it's just an array of chars (they program with full awareness of the implementation), so it doesn't feel like there's much abstraction happening there. Other higher-level data structures for text manipulation probably do implement some useful abstractions; I haven't worked with those.

Sol- 5y ago

Interesting, didn't even know you can cut files from a zip in that Windows zip file viewer. I would have thought it's some read-only filesystem like viewing a mounted CD or so. In that context I wonder if you could cut from rewritable CD-RWs as well back in the day (can't remember) - that seems like another abstraction that's similarly slow in reality.

jakub_g 5y ago
I never tried cutting from CD-RW, but AFAIR each burning would append a non-trivial header (like 20MBs or so) so that would be a pretty expensive thing to do :) AFAIR when you "copied" into CD-RW the files would show up semi-transparent (pending) and you'd have to click a button to process with burning. Probably same for cutting I guess.
- bluGill 5y ago
  Been years, but the ISO file system says 16 sectors past the start of the last track is where the file system index starts. (the first 15 sectors are reserved for booting) Then however many sectors you need for the index - which depends on how many files you have, and how long the filenames are. Note that both CDR and CDRW don't allow you to write sectors in the middle of a track, so you need to know where every file will be on disk before you can write anything - you can't build the index on the fly. (I remember this part because I was writing backups and so I didn't know how big the file would be until I was done and thus couldn't write the index first - I eventually got around the rule by creating a new track with just the index)
- TeMPOraL 5y ago
  Speaking of old Windows and CDs, Windows had some crazy trick to turn CD-R (not CD-RW) into rewriteable medium. I'm guessing they simulated a regular file system on top of an append-only representation. I never dug into the details back in the day, because I never used this feature. Unfortunately, IIRC, this trickery was enabled by default when copying files to CDs - which was a problem, because nothing else could read it. This caused me an unending stream of calls from friends and relatives, who all tried to burn some family/vacation photos onto a CD-R, in order to view them on their TVs, but the CD readers for TVs couldn't parse the format.

dsego 5y ago

Unless I'm mistaken, this seems to be the original programmer on youtube: Dave's Garage - Secret History of Windows ZIPFolders https://youtu.be/aQUtUQ_L8Yk https://youtu.be/aQUtUQ_L8Yk

codetrotter 5y ago
It is. I have seen a few of his videos before. They are generally interesting and worth checking out.
ericlaw 5y ago
As noted in the blog post, he's the author of the Shell Namespace code, but not the ZIP code beneath.
- dsego 5y ago
  Oh, I can see now it's mentioned, I must've skipped it on first reading.

mrkramer 5y ago

"Unfortunately, the code hasn’t really been updated in a while. A long while. The timestamp in the module claims it was last updated on Valentine’s Day 1998" When I saw this timestamp sometime ago on my PC I thought it was a joke?! C'mon 1998 like WTF!

sachinjoseph 5y ago

A bug has been filed about this on the WinDev repo, might help resolve it faster if it gets some upvotes: https://github.com/microsoft/Windows-Dev-Performance/issues/91 https://github.com/microsoft/Windows-Dev-Performance/issues/...

amichal 5y ago

This article is about the details of zip file handling in the Shell Extension... Reading the first few paras I thought they would talk about the struggle it was to implement a shell namespace extension. I did a somewhat crappy one 20-something years ago and I still remember how confusing it was. At the time is spent all my days writing various COM things. I was very comfortable doing things like hand editing IDL headers to avoid changing GUIDs. NSEs were still the most baffling, brittle and confusing interfaces I've ever seen

ping_pong 5y ago

25 years without major bugs and still being extremely useful is a complete win in my book! This is definitely a bug, but I wonder if Microsoft will think it's worth it to even update it, it doesn't seem like it happens enough to cause widespread issues.

foxbee 5y ago
This was my exact thinking!

deleted 5y ago

[deleted]

monocasa 5y ago

Having written a shell namespace extension, I now use the concept as a great example of making the wrong abstraction interestingly enough. Shell namespaces are a odd win32 concept rather than a filesystem concept and cause a lot of headaches (read support calls) when people see that they work in explorer, but not from arbitrary applications depending on how they use the underlying APIs. If you're trying to abstract the filesystem with pluggable components... abstract _the filesystem_ with pluggable components. It'd be interesting to now what lead to the architectural decision of wear to cut that abstraction. It feels like a Microsoft team balkanization issue (the explorer team and POs being different from the NT kernel ones, I assume?), but it's near impossible to know without inside knowledge.

bhawks 5y ago
The traditional posix filesystem is yet another leaky abstraction. Inflexible ACLS and metadata, a horrible and confusing consistency story, an inability to reliably / cleanly rename files, interfaces that are painful to implement performantly without surprises, don't even get started on fsync... I just want to create, read, update and delete some resources :)
- monocasa 5y ago
  But at least there I can expect programs run as the same user to see the same files. "Why can I see the files in explorer and not this program I run to use those files?" isn't an answer you have to burn support time on because it's not ultimately implementing an FS layer in special GUI components. It's not that I don't like their FS abstraction (there's a lot of abstractions I don't like; as an engineer that's not an interesting topic for discussion but instead basically expected). It's that they broke their own abstraction a decade later by implementing the multiplextion in a layer totally contrary to what the user expects. Like if FUSE was implemented entirely in Gnome components and anything that used open(2) broke as not seeing the veneer files but you could see them in every system application.
  - teddyh 5y ago
    > Like if FUSE was implemented entirely in Gnome components and anything that used open(2) broke as not seeing the veneer files but you could see them in every system application. Oh, you mean exactly like Gnome GVfs?

londons_explore 5y ago

I'd like to see API's like this implement deferred operations, with help from the kernel. For example, the zip folder could realise that deletion is an expensive task, and defer the job till later. It would tell the kernel that the raw bytes of the zip file are only available to other applications after this operation is complete (so that emailing someone the zip file can't represent the pre-deferred-operations state). The tree of deferred operations can then be optimized - for example, deleting multiple files in a zip file could be combined into one. Deleting stuff in a zip file, and then deleting the whole file can likewise be combined. Kinda like peephole optimisations for stuff your OS does.

redsparrow 5y ago

I don't know why but "your CAB archive is a folder" really took me back. It's nice to read something that makes me nostalgic for a time when Windows was a big part of my life. I don't know that I ever loved Windows, but we were intimate.

4gotunameagain 5y ago
Stockholm syndrome ♡
- creeble 5y ago
  Totally, I still have ‘fond’ memories of Hungarian notation variables starting with ‘h’ and ‘w’. What I can’t remember is what they stood for - ‘w’ was a word, and I think ‘h’ a half-word. But was a word 16 bits when I was writing that code? Thanks for the PTSD.

BitPirate 5y ago

Hussein Nasser just recently created a video on the topic of leaky abstractions in our everday tech stack. https://youtube.com/watch?v=4a3bI7AYsy4 https://youtube.com/watch?v=4a3bI7AYsy4

cryptica 5y ago

It's an interesting article but the term 'Leaky abstractions' is not suitable here. This abstraction is not leaky at all. It sounds like a great abstraction in fact. It's just the implementation that's slow and it's not because of the abstraction. The author himself admits "it’s relatively easy to think of ways to dramatically improve the performance of this scenario". That fact that the issue can be easily resolved without changing the abstraction (the plugin system) is proof that the abstraction is not leaky. A leaky abstraction would leak implementation details from the plugin to the caller and thus make it impossible to replace or modify the plugin without modifying the caller's logic. It doesn't appear to be the case here. It seems like the issue can be resolved just by changing the plugin's logic; the caller's logic would stay the same. This sounds like an excellent abstraction and could be the reason why the core logic hasn't had to be changed in 1998... It should be hailed as an achievement, not shunned as 'leaky'.

ericlaw 5y ago
The abstraction leaks because the performance characteristics are inherently impossible to mask from the higher layers. There are other places where the abstraction leaks as well (try to add a file with a Unicode name to a Compressed Folder). It's true that you could add more and more code to try to make the abstraction leak less.
- cryptica 5y ago
  It's not impossible because the author of the article themselves admit that there are many ways to fix this. I can already think of one possible way they could speed up the copy-paste from the zip file; by waiting until all files have been copied to the file system before deleting files from the zip in one go. The fact that solutions do not require a change to the interface is a sign that this is not a problem with the abstraction. Is it the abstraction which is forcing the files to be copied 1 byte at a time? It doesn't seem like it. It's an implementation issue. The interface allowed the plugin to have been implemented in many other ways, the implementer of the plugin just happened to go about it the wrong way.

teknopaul 5y ago

leaky abstracttion is a... polite way of describing the Windows filesystem. :) Unix particularly Linux show how the filesystem abstraction can be sufficient for a huge range of problems, it's such a shame Pottering et al. fail to grok this. System services is a virus.

teknopaul 5y ago
Anyone interested in working on a library libportfs that allowed userland apps (in a cross platform manner) to create fs like /mnt/apache/connections and /mnt/apache/enabled
- teknopaul 5y ago
  First job would be to expose the entire windows registry as a filesystem

JohnL4 5y ago

On a quick scan, I'm not seeing any mention of OS/2, but I feel like this extensible behavior of the shell first showed up there. Not sure if that was an IBM thing or a Microsoft thing (or both), but it was WAY WAY WAY different from what came before.

ivanche 5y ago

To this very day I wonder why people torture themselves with Windows Explorer instead of just using Total Commander and pressing Alt+F9.