7 ms·
I do, but it's not a popular opinion. POSIX, and by extension, the classic 1960s-1980s era UNIX way of doing things just needs die a long overdue death. This
by jiggawatts 7y ago
I do, but it's not a popular opinion.
POSIX, and by extension, the classic 1960s-1980s era UNIX way of doing things just needs die a long overdue death.
This stuff was designed at a time when every CPU instruction mattered, everything was optimised to death for frugality, and commands were abbreviated from "copy" to "cp" because ermahgerd two bytes is a huge saving! That mentality got us Y2K. This is an era where latencies were not the bottleneck, CPU cycles and memory bytes were.
A lot of stuff in filesystems is just plain stupid. For example, why do applications install their files. one. at. a. time? Like... what the fuck? How does it make any sense for an application to be partially installed? Who actually codes their application with 500 modules and dynamic libraries to be able to handle the scenario where one of them is inaccessible due to an ACL or a mismatched version because of an overwrite by something or someone else? NOBODY, that's who. Meanwhile, I can make a cup of tea while Adobe Lightroom launches on an SSD drive because it is 99% OS API overhead and 1% usermode action.
This is why Docker is popular. Not because Docker is good, but because OS APIs are retarded.
Every application install should be a union fs. This union fs should be entirely user-mode, so that if an application has 10,000 files, it doesn't take 10,000 round-trips to the OS kernel with the Intel mitigations, context switches, and cache flushes that all brings with it.
Copying a file shouldn't require a user-mode buffer to feed the data through, forcing it to come down WAN links just to go back up the same WAN link again on the way out.
Overwriting a file shouldn't require more than a single API call, because it's nearly 2020, and we should have long since realised that kernel transitions are expensive, so we should optimise to minimise the number of round-trips. open(), write(), write(), write(), flush(), sync(), close(), poke(), prod(), jesusfuckingchrist(). Just take a buffer bigger than 4KB, or better yet, standardise an API to take a stream from user mode.
Just take a buffer and a filename, and atomically replace. Done. Bang. No lost data, not torn writes, just DO IT. How hard can this be? Is it impossible to do this? Are we forever stuck with POSIX, which was created in 1988, before most of its modern users were born?
- eyelidlessness 7y agoSpeaking as someone whose job is primarily building high level APIs, this is unachievable. Those high level APIs satisfy the work the vast majority of people want or need to achieve, but some tasks require greater control over the details those APIs achieve, and all tasks require those details to otherwise be right. POSIX might not be the right lower level API, and maybe it can be replaced with something that achieves those details better. But "done bang just do it" is simply not how computers work. Software is how you achieve that, and generally by building simpler abstractions on top of more brittle ones. "How hard can this be?" is the right question. It's very damn hard.
- deeblering4 7y agoWith respect that wasn’t solely a design mentality. Resource limitations were a factor as well.
- blt 7y agoYour comment reminds me of the situation of game engines and 3D graphics APIs a few years ago before Vulkan and Metal were released. Too high-level for developers who want control (and understand how the hardware actually works), but too low-level for developers who want to minimize complexity. Now, Vulkan and Metal offer the detailed control for library developers, and everyone else uses some higher-level wrapper. Does it make sense to split the file system in a similar way? I guess the main challenge is avoiding too many competing wrappers.
- TazeTSchnitzel 7y agoThe problem with Vulkan is even graphics driver developers struggle to use it correctly.
- Shish2k 7y agoI give it 2-3 more years of people saying "this is impossible" before Lennart does it and makes everyone mad
- michaelmrose 7y agoNobody said it was impossible
- heavenlyblue 7y agoAll of what you are describing is already a product on top of the APIs you are describing?
- pas 7y agoThis already exists. O_DIRECT exists, now there's io_urig too (plus the libaio1 which uses io_submit the old async API). The problem is not with files. Kernel transitions are expensive, but the problem is not that. It's metadata sync (file inode + directory inode) and so on. Install is slow on Windows because braindeadness of vendors (cygwin is fast to install). Package stuff is slow on Linux, because apt calls out to dpkg, which first reads the package db. Which is a plain text file with newlines as separators and it has to be parsed. Yum is slow because by default it syncs repos on install, etc.
- asveikau 7y agoWindows filesystem performance is also much poorer than Linux for lots of small files.
- saagarjha 7y ago> commands were abbreviated from "copy" to "cp" because ermahgerd two bytes is a huge saving Considering the number of times I type that command, I wouldn't be as quick to throw out the savings here.
- AnIdiotOnTheNet 7y ago> Every application install should be a union fs. This union fs should be entirely user-mode, so that if an application has 10,000 files, it doesn't take 10,000 round-trips to the OS kernel with the Intel mitigations, context switches, and cache flushes that all brings with it. I disagree with this, as you've already over-complicated it by assuming some kind of unix-like-scatter-everything-all-over-the-file-tree-for-no-good-reason application installation. Just do what MacOS, RiscOS, DOS, etc did and have applications be a single file (or folder). There, install is just copy, can place it on any media you want, can carry it around with you, keep multiple versions, etc. It even keeps the abstraction of the application actually existing where it appears to exist.