5 ms·

I think he's got a point. As a developer I find myself in a different scenario. I'm usually trying to find out what exactly the 100% guaranteed way to do somet

by bvinc 7y ago

I think he's got a point. As a developer I find myself in a different scenario. I'm usually trying to find out what exactly the 100% guaranteed way to do something is. Instead, I find incomplete documentation and different people with different opinions on what the guarantees are, and most people writing bad code that they assume will usually work. Just modifying a file in an atomic way requires a complicated dance of multiple files and multiple syncs and a rarely tested cleanup routine the next time the file is opened. No one does this. I don't know what the solution is.

jiggawatts 7y ago
I do, but it's not a popular opinion. POSIX, and by extension, the classic 1960s-1980s era UNIX way of doing things just needs die a long overdue death. This stuff was designed at a time when every CPU instruction mattered, everything was optimised to death for frugality, and commands were abbreviated from "copy" to "cp" because ermahgerd two bytes is a huge saving! That mentality got us Y2K. This is an era where latencies were not the bottleneck, CPU cycles and memory bytes were. A lot of stuff in filesystems is just plain stupid. For example, why do applications install their files. one. at. a. time? Like... what the fuck? How does it make any sense for an application to be partially installed? Who actually codes their application with 500 modules and dynamic libraries to be able to handle the scenario where one of them is inaccessible due to an ACL or a mismatched version because of an overwrite by something or someone else? NOBODY, that's who. Meanwhile, I can make a cup of tea while Adobe Lightroom launches on an SSD drive because it is 99% OS API overhead and 1% usermode action. This is why Docker is popular. Not because Docker is good, but because OS APIs are retarded. Every application install should be a union fs. This union fs should be entirely user-mode, so that if an application has 10,000 files, it doesn't take 10,000 round-trips to the OS kernel with the Intel mitigations, context switches, and cache flushes that all brings with it. Copying a file shouldn't require a user-mode buffer to feed the data through, forcing it to come down WAN links just to go back up the same WAN link again on the way out. Overwriting a file shouldn't require more than a single API call, because it's nearly 2020, and we should have long since realised that kernel transitions are expensive, so we should optimise to minimise the number of round-trips. open(), write(), write(), write(), flush(), sync(), close(), poke(), prod(), jesusfuckingchrist(). Just take a buffer bigger than 4KB, or better yet, standardise an API to take a stream from user mode. Just take a buffer and a filename, and atomically replace. Done. Bang. No lost data, not torn writes, just DO IT. How hard can this be? Is it impossible to do this? Are we forever stuck with POSIX, which was created in 1988, before most of its modern users were born?
- eyelidlessness 7y ago
  Speaking as someone whose job is primarily building high level APIs, this is unachievable. Those high level APIs satisfy the work the vast majority of people want or need to achieve, but some tasks require greater control over the details those APIs achieve, and all tasks require those details to otherwise be right. POSIX might not be the right lower level API, and maybe it can be replaced with something that achieves those details better. But "done bang just do it" is simply not how computers work. Software is how you achieve that, and generally by building simpler abstractions on top of more brittle ones. "How hard can this be?" is the right question. It's very damn hard.
- deeblering4 7y ago
  With respect that wasn’t solely a design mentality. Resource limitations were a factor as well.
- blt 7y ago
  Your comment reminds me of the situation of game engines and 3D graphics APIs a few years ago before Vulkan and Metal were released. Too high-level for developers who want control (and understand how the hardware actually works), but too low-level for developers who want to minimize complexity. Now, Vulkan and Metal offer the detailed control for library developers, and everyone else uses some higher-level wrapper. Does it make sense to split the file system in a similar way? I guess the main challenge is avoiding too many competing wrappers.
  - TazeTSchnitzel 7y ago
    The problem with Vulkan is even graphics driver developers struggle to use it correctly.
- Shish2k 7y ago
  I give it 2-3 more years of people saying "this is impossible" before Lennart does it and makes everyone mad
  - 7y ago
randompi 7y ago
As a developer, no one seems to know what's the best solution. But if they're not the one implementing it, then suddenly everyone's an expert and has an opinion.

by bvinc 7y ago

jiggawatts 7y ago

I do, but it's not a popular opinion. POSIX, and by extension, the classic 1960s-1980s era UNIX way of doing things just needs die a long overdue death. This stuff was designed at a time when every CPU instruction mattered, everything was optimised to death for frugality, and commands were abbreviated from "copy" to "cp" because ermahgerd two bytes is a huge saving! That mentality got us Y2K. This is an era where latencies were not the bottleneck, CPU cycles and memory bytes were. A lot of stuff in filesystems is just plain stupid. For example, why do applications install their files. one. at. a. time? Like... what the fuck? How does it make any sense for an application to be partially installed? Who actually codes their application with 500 modules and dynamic libraries to be able to handle the scenario where one of them is inaccessible due to an ACL or a mismatched version because of an overwrite by something or someone else? NOBODY, that's who. Meanwhile, I can make a cup of tea while Adobe Lightroom launches on an SSD drive because it is 99% OS API overhead and 1% usermode action. This is why Docker is popular. Not because Docker is good, but because OS APIs are retarded. Every application install should be a union fs. This union fs should be entirely user-mode, so that if an application has 10,000 files, it doesn't take 10,000 round-trips to the OS kernel with the Intel mitigations, context switches, and cache flushes that all brings with it. Copying a file shouldn't require a user-mode buffer to feed the data through, forcing it to come down WAN links just to go back up the same WAN link again on the way out. Overwriting a file shouldn't require more than a single API call, because it's nearly 2020, and we should have long since realised that kernel transitions are expensive, so we should optimise to minimise the number of round-trips. open(), write(), write(), write(), flush(), sync(), close(), poke(), prod(), jesusfuckingchrist(). Just take a buffer bigger than 4KB, or better yet, standardise an API to take a stream from user mode. Just take a buffer and a filename, and atomically replace. Done. Bang. No lost data, not torn writes, just DO IT. How hard can this be? Is it impossible to do this? Are we forever stuck with POSIX, which was created in 1988, before most of its modern users were born?

eyelidlessness 7y ago
Speaking as someone whose job is primarily building high level APIs, this is unachievable. Those high level APIs satisfy the work the vast majority of people want or need to achieve, but some tasks require greater control over the details those APIs achieve, and all tasks require those details to otherwise be right. POSIX might not be the right lower level API, and maybe it can be replaced with something that achieves those details better. But "done bang just do it" is simply not how computers work. Software is how you achieve that, and generally by building simpler abstractions on top of more brittle ones. "How hard can this be?" is the right question. It's very damn hard.
deeblering4 7y ago
With respect that wasn’t solely a design mentality. Resource limitations were a factor as well.
blt 7y ago
Your comment reminds me of the situation of game engines and 3D graphics APIs a few years ago before Vulkan and Metal were released. Too high-level for developers who want control (and understand how the hardware actually works), but too low-level for developers who want to minimize complexity. Now, Vulkan and Metal offer the detailed control for library developers, and everyone else uses some higher-level wrapper. Does it make sense to split the file system in a similar way? I guess the main challenge is avoiding too many competing wrappers.
- TazeTSchnitzel 7y ago
  The problem with Vulkan is even graphics driver developers struggle to use it correctly.
Shish2k 7y ago
I give it 2-3 more years of people saying "this is impossible" before Lennart does it and makes everyone mad
- 7y ago

randompi 7y ago

As a developer, no one seems to know what's the best solution. But if they're not the one implementing it, then suddenly everyone's an expert and has an opinion.