7 ms·

Filesystem devs should aim to make “badly written” app code “just work” (2009)

195 points by pcr910303 7y ago

justinator 7y ago
Told you: Perl is built by the gods.
- peterwwillis 7y ago
  Perl has subtle complexity just like everything else. But it also has taint mode, which depending on the way you squint is either a great example of "security that just works", or a great example of "you must use X method to get good security".
  - mapgrep 7y ago
    I vote for the second option :-) IMO taint mode is a great example of what Linus is talking about. Too subtle. I cut my teeth on Perl and actually believed the books when they said you should always use taint mode when touching data that came e.g. over the network. I wrote all my web code under -T (-wT actually but you get the idea). Then one day I went to drop in some full text search via a then popular library (Plucene). And what do you know, Plucene would not work under taint mode, because it was not developed under taint mode. The maintainer would not accept my simple patch essentially because he did not understand that you could not untaint without a regex somewhere (i.e. he did not know how taint mode worked at a basic level). So I maintained a patched version of that lib privately. Only to later hit the same issue with another popular library. So I had to stop using taint mode. If it’s just an option — even one aggressively marketed in O’Reilly books back when people actually mostly read O’Reilly books to learn various systems — it’s not going to win much adoption.
- justinator 7y ago
  My dudes: it's Saturday night, and this was just a subtle ribbing on a programming language. Save your downvotes.
slimsag 7y ago
Suggestion: I think the title should be: > Linus: Theory and practice sometimes clash. And when that happens, theory loses. Every single time. His point seems to be more about accepting reality and allowing the practice (which in this case is theoretically "badly written" code) to "just work". As the title stands now: > Linus: People should aim to make "badly written" code "just work" One might incorrectly assume Linus is suggesting that defensive programming should be practiced heavily -- but that does not appear to be what he is saying here.
- sophiebits 7y ago
  Or perhaps “Filesystems should aim…”. Which is what he actually seemed to mean.
  - slimsag 7y ago
    Yeah, that seems a bit more precise!
mehrdadn 7y ago
Quite a bizarre read considering Linux's decision to e.g. overcommit memory makes 100% correctly written code break nondeterministically...
- perl4ever 7y ago
  It seems to me that a basic requirement for not being an idiot is that you recognize hard problems as being hard, and I see a lot of supposedly smart people declaring hard problems are easy because they're just ignoring tradeoffs or aspects of an approach that undermine it. I'd think "everybody" knows, certainly I would expect Linus to know, about Postel's law and the subtle ways it ends up causing problems. Whenever you make things easy or difficult, you shape the evolution of how people do those things. There's no simple universal answer to "do we make things easy or difficult". Or "do we blame the user or the toolmaker?" I don't really understand the psychology of going around arguing one side of an insoluble problem, observing that others are totally convinced of the other side, and occasionally flipping sides, but never acknowledging the meta-problem of integrating both or deciding when to apply each.
  - dooglius 7y ago
    What do you think the "insoluble" problem is here?
    - perl4ever 7y ago
      Deciding whether and how to influence the way people use a tool or product. Do you say "you're holding it wrong", or do you adjust to what people seem to be like? I mean, it's insoluble if treated as a single binary decision and not contextual.
- teddyh 7y ago
  The difference is that people do tend to check return values of write() (in cases where it actually matters), but not check the return value of malloc(). Since Linux must work with the code which exists, and since most programs tend to malloc() memory but never use it, the overcommitting practice begins to make sense. A twisted and ugly sense, but that is the world we live in.
zer00eyz 7y ago
"Anybody who wants more complex and subtle filesystem interfaces is just crazy. Not only will they never get used, they'll definitely not be stable." I think there is a more universal truism here - that "complex and subtle" are sources of pain, problems and headaches. I want to write "cool" and "magical" code as much as the next person, but that's the stuff that I look at later WTF because I am no longer in the same state of mind. Clear, simple, straight forward, plain as day are better than anything else. And if you do have to do something "magical" something "odd" or hard to understand for the love of god please leave notes explaining what, why and how you did what you did. And if you are replacing a "clean" and "readable" version leave it there commented out. Sure it is "in the repo" but almost no one ever looks and you are just making it harder for me to figure out the original intent of whatever was there.
- sillysaurusx 7y ago
  I am currently trying to wring maximum performance from Cloud TPUs. This comment really resonated with me, because Cloud TPUs are complex, subtle, overly complicated, and have twenty opaque ways that you can ruin your performance. Compare that with a GPU. It took decades, but (at least for nvidia GPUs) they have finally reached the point where you can be at least reasonably certain that high-performance is the default. Sure, you can ruin your performance by doing some stupid things, but it's at least harder to do those stupid things. The defaults are high-performance. It's exactly the opposite for Cloud TPUs. By default, it only uses one core. You have to use their TPUEstimator API, which is a byzantine mess. Since two weeks ago, I've spent roughly 3 solid working days of effort solely trying to read and understand (a) what is the TPUEstimator actually doing? (b) why is it supposedly so much faster? I have some half-hearted justifications -- the answer seems to be "you have to colocate your gradients with the device; you have to scope your tensorflow graph to a specific device; and each TPU core is a separate device." But there are unanswered questions. For example, you're supposed to pass your tensorflow computation to tpu.rewrite(). Yet I've never done that. The defaults seem to just work. So does that mean it does the rewriting for me automatically? Is it being emulated in software, and I'm ruining my performance? Tensorflow, why don't you just crash instead of being so damn slow on TPUs? That would at least let me aim my optimizations! If only I had low level access to the actual TPU operations, I could just write a compiler that specifically emits instructions to give me the performance I need. But this tensorflow graph abstraction makes everything "easy" yet exponentially more complicated. Anyway. Yes. More of your mindset, please. Simplicity is such a lovely metric seldom optimized for. (Anyone who's curious can see a dramatic 11x difference in performance on GPUs vs TPUs in Colab: https://twitter.com/theshawwn/status/1196593451174891520 https://twitter.com/theshawwn/status/1196593451174891520 ... this was very surprising, since Google's marketing would have you believe that TPUs are the bee's knees. Yet by default a TPUv2 is 11x slower than a K80 GPU for some basic matrix multiplications.)
NathanOsullivan 7y ago
(2009)
ymerej 7y ago
I think the message is that we should make the basic stuff "just work" before we start adding the complexity of more bells and whistles.
- perl4ever 7y ago
  More features don't have to be more complex or "subtle" (I wish).
pdonis 7y ago
The thread title omits a crucial word: "Filesystem" (as in "filesystem people", not just "people"). The point he is making is that filesystems are supposed to be utterly reliable; applications should not have to take extreme precautions to avoid having the filesystem lose their data. And the fact that practically nobody actually takes any precautions, let alone extreme ones, is strong evidence that programmers do in fact expect filesystems to be that reliable.
- pcr910303 7y ago
  I personally thought that this opinion is applicable to not only file systems but all ‘base’ systems that can be built upon. Hence the omission of the ‘filesystem’ word. But if most people think that this doesn’t apply outside of FS, I have the will to change the title :-)
  - reitzensteinm 7y ago
    It applies broadly to people building APIs, but your title reads as though Linus is giving general coding advice, saying not to be fussy about code quality but focus on correctness. At least one comment here is replying to that notion.
- blazespin 7y ago
  “File system people should make badly written application code just work.”
  - sn41 7y ago
    That makes much more sense. And it turns out to be the opposite of what I thought when I read the title. I had initially thought - just write bad code whose only merit is that it works.
- Gibbon1 7y ago
  More to the point he's saying crappy code that uses the filesystem should just work if possible. Don't make it worse than it is. And don't expect anyone's going to fix their code to use your new and improved API. Because they won't.
capdeck 7y ago
Is there a point though, when badly written code becomes so hard to maintain and improve, that people would just avoid doing that altogether? The reason "badly written" works for Open Source is that if code is useful - there will be someone in the future who will refactor it. In proprietary setting that only happens when the fate of the company itself (or a large chunk of the business) is at stake. Otherwise stagnation is king.
- diminoten 7y ago
  It's easy to pretend that "badly written" code is intentional or desirable, because then when we write it, we can excuse ourselves by saying, "BUT IT WORKS!" That's why this idea is popular, and your comment is controversial. You're taking away a crutch that many of our peers cling desperately to as a way to justify their shortcuts and poor decision making.
  - deleted 7y ago
    [deleted]
- naniwaduni 7y ago
  > there will be someone in the future who will refactor it maybe abstractly in practice, this never happens because the code will cease to be considered useful first
tomthumb 7y ago
Click bait title. Whenever Linus says something, he is correct, succinct and to the point, like his code. No fluff.
1gor 7y ago
APIs you provide to consumers should aim to make their badly written code just work. That's what Linus said.
- blazespin 7y ago
  Yeah. So many api writers aim to force clients do all the heavy lifting. The whole point of a good api is that it reduces heavy lifting. Anyone can write pass through apis that don’t do anything.
  - dpark 7y ago
    > The whole point of a good api is that it reduces heavy lifting. Isn’t that the opposite of what Torvalds is saying? He seems to be arguing for simplicity. APIs that do a bunch of magic for you are the opposite of simple and tend to be mountains of subtle bugs and unexpected behavior.
    - jasonjayr 7y ago
      I think that lies in the art of developing the API in the first place. It should give you enough primitives that it gets the job done balancing the responsibilities it advertises with the cognitive load on the developer to use it correctly.
    - Const-me 7y ago
      > APIs that do a bunch of magic for you are the opposite of simple You're mixing simplicity of API with simplicity of implementation. More often than not, you can only have one but not both. Modern Linux or Windows do huge amount of magic when you call kernel API like open (POSIX) / CreateFile (Windows), yet the API is simple and easy. You can expose all implementation details, your code will be simple, but hard to build upon. Speaking about data storage, once upon a time I programmed Nintendo consoles, their file system API probably was very simple for Nintendo to implement, but using it wasn't fun: SDK documentation specified delays, specified how to deal with corrupt flash memory, etc. You can do the other way, you'll have to do lot of work handling all the edge cases, your code will be very complex, but this way you might make a system that's actually useful. About data storage, SQL servers have tons of internal complexity, even sqlight does, but API, the SQL, is high level and easy to use even by non-programmers.
l1g2d5 7y ago
At the end of the day, the client/users/etc just want it to work. They don't care about the quality of the code.
RcouF1uZ4gsC 7y ago
> The undeniable FACT that people don't tend to check errors from close() should, for example, mean that delayed allocation must still track disk full conditions, for example. If your filesystem returns ENOSPC at close() rather than at write(), you just lost error coverage for disk full cases from 90% of all apps. It's that simple. Most programmers abstraction of a computer system is synchronous and consistent. If you make the simple synchronous cases do the wrong thing, like write returning success on disk full when a write did not happen, you are going to break people’s code. This also explains why when you when you choose a database, a relational DB with strong consistency and linearizability should be your default. Going for eventually consistent in the database layer and expecting the application logic to deal with it will lead to grief in many, many cases.
- TazeTSchnitzel 7y ago
  This could be summarised as: don't assume the responsibility to do something correctly if you don't have to, and whoever has to do it otherwise is more competent than you.
deleted 7y ago
[deleted]
bvinc 7y ago
I think he's got a point. As a developer I find myself in a different scenario. I'm usually trying to find out what exactly the 100% guaranteed way to do something is. Instead, I find incomplete documentation and different people with different opinions on what the guarantees are, and most people writing bad code that they assume will usually work. Just modifying a file in an atomic way requires a complicated dance of multiple files and multiple syncs and a rarely tested cleanup routine the next time the file is opened. No one does this. I don't know what the solution is.
- jiggawatts 7y ago
  I do, but it's not a popular opinion. POSIX, and by extension, the classic 1960s-1980s era UNIX way of doing things just needs die a long overdue death. This stuff was designed at a time when every CPU instruction mattered, everything was optimised to death for frugality, and commands were abbreviated from "copy" to "cp" because ermahgerd two bytes is a huge saving! That mentality got us Y2K. This is an era where latencies were not the bottleneck, CPU cycles and memory bytes were. A lot of stuff in filesystems is just plain stupid. For example, why do applications install their files. one. at. a. time? Like... what the fuck? How does it make any sense for an application to be partially installed? Who actually codes their application with 500 modules and dynamic libraries to be able to handle the scenario where one of them is inaccessible due to an ACL or a mismatched version because of an overwrite by something or someone else? NOBODY, that's who. Meanwhile, I can make a cup of tea while Adobe Lightroom launches on an SSD drive because it is 99% OS API overhead and 1% usermode action. This is why Docker is popular. Not because Docker is good, but because OS APIs are retarded. Every application install should be a union fs. This union fs should be entirely user-mode, so that if an application has 10,000 files, it doesn't take 10,000 round-trips to the OS kernel with the Intel mitigations, context switches, and cache flushes that all brings with it. Copying a file shouldn't require a user-mode buffer to feed the data through, forcing it to come down WAN links just to go back up the same WAN link again on the way out. Overwriting a file shouldn't require more than a single API call, because it's nearly 2020, and we should have long since realised that kernel transitions are expensive, so we should optimise to minimise the number of round-trips. open(), write(), write(), write(), flush(), sync(), close(), poke(), prod(), jesusfuckingchrist(). Just take a buffer bigger than 4KB, or better yet, standardise an API to take a stream from user mode. Just take a buffer and a filename, and atomically replace. Done. Bang. No lost data, not torn writes, just DO IT. How hard can this be? Is it impossible to do this? Are we forever stuck with POSIX, which was created in 1988, before most of its modern users were born?
etxm 7y ago
That title would explain most of the industry. Slaaaaaaaammmmmmmmm.
iradik 7y ago
Kyle wants to add a new API called barrier() which will improve consistency and remove the need for fsync. Linus makes the point here is that the file system API is already complicated to the point that few use or implement it correctly. Further complicating the API will likely create more problems than it fixes.
ijiiijji1 7y ago
A robust interface does what the user intended, not just the bare minimum to fulfill a specification. And it introduces the fewest novel quirks, rough edges and gotchas possible. Because that's what people want: things to be handled lower down with least surprises, including "just works" gracefully as possible in the face of real-world errors. This usually means handling some corner-cases lower down with special treatment so as to not burden the caller.
Animats 7y ago
I'd argue that UNIX-type file systems should offer several types of files: * Unit files. When you create a file and write it, it's not visible for other opens until you close it. If you open a file with O_CREAT|O_WRONLY|O_TRUNC, you create a new file, which replaces the old one on close. In the event of a program or system crash, or exiting via "abort" without closing first, the old file remains. So there's always one completely written file. Creating a unit file is an atomic operation. Most files are unit files. (This isn't original with me; it comes from a distributed UNIX variant developed at UCLA in the 1980s.) Replacing an existing file currently requires elaborate renaming gyrations which vary from OS to OS and file system to file system. At least for Linux, this should Just Work in the normal case. * Log files. If you open a file with O_APPEND, you can only write at the end. The file system should guarantee that, even after a crash, you get the file as written out to some previous write. If you call "fsync", the recovery guarantee should include everything written up to the "fsync" point. No seeking backwards and overwriting on a log file. * Temporary files. You can do all the file operations, and the file disappears on a reboot. * Managed files. These are for databases and such. They have some extra API functions, for async reads and writes and commits. Async I/O should have two callbacks - "the data has been taken and you can now reuse the buffer", and "this write is definitely committed and will survive a system crash" That's what databases really need, and try to fake with "fsync". Only a few programs will use this, but those are important programs.
- ianai 7y ago
  How about sandboxed files? Ie can be assigned to a process - full acl per process/application binary. Maybe even checksum the binary before loading to ram and granting access.
- skissane 7y ago
  > * Temporary files. You can do all the file operations, and the file disappears on a reboot. It would be nice to be able to have a process tree own a temporary file, such that when the last process in the tree exits (not necessarily the process which created the file), the file is automatically deleted, rather than having to wait for the next reboot.
ianai 7y ago
Slightly OT: is there any new emerging tech for Linux file systems that aren’t ext4/xfs/zfs? I was surprised the other day by those being the options for / on centos.
- Lammy 7y ago
  Btrfs was included as experimental in CentOS 6 and 7, but removed for CentOS 8.
- wmf 7y ago
  bcachefs looks pretty cool but who knows how long it will take to get upstream.
bsder 7y ago
I wonder if ZFS had these issues in the same timeframe? These strike me more as "Linux doesn't believe in actual testing" rather than inherent bugs because its a filesystem.
newnewpdro 7y ago
I feel like Linus is basically just arguing in favor of the principle of least surprise. There are definitely some serious pitfalls when it comes to unix file io, just look at for how long we lived with postgres and its broken assumptions surrounding fsync behavior on linux.
- rodgerd 7y ago
  > how long we lived with postgres and its broken assumptions surrounding fsync behavior on linux. You mean its assumption that the API wasn't lying about the integrity of the data it claimed to be writing? Very broken indeed.
kstenerud 7y ago
To put this more generally: a properly designed API makes is easy and natural to do things right, and difficult (but not impossible) to do things wrong.
purplezooey 7y ago
Missing the W. Richard Stevens books right about now...

195 points by pcr910303 7y ago

justinator 7y ago

Told you: Perl is built by the gods.

peterwwillis 7y ago
Perl has subtle complexity just like everything else. But it also has taint mode, which depending on the way you squint is either a great example of "security that just works", or a great example of "you must use X method to get good security".
- mapgrep 7y ago
  I vote for the second option :-) IMO taint mode is a great example of what Linus is talking about. Too subtle. I cut my teeth on Perl and actually believed the books when they said you should always use taint mode when touching data that came e.g. over the network. I wrote all my web code under -T (-wT actually but you get the idea). Then one day I went to drop in some full text search via a then popular library (Plucene). And what do you know, Plucene would not work under taint mode, because it was not developed under taint mode. The maintainer would not accept my simple patch essentially because he did not understand that you could not untaint without a regex somewhere (i.e. he did not know how taint mode worked at a basic level). So I maintained a patched version of that lib privately. Only to later hit the same issue with another popular library. So I had to stop using taint mode. If it’s just an option — even one aggressively marketed in O’Reilly books back when people actually mostly read O’Reilly books to learn various systems — it’s not going to win much adoption.
justinator 7y ago
My dudes: it's Saturday night, and this was just a subtle ribbing on a programming language. Save your downvotes.

slimsag 7y ago

Suggestion: I think the title should be: > Linus: Theory and practice sometimes clash. And when that happens, theory loses. Every single time. His point seems to be more about accepting reality and allowing the practice (which in this case is theoretically "badly written" code) to "just work". As the title stands now: > Linus: People should aim to make "badly written" code "just work" One might incorrectly assume Linus is suggesting that defensive programming should be practiced heavily -- but that does not appear to be what he is saying here.

sophiebits 7y ago
Or perhaps “Filesystems should aim…”. Which is what he actually seemed to mean.
- slimsag 7y ago
  Yeah, that seems a bit more precise!

mehrdadn 7y ago

Quite a bizarre read considering Linux's decision to e.g. overcommit memory makes 100% correctly written code break nondeterministically...

perl4ever 7y ago
It seems to me that a basic requirement for not being an idiot is that you recognize hard problems as being hard, and I see a lot of supposedly smart people declaring hard problems are easy because they're just ignoring tradeoffs or aspects of an approach that undermine it. I'd think "everybody" knows, certainly I would expect Linus to know, about Postel's law and the subtle ways it ends up causing problems. Whenever you make things easy or difficult, you shape the evolution of how people do those things. There's no simple universal answer to "do we make things easy or difficult". Or "do we blame the user or the toolmaker?" I don't really understand the psychology of going around arguing one side of an insoluble problem, observing that others are totally convinced of the other side, and occasionally flipping sides, but never acknowledging the meta-problem of integrating both or deciding when to apply each.
- dooglius 7y ago
  What do you think the "insoluble" problem is here?
  - perl4ever 7y ago
    Deciding whether and how to influence the way people use a tool or product. Do you say "you're holding it wrong", or do you adjust to what people seem to be like? I mean, it's insoluble if treated as a single binary decision and not contextual.
teddyh 7y ago
The difference is that people do tend to check return values of write() (in cases where it actually matters), but not check the return value of malloc(). Since Linux must work with the code which exists, and since most programs tend to malloc() memory but never use it, the overcommitting practice begins to make sense. A twisted and ugly sense, but that is the world we live in.

zer00eyz 7y ago

"Anybody who wants more complex and subtle filesystem interfaces is just crazy. Not only will they never get used, they'll definitely not be stable." I think there is a more universal truism here - that "complex and subtle" are sources of pain, problems and headaches. I want to write "cool" and "magical" code as much as the next person, but that's the stuff that I look at later WTF because I am no longer in the same state of mind. Clear, simple, straight forward, plain as day are better than anything else. And if you do have to do something "magical" something "odd" or hard to understand for the love of god please leave notes explaining what, why and how you did what you did. And if you are replacing a "clean" and "readable" version leave it there commented out. Sure it is "in the repo" but almost no one ever looks and you are just making it harder for me to figure out the original intent of whatever was there.

sillysaurusx 7y ago
I am currently trying to wring maximum performance from Cloud TPUs. This comment really resonated with me, because Cloud TPUs are complex, subtle, overly complicated, and have twenty opaque ways that you can ruin your performance. Compare that with a GPU. It took decades, but (at least for nvidia GPUs) they have finally reached the point where you can be at least reasonably certain that high-performance is the default. Sure, you can ruin your performance by doing some stupid things, but it's at least harder to do those stupid things. The defaults are high-performance. It's exactly the opposite for Cloud TPUs. By default, it only uses one core. You have to use their TPUEstimator API, which is a byzantine mess. Since two weeks ago, I've spent roughly 3 solid working days of effort solely trying to read and understand (a) what is the TPUEstimator actually doing? (b) why is it supposedly so much faster? I have some half-hearted justifications -- the answer seems to be "you have to colocate your gradients with the device; you have to scope your tensorflow graph to a specific device; and each TPU core is a separate device." But there are unanswered questions. For example, you're supposed to pass your tensorflow computation to tpu.rewrite(). Yet I've never done that. The defaults seem to just work. So does that mean it does the rewriting for me automatically? Is it being emulated in software, and I'm ruining my performance? Tensorflow, why don't you just crash instead of being so damn slow on TPUs? That would at least let me aim my optimizations! If only I had low level access to the actual TPU operations, I could just write a compiler that specifically emits instructions to give me the performance I need. But this tensorflow graph abstraction makes everything "easy" yet exponentially more complicated. Anyway. Yes. More of your mindset, please. Simplicity is such a lovely metric seldom optimized for. (Anyone who's curious can see a dramatic 11x difference in performance on GPUs vs TPUs in Colab: https://twitter.com/theshawwn/status/1196593451174891520 https://twitter.com/theshawwn/status/1196593451174891520 ... this was very surprising, since Google's marketing would have you believe that TPUs are the bee's knees. Yet by default a TPUv2 is 11x slower than a K80 GPU for some basic matrix multiplications.)

NathanOsullivan 7y ago

(2009)

ymerej 7y ago

I think the message is that we should make the basic stuff "just work" before we start adding the complexity of more bells and whistles.

perl4ever 7y ago
More features don't have to be more complex or "subtle" (I wish).

pdonis 7y ago

The thread title omits a crucial word: "Filesystem" (as in "filesystem people", not just "people"). The point he is making is that filesystems are supposed to be utterly reliable; applications should not have to take extreme precautions to avoid having the filesystem lose their data. And the fact that practically nobody actually takes any precautions, let alone extreme ones, is strong evidence that programmers do in fact expect filesystems to be that reliable.

pcr910303 7y ago
I personally thought that this opinion is applicable to not only file systems but all ‘base’ systems that can be built upon. Hence the omission of the ‘filesystem’ word. But if most people think that this doesn’t apply outside of FS, I have the will to change the title :-)
- reitzensteinm 7y ago
  It applies broadly to people building APIs, but your title reads as though Linus is giving general coding advice, saying not to be fussy about code quality but focus on correctness. At least one comment here is replying to that notion.
blazespin 7y ago
“File system people should make badly written application code just work.”
- sn41 7y ago
  That makes much more sense. And it turns out to be the opposite of what I thought when I read the title. I had initially thought - just write bad code whose only merit is that it works.
Gibbon1 7y ago
More to the point he's saying crappy code that uses the filesystem should just work if possible. Don't make it worse than it is. And don't expect anyone's going to fix their code to use your new and improved API. Because they won't.

capdeck 7y ago

Is there a point though, when badly written code becomes so hard to maintain and improve, that people would just avoid doing that altogether? The reason "badly written" works for Open Source is that if code is useful - there will be someone in the future who will refactor it. In proprietary setting that only happens when the fate of the company itself (or a large chunk of the business) is at stake. Otherwise stagnation is king.

diminoten 7y ago
It's easy to pretend that "badly written" code is intentional or desirable, because then when we write it, we can excuse ourselves by saying, "BUT IT WORKS!" That's why this idea is popular, and your comment is controversial. You're taking away a crutch that many of our peers cling desperately to as a way to justify their shortcuts and poor decision making.
- deleted 7y ago
  [deleted]
naniwaduni 7y ago
> there will be someone in the future who will refactor it maybe abstractly in practice, this never happens because the code will cease to be considered useful first

tomthumb 7y ago

Click bait title. Whenever Linus says something, he is correct, succinct and to the point, like his code. No fluff.

1gor 7y ago

APIs you provide to consumers should aim to make their badly written code just work. That's what Linus said.

blazespin 7y ago
Yeah. So many api writers aim to force clients do all the heavy lifting. The whole point of a good api is that it reduces heavy lifting. Anyone can write pass through apis that don’t do anything.
- dpark 7y ago
  > The whole point of a good api is that it reduces heavy lifting. Isn’t that the opposite of what Torvalds is saying? He seems to be arguing for simplicity. APIs that do a bunch of magic for you are the opposite of simple and tend to be mountains of subtle bugs and unexpected behavior.
  - jasonjayr 7y ago
    I think that lies in the art of developing the API in the first place. It should give you enough primitives that it gets the job done balancing the responsibilities it advertises with the cognitive load on the developer to use it correctly.
  - Const-me 7y ago
    > APIs that do a bunch of magic for you are the opposite of simple You're mixing simplicity of API with simplicity of implementation. More often than not, you can only have one but not both. Modern Linux or Windows do huge amount of magic when you call kernel API like open (POSIX) / CreateFile (Windows), yet the API is simple and easy. You can expose all implementation details, your code will be simple, but hard to build upon. Speaking about data storage, once upon a time I programmed Nintendo consoles, their file system API probably was very simple for Nintendo to implement, but using it wasn't fun: SDK documentation specified delays, specified how to deal with corrupt flash memory, etc. You can do the other way, you'll have to do lot of work handling all the edge cases, your code will be very complex, but this way you might make a system that's actually useful. About data storage, SQL servers have tons of internal complexity, even sqlight does, but API, the SQL, is high level and easy to use even by non-programmers.

l1g2d5 7y ago

At the end of the day, the client/users/etc just want it to work. They don't care about the quality of the code.

RcouF1uZ4gsC 7y ago

> The undeniable FACT that people don't tend to check errors from close() should, for example, mean that delayed allocation must still track disk full conditions, for example. If your filesystem returns ENOSPC at close() rather than at write(), you just lost error coverage for disk full cases from 90% of all apps. It's that simple. Most programmers abstraction of a computer system is synchronous and consistent. If you make the simple synchronous cases do the wrong thing, like write returning success on disk full when a write did not happen, you are going to break people’s code. This also explains why when you when you choose a database, a relational DB with strong consistency and linearizability should be your default. Going for eventually consistent in the database layer and expecting the application logic to deal with it will lead to grief in many, many cases.

TazeTSchnitzel 7y ago
This could be summarised as: don't assume the responsibility to do something correctly if you don't have to, and whoever has to do it otherwise is more competent than you.

deleted 7y ago

[deleted]

bvinc 7y ago

I think he's got a point. As a developer I find myself in a different scenario. I'm usually trying to find out what exactly the 100% guaranteed way to do something is. Instead, I find incomplete documentation and different people with different opinions on what the guarantees are, and most people writing bad code that they assume will usually work. Just modifying a file in an atomic way requires a complicated dance of multiple files and multiple syncs and a rarely tested cleanup routine the next time the file is opened. No one does this. I don't know what the solution is.

jiggawatts 7y ago
I do, but it's not a popular opinion. POSIX, and by extension, the classic 1960s-1980s era UNIX way of doing things just needs die a long overdue death. This stuff was designed at a time when every CPU instruction mattered, everything was optimised to death for frugality, and commands were abbreviated from "copy" to "cp" because ermahgerd two bytes is a huge saving! That mentality got us Y2K. This is an era where latencies were not the bottleneck, CPU cycles and memory bytes were. A lot of stuff in filesystems is just plain stupid. For example, why do applications install their files. one. at. a. time? Like... what the fuck? How does it make any sense for an application to be partially installed? Who actually codes their application with 500 modules and dynamic libraries to be able to handle the scenario where one of them is inaccessible due to an ACL or a mismatched version because of an overwrite by something or someone else? NOBODY, that's who. Meanwhile, I can make a cup of tea while Adobe Lightroom launches on an SSD drive because it is 99% OS API overhead and 1% usermode action. This is why Docker is popular. Not because Docker is good, but because OS APIs are retarded. Every application install should be a union fs. This union fs should be entirely user-mode, so that if an application has 10,000 files, it doesn't take 10,000 round-trips to the OS kernel with the Intel mitigations, context switches, and cache flushes that all brings with it. Copying a file shouldn't require a user-mode buffer to feed the data through, forcing it to come down WAN links just to go back up the same WAN link again on the way out. Overwriting a file shouldn't require more than a single API call, because it's nearly 2020, and we should have long since realised that kernel transitions are expensive, so we should optimise to minimise the number of round-trips. open(), write(), write(), write(), flush(), sync(), close(), poke(), prod(), jesusfuckingchrist(). Just take a buffer bigger than 4KB, or better yet, standardise an API to take a stream from user mode. Just take a buffer and a filename, and atomically replace. Done. Bang. No lost data, not torn writes, just DO IT. How hard can this be? Is it impossible to do this? Are we forever stuck with POSIX, which was created in 1988, before most of its modern users were born?

etxm 7y ago

That title would explain most of the industry. Slaaaaaaaammmmmmmmm.

iradik 7y ago

Kyle wants to add a new API called barrier() which will improve consistency and remove the need for fsync. Linus makes the point here is that the file system API is already complicated to the point that few use or implement it correctly. Further complicating the API will likely create more problems than it fixes.

ijiiijji1 7y ago

A robust interface does what the user intended, not just the bare minimum to fulfill a specification. And it introduces the fewest novel quirks, rough edges and gotchas possible. Because that's what people want: things to be handled lower down with least surprises, including "just works" gracefully as possible in the face of real-world errors. This usually means handling some corner-cases lower down with special treatment so as to not burden the caller.

Animats 7y ago

I'd argue that UNIX-type file systems should offer several types of files: * Unit files. When you create a file and write it, it's not visible for other opens until you close it. If you open a file with O_CREAT|O_WRONLY|O_TRUNC, you create a new file, which replaces the old one on close. In the event of a program or system crash, or exiting via "abort" without closing first, the old file remains. So there's always one completely written file. Creating a unit file is an atomic operation. Most files are unit files. (This isn't original with me; it comes from a distributed UNIX variant developed at UCLA in the 1980s.) Replacing an existing file currently requires elaborate renaming gyrations which vary from OS to OS and file system to file system. At least for Linux, this should Just Work in the normal case. * Log files. If you open a file with O_APPEND, you can only write at the end. The file system should guarantee that, even after a crash, you get the file as written out to some previous write. If you call "fsync", the recovery guarantee should include everything written up to the "fsync" point. No seeking backwards and overwriting on a log file. * Temporary files. You can do all the file operations, and the file disappears on a reboot. * Managed files. These are for databases and such. They have some extra API functions, for async reads and writes and commits. Async I/O should have two callbacks - "the data has been taken and you can now reuse the buffer", and "this write is definitely committed and will survive a system crash" That's what databases really need, and try to fake with "fsync". Only a few programs will use this, but those are important programs.

ianai 7y ago
How about sandboxed files? Ie can be assigned to a process - full acl per process/application binary. Maybe even checksum the binary before loading to ram and granting access.
skissane 7y ago
> * Temporary files. You can do all the file operations, and the file disappears on a reboot. It would be nice to be able to have a process tree own a temporary file, such that when the last process in the tree exits (not necessarily the process which created the file), the file is automatically deleted, rather than having to wait for the next reboot.

ianai 7y ago

Slightly OT: is there any new emerging tech for Linux file systems that aren’t ext4/xfs/zfs? I was surprised the other day by those being the options for / on centos.

Lammy 7y ago
Btrfs was included as experimental in CentOS 6 and 7, but removed for CentOS 8.
wmf 7y ago
bcachefs looks pretty cool but who knows how long it will take to get upstream.

bsder 7y ago

I wonder if ZFS had these issues in the same timeframe? These strike me more as "Linux doesn't believe in actual testing" rather than inherent bugs because its a filesystem.

newnewpdro 7y ago

I feel like Linus is basically just arguing in favor of the principle of least surprise. There are definitely some serious pitfalls when it comes to unix file io, just look at for how long we lived with postgres and its broken assumptions surrounding fsync behavior on linux.

rodgerd 7y ago
> how long we lived with postgres and its broken assumptions surrounding fsync behavior on linux. You mean its assumption that the API wasn't lying about the integrity of the data it claimed to be writing? Very broken indeed.

kstenerud 7y ago

To put this more generally: a properly designed API makes is easy and natural to do things right, and difficult (but not impossible) to do things wrong.

purplezooey 7y ago

Missing the W. Richard Stevens books right about now...