9 ms·

Files Are Fraught with Peril (2019)

103 points by bshanks 6y ago

helmholtz 6y ago
I know this guy is some kind of hacker news celebrity, but that website is an abomination. It's undreadably ugly. Is it a thing now? "Look at me, I'm too cool to think on behalf of my users. They'll adjust it for themselves. They'll use reader mode. I'm 1337 and styling is for fucktards"?? It's like Norvig's website, only worse.
- dxdm 6y ago
  This actually looks fine to me on mobile. Is it bad on a large screen? What a nice surprise, usually it's the other way round, and then reader mode often also doesn't work.
  - _0ffh 6y ago
    Large screen user here, site looks fine to me.
  - netmare 6y ago
    It's a mofo™-styled website, with eye-piercing white backgrounds and mile long lines. It's as minimalist as it gets, unless you consider vowels superfluous in prose. Joking aside, yes, it looks bland on a PC. However, I would rather have more sites like this, where I don't have to remove 97% of fluff just to read the content.
    - BlueTemplar 6y ago
      > with eye-piercing white backgrounds I recommend the Dark Reader extension.
  - snazz 6y ago
    The long lines significantly harm readability. Luckily, it's a very easy site for Reader View to parse (at least in Safari). I'm fine with this type of site, although a simple max-width would make it much easier to read without Reader View.
pvg 6y ago
Previously: https://news.ycombinator.com/item?id=20504734 https://news.ycombinator.com/item?id=20504734
- mjevans 6y ago
  One of the sub-parts of the previous discussion covered what I too thought was the recommended method. https://news.ycombinator.com/item?id=20509533 https://news.ycombinator.com/item?id=20509533 A temp file plus rename seems to be close but not fully enough. This is exactly the sort of thing that should be abstracted by a standard library so that whatever needs to be done is correctly done.
  - nh2 6y ago
    That is also the most important part to me. But I don't understand it. It says: > I was also (incorrectly) assuming that renaming a file after having fsynced the contents is a valid mechanism Where does anyone say that this doesn't work? I read the article twice and it suggests that open+write+fsync+close+dir-fsync is safe ("We've now seen what we have to do to write a file safely"). I've also read the reference "Bornholt et al., ASPLOS’16" mentioned in the "Rename" section; it says "One possible fix is to add an fsync(fd)". I cannot find another place in that paper that says otherwise. The later analysed Atomic-replace-via-rename (ARVR) and Atomic-create-via-rename (ACVR) approaches skip the fsync. That said, user "tfha" replied to your linked thread > I can confirm experimentally that the rename trick is not good enough. but unfortunately without any data on whether the directory was fsynced, or on which file system. So I'd like to ask again: Is there any evidence that open+write+fsync+close+dir-fsync does not work?
    - buckminster 6y ago
      It "works" in the sense that if nothing goes wrong you get the expected result. But you can omit the fsyncs and it still "works" to this standard. The article gives a concrete example of a failure mode: fsync can silently fail. If this happens then "open+write+fsync+close+dir+fsync" does not work.
josephg 6y ago
This situation is so terrible, and its embarrassing that despite decades of work, our industry doesn't seem to be able to solve this problem in any modern operating system. (Edit: Apparently except for windows vista!) It seems obvious to me that we need a transactional filesystem API with methods like: fbegin() fwrite() / rename / etc fcommit() Most of the write calls wouldn't return errors directly. Then the fcommit() call would atomically flush the locally buffered changes to the filesystem. If an error happened, the API should guarantee that the final state either contains all changes or none of them. The commit call should either block the thread until changes are safely stored on disk, or use a completion callback. This API probably couldn't span multiple filesystems, but thats fine in practice. And bonus points if the API supports snapshot reads within the transaction. This would dramatically simplify the code in every program that wants to safely interact with the filesystem. And it would remove the perennial, justified anxiety database engineers carry that they're doing something subtly wrong without even knowing. Because fsync() is so coarse, I suspect performance would improve with this API, too. (So long as the transactions are mostly small.) But why don't we have this API already? My theory is that kernel developers don't write enough userspace software to understand how bad the status quo is. And most people writing software in userland don't understand the kernel well enough to feel empowered to change it. Adding an API like this might also need changes in the filesystems. But wow, it would be a huge boon to humanity to have a filesystem API which isn't so braindead. How many more decades do we need to have our files corrupt by default when errors happen? I wince every time I'm playing a video game and it has that "saving - don't turn off your console!" animation. Or during OS updates. Data corruption happens because we implemented the wrong abstractions 50 years ago and somehow forgot that code is mutable.
- jfim 6y ago
  Interestingly enough, that feature already exists and is being removed from Windows. From the transactional NTFS Wikipedia page[0]: > Microsoft has deprecated TxF and stated that it may be removed in a future version of Windows [0] https://en.wikipedia.org/wiki/Transactional_NTFS https://en.wikipedia.org/wiki/Transactional_NTFS
  - josephg 6y ago
352356 6y ago
ITT: HN hipster programmers who've never read a manpage learn that all of the file manipulation code they've ever written was broken. There are too many people writing software, who have no business doing so.
- sabellito 6y ago
  Which man page should the hipster programmers read?
- BlueTemplar 6y ago
  Hipsters are people boasting that they were doing something long before it became 'hip' and 'cool' (and often have moved on to another in their opinion soon-to-be 'hip' and 'cool' thing). > There are too many people writing software, who have no business doing so. Sounds like something a hipster would say ?
- dang 6y ago
  Could you please stop posting unsubstantive comments to HN? We're trying for something better than that here. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and sticking to the rules when posting, we'd be grateful. Note that they include: Please don't sneer. Throwaway accounts are ok for sensitive information, but please don't create accounts routinely. Re the latter: could you please stop creating accounts for every few comments you post? We ban accounts that do that. You needn't use your real name, of course, but for HN to be a community, users need some identity for other users to relate to. Otherwise we may as well have no usernames and no community, and that would be a different kind of forum. https://hn.algolia.com/?sort=byDate&dateRange=all&type=comment&storyText=false&prefix&page=0&query=community%20identity%20by:dang https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...
tyingq 6y ago
I always wondered why Unix didn't come with a standard record based file in addition to the stream based file. Mainframes and other OSs, like VMS, for example, have both fixed width and variable width record based files. So, if you're writing a database, there's already a bullet proof and fast backend to store things that comes with the OS and is maintained. No need to evaluate various layers on top of a stream (RocksDB, LMDB, etc).
- rakoo 6y ago
  Because then everything is not a file, it's either a stream-based file or a record-based file. So you don't have a uniform API for everything. The UNIX way would be to have a filesystem that does all the internal logic of checking line width, and allow the user to mount it. One API, different functionality
  - tyingq 6y ago
    Well, we re-invent it anyway on top of stream files, so I'm not sure I get the difference other than fragmentation of solutions.
    - rakoo 6y ago
      Because in the UNIX way you'd have one process for providing record-based files, one process to write to/read from them, one process to parse the query language into an AST, one process to maintain indexes, etc... and all of this would be bound by a big jar of glue-shell. It's much easier to just do everything inside a single software as a monolith than to distribute functionalities, I don't think one needs to look further than that

103 points by bshanks 6y ago

I know this guy is some kind of hacker news celebrity, but that website is an abomination. It's undreadably ugly. Is it a thing now? "Look at me, I'm too cool to think on behalf of my users. They'll adjust it for themselves. They'll use reader mode. I'm 1337 and styling is for fucktards"?? It's like Norvig's website, only worse.

dxdm 6y ago
This actually looks fine to me on mobile. Is it bad on a large screen? What a nice surprise, usually it's the other way round, and then reader mode often also doesn't work.
- _0ffh 6y ago
  Large screen user here, site looks fine to me.
- netmare 6y ago
  It's a mofo™-styled website, with eye-piercing white backgrounds and mile long lines. It's as minimalist as it gets, unless you consider vowels superfluous in prose. Joking aside, yes, it looks bland on a PC. However, I would rather have more sites like this, where I don't have to remove 97% of fluff just to read the content.
  - BlueTemplar 6y ago
    > with eye-piercing white backgrounds I recommend the Dark Reader extension.
- snazz 6y ago
  The long lines significantly harm readability. Luckily, it's a very easy site for Reader View to parse (at least in Safari). I'm fine with this type of site, although a simple max-width would make it much easier to read without Reader View.

pvg 6y ago

Previously: https://news.ycombinator.com/item?id=20504734 https://news.ycombinator.com/item?id=20504734

mjevans 6y ago
One of the sub-parts of the previous discussion covered what I too thought was the recommended method. https://news.ycombinator.com/item?id=20509533 https://news.ycombinator.com/item?id=20509533 A temp file plus rename seems to be close but not fully enough. This is exactly the sort of thing that should be abstracted by a standard library so that whatever needs to be done is correctly done.
- nh2 6y ago
  That is also the most important part to me. But I don't understand it. It says: > I was also (incorrectly) assuming that renaming a file after having fsynced the contents is a valid mechanism Where does anyone say that this doesn't work? I read the article twice and it suggests that open+write+fsync+close+dir-fsync is safe ("We've now seen what we have to do to write a file safely"). I've also read the reference "Bornholt et al., ASPLOS’16" mentioned in the "Rename" section; it says "One possible fix is to add an fsync(fd)". I cannot find another place in that paper that says otherwise. The later analysed Atomic-replace-via-rename (ARVR) and Atomic-create-via-rename (ACVR) approaches skip the fsync. That said, user "tfha" replied to your linked thread > I can confirm experimentally that the rename trick is not good enough. but unfortunately without any data on whether the directory was fsynced, or on which file system. So I'd like to ask again: Is there any evidence that open+write+fsync+close+dir-fsync does not work?
  - buckminster 6y ago
    It "works" in the sense that if nothing goes wrong you get the expected result. But you can omit the fsyncs and it still "works" to this standard. The article gives a concrete example of a failure mode: fsync can silently fail. If this happens then "open+write+fsync+close+dir+fsync" does not work.

josephg 6y ago

This situation is so terrible, and its embarrassing that despite decades of work, our industry doesn't seem to be able to solve this problem in any modern operating system. (Edit: Apparently except for windows vista!) It seems obvious to me that we need a transactional filesystem API with methods like: fbegin() fwrite() / rename / etc fcommit() Most of the write calls wouldn't return errors directly. Then the fcommit() call would atomically flush the locally buffered changes to the filesystem. If an error happened, the API should guarantee that the final state either contains all changes or none of them. The commit call should either block the thread until changes are safely stored on disk, or use a completion callback. This API probably couldn't span multiple filesystems, but thats fine in practice. And bonus points if the API supports snapshot reads within the transaction. This would dramatically simplify the code in every program that wants to safely interact with the filesystem. And it would remove the perennial, justified anxiety database engineers carry that they're doing something subtly wrong without even knowing. Because fsync() is so coarse, I suspect performance would improve with this API, too. (So long as the transactions are mostly small.) But why don't we have this API already? My theory is that kernel developers don't write enough userspace software to understand how bad the status quo is. And most people writing software in userland don't understand the kernel well enough to feel empowered to change it. Adding an API like this might also need changes in the filesystems. But wow, it would be a huge boon to humanity to have a filesystem API which isn't so braindead. How many more decades do we need to have our files corrupt by default when errors happen? I wince every time I'm playing a video game and it has that "saving - don't turn off your console!" animation. Or during OS updates. Data corruption happens because we implemented the wrong abstractions 50 years ago and somehow forgot that code is mutable.

jfim 6y ago
Interestingly enough, that feature already exists and is being removed from Windows. From the transactional NTFS Wikipedia page[0]: > Microsoft has deprecated TxF and stated that it may be removed in a future version of Windows [0] https://en.wikipedia.org/wiki/Transactional_NTFS https://en.wikipedia.org/wiki/Transactional_NTFS
- josephg 6y ago

352356 6y ago

ITT: HN hipster programmers who've never read a manpage learn that all of the file manipulation code they've ever written was broken. There are too many people writing software, who have no business doing so.

sabellito 6y ago
Which man page should the hipster programmers read?
BlueTemplar 6y ago
Hipsters are people boasting that they were doing something long before it became 'hip' and 'cool' (and often have moved on to another in their opinion soon-to-be 'hip' and 'cool' thing). > There are too many people writing software, who have no business doing so. Sounds like something a hipster would say ?
dang 6y ago
Could you please stop posting unsubstantive comments to HN? We're trying for something better than that here. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and sticking to the rules when posting, we'd be grateful. Note that they include: Please don't sneer. Throwaway accounts are ok for sensitive information, but please don't create accounts routinely. Re the latter: could you please stop creating accounts for every few comments you post? We ban accounts that do that. You needn't use your real name, of course, but for HN to be a community, users need some identity for other users to relate to. Otherwise we may as well have no usernames and no community, and that would be a different kind of forum. https://hn.algolia.com/?sort=byDate&dateRange=all&type=comment&storyText=false&prefix&page=0&query=community%20identity%20by:dang https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...

tyingq 6y ago

I always wondered why Unix didn't come with a standard record based file in addition to the stream based file. Mainframes and other OSs, like VMS, for example, have both fixed width and variable width record based files. So, if you're writing a database, there's already a bullet proof and fast backend to store things that comes with the OS and is maintained. No need to evaluate various layers on top of a stream (RocksDB, LMDB, etc).

rakoo 6y ago
Because then everything is not a file, it's either a stream-based file or a record-based file. So you don't have a uniform API for everything. The UNIX way would be to have a filesystem that does all the internal logic of checking line width, and allow the user to mount it. One API, different functionality
- tyingq 6y ago
  Well, we re-invent it anyway on top of stream files, so I'm not sure I get the difference other than fragmentation of solutions.
  - rakoo 6y ago
    Because in the UNIX way you'd have one process for providing record-based files, one process to write to/read from them, one process to parse the query language into an AST, one process to maintain indexes, etc... and all of this would be bound by a big jar of glue-shell. It's much easier to just do everything inside a single software as a monolith than to distribute functionalities, I don't think one needs to look further than that