3 ms·

Files work fine and there are abstraction layers on top of the file system that handle more of it.

josephg 4y ago
They don't work fine for everyone. I'm building a custom storage engine for a CRDT at the moment and I don't want my data corrupted under any circumstances. Writing code to do this reliably is painful. Even figuring out the rules is difficult - is fsync enough? fdatasync? Will it still corrupt on power loss events in modern consumer SSDs? How do I test / simulate my software under these failure conditions? If I get an error when I fsync, what assumptions can I make about the state of my data? Some atomicity rules seem to be page-size specific, but I can't find any way to query the OS for the kernel's page size, or the page size of the underlying device. The status quo is horrid.
- mangamadaiyan 4y ago
  What operating system are you on?
  - josephg 4y ago
    I'm developing on linux, but my code should work on any modern platform. (Windows, MacOS, iOS, Android, Linux). On Apple's hardware, fsync (the "no really" version) actually does durably flush data to disk when you ask it to. And darwin has a write barrier command, which is great because I don't need to fsync at all in order to append to my transaction log. But linux is missing a barrier command, and I've heard most consumer grade SSDs lie when you ask them to fsync. Generally, linux doesn't really seems designed to preserve your data when the system crashes or experiences a power loss. The single API it gives you to help (fsync) is badly designed, coarse grained, slow, badly documented and its hard to use correctly. And worst of all, it often doesn't work anyway thanks to crap SSDs. There's lots of better APIs out there. IO completion ports on windows are a better design. Likewise, adding write barriers (like macos has done) would help a lot. The filesystem APIs on linux are long in need of being overhauled. Its sorely lacking if you want your software to be reliable.
    - mangamadaiyan 4y ago
      Fair enough :) There's the problem of badly designed APIs, and there's the problem of crap hardware that lies to you. IME, it's difficult to ensure that data "written" to disk has actually reached the disk, without disabling the drive's cache. This was true even with spinning rust, and continues to be true with SSDs. I'd rather not get started on read-after-write, with inline checksums in order to verify that your disk isn't lying to you. In an ideal world, we'd have both well-designed APIs as well as hardware that isn't crap. If I had to choose between the two, I'd go with good hardware, because the API problem can at least be worked around - albeit with a significant investment in time, effort and pain. I feel your pain, and wish you good luck in your efforts.
      - josephg 4y ago
        Thanks! But as for the two problems (bad APIs and bad firmware on SSDs), we should do both! Personally I don't have influence over my solid state devices, but linux is opensource. We should absolutely fix linux to have better APIs here, even if the underlying devices lie sometimes. It might even help with the firmware. The reason for SSDs to lie about flush commands is so they look better in benchmarks. If linux had a better API (with barriers, write groups, transactions, whatever), then benchmarks would probably depend less on fsync. Maybe, eventually, we can solve that problem too.
- hyc_symas 4y ago
  kernel pagesize is returned by sysconf(_SC_PAGESIZE)
eru 4y ago
It's the opposite, you want less layers of abstractions. At least you want your OS to give you less abstractions, so you can choose your own. See 'exokernels' for that: their idea was that the OS should concentrate on safely multiplexing hardware, and leave abstracting to user-space libraries. Do one thing, and do it well; instead of serving multiple masters badly.

They don't work fine for everyone. I'm building a custom storage engine for a CRDT at the moment and I don't want my data corrupted under any circumstances. Writing code to do this reliably is painful. Even figuring out the rules is difficult - is fsync enough? fdatasync? Will it still corrupt on power loss events in modern consumer SSDs? How do I test / simulate my software under these failure conditions? If I get an error when I fsync, what assumptions can I make about the state of my data? Some atomicity rules seem to be page-size specific, but I can't find any way to query the OS for the kernel's page size, or the page size of the underlying device. The status quo is horrid.

mangamadaiyan 4y ago
What operating system are you on?
- josephg 4y ago
  I'm developing on linux, but my code should work on any modern platform. (Windows, MacOS, iOS, Android, Linux). On Apple's hardware, fsync (the "no really" version) actually does durably flush data to disk when you ask it to. And darwin has a write barrier command, which is great because I don't need to fsync at all in order to append to my transaction log. But linux is missing a barrier command, and I've heard most consumer grade SSDs lie when you ask them to fsync. Generally, linux doesn't really seems designed to preserve your data when the system crashes or experiences a power loss. The single API it gives you to help (fsync) is badly designed, coarse grained, slow, badly documented and its hard to use correctly. And worst of all, it often doesn't work anyway thanks to crap SSDs. There's lots of better APIs out there. IO completion ports on windows are a better design. Likewise, adding write barriers (like macos has done) would help a lot. The filesystem APIs on linux are long in need of being overhauled. Its sorely lacking if you want your software to be reliable.
  - mangamadaiyan 4y ago
    Fair enough :) There's the problem of badly designed APIs, and there's the problem of crap hardware that lies to you. IME, it's difficult to ensure that data "written" to disk has actually reached the disk, without disabling the drive's cache. This was true even with spinning rust, and continues to be true with SSDs. I'd rather not get started on read-after-write, with inline checksums in order to verify that your disk isn't lying to you. In an ideal world, we'd have both well-designed APIs as well as hardware that isn't crap. If I had to choose between the two, I'd go with good hardware, because the API problem can at least be worked around - albeit with a significant investment in time, effort and pain. I feel your pain, and wish you good luck in your efforts.
    - josephg 4y ago
      Thanks! But as for the two problems (bad APIs and bad firmware on SSDs), we should do both! Personally I don't have influence over my solid state devices, but linux is opensource. We should absolutely fix linux to have better APIs here, even if the underlying devices lie sometimes. It might even help with the firmware. The reason for SSDs to lie about flush commands is so they look better in benchmarks. If linux had a better API (with barriers, write groups, transactions, whatever), then benchmarks would probably depend less on fsync. Maybe, eventually, we can solve that problem too.
hyc_symas 4y ago
kernel pagesize is returned by sysconf(_SC_PAGESIZE)

eru 4y ago

It's the opposite, you want less layers of abstractions. At least you want your OS to give you less abstractions, so you can choose your own. See 'exokernels' for that: their idea was that the OS should concentrate on safely multiplexing hardware, and leave abstracting to user-space libraries. Do one thing, and do it well; instead of serving multiple masters badly.