4 ms·

I guess I'm not gorking how the behavior of write() blocking when fsync is being called is considered a problem. write() is documented and well known to be a p

by pquerna 16y ago

I guess I'm not gorking how the behavior of write() blocking when fsync is being called is considered a problem. write() is documented and well known to be a potentially blocking function, for potentially a long time. If you are writing a small-number-of-threads process, that is doing Disk and Network IO in the same threads, you can definitely starve out the network IO when the disk system starts thrashing, but generally that is fine, because you can't keep up with new requests at that point anyways. For that matter, read() can block too, people are just 'used' to their Kernels being smart and most important data being in cache. If you want your event-style threads servicing network clients to be able to use write() or read without 'blocking' up a whole thread for apparently random times, consider using the aio_* style write/fsync, although you definitely would want to benchmark their impacts on general performance. If you don't want to use aio_*, which is likely to be.. dangerous ground considering how few people use them, you end up adding threads to provide more isolation and parallelism to IO.

antirez 16y ago
Hello. Write in non real time Linux is not able to guarantee that it will return in a given amount of time, still all the system calls have more or less predictable timing behavior when the disk and the CPU are not busy. What I mean is that if you remove the fsync() call from the other thread, what you get is a constant stream of "13 microseconds" delays. So when fsync() is not into the mix the kernel will do the right thing, will use buffers and will make the write calls very cheap. This is important for many applications. But when there are more strict durability requirements this is no longer true and care must be used. Non blocking I/O (aio_*) is an interesting alternative in some application, but in the case of Redis it is important to return "OK" only after we got an acknowledge from write(2). Doing this suspending the client and resuming it when the write was performed will turn Redis from a 140k operations/second database into a 10k operations second database, so this is not going to be the solution. Real world software is written not reading manual pages, but checking how the underlying OS actually works IMHO. For instance Redis persistence uses fork() copy-on-write semantic of modern operation system. Also the fact that write(2) can block per semantic, does not mean you'll be happy to know your kernel is blocking a process for seconds many times as a result of a write(2) operation.
- xpaulbettsx 16y ago
  Kernel developer here - honestly, I'm not surprised that fsync() stops the world, and furthermore I suspect that even if you got past the vfs layer, you would see different effects on different filesystems (i.e. you still couldn't bet that fsync() would act like you want it to). The semantics of fsync mean "Please guarantee that everything is written to disk, flush all caches now". The kernel doesn't keep a 2nd queue for post-fsync writes that it will then swap into the "real" one - think about what happens if someone else calls fsync(); does it spin up a 3rd queue for that one? Does the fsync block? I think it would quickly descend into Crazyville.
  - antirez 16y ago
    Hello xpaulbettsx, thanks for your comment! Yes I guess the implementation may get more complex, not sure about the actual implementation it's just a linked list of operations to flush like it happens looking at a few source code fragments, then it's just possible to put a "sentinel" in the list that will block the first fsync() when the first sentinel is found and so forth. I mean, I'm all against the complexity myself in the code I write, so I can't really question this behavior and the "fsync every second" policy is not a huge use case indeed, but still it's important to now that. Googling a bit around there are tons of people that appear to be pretty confident that moving fsync() into another thread is the way to go, while instead stuff are working in a different way.
  - houseabsolute 16y ago
    A pair of queues should be enough. The first queue is the one being fsynced, the second queue is open for writes. The first fsync call starts the first queue flushing. The second fsync call marks the second queue as read to flush once the first one is done. The third has no additional effect. Writes can still proceed against the second until it actually starts flushing. At which point the queues swap and you're back where you started.
  - doty 16y ago
    Isn't this just good old group commit? I'm surprised the kernel wouldn't do this optimization for you.
gaius 16y ago
If you don't want to use aio_, which is likely to be.. dangerous ground considering how few people use them* Oracle uses AIO (you can strace it and see for yourself) and if it's safe for them, it's probably safe for you.

by pquerna 16y ago

antirez 16y ago

Hello. Write in non real time Linux is not able to guarantee that it will return in a given amount of time, still all the system calls have more or less predictable timing behavior when the disk and the CPU are not busy. What I mean is that if you remove the fsync() call from the other thread, what you get is a constant stream of "13 microseconds" delays. So when fsync() is not into the mix the kernel will do the right thing, will use buffers and will make the write calls very cheap. This is important for many applications. But when there are more strict durability requirements this is no longer true and care must be used. Non blocking I/O (aio_*) is an interesting alternative in some application, but in the case of Redis it is important to return "OK" only after we got an acknowledge from write(2). Doing this suspending the client and resuming it when the write was performed will turn Redis from a 140k operations/second database into a 10k operations second database, so this is not going to be the solution. Real world software is written not reading manual pages, but checking how the underlying OS actually works IMHO. For instance Redis persistence uses fork() copy-on-write semantic of modern operation system. Also the fact that write(2) can block per semantic, does not mean you'll be happy to know your kernel is blocking a process for seconds many times as a result of a write(2) operation.

xpaulbettsx 16y ago
Kernel developer here - honestly, I'm not surprised that fsync() stops the world, and furthermore I suspect that even if you got past the vfs layer, you would see different effects on different filesystems (i.e. you still couldn't bet that fsync() would act like you want it to). The semantics of fsync mean "Please guarantee that everything is written to disk, flush all caches now". The kernel doesn't keep a 2nd queue for post-fsync writes that it will then swap into the "real" one - think about what happens if someone else calls fsync(); does it spin up a 3rd queue for that one? Does the fsync block? I think it would quickly descend into Crazyville.
- antirez 16y ago
  Hello xpaulbettsx, thanks for your comment! Yes I guess the implementation may get more complex, not sure about the actual implementation it's just a linked list of operations to flush like it happens looking at a few source code fragments, then it's just possible to put a "sentinel" in the list that will block the first fsync() when the first sentinel is found and so forth. I mean, I'm all against the complexity myself in the code I write, so I can't really question this behavior and the "fsync every second" policy is not a huge use case indeed, but still it's important to now that. Googling a bit around there are tons of people that appear to be pretty confident that moving fsync() into another thread is the way to go, while instead stuff are working in a different way.
- houseabsolute 16y ago
  A pair of queues should be enough. The first queue is the one being fsynced, the second queue is open for writes. The first fsync call starts the first queue flushing. The second fsync call marks the second queue as read to flush once the first one is done. The third has no additional effect. Writes can still proceed against the second until it actually starts flushing. At which point the queues swap and you're back where you started.
- doty 16y ago
  Isn't this just good old group commit? I'm surprised the kernel wouldn't do this optimization for you.

gaius 16y ago

If you don't want to use aio_, which is likely to be.. dangerous ground considering how few people use them* Oracle uses AIO (you can strace it and see for yourself) and if it's safe for them, it's probably safe for you.