4 ms·

Fsync() on a different thread: apparently a useless trick

55 points by antirez 17y ago

azim 17y ago
One thing the author of this post is doing here is calling gettimeofday() in every iteration through the loops. That's an awful way to benchmark something. gettimeofday() issues a serializing instruction, which can potentially flush the entire pipeline. It can also cause cores to sync their clocks. Either way, the net result of calling gettimeofday() too often is awful multithreaded performance. Calling it much less often or using a profiler would give very different results than reported.
russell_h 17y ago
Is there a reason to call fsync() all the time instead of using O_SYNC?
- antirez 17y ago
  This is what I'm considering for "fsync always", but for "fsync everysec" this is now ideal unfortunately. Still O_SYNC can be able to fix at least one problem, that's cool :) Thanks for the comment. Edit: worth noting that instead O_DIRECT can't be used for an append only file, because of alignment requirements. Edit 2: yep O_SYNC helps a lot. Just updated the post with the results.
  - russell_h 17y ago
    Ah, yeah, thats what I meant, I probably should have specified that.
- btmorex 17y ago
  Correct me if I'm wrong, but O_SYNC is the equivalent of calling fsync() after every single write, so even if you manage to omit one fsync() you're going to come out ahead.
  - antirez 17y ago
    Semantically it's the same, but maybe it is somewhat optimized? Testing right now... Edit: indeed it's different, this is the result: ... Write in 219 microseconds Write in 253 microseconds Write in 264 microseconds Write in 271 microseconds Write in 246 microseconds ... 250 us instead of 13, but no big stops. So much better indeed... Even summing the times, every 10 writes this take 2500 us compared to 40000 us of fsync(). Definitely an improvement. p.s. all this, in ext4 FS.
    - pquerna 17y ago
      its not the same on filesystems like ext3 with ordered data mode (the default). fsync on those kind of filesystems will flush everything, not just a single file, though I believe there has been some more recent work in 2.6 to improve this behavior.
      - jedbrown 17y ago
        > recent work in 2.6 So, since 2003?
deleted 17y ago
[deleted]
- antirez 17y ago
  Hello, I already use fdatasync() in the actual Redis source code. Unfortunately it has the same problem about blocking write() but still most of the times it's much faster.
pquerna 17y ago
I guess I'm not gorking how the behavior of write() blocking when fsync is being called is considered a problem. write() is documented and well known to be a potentially blocking function, for potentially a long time. If you are writing a small-number-of-threads process, that is doing Disk and Network IO in the same threads, you can definitely starve out the network IO when the disk system starts thrashing, but generally that is fine, because you can't keep up with new requests at that point anyways. For that matter, read() can block too, people are just 'used' to their Kernels being smart and most important data being in cache. If you want your event-style threads servicing network clients to be able to use write() or read without 'blocking' up a whole thread for apparently random times, consider using the aio_* style write/fsync, although you definitely would want to benchmark their impacts on general performance. If you don't want to use aio_*, which is likely to be.. dangerous ground considering how few people use them, you end up adding threads to provide more isolation and parallelism to IO.
- antirez 17y ago
  Hello. Write in non real time Linux is not able to guarantee that it will return in a given amount of time, still all the system calls have more or less predictable timing behavior when the disk and the CPU are not busy. What I mean is that if you remove the fsync() call from the other thread, what you get is a constant stream of "13 microseconds" delays. So when fsync() is not into the mix the kernel will do the right thing, will use buffers and will make the write calls very cheap. This is important for many applications. But when there are more strict durability requirements this is no longer true and care must be used. Non blocking I/O (aio_*) is an interesting alternative in some application, but in the case of Redis it is important to return "OK" only after we got an acknowledge from write(2). Doing this suspending the client and resuming it when the write was performed will turn Redis from a 140k operations/second database into a 10k operations second database, so this is not going to be the solution. Real world software is written not reading manual pages, but checking how the underlying OS actually works IMHO. For instance Redis persistence uses fork() copy-on-write semantic of modern operation system. Also the fact that write(2) can block per semantic, does not mean you'll be happy to know your kernel is blocking a process for seconds many times as a result of a write(2) operation.
geocar 17y ago
This has nothing to do with fsync(). If you lseek() back to the beginning after each write(), this behavior should never occur. Growing a file is complicated, and may require extra unexpected disk accesses while the directory entry is modified or the free blocks are reassigned. These are what are blocking the write(), and write() will do them eventually, on a busy enough system, whether you call fsync() or not.
rbranson 17y ago
Have you considered timing the average wait time for fsync() and continuously adjusting a timer to fsync() every N milliseconds based on this data? You could queue all of your response-to-client messages to wait on this fsync(), and do tens of these per second, even on a slow drive. This technique can potentially get client responses down to fractions of a second, even in high throughput scenarios.
houseabsolute 17y ago
Seems like a userspace write queue would solve the write-delay problem. On the other hand, it's very dangerous to report success to the user before the changes actually hit the disk, because they might make some other change to the world that depends on the contents of that file being on disk, leading to an inconsistent state.
alecco 17y ago
Using a complex generic filesystem with too much metadata is a bit part of the problem. If your app is write oriented try a logfs-style filesystem. Or perhaps it's time for a new minimal filesystem. Direct I/O is another option but too crass, IMHO.

55 points by antirez 17y ago

azim 17y ago

One thing the author of this post is doing here is calling gettimeofday() in every iteration through the loops. That's an awful way to benchmark something. gettimeofday() issues a serializing instruction, which can potentially flush the entire pipeline. It can also cause cores to sync their clocks. Either way, the net result of calling gettimeofday() too often is awful multithreaded performance. Calling it much less often or using a profiler would give very different results than reported.

russell_h 17y ago

Is there a reason to call fsync() all the time instead of using O_SYNC?

antirez 17y ago
This is what I'm considering for "fsync always", but for "fsync everysec" this is now ideal unfortunately. Still O_SYNC can be able to fix at least one problem, that's cool :) Thanks for the comment. Edit: worth noting that instead O_DIRECT can't be used for an append only file, because of alignment requirements. Edit 2: yep O_SYNC helps a lot. Just updated the post with the results.
- russell_h 17y ago
  Ah, yeah, thats what I meant, I probably should have specified that.
btmorex 17y ago
Correct me if I'm wrong, but O_SYNC is the equivalent of calling fsync() after every single write, so even if you manage to omit one fsync() you're going to come out ahead.
- antirez 17y ago
  Semantically it's the same, but maybe it is somewhat optimized? Testing right now... Edit: indeed it's different, this is the result: ... Write in 219 microseconds Write in 253 microseconds Write in 264 microseconds Write in 271 microseconds Write in 246 microseconds ... 250 us instead of 13, but no big stops. So much better indeed... Even summing the times, every 10 writes this take 2500 us compared to 40000 us of fsync(). Definitely an improvement. p.s. all this, in ext4 FS.
  - pquerna 17y ago
    its not the same on filesystems like ext3 with ordered data mode (the default). fsync on those kind of filesystems will flush everything, not just a single file, though I believe there has been some more recent work in 2.6 to improve this behavior.
    - jedbrown 17y ago
      > recent work in 2.6 So, since 2003?

deleted 17y ago

[deleted]

antirez 17y ago
Hello, I already use fdatasync() in the actual Redis source code. Unfortunately it has the same problem about blocking write() but still most of the times it's much faster.

pquerna 17y ago

I guess I'm not gorking how the behavior of write() blocking when fsync is being called is considered a problem. write() is documented and well known to be a potentially blocking function, for potentially a long time. If you are writing a small-number-of-threads process, that is doing Disk and Network IO in the same threads, you can definitely starve out the network IO when the disk system starts thrashing, but generally that is fine, because you can't keep up with new requests at that point anyways. For that matter, read() can block too, people are just 'used' to their Kernels being smart and most important data being in cache. If you want your event-style threads servicing network clients to be able to use write() or read without 'blocking' up a whole thread for apparently random times, consider using the aio_* style write/fsync, although you definitely would want to benchmark their impacts on general performance. If you don't want to use aio_*, which is likely to be.. dangerous ground considering how few people use them, you end up adding threads to provide more isolation and parallelism to IO.

antirez 17y ago
Hello. Write in non real time Linux is not able to guarantee that it will return in a given amount of time, still all the system calls have more or less predictable timing behavior when the disk and the CPU are not busy. What I mean is that if you remove the fsync() call from the other thread, what you get is a constant stream of "13 microseconds" delays. So when fsync() is not into the mix the kernel will do the right thing, will use buffers and will make the write calls very cheap. This is important for many applications. But when there are more strict durability requirements this is no longer true and care must be used. Non blocking I/O (aio_*) is an interesting alternative in some application, but in the case of Redis it is important to return "OK" only after we got an acknowledge from write(2). Doing this suspending the client and resuming it when the write was performed will turn Redis from a 140k operations/second database into a 10k operations second database, so this is not going to be the solution. Real world software is written not reading manual pages, but checking how the underlying OS actually works IMHO. For instance Redis persistence uses fork() copy-on-write semantic of modern operation system. Also the fact that write(2) can block per semantic, does not mean you'll be happy to know your kernel is blocking a process for seconds many times as a result of a write(2) operation.

geocar 17y ago

This has nothing to do with fsync(). If you lseek() back to the beginning after each write(), this behavior should never occur. Growing a file is complicated, and may require extra unexpected disk accesses while the directory entry is modified or the free blocks are reassigned. These are what are blocking the write(), and write() will do them eventually, on a busy enough system, whether you call fsync() or not.

rbranson 17y ago

Have you considered timing the average wait time for fsync() and continuously adjusting a timer to fsync() every N milliseconds based on this data? You could queue all of your response-to-client messages to wait on this fsync(), and do tens of these per second, even on a slow drive. This technique can potentially get client responses down to fractions of a second, even in high throughput scenarios.

houseabsolute 17y ago

Seems like a userspace write queue would solve the write-delay problem. On the other hand, it's very dangerous to report success to the user before the changes actually hit the disk, because they might make some other change to the world that depends on the contents of that file being on disk, leading to an inconsistent state.

alecco 17y ago

Using a complex generic filesystem with too much metadata is a bit part of the problem. If your app is write oriented try a logfs-style filesystem. Or perhaps it's time for a new minimal filesystem. Direct I/O is another option but too crass, IMHO.