6 ms·

Achieving cutting-edge storage performance tends to require bypassing the filesystem anyways. Traditionally, that meant using SPDK. Nowadays, opening /dev/nvme*

Achieving cutting-edge storage performance tends to require bypassing the filesystem anyways. Traditionally, that meant using SPDK. Nowadays, opening /dev/nvme* with O_DIRECT and operating on it with io_uring will get you most of the way there. In either case, the advice given in the article and by the OP is filesystem agnostic.

quotemstr 5y ago
Why would you want to bypass the filesystem by talking to the block device directly? Doesn't O_DIRECT on a preallocated regular file accomplish the same thing with less management complexity and special OS permissions? Granted, the file extents might be fragmented a bit, but that can be fixed.
- lazide 5y ago
  If you understand your workload and the hardware well enough to understand how doing direct I/O on a file will help - then you’re going to generally do better against a direct block device because there are fewer intermediate layers doing the wrong optimizations or otherwise messing you up. From a pure performance perspective anyway. Extents are one part of the issue, flushes to disk (and how/when they happen), caching, etc. Doesn’t mean it isn’t easier to deal with as a file from an administration perspective (and you can do snapshots, or whatever!), but Lvm can do that too for a block device, and many other things.
  - quotemstr 5y ago
    With O_DIRECT though you're opting out of the filesystem's caching (well, VFS's), forced flushes, and most FS level optimizations, so I'd expect it to perform on par with direct partition access. Do you have numbers showing an advantage of going directly to the block device? Personally, I'd consider the management advantages of a filesystem compelling absent specific performance numbers showing the benefit of direct partition access.
    - lazide 5y ago
      You do when it does that/respects it which isn’t always. The point is that you have more layers. If you’re trying to be as direct as possible, more layers is unhelpful. Since you get most of the same advantages management wise with lvm while using the block interface (including snapshots, resizing, and all the other management goodies), you’re not exactly getting much extra functionality either.
      - quotemstr 5y ago
        Your concerns are all theoretical and the management disadvantages of direct partition access are real with or without LVM (which itself is exactly the sort of middle layer you claim to be worried about.) Do you have numbers or not?
        lazide 5y ago
        Ah, but now you’re moving the goalposts it seems? Since most of what we’re talking about is unnecessary complexity for no real gain, what concrete metric do you think would be useful exactly? I just pointed out that you can get the same management advantages without it (say for a dev environment or rollbacks or whatever). And you get a simpler, cleaner story without extra layers if you don’t want to use lvm (such as in production), which you can’t get from O_DIRECT. I also have this thread from Linus calling O_DIRECT brain damaged and to never use it. [https://lkml.org/lkml/2007/1/10/233 https://lkml.org/lkml/2007/1/10/233]
        josefx 5y ago
        The problem is some of the alternatives seem to be suggested by way of "if we had any support for this it would be better than O_DIRECT". So don't use O_DIRECT, use the alternative which doesn't exist, is still too slow, only covers parts of what you need, etc. .
        quotemstr 5y ago
        I respect Linus, but he has a problem where he never ever backtracks and admits he was wrong about something. Take C++ for example.
- 10000truths 5y ago
  A "regular file" might reside in multiple locations on disk for redundancy, or might have a checksum that needs to be maintained alongside it for integrity. Or, as you say, its contents might not reside in contiguous sectors - or you might be writing to a hole in a sparse file. There's a lot of "magic" that could go on behind the scenes when operating on "regular files", depending on what filesystem you're using with what options. Directly operating on the block device makes it easier to reason about the performance guarantees, since your reads and writes map more cleanly to the underlying SCSI/ATA/NVME commands issued.
nyanpasu64 5y ago
Will an end user downloading a video editing app (or similar) have a NVME drive, know how to give your app direct access to a NVME drive, and will your app not corrupt the rest of the files on the drive?
- 10000truths 5y ago
  Extreme performance requires extreme tradeoffs. As with anything else, you have to evaluate your use cases and determine for yourself whether the tradeoffs are worth it. For a mass-market application that has to play nice with other applications and work with a wide variety of commodity hardware, it's probably not worthwhile. For a state-of-the-art high performance data store that expects low latencies and high throughput (à la ScyllaDB), it may very well be.
  - naikrovek 5y ago
    at that point just use a RAM disk and periodically write that data to physical disk or SSD. no extreme tradeoff required, because RAM disks are WAY faster than SSDs. manhandling /dev/nvme0 seems equally likely to corrupt data in the event of a power failure.
    - wtallis 5y ago
      > manhandling /dev/nvme0 seems equally likely to corrupt data in the event of a power failure. If we make the reasonable assumption that this subthread is discussing a server use case, then we can assume that the SSD is tolerant of power failures and has the capacitors necessary to finish any cached writes it has reported as complete. Thus, having fewer layers between the hardware and the application means there are fewer opportunities for some layer to lie to those above it about whether the data has made it to persistent storage. Whether or not you're bypassing large parts of the operating system's IO stack, the application needs to have a clear idea of what data needs to be flushed to persistent storage at what times in order to properly survive unexpected power loss without unnecessary data loss or corruption.
    - 10000truths 5y ago
Unklejoe 5y ago
I'm wondering if it's really necessary to get at the block device directly. I'm able to saturate a PCIe 3.0 x4 link doing direct IO to an NVMe drive with a single 1.7 GHz Power PC core without breaking a sweat. This is through ext4. My accesses are sequential though. Maybe there's more of a penalty with random IO.

quotemstr 5y ago

Why would you want to bypass the filesystem by talking to the block device directly? Doesn't O_DIRECT on a preallocated regular file accomplish the same thing with less management complexity and special OS permissions? Granted, the file extents might be fragmented a bit, but that can be fixed.

lazide 5y ago
If you understand your workload and the hardware well enough to understand how doing direct I/O on a file will help - then you’re going to generally do better against a direct block device because there are fewer intermediate layers doing the wrong optimizations or otherwise messing you up. From a pure performance perspective anyway. Extents are one part of the issue, flushes to disk (and how/when they happen), caching, etc. Doesn’t mean it isn’t easier to deal with as a file from an administration perspective (and you can do snapshots, or whatever!), but Lvm can do that too for a block device, and many other things.
- quotemstr 5y ago
  With O_DIRECT though you're opting out of the filesystem's caching (well, VFS's), forced flushes, and most FS level optimizations, so I'd expect it to perform on par with direct partition access. Do you have numbers showing an advantage of going directly to the block device? Personally, I'd consider the management advantages of a filesystem compelling absent specific performance numbers showing the benefit of direct partition access.
  - lazide 5y ago
    You do when it does that/respects it which isn’t always. The point is that you have more layers. If you’re trying to be as direct as possible, more layers is unhelpful. Since you get most of the same advantages management wise with lvm while using the block interface (including snapshots, resizing, and all the other management goodies), you’re not exactly getting much extra functionality either.
    - quotemstr 5y ago
      Your concerns are all theoretical and the management disadvantages of direct partition access are real with or without LVM (which itself is exactly the sort of middle layer you claim to be worried about.) Do you have numbers or not?
      - lazide 5y ago
        Ah, but now you’re moving the goalposts it seems? Since most of what we’re talking about is unnecessary complexity for no real gain, what concrete metric do you think would be useful exactly? I just pointed out that you can get the same management advantages without it (say for a dev environment or rollbacks or whatever). And you get a simpler, cleaner story without extra layers if you don’t want to use lvm (such as in production), which you can’t get from O_DIRECT. I also have this thread from Linus calling O_DIRECT brain damaged and to never use it. [https://lkml.org/lkml/2007/1/10/233 https://lkml.org/lkml/2007/1/10/233]
        josefx 5y ago
        The problem is some of the alternatives seem to be suggested by way of "if we had any support for this it would be better than O_DIRECT". So don't use O_DIRECT, use the alternative which doesn't exist, is still too slow, only covers parts of what you need, etc. .
        quotemstr 5y ago
        I respect Linus, but he has a problem where he never ever backtracks and admits he was wrong about something. Take C++ for example.
10000truths 5y ago
A "regular file" might reside in multiple locations on disk for redundancy, or might have a checksum that needs to be maintained alongside it for integrity. Or, as you say, its contents might not reside in contiguous sectors - or you might be writing to a hole in a sparse file. There's a lot of "magic" that could go on behind the scenes when operating on "regular files", depending on what filesystem you're using with what options. Directly operating on the block device makes it easier to reason about the performance guarantees, since your reads and writes map more cleanly to the underlying SCSI/ATA/NVME commands issued.

nyanpasu64 5y ago

Will an end user downloading a video editing app (or similar) have a NVME drive, know how to give your app direct access to a NVME drive, and will your app not corrupt the rest of the files on the drive?

10000truths 5y ago
Extreme performance requires extreme tradeoffs. As with anything else, you have to evaluate your use cases and determine for yourself whether the tradeoffs are worth it. For a mass-market application that has to play nice with other applications and work with a wide variety of commodity hardware, it's probably not worthwhile. For a state-of-the-art high performance data store that expects low latencies and high throughput (à la ScyllaDB), it may very well be.
- naikrovek 5y ago
  at that point just use a RAM disk and periodically write that data to physical disk or SSD. no extreme tradeoff required, because RAM disks are WAY faster than SSDs. manhandling /dev/nvme0 seems equally likely to corrupt data in the event of a power failure.
  - wtallis 5y ago
    > manhandling /dev/nvme0 seems equally likely to corrupt data in the event of a power failure. If we make the reasonable assumption that this subthread is discussing a server use case, then we can assume that the SSD is tolerant of power failures and has the capacitors necessary to finish any cached writes it has reported as complete. Thus, having fewer layers between the hardware and the application means there are fewer opportunities for some layer to lie to those above it about whether the data has made it to persistent storage. Whether or not you're bypassing large parts of the operating system's IO stack, the application needs to have a clear idea of what data needs to be flushed to persistent storage at what times in order to properly survive unexpected power loss without unnecessary data loss or corruption.
  - 10000truths 5y ago

Unklejoe 5y ago

I'm wondering if it's really necessary to get at the block device directly. I'm able to saturate a PCIe 3.0 x4 link doing direct IO to an NVMe drive with a single 1.7 GHz Power PC core without breaking a sweat. This is through ext4. My accesses are sequential though. Maybe there's more of a penalty with random IO.