5 ms·

> Both Chinner and Ts'o, along with others, said that the proper solution is for PostgreSQL to move to direct I/O (DIO) instead. Wait, is "direct I/O" the same

> Both Chinner and Ts'o, along with others, said that the proper solution is for PostgreSQL to move to direct I/O (DIO) instead. Wait, is "direct I/O" the same as O_DIRECT? The same O_DIRECT that Linus skewered in 2007? > There really is no valid reason for EVER using O_DIRECT. You need a buffer whatever IO you do, and it might as well be the page cache. There are better ways to control the page cache than play games and think that a page cache isn't necessary. https://lkml.org/lkml/2007/1/10/233 https://lkml.org/lkml/2007/1/10/233 > Side note: the only reason O_DIRECT exists is because database people are too used to it, because other OS's haven't had enough taste to tell them to do it right, so they've historically hacked their OS to get out of the way. https://lkml.org/lkml/2007/1/10/235 https://lkml.org/lkml/2007/1/10/235 More background from 2002-2007: https://yarchive.net/comp/linux/o_direct.html https://yarchive.net/comp/linux/o_direct.html

anarazel 8y ago
Turns out sometimes people other than Linus have more experience with IO than Linus. I think there's pretty good reasons to go for DIO for a database. But only when there's a good sysadmin/DBA and when the system is dedicated to the database. There's considerable performance gains in going for DIO (at the cost of significant software complexity), but it's much more sensitive to bad tuning and isn't at all adaptive to overall system demands.
- pgaddict 8y ago
  Yeah, which is one of the reasons PostgreSQL went with buffered I/O, not to have to deal with this complexity. And it served us pretty well over time, I think.
  - anarazel 8y ago
    I don't think that's really true. It worked well enough, true, but I think it allowed us to not fix deficiencies in a number of areas that we should just have fixed. IOW, I think we survived despite not offering DIO (because other things are good), rather than because of it.
    - pgaddict 8y ago
      I don't think we disagree, actually. Yes - from a purely technical point of view, DIO is superior in various ways. It allows tuning to specific I/O patterns, etc. But it's also quite laborious to get right - not only does it require a fair amount of new code, but AFAIK there is significant variability between platforms and storage devices. I'm not sure the project had enough developer bandwidth back then, or differently - it was more efficient to spend the developer time on other stuff, with better cost/benefit ratio.
- haberman 8y ago
  Yes, more than anything I'm amused by the fact that when you do the Linus-approved thing in 2007 it leaves you in this terrible situation in 2019, and when the other kernel experts rub their heads together their solution is to abandon the gentle advice from 10 years earlier.
icebraining 8y ago
> Wait, is "direct I/O" the same as O_DIRECT? No, there are other ways of doing DIO besides that particular interface.
- guenthert 8y ago
  Care to elaborate? And won't use those different interfaces the same direct i/o implementation, much like e.g. raw devices? I take above question as rhetorical and would think the discussion above as well as Linus' rant apply to those too.
Unklejoe 8y ago
I have to say he’s wrong when he says there’s no reason for EVER using O_DIRECT. One HUGE reason is performance for large sequential writes when using fast media and slow processors. Specifically, when the write speed of the media is in the same ballpark as the effective memcpy() speed of the processor itself, which, believe it or not, is very possible today (but was probably more unlikely in 2007) when working with embedded systems and modern NVMe media. Consider a system with an SoC like the NXP T2080 that has a high speed internal memory bus, DDR3, and PCIe 3.0 support. The processor cores themselves are relatively slow PowerPC cores, but the chip has a ton of memory bandwidth. Now assume you had a contiguous 512 MiB chunk of data to write to an NVMe drive capable of a sustained write rate of 2000 MB/s. The processor core itself can barely move 2000 MB/s of data, so it’s clear why direct IO would perform better since you’re telling the drive to pull the data directly from the buffer instead of memcpy-ing into into an intermediate kernel buffer first. With direct IO, you can perform zero-copy writes to the media. This is why I’m able to achieve higher sequential write performance on some modern MVMe drives than most benchmark sites report, all while using a 1.2 GHz PowerPC system.

anarazel 8y ago

Turns out sometimes people other than Linus have more experience with IO than Linus. I think there's pretty good reasons to go for DIO for a database. But only when there's a good sysadmin/DBA and when the system is dedicated to the database. There's considerable performance gains in going for DIO (at the cost of significant software complexity), but it's much more sensitive to bad tuning and isn't at all adaptive to overall system demands.

pgaddict 8y ago
Yeah, which is one of the reasons PostgreSQL went with buffered I/O, not to have to deal with this complexity. And it served us pretty well over time, I think.
- anarazel 8y ago
  I don't think that's really true. It worked well enough, true, but I think it allowed us to not fix deficiencies in a number of areas that we should just have fixed. IOW, I think we survived despite not offering DIO (because other things are good), rather than because of it.
  - pgaddict 8y ago
    I don't think we disagree, actually. Yes - from a purely technical point of view, DIO is superior in various ways. It allows tuning to specific I/O patterns, etc. But it's also quite laborious to get right - not only does it require a fair amount of new code, but AFAIK there is significant variability between platforms and storage devices. I'm not sure the project had enough developer bandwidth back then, or differently - it was more efficient to spend the developer time on other stuff, with better cost/benefit ratio.
haberman 8y ago
Yes, more than anything I'm amused by the fact that when you do the Linus-approved thing in 2007 it leaves you in this terrible situation in 2019, and when the other kernel experts rub their heads together their solution is to abandon the gentle advice from 10 years earlier.

icebraining 8y ago

> Wait, is "direct I/O" the same as O_DIRECT? No, there are other ways of doing DIO besides that particular interface.

guenthert 8y ago
Care to elaborate? And won't use those different interfaces the same direct i/o implementation, much like e.g. raw devices? I take above question as rhetorical and would think the discussion above as well as Linus' rant apply to those too.

Unklejoe 8y ago

I have to say he’s wrong when he says there’s no reason for EVER using O_DIRECT. One HUGE reason is performance for large sequential writes when using fast media and slow processors. Specifically, when the write speed of the media is in the same ballpark as the effective memcpy() speed of the processor itself, which, believe it or not, is very possible today (but was probably more unlikely in 2007) when working with embedded systems and modern NVMe media. Consider a system with an SoC like the NXP T2080 that has a high speed internal memory bus, DDR3, and PCIe 3.0 support. The processor cores themselves are relatively slow PowerPC cores, but the chip has a ton of memory bandwidth. Now assume you had a contiguous 512 MiB chunk of data to write to an NVMe drive capable of a sustained write rate of 2000 MB/s. The processor core itself can barely move 2000 MB/s of data, so it’s clear why direct IO would perform better since you’re telling the drive to pull the data directly from the buffer instead of memcpy-ing into into an intermediate kernel buffer first. With direct IO, you can perform zero-copy writes to the media. This is why I’m able to achieve higher sequential write performance on some modern MVMe drives than most benchmark sites report, all while using a 1.2 GHz PowerPC system.