4 ms·

Casual disk writes like the ones mentioned in the article rarely effect overall performance, because the operating system doesn't actually do them right now.

Casual disk writes like the ones mentioned in the article rarely effect overall performance, because the operating system doesn't actually do them right now. They're buffered by the OS and happen behind the scenes whenever the disk is free. If you profile a complex program that uses much disk IO, rarely will you find that write calls are hot. Rather, it's usually reads (and consequently seeks) that matter. The exception to this is when you're wanting to fsync or fdatasync on every write. Similarly, the situation of writing an entire new file when you only need to append a few lines to the end may sometimes be desirable, depending on how often that file is read. If it's a fairly short file that's read more frequently than written, it's often better to just write out the whole new file without fragmentation, rather that appending to the end.

gwern 14y ago
> They're buffered by the OS and happen behind the scenes whenever the disk is free. I don't think you're cooperating with the thought experiment here. Yes, buffering will help a lot, but buffering doesn't conjure up capacity that doesn't exist. If you need to write a megabyte and your throughput is just 5kb/sec, then buffering isn't going to make a whole lot of difference: you'll need to use up a full 2000 seconds of throughput at some point. If your written logs and files over a day average out to be more than 5kb/sec, you're still in for a world of pain no matter how nicely your buffering smooths out the load.
- Evbn 14y ago
  You have to consider relative scale. If you have 5mb of log data, and you have 1000mb of app data that triggered that log data, the logging isn't your problem.
- nkurz 14y ago
  Rarrrrrr may not be cooperative, but I think he's right. Sure, if you actually need to write a giant file, beyond a certain point buffering isn't going to help you. But two of the three examples in the text are rapidly rewriting short history files and avoiding intermediate files that will immediately be deleted. In both of these cases, the VM system of a modern operating system completely solves the problem. The third example is logging, which feels more like a question of defaults. He's probably right, and a lot of logged messages are never used, but this is a matter of configuring syslog rather than changing the way we think about disk IO. Removing the ability to turn on debug messages doesn't feel like a good optimization. The overall thought exercise is good, but I think it's misguided. I think the real frontier is optimizing access to RAM: RAM is the new disk. Assume every TLB miss costs a second: how would you lay out your data differently? If (as is approximately the case) access to main memory is 100x as expensive as fetching from L1, do you really want to be using a linked list? Or one that I'd love to have a good solution to: given the flexibility of the mapping between virtual memory and physical memory, why is there no good way to insert a 4K page at the head or in the middle of a multi-megabyte (virtually) contiguous array? http://stackoverflow.com/questions/11621606/faster-way-to-move-memory-page-than-mremap http://stackoverflow.com/questions/11621606/faster-way-to-mo...

gwern 14y ago

> They're buffered by the OS and happen behind the scenes whenever the disk is free. I don't think you're cooperating with the thought experiment here. Yes, buffering will help a lot, but buffering doesn't conjure up capacity that doesn't exist. If you need to write a megabyte and your throughput is just 5kb/sec, then buffering isn't going to make a whole lot of difference: you'll need to use up a full 2000 seconds of throughput at some point. If your written logs and files over a day average out to be more than 5kb/sec, you're still in for a world of pain no matter how nicely your buffering smooths out the load.

Evbn 14y ago
You have to consider relative scale. If you have 5mb of log data, and you have 1000mb of app data that triggered that log data, the logging isn't your problem.
nkurz 14y ago
Rarrrrrr may not be cooperative, but I think he's right. Sure, if you actually need to write a giant file, beyond a certain point buffering isn't going to help you. But two of the three examples in the text are rapidly rewriting short history files and avoiding intermediate files that will immediately be deleted. In both of these cases, the VM system of a modern operating system completely solves the problem. The third example is logging, which feels more like a question of defaults. He's probably right, and a lot of logged messages are never used, but this is a matter of configuring syslog rather than changing the way we think about disk IO. Removing the ability to turn on debug messages doesn't feel like a good optimization. The overall thought exercise is good, but I think it's misguided. I think the real frontier is optimizing access to RAM: RAM is the new disk. Assume every TLB miss costs a second: how would you lay out your data differently? If (as is approximately the case) access to main memory is 100x as expensive as fetching from L1, do you really want to be using a linked list? Or one that I'd love to have a good solution to: given the flexibility of the mapping between virtual memory and physical memory, why is there no good way to insert a 4K page at the head or in the middle of a multi-megabyte (virtually) contiguous array? http://stackoverflow.com/questions/11621606/faster-way-to-move-memory-page-than-mremap http://stackoverflow.com/questions/11621606/faster-way-to-mo...