4 ms·
Short writes to (local) filesystems are rare in practice, unless you run into kernel limits (e.g., Linux stops a couple KB short of 2GB). When they do happen, i
by pkhuong 6y ago
Short writes to (local) filesystems are rare in practice, unless you run into kernel limits (e.g., Linux stops a couple KB short of 2GB). When they do happen, it's often due to issues that are hard to recover from, like failing media or lack of disk space.
You're right that short writes can cause data loss; however, self-synchronisation guarantees that only the two records immediately adjacent to the short write can be affected (with a reasonable implementation, at most one... and if you prefix each record with the 2-byte separator, only if the write is cut short after the sepearator's first byte).
- anarazel 6y agoI see short writes pretty regularly under memory pressure. For writes well under 2GB. With io_uring there are extremely common (but got less so in very recent kernel versions). For different reasons for direct and buffered IO IIRC.
- anarazel 6y agoOh, aber not just memory pressure, a busy IO stack too. When there's multiple requests in flight already (concurrent processes, AIO, writeback) a part of the write may be issued as a separate (block layer) request, and IIRC there's a chance to get interrupted before sending the next parts.
- pkhuong 6y agoI can see how it might be different with io_uring; I've not observed short writes with O_APPEND and regular buffered I/O.