3 ms·
Atomic writes are pretty easy - as this article points out, PIPE_BUF sized writes to pipes are atomic. It seems like atomic reads are harder. You want to read
by catern 10y ago
Atomic writes are pretty easy - as this article points out, PIPE_BUF sized writes to pipes are atomic.
It seems like atomic reads are harder. You want to read a record from a shared input, but you don't know how big a record might be. Any thoughts?
- GauntletWizard 10y agoNot just pipes - Files, too (actually, all write calls). Assuring your records are always smaller than PIPE_BUF and just using O_APPEND works pretty well.
- fnord123 10y agoWrites to files less than PIPE_BUF aren't atomic on parallel file systems. e.g. Panasas, Lustre, GPFS, etc.
- fnord123 10y agoClarification: writes to files less than PIPE_BUFF may be atomic, but they aren't necessarily consistent. This is another important property as you want reads following writes to always have the written values. I think I tripped up over myself since in atomicity you expect the 'all written' and 'nothing written' states, but there's also 'all written but nothing visible' which is a third state which feels like it's part of atomicity, but I guess it's really part of consistency.
- rpcope1 10y agoIf you have the ability to (minimally) parse what is coming out of your pipe, and you know that your writes are not interleaved, and you know the size of the record you're writing before you write it, you can prefix each record with the size, and use this on the reader side to fetch only upto the record length (read(2) syscall and friends allow you to specify a maximum number of bytes to fetch), for each record. Does this answer your question?
- catern 10y agoI don't believe what you're describing works. You're suggesting reading first the size of a record (stored itself in a fixed-size field), then reading the rest of it? But unfortunately that is not atomic. Another process could read into the data between your first and second reads.