6 ms·
Persistent “pipes” in Linux
- porlw 11y agoSoooo... What happens when an errant grep wanders in to the directory holding these persistent pipes?
- _yy 11y agoNothing, since those "persistent pipes" are just regular sparse files (just like VM images). Clever idea.
- zimbatm 11y agoI wonder how it behaves in terms of performance and disk fragmentation. Potentially log-oriented databases could adopt that technique instead of having lots of files on disk.
- joshbaptiste 11y agoSome previous thoughts: https://news.ycombinator.com/item?id=10068591 https://news.ycombinator.com/item?id=10068591
- otterley 11y agoSee also Varnish's log implementation, which employs a similar technique but uses a shared memory region instead: http://varnish-cache.readthedocs.org/en/latest/users-guide/operation-logging.html http://varnish-cache.readthedocs.org/en/latest/users-guide/o...
- teddyh 11y agoI would like to know what the file system implementors have to say about this technique. Is their reaction “Aargh, why would you do that, it wrecks all our optimizations!” or is it “Huh, OK, that’s kind of odd but I guess it would technically work” or even “Yes, finally someone who takes advantage of all our nice features like we always wanted!”?
- planckscnst 11y agoOr maybe, "Huh. That's interesting. Maybe I'll make a new filesystem object type or add an optional disk buffer for pipes."
- ksherlock 11y agoAnother approach (used by qmail and I'm sure other programs) is to use the file system as a database. Process 1 creates a file into a temp directory. Once writing is finished, the file is atomically moved to the working directory where Process 2 deals with it. That wouldn't work so well if you need them to be processed in a specific order, though.
- djcapelis 11y agoHmm, sparse files get a little annoying after the offset gets high enough. I've had weirdness with sparse files that had offsets in the 64bit range. So... eventually if you don't compact and start from a low offset ring buffer style, this solution will stop working after you pass enough data through the "pipe". Yes. That means a lot of data, but it's a boundary case worth knowing about.