5 ms·
You don’t even need a database to implement an effective queueing mechanism. Linux file move is atomic, which means file system based queues are perfectly viab
by 00000000005 5y ago
You don’t even need a database to implement an effective queueing mechanism.
Linux file move is atomic, which means file system based queues are perfectly viable. Just save one message per file. Move the file to a different directory when it changes state.
I built a prototype queue around this mechanism. Performance was bound to the disks ability to create small files, around 30,000 messages per second.
This sort of performance rivals some of much more sophisticated and complex queuing systems, but has zero configuration.
- slashdev 5y agoIt has complex failure modes though, unless you're not at all concerned about what happens if the program crashes, power goes out, file gets corrupted, etc. It's surprisingly difficult to write something that stores changing data on disk correctly and durably.
- 00000000005 5y agoDatabases have the same problems when disks fail, power goes out etc.
- eloff 5y agoIf you mean they have to solve the same problems with very careful design and programming, yes. It's non trivial. Even Postrgres had a correctness issue around fsync for nearly twenty years. If you roll your own, that's very hard to get right. I wouldn't have confidence in such a system that I've built myself.
- kumarvvr 5y agoThis mechanism has multiple points of failure, and too many disk writes. Batching up writes saves the hardware.
- 00000000005 5y agoBatching up writes means greater risk if data loss compared to immediate writes.
- RedShift1 5y agoWould you mind sharing the code? How do you know subscribers have messages waiting for them, inotify or polling or something else?