5 ms·
This is a very sensible design for small-scale queueing systems. A few implementation notes I’ve learned along the way: - You’ll want writers to create a temp
by wolfgang42 3y ago
This is a very sensible design for small-scale queueing systems. A few implementation notes I’ve learned along the way:
- You’ll want writers to create a temp file and then rename it into place; otherwise your reader might pick up a file that’s only half-written. (POSIX rename() is atomic within a filesystem.)
- It’s also helpful to have an `in_process` folder, and a cronjob to send alerts if a task sits in that place too long. That way you can quickly catch crashes and other failures.
- You can have multiple readers; they cooperate by rename()ing the input file into in_process/ before they start working on it, and ignoring ENOENT (which indicates some other process got to it first).
- This pattern is great for loosely-coupled systems. In my case I was importing orders into a system that ran very slowly; once a reliable import queue was available new use cases kept presenting themselves, and it was very easy to add them; since the API was just “write a file in the correct format”, individual scripts could be written in whatever way was easiest for the particular task at hand. (It helped that the input format hadn’t changed in a couple decades; if it had needed changing a lot you’d want a more structured system. But this is the same as any distributed systems interface design.)
- vidarh 3y agoWith respect to how to move things into folders, the simplest advice is to tell people to just read the Maildir spec. And read Qmail as well. While Qmail is dated as a mailsystem, as a description of how to compose a decoupled system that reliably processes filesystem based queues, it contains lots of little nuggets.
- otagekki 3y ago3 years ago, I extensively used the all-company's shared filesystem to pass information between 2 independent Jenkins instances (One on Windows for jobs that worked best under a Windows machine and one on Redhat which was considered as the "main" instance); and between the Jenkins and target application servers (Windows and Redhat) which all had the company's filesystem mounted. It took time to perfect, but worked wonderfully once I adopted the rename-in-place technique as described by parent. I wonder if the system is still in place. Last time I checked the Jenkins folder was occupying 270 GB (!) of the 10 TB shared FS, most likely because FS block size was 1 MB.