4 ms·

As a junior dev I used folder pairs (to_process and processed) as a way to move messages between loosely coupled systems with file system watchers picking up ne

by mikece 3y ago

As a junior dev I used folder pairs (to_process and processed) as a way to move messages between loosely coupled systems with file system watchers picking up new files in the to_process folder. Very light weight, got the job done, and was told it was in production for over a decade (even after better best practices came into use).

derefr 3y ago
FYI, this is an old Unixism called a "spool directory" :)
- WeAddValue 3y ago
  FYI, which was probably derived from the old mainframe-ism which had spooling before Unix was written. See https://en.wikipedia.org/wiki/Spooling https://en.wikipedia.org/wiki/Spooling
  - derefr 3y ago
    Literally spooling message-filled tape onto a tape reel! I never even thought about that. On the topic of message queues, I wonder if anyone ever designed a mechanical, asynchronous hardware IPC mechanism based on tape. Imagine: an endless-loop tape that stretches between a "sender" tape-drive on one machine, and a "receiver" tape-drive on another machine; with each tape drive locally tensioning the tape as it passes through, but letting it hang slack outside the drive; with two tape-width buffer boxes in between the two drives, for taking up the slack; and where each tape-drive has a strain gauge on its input-end capstan. With this design, you'd never need to seek the tape (backward, at least) — the receive head would just read the tape forward at whatever speed it could until it ran out of slack in the tape (i.e. blocked on read); and the send head would write the tape forward at whatever speed it could until it ran out of slack (i.e. blocked on write.) The tape itself would be a literal, physical ring buffer.
    - wolfgang42 3y ago
      Western Union built a store-and-forward message switching system in pretty much exactly the way you describe (but with paper tape rather than magnetic): https://en.wikipedia.org/wiki/Plan_55-A https://en.wikipedia.org/wiki/Plan_55-A
    - robotresearcher 3y ago
      Analog audio delay/echo machines work like this. They could have multiple read heads and you could move them relative to each other and adjust the speed of the tape to get various cool effects.
- mikece 3y ago
  So having "discovered" this approach as a young .NET dev retroactively gives me a smug smile that I came up with such a battle-tested pattern. Then again, it's also quite an obvious and simple pattern which is why it was used in the first place, I'm sure.
florbo 3y ago
Sometimes a filesystem-based queue like that is all you really need. Recently I used S3 API compatible storage for something similar, as introducing AMQP or pubsub would have been immensely overkill for this low volume component.
wolfgang42 3y ago
This is a very sensible design for small-scale queueing systems. A few implementation notes I’ve learned along the way: - You’ll want writers to create a temp file and then rename it into place; otherwise your reader might pick up a file that’s only half-written. (POSIX rename() is atomic within a filesystem.) - It’s also helpful to have an `in_process` folder, and a cronjob to send alerts if a task sits in that place too long. That way you can quickly catch crashes and other failures. - You can have multiple readers; they cooperate by rename()ing the input file into in_process/ before they start working on it, and ignoring ENOENT (which indicates some other process got to it first). - This pattern is great for loosely-coupled systems. In my case I was importing orders into a system that ran very slowly; once a reliable import queue was available new use cases kept presenting themselves, and it was very easy to add them; since the API was just “write a file in the correct format”, individual scripts could be written in whatever way was easiest for the particular task at hand. (It helped that the input format hadn’t changed in a couple decades; if it had needed changing a lot you’d want a more structured system. But this is the same as any distributed systems interface design.)
- vidarh 3y ago
  With respect to how to move things into folders, the simplest advice is to tell people to just read the Maildir spec. And read Qmail as well. While Qmail is dated as a mailsystem, as a description of how to compose a decoupled system that reliably processes filesystem based queues, it contains lots of little nuggets.
- otagekki 3y ago
  3 years ago, I extensively used the all-company's shared filesystem to pass information between 2 independent Jenkins instances (One on Windows for jobs that worked best under a Windows machine and one on Redhat which was considered as the "main" instance); and between the Jenkins and target application servers (Windows and Redhat) which all had the company's filesystem mounted. It took time to perfect, but worked wonderfully once I adopted the rename-in-place technique as described by parent. I wonder if the system is still in place. Last time I checked the Jenkins folder was occupying 270 GB (!) of the 10 TB shared FS, most likely because FS block size was 1 MB.
fukawi2 3y ago
I'm doing the same right now. The incoming messages are coming from a radio network at 1200 baud, so I don't foresee capacity problems :D
albrewer 3y ago
At my old company, we had a system where we had a limited number of physical machines that multiple testers would need to remote into. We didn't want to disallow two people logging in simultaneously because sometimes that was necessary, and if someone stayed logged in then wen to lunch, where was no way to kick them off to get access without waling to the server closet, pulling up the machine on the KVM, and kicking them that way. I wrote some VBScript that wrote a locking folder into a sub-folder in a network folder, then launched VNC targeting the proper VM. When VNC was closed, the script would complete and delete the locking folder. That let us know what was available and when. I always thought it was hacky, but at ~60 lines it's incredibly simple and has never really failed.

by mikece 3y ago

derefr 3y ago

FYI, this is an old Unixism called a "spool directory" :)

WeAddValue 3y ago
FYI, which was probably derived from the old mainframe-ism which had spooling before Unix was written. See https://en.wikipedia.org/wiki/Spooling https://en.wikipedia.org/wiki/Spooling
- derefr 3y ago
  Literally spooling message-filled tape onto a tape reel! I never even thought about that. On the topic of message queues, I wonder if anyone ever designed a mechanical, asynchronous hardware IPC mechanism based on tape. Imagine: an endless-loop tape that stretches between a "sender" tape-drive on one machine, and a "receiver" tape-drive on another machine; with each tape drive locally tensioning the tape as it passes through, but letting it hang slack outside the drive; with two tape-width buffer boxes in between the two drives, for taking up the slack; and where each tape-drive has a strain gauge on its input-end capstan. With this design, you'd never need to seek the tape (backward, at least) — the receive head would just read the tape forward at whatever speed it could until it ran out of slack in the tape (i.e. blocked on read); and the send head would write the tape forward at whatever speed it could until it ran out of slack (i.e. blocked on write.) The tape itself would be a literal, physical ring buffer.
  - wolfgang42 3y ago
    Western Union built a store-and-forward message switching system in pretty much exactly the way you describe (but with paper tape rather than magnetic): https://en.wikipedia.org/wiki/Plan_55-A https://en.wikipedia.org/wiki/Plan_55-A
  - robotresearcher 3y ago
    Analog audio delay/echo machines work like this. They could have multiple read heads and you could move them relative to each other and adjust the speed of the tape to get various cool effects.
mikece 3y ago
So having "discovered" this approach as a young .NET dev retroactively gives me a smug smile that I came up with such a battle-tested pattern. Then again, it's also quite an obvious and simple pattern which is why it was used in the first place, I'm sure.

florbo 3y ago

Sometimes a filesystem-based queue like that is all you really need. Recently I used S3 API compatible storage for something similar, as introducing AMQP or pubsub would have been immensely overkill for this low volume component.

wolfgang42 3y ago

This is a very sensible design for small-scale queueing systems. A few implementation notes I’ve learned along the way: - You’ll want writers to create a temp file and then rename it into place; otherwise your reader might pick up a file that’s only half-written. (POSIX rename() is atomic within a filesystem.) - It’s also helpful to have an `in_process` folder, and a cronjob to send alerts if a task sits in that place too long. That way you can quickly catch crashes and other failures. - You can have multiple readers; they cooperate by rename()ing the input file into in_process/ before they start working on it, and ignoring ENOENT (which indicates some other process got to it first). - This pattern is great for loosely-coupled systems. In my case I was importing orders into a system that ran very slowly; once a reliable import queue was available new use cases kept presenting themselves, and it was very easy to add them; since the API was just “write a file in the correct format”, individual scripts could be written in whatever way was easiest for the particular task at hand. (It helped that the input format hadn’t changed in a couple decades; if it had needed changing a lot you’d want a more structured system. But this is the same as any distributed systems interface design.)

vidarh 3y ago
With respect to how to move things into folders, the simplest advice is to tell people to just read the Maildir spec. And read Qmail as well. While Qmail is dated as a mailsystem, as a description of how to compose a decoupled system that reliably processes filesystem based queues, it contains lots of little nuggets.
otagekki 3y ago
3 years ago, I extensively used the all-company's shared filesystem to pass information between 2 independent Jenkins instances (One on Windows for jobs that worked best under a Windows machine and one on Redhat which was considered as the "main" instance); and between the Jenkins and target application servers (Windows and Redhat) which all had the company's filesystem mounted. It took time to perfect, but worked wonderfully once I adopted the rename-in-place technique as described by parent. I wonder if the system is still in place. Last time I checked the Jenkins folder was occupying 270 GB (!) of the 10 TB shared FS, most likely because FS block size was 1 MB.

fukawi2 3y ago

I'm doing the same right now. The incoming messages are coming from a radio network at 1200 baud, so I don't foresee capacity problems :D

albrewer 3y ago

At my old company, we had a system where we had a limited number of physical machines that multiple testers would need to remote into. We didn't want to disallow two people logging in simultaneously because sometimes that was necessary, and if someone stayed logged in then wen to lunch, where was no way to kick them off to get access without waling to the server closet, pulling up the machine on the KVM, and kicking them that way. I wrote some VBScript that wrote a locking folder into a sub-folder in a network folder, then launched VNC targeting the proper VM. When VNC was closed, the script would complete and delete the locking folder. That let us know what was available and when. I always thought it was hacky, but at ~60 lines it's incredibly simple and has never really failed.