4 ms·
People are coming out with their own cases where they've implemented this pattern, and I'm sure there are a lot more that haven't bothered to note their cases.
by kbenson 9y ago
People are coming out with their own cases where they've implemented this pattern, and I'm sure there are a lot more that haven't bothered to note their cases. That's because it's actually the simplest (and possibly most obvious) solution whenever you have disjoint producers and consumers. I use it myself through the disk. I have a time based directory structure, and within that compressed blobs of data that are timestamped and have whatever other uniquely identifying information is required.[1] In fact, I have no less than seven distinct sets of data like this, some JSON, some HTML from scraped pages, etc
One set of processes retrieves and stores the data in the normalized structure, another set of processes reads and processes them. If there's ever a problem in the processing, the system can fail until someone looks into the error and adds an exception or fixes the processing. This is useful for when you can't be sure the remote side will continue to work as expected (I've found API's to be only slightly more stable than random web scraping. Sure, they may version their APIs, but that doesn't help when they don't communicate new versions or when they are deprecating the old ones and you're hit with a dead API endpoint it out of the blue).
Not only is the processing asynchronous, but the development is so asynchronous, I've added data sources that I knew I would eventually want to process (as adding a data source is fairly easy), only to get around to writing the processor six months later, and the nature of the system means that it's trivial to process the old data. In fact, the normal operation would likely note it is that far out of date and process the old data. The only consideration that needs to be made is whether you want to optimize the back-processing by aggressively caching, since that can greatly speed up the process.
I really think the pattern it is the natural conclusion most people would come to when presented with the right requirements. The real art comes from recognizing when your current problem can map to a pattern such as this when it doesn't appear to initially.
1: e.g. "/$base/$logname/%Y-%m-%d/%H/%Y%m%d%H%M%S_${unique}.json.gz"