3 ms·
"tailing" is a pretty cool mechanism. An MQ is possibly too general. For "tailing" you need to be able to resume, and that requires that you have a stable "of
by cryptonector 3y ago
"tailing" is a pretty cool mechanism. An MQ is possibly too general.
For "tailing" you need to be able to resume, and that requires that you have a stable "offset" identifier, which in HTTP-speak would be: {URI, weak ETag, byte-range offset}, and then you can use conditional requests and local-part naming conventions to deal with things like:
- detection of rollover (which is where you can move older content to tiered storage),
- recovery from that (resumption), and
- detection of lost content (e.g., you resumed a tail a month later and [surprise!] you can't catch up and you need to recover in some other way.
In fact, I've written an HTTP server I call tailfhttpd that supports all of that and, if you GET w/ `Range: bytes=${offset}-` (for some offset) then the GET doesn't complete until the file is unlinked or renamed away. Poor-person's-Kafka. This is generally quite useful since you can use it to tail both structured and unstructured log files, as long as you only ever append to them.