3 ms·

> taken to its logical conclusion, tiered storage could turn Kafka into [...] A message broker sitting in front of an RDBMS. I mean, if we're now basically 'ta

• datavirtue 2y ago
  Kind of difficult if all you know is a cloud Kafka and have no idea that RabbitMQ exists or how to use it. I went to a lunch and learn where a young engineer was demoing some Kafka-based solution that was spun up in AWS. I asked about some of the functionality in comparison with RabbitMQ. Deer in the headlights.
• cryptonector 2y ago
  "tailing" is a pretty cool mechanism. An MQ is possibly too general. For "tailing" you need to be able to resume, and that requires that you have a stable "offset" identifier, which in HTTP-speak would be: {URI, weak ETag, byte-range offset}, and then you can use conditional requests and local-part naming conventions to deal with things like: - detection of rollover (which is where you can move older content to tiered storage), - recovery from that (resumption), and - detection of lost content (e.g., you resumed a tail a month later and [surprise!] you can't catch up and you need to recover in some other way. In fact, I've written an HTTP server I call tailfhttpd that supports all of that and, if you GET w/ `Range: bytes=${offset}-` (for some offset) then the GET doesn't complete until the file is unlinked or renamed away. Poor-person's-Kafka. This is generally quite useful since you can use it to tail both structured and unstructured log files, as long as you only ever append to them.