4 ms·

For the folks using kafka or kinesis or other products with streaming event architectures - do you have a replay capability? For example, I am using kinesis and

For the folks using kafka or kinesis or other products with streaming event architectures - do you have a replay capability? For example, I am using kinesis and I have a lambda that processes events and writes to a postgres database. so say 3 months from now I want to also create a lambda that firehoses to redshift. How do I get the first 3 months of my data into redshift? Right now I have a lambda that writes all events to S3, so I can replay them for other stream consumers (or even for debugging existing consumers). This seems like a reasonable (if naive) solution, but I don't see a lot of talk out there in the first place about ensuring this capability exists, and definitely haven't seen any more robust or thoughtful options discussed. Are people just not doing this generally?

breischl 9y ago
IDK about Kinesis, but for Kafka there are two major cases here: - If your Kafka retention period is long enough, then the data will still be sitting in Kafka. You don't need to replay, just have the new consumer start from the beginning. - If the retention period is not long enough, then you'll have to replay data. There's not any provision for that in Kafka itself - it's really just a somewhat-smart logfile. But whatever you're using to write into Kafka may have some way to do it. This is similar to most message queuing or eventing systems. eg, RabbitMQ.
manigandham 9y ago
There's nothing special about replay when it comes to a log. The only issue is retention and many managed services have limited retention based on data size or time frame. S3 archiving is common to keep the operational dataset small so reading it back and refilling Kafka for another consumer is actually pretty normal. If you run Kafka yourself, you can set an infinite retention and keep all the messages, and also use the compaction feature to keep the latest value for every key to reduce some of the bloat if that applies to your data, but if you don't need it instantly accessible then it's probably worth archiving to cheaper storage.
wenc 9y ago
This replayability is discussed in Designing Data Intensive Applications (DDIA), a book by Martin Kleppmann. Essentially you can use the Change Data Capture (CDC) information in your primary Postgres database, and pipe it through Kafka and replay it on any other data store.[0] This is also the basis of traditional database replication technology, where the change logs are replayed on other databases. Is this architecture common? Well, I suspect it is overkill for most smaller organizations due to increased infrastructure complexity. I wouldn't do this just for the sake of doing it -- you may find yourself saddled with an increased maintenance workload just keeping the infrastructure running. But if you truly have this use case, this is a well-known method for syncing data across various types of datastores (ie. so-called polyglot persistence). [0] https://www.confluent.io/blog/bottled-water-real-time-integration-of-postgresql-and-kafka/ https://www.confluent.io/blog/bottled-water-real-time-integr...
teej 9y ago
I have almost exactly your use-case and I solved the problem very similarly. I use a lambda to write the events to S3. I have another lambda that can read the files in S3 and produce them to replay. It's designed so I can replay many years of events if necessary, which is way beyond what I would want Kafka's retention period to be.
etxm 9y ago
We have something overly dumb and simple. We generate about 400GB of logs a day through kinesis to lambda to s3. And a second lambda that reads of s3 and forwards the stuff to wherever using an s3:objectCreated event ... If we need to replay something we just move the group of files in and out of the folder on s3 (bucket/logs -> bucket/temp -> bucket/logs) and it triggers a new objectCreated and fires the lambda. We execute the 2 move operations with the AWS cli tool. The one other thing we do is we have a config file in bucket/config.json that’s says event type X should go to redshift, pg, es, where ever. So we can tweak that before a move to send data to additional data stores. We follow ELT instead of ETL for streaming into data stores.
- connormcarthur 9y ago
  Do you find that you have to replay data often? I'd imagine that 99% or more of that 400GB/day gets thrown away before ending up in Redshift.
  - odammit 9y ago
    Rarely. We visualize a lot of it in Kibana and our kibana indexes are pruned after N days (configured for each event type). We usually only replay when we want to restore data for querying stuff outside the pruning range. We keep all that data forever.
- angersock 9y ago
  What are you doing that makes a half-terabyte of logs per day? That seems kinda nuts.
  - odammit 9y ago
    Ads ads ads! That’s just one system. Our Nginx access logs alone are about 250GB per day (they use that system).
new_question224 9y ago
Do you go kinesis-->lambda-->S3? How do you write multiple events at once into S3? Lambdas are stateless. Where would I save the last saved event?
emidln 9y ago
I journal everything to s3 after batching and compressing the batches. I have minute-resolution batches per machine (so you read the prefix in S3 for ROLE/YYYY/MM/DD/HH/MM/* to get all of the machines who wrote for that time period). I have replay capability as well as a utility library built up for my company's use cases to find specific events. Fundamentally I use an architecture similar to s3-journal[0]. There are some stream systems that can handle writing to s3 and reading from s3 natively such as Onyx[1][2] as well as checkpointing stateful operations against s3 [3]. We use replay capability to fix bugs, add new features to existing data, and to load qa environments. [0] https://github.com/Factual/s3-journal https://github.com/Factual/s3-journal [1] http://www.onyxplatform.org/ http://www.onyxplatform.org/ [2] https://github.com/onyx-platform/onyx-amazon-s3 https://github.com/onyx-platform/onyx-amazon-s3 [3] http://www.onyxplatform.org/docs/api/0.10.x/onyx.storage.s3.html http://www.onyxplatform.org/docs/api/0.10.x/onyx.storage.s3....
Buttes 9y ago
This is like the whole point of kakfa. You make all your workers idempotent and retain N days/weeks/months/years of messages. The design of kafka follows, clients track their offset, messages aren't "delivered". The docs are rly simple and if you spent 5 minutes reading the overview it would answer all your questions.
cobookman 9y ago
Google cloud solved this by offering a pubsub to GCS and GCS to pubsub pipeline. Letting you save and replay events. https://cloud.google.com/dataflow/docs/templates/provided-templates#cloud-storage-text-to-cloud-pubsub https://cloud.google.com/dataflow/docs/templates/provided-te...
elyahou 9y ago
For all folks that forward kinesis messages to s3 via lambda. Kinesis stream can be forwarded to a kinesis firehose that will persist messages to s3

breischl 9y ago

IDK about Kinesis, but for Kafka there are two major cases here: - If your Kafka retention period is long enough, then the data will still be sitting in Kafka. You don't need to replay, just have the new consumer start from the beginning. - If the retention period is not long enough, then you'll have to replay data. There's not any provision for that in Kafka itself - it's really just a somewhat-smart logfile. But whatever you're using to write into Kafka may have some way to do it. This is similar to most message queuing or eventing systems. eg, RabbitMQ.

manigandham 9y ago

There's nothing special about replay when it comes to a log. The only issue is retention and many managed services have limited retention based on data size or time frame. S3 archiving is common to keep the operational dataset small so reading it back and refilling Kafka for another consumer is actually pretty normal. If you run Kafka yourself, you can set an infinite retention and keep all the messages, and also use the compaction feature to keep the latest value for every key to reduce some of the bloat if that applies to your data, but if you don't need it instantly accessible then it's probably worth archiving to cheaper storage.

wenc 9y ago

This replayability is discussed in Designing Data Intensive Applications (DDIA), a book by Martin Kleppmann. Essentially you can use the Change Data Capture (CDC) information in your primary Postgres database, and pipe it through Kafka and replay it on any other data store.[0] This is also the basis of traditional database replication technology, where the change logs are replayed on other databases. Is this architecture common? Well, I suspect it is overkill for most smaller organizations due to increased infrastructure complexity. I wouldn't do this just for the sake of doing it -- you may find yourself saddled with an increased maintenance workload just keeping the infrastructure running. But if you truly have this use case, this is a well-known method for syncing data across various types of datastores (ie. so-called polyglot persistence). [0] https://www.confluent.io/blog/bottled-water-real-time-integration-of-postgresql-and-kafka/ https://www.confluent.io/blog/bottled-water-real-time-integr...

teej 9y ago

I have almost exactly your use-case and I solved the problem very similarly. I use a lambda to write the events to S3. I have another lambda that can read the files in S3 and produce them to replay. It's designed so I can replay many years of events if necessary, which is way beyond what I would want Kafka's retention period to be.

etxm 9y ago

We have something overly dumb and simple. We generate about 400GB of logs a day through kinesis to lambda to s3. And a second lambda that reads of s3 and forwards the stuff to wherever using an s3:objectCreated event ... If we need to replay something we just move the group of files in and out of the folder on s3 (bucket/logs -> bucket/temp -> bucket/logs) and it triggers a new objectCreated and fires the lambda. We execute the 2 move operations with the AWS cli tool. The one other thing we do is we have a config file in bucket/config.json that’s says event type X should go to redshift, pg, es, where ever. So we can tweak that before a move to send data to additional data stores. We follow ELT instead of ETL for streaming into data stores.

connormcarthur 9y ago
Do you find that you have to replay data often? I'd imagine that 99% or more of that 400GB/day gets thrown away before ending up in Redshift.
- odammit 9y ago
  Rarely. We visualize a lot of it in Kibana and our kibana indexes are pruned after N days (configured for each event type). We usually only replay when we want to restore data for querying stuff outside the pruning range. We keep all that data forever.
angersock 9y ago
What are you doing that makes a half-terabyte of logs per day? That seems kinda nuts.
- odammit 9y ago
  Ads ads ads! That’s just one system. Our Nginx access logs alone are about 250GB per day (they use that system).

new_question224 9y ago

Do you go kinesis-->lambda-->S3? How do you write multiple events at once into S3? Lambdas are stateless. Where would I save the last saved event?

emidln 9y ago

I journal everything to s3 after batching and compressing the batches. I have minute-resolution batches per machine (so you read the prefix in S3 for ROLE/YYYY/MM/DD/HH/MM/* to get all of the machines who wrote for that time period). I have replay capability as well as a utility library built up for my company's use cases to find specific events. Fundamentally I use an architecture similar to s3-journal[0]. There are some stream systems that can handle writing to s3 and reading from s3 natively such as Onyx[1][2] as well as checkpointing stateful operations against s3 [3]. We use replay capability to fix bugs, add new features to existing data, and to load qa environments. [0] https://github.com/Factual/s3-journal https://github.com/Factual/s3-journal [1] http://www.onyxplatform.org/ http://www.onyxplatform.org/ [2] https://github.com/onyx-platform/onyx-amazon-s3 https://github.com/onyx-platform/onyx-amazon-s3 [3] http://www.onyxplatform.org/docs/api/0.10.x/onyx.storage.s3.html http://www.onyxplatform.org/docs/api/0.10.x/onyx.storage.s3....

Buttes 9y ago

This is like the whole point of kakfa. You make all your workers idempotent and retain N days/weeks/months/years of messages. The design of kafka follows, clients track their offset, messages aren't "delivered". The docs are rly simple and if you spent 5 minutes reading the overview it would answer all your questions.

cobookman 9y ago

Google cloud solved this by offering a pubsub to GCS and GCS to pubsub pipeline. Letting you save and replay events. https://cloud.google.com/dataflow/docs/templates/provided-templates#cloud-storage-text-to-cloud-pubsub https://cloud.google.com/dataflow/docs/templates/provided-te...

elyahou 9y ago

For all folks that forward kinesis messages to s3 via lambda. Kinesis stream can be forwarded to a kinesis firehose that will persist messages to s3