4 ms·
Redis streams have been a phenomenal addition to my toolbelt in designing realtime ETL/ELT pipelines. Before I had to make do with a way more complicated Pub/Su
by subhobroto 6y ago
Redis streams have been a phenomenal addition to my toolbelt in designing realtime ETL/ELT pipelines. Before I had to make do with a way more complicated Pub/Sub + job q (Tasktiger). That all became redundant thanks to Redis streams.
Thank you!
It would really be awesome if there was a built in way to attach/spill/persist individual streams to external data volumes (older/busy streams could run out of memory) and have it support hot swapping.
> Btw this Kafka monster is a total of 2775 lines of code, including comments. 1731 lines of code without comments. In other systems this is the prelude in the copyright notice.
Hilarious, funny and informative :D Upvoted!
- itamarhaber 6y agoGreat to hear you enjoying streams - you got my upvote :)
- leandot 6y agoDo you mind sharing more how you use Redis streams for ETL pipelines?
- subhobroto 6y agoHappy to talk shop anytime, feel free to reach out. In short - I like to have audit-able dataflows in my pipelines. Streams are inherently timestamped and provide a great way to see how data changed over time. For one, if you had a bug or regression in the pipeline, you can precisely track down the impacted time window - no guessing needed.