4 ms·
> tradeoffs in our operation of Kafka have lossy effects on stream-ability. Balancing costs and operational feasibility, we ask Kafka to forget older data throu
by dominotw 9y ago
> tradeoffs in our operation of Kafka have lossy effects on stream-ability. Balancing costs and operational feasibility, we ask Kafka to forget older data through retention policies.
What does ' lossy effects on stream-ability. ' mean here. Stream slows down, data loss or something else?
- deleted 9y ago[deleted]
- lbradstreet 9y agoPyrostore co-founder here. When practitioners archive their data from Kafka to other storage products (S3, SQL database, etc) today, they are giving up on the log ordered structure of the data their ability to consume their data in its original ordering, with its original offsets and timestamps. Pyrostore structures and indexes your data in S3 in order to provide a consumer that implements the Kafka consumer interfaces, ensuring you are always able to stream from hot and cold storage alike.