28 ms·
Just curious how does Kafka handle data rentention though? Can it be easily configured? Or you need to build something from scratch?
by eva1984 12y ago
Just curious how does Kafka handle data rentention though? Can it be easily configured? Or you need to build something from scratch?
- mountaineer 12y agoIt can be easily configured. Default is to retain for 7 days. See log.retention.hours[1] [1] https://kafka.apache.org/08/configuration.html https://kafka.apache.org/08/configuration.html
- erichmond 12y agoAnswer: Very well. Kafka supports replication and fault-tolerance, runs on cheap, commodity hardware, and is glad to store many TBs of data per machine. So, retaining large amounts of data is a perfectly natural and economical thing to do and won’t hurt performance. LinkedIn keeps more than a petabyte of Kafka storage online, and a number of applications make good use of this long retention pattern for exactly this purpose. From http://radar.oreilly.com/2014/07/questioning-the-lambda-architecture.html http://radar.oreilly.com/2014/07/questioning-the-lambda-arch...