3 ms·
How to use Apache Kafka to transform a batch pipeline into a real-time one
- sidlls 8y agoThe examples in these articles are constrained ("toys") for demonstration purposes, but too often are taken as a recipe by people who should know better but want to build their resume. LinkedIn didn't develop Kafka just for real-time processing: dozens of solutions for that existed for years before LinkedIn existed. They developed Kafka for a problem at huge scale. A little 100MB/day of traffic doesn't need Kafka for real time processing, and neither does a 100GB monthly data dump/ETL batch job.