3 ms·
Lots of good suggestions here. I won't say anything new but just wanted to stress on the data ingestion part. DO NOT write to txt files and read them again. Th
by agnivade 11y ago
Lots of good suggestions here. I won't say anything new but just wanted to stress on the data ingestion part.
DO NOT write to txt files and read them again. This is unnecessary disk IO and you will run into a lot of problems later on. Instead, have an agent which writes into Kafka (like everyone mentioned), preferably using protobuff.
Then have an aggregator which does the data extraction and analysis and puts them in some sort of storage. You can browse this thread to look for and decide what sort of storage is suitable for you.