8 ms·
No, this is often used for logging. In my last job, we were generating on the order of a million lines like this per hour in the log I worked with most often (m
by rented_mule 3y ago
No, this is often used for logging. In my last job, we were generating on the order of a million lines like this per hour in the log I worked with most often (many millions per hour if you count all the logs in the company). Each line of that log was between a few hundred bytes and ~100 KB. It would take a lot of work to support that in a database. That log was processed by many Spark jobs (some in streaming fashion, others in batch). "Small" summaries were added to a columnar database each night. The main table in that database contained hundreds of columns and billions of rows for more interactive analysis.