5 ms·
If we're having milions of line of anything, just go with an actual db, no?
by zipping1549 3y ago
If we're having milions of line of anything, just go with an actual db, no?
- masklinn 3y agoHow does "an actual db" help you with an event stream e.g. over a socket? Are you going to stream WAL records and ask the client to merge that into their local database?
- rented_mule 3y agoNo, this is often used for logging. In my last job, we were generating on the order of a million lines like this per hour in the log I worked with most often (many millions per hour if you count all the logs in the company). Each line of that log was between a few hundred bytes and ~100 KB. It would take a lot of work to support that in a database. That log was processed by many Spark jobs (some in streaming fashion, others in batch). "Small" summaries were added to a columnar database each night. The main table in that database contained hundreds of columns and billions of rows for more interactive analysis.
- baq 3y agoWhy would I need a db for a meager million rows? Even excel handles a million rows
- DonHopkins 3y agoAn actual db like BigQuery supports importing jsonl directly. https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-json https://cloud.google.com/bigquery/docs/loading-data-cloud-st...
- zX41ZdbW 3y agoClickHouse supports it for actual INSERT queries. So instead of INSERT ... VALUES, you write INSERT ... FORMAT JSONEachRow and stream your data. https://clickhouse.com/docs/en/sql-reference/statements/insert-into https://clickhouse.com/docs/en/sql-reference/statements/inse...