4 ms·
> Moving 1 billion rows is no easy task This isn't an accurate premise. Modern OLAP databases make dealing with billions to trillions of rows manageable, inclu
by hipadev23 3y ago
> Moving 1 billion rows is no easy task
This isn't an accurate premise. Modern OLAP databases make dealing with billions to trillions of rows manageable, including on a single server. Exporting "select * from table" from an OLTP such as Postgres or MySQL into an OLAP is trivial and quite fast, and if 100M rows/sec on commodity servers isn't fast enough, there's always performance tuning [1].
[1] https://altinity.com/blog/loading-100b-rows-in-minutes-in-altinity-cloud https://altinity.com/blog/loading-100b-rows-in-minutes-in-al...
- saisrirampur 3y agoThat sentence is more in the context of the blog of moving a billion rows across data-stores. We will edit it to make that more clear. Thanks for the feedback.
- CyberDildonics 3y agodata stores or data lakes with data rivers?
- ssss11 3y agoYou’re forgetting the data dams on those data rivers
- collingreen 3y agoPlease can we have data fish ladders next?
- davidw 3y agoHere in Oregon we are removing the data dams to have more natural data flows.
- datadrivenangel 3y agoData dams adds to the visual metaphor of data breaches...
- forgingahead 3y agogotta stick to the rivers and lakes that you're used to
- hipadev23 3y agoI can't speak for Snowflake, but on Clickhouse it's quite literally [1]: insert into new_table select * from postgresql('postgres:5432', 'db', 'table', 'user', 'pass'); I assume it's similarly easy on Snowflake, Databricks, SingleStore, and the rest. [1] https://clickhouse.com/docs/en/sql-reference/table-functions/postgresql https://clickhouse.com/docs/en/sql-reference/table-functions...
- notjoemama 3y agoYes, and there’s loads of opportunity to over engineer every step of the process.
- vmfunction 3y agoOr wouldn't COPY from CSV be much faster?
- killingtime74 3y agoCSV based export/import would involve writing to disk, sending over network then reading back from file. Select is just one copy over network, no intermediate CSV.
- subleq 3y agoYou can stream CSV without writing it to a disk.
- edgyquant 3y agoWhat would be the point in that?
- vmfunction 3y agoCOPY is way faster than INSERT, at least in PG.
- jasonjmcghee 3y agoFwiw you'd lose type information. Parquet / arrow are much more appropriate here