3 ms·
COPY trips FROM '/Users/mark/taxi_csv/*.gz' WITH (HEADER='false'); > The above managed to complete in 31 minutes and 40 seconds. The resulting import p
by danso 6y ago
COPY trips
FROM '/Users/mark/taxi_csv/*.gz'
WITH (HEADER='false');
> The above managed to complete in 31 minutes and 40 seconds. The resulting import produced 294 GB of data in OmniSciDB's internal format.
I’m really curious how a simple import (no indexes or data typing) into SQLite would compare. But I don’t have 700GB of free SSD space to spare.
- momonga 6y agoThe same blog has other benchmarks[1], one of which is SQLite[2]. [1] https://tech.marksblogg.com/benchmarks.html https://tech.marksblogg.com/benchmarks.html [2] https://tech.marksblogg.com/billion-nyc-taxi-rides-sqlite-parquet-hdfs.html https://tech.marksblogg.com/billion-nyc-taxi-rides-sqlite-pa...
- tmostak 6y agoMark did a benchmark of SQLite using its internal file format a few years ago (https://tech.marksblogg.com/billion-nyc-taxi-rides-sqlite-parquet-hdfs.html#comparing-to-sqlite-s-internal-format https://tech.marksblogg.com/billion-nyc-taxi-rides-sqlite-pa...), clocking the import at 5.5 hours. It looks like this was done though on a spinning disk, so given a proper SSD, and a newer version of SQLite, it might be much faster.