3 ms·
If you need to do an indexing step anyways, why not simply import the data into a SQL database, or build this as a wrapper that introspects the CSV file, builds
by btown 12y ago
If you need to do an indexing step anyways, why not simply import the data into a SQL database, or build this as a wrapper that introspects the CSV file, builds a database schema, and does the import for you? Is the issue limited scratch space?
- burntsushi 12y agoSee the "Motivation" section: https://github.com/BurntSushi/xsv#motivation https://github.com/BurntSushi/xsv#motivation There's a line somewhere between "conveniently play with large CSV data on the CLI" and "the full power of a RDBMS." It's blurry and we won't all agree on where it lays, but it certainly exists for me. (And based on feedback, it exists for lots of others too.) Also, there are already tools that look at a CSV file and figure out a schema. No need to replicate that. Finally, the indexing step is blindingly fast and only uses `N * 8` bytes, where `N` is the number of records.