3 ms·
I have a gut feeling that this would be far, far faster with a sqlite import, including the time spent creating the sql table structures. select count(*) group
by vessenes 11y ago
I have a gut feeling that this would be far, far faster with a sqlite import, including the time spent creating the sql table structures.
select count(*) group by queries in sqlite are significantly faster than the times he's showing here, at the very least.
EDIT: I went ahead and ran some tests and added the results to another comment. In short, sqlite is significantly faster and easier to debug.
- gglitch 11y agoI'm on a phone and no longer have the essay up but I too am puzzled by why sql isn't a good fit for this work.
- okket 11y agoMaybe some misguided "SQL is black magic, avoid at any cost" bias? I'd even pull out PostgreSQL with its mighty scripting and analyzing capabilities. Ok, speed may come at cost of HD space and memory consumption and would require a not so fast import...
- vinceguidry 11y agoThe time it takes to get familiar with a tool shouldn't be too heavily discounted. Bash is remarkably easy to get started with.
- peterwwillis 11y agoYou disable stuff that slows down processing and then import the data using the SQL engine's own LOAD DATA commands. Not that slow either. Really though, you first have to ensure your documents all match a standard, since a lot of people just expect csv, tsv and other documents to all be the same, and they are not. So you have to do extra work anyway. Might as well use SQL too.