4 ms·
Or perhaps SQLite, since it should perform similar to writing to a CSV file and you get querying out of the box afterwards.
by Mahn 10y ago
Or perhaps SQLite, since it should perform similar to writing to a CSV file and you get querying out of the box afterwards.
- saosebastiao 10y agoSQLite is a major powerhouse for stuff like this, and if your analysis is on a single computer with a single disk, analysis is almost always faster than any sort of highly parallel big data setup. I've loaded, analyzed, and reported on 2TB of data using SQLite in less time than it took for an 8 machine spark cluster took to load the data.