4 ms·
> I used to do a fair amount of ad hoc analysis with large (more than Excel can handle is large enough) csv files. Sqlite3 was my saviour. This was probably th
by speedplane 7y ago
> I used to do a fair amount of ad hoc analysis with large (more than Excel can handle is large enough) csv files. Sqlite3 was my saviour.
This was probably the right decision 5 years ago, but I'm not sure Sqllite is the proper solution for "ad hoc" analysis now. Excel now has plugins to handle very large datasources. And of course, the company leading this charge here is Tableau, with many others doing something similar.
Sqllite can still do all the manipulation that these others can, but the others can also be used by non developers and come with pretty graphs and pictures. For large CSV analysis on an ad hoc basis, SQLlite doesn't stack up well to the competition.
- snthpy 7y agoAs someone who has to use Excel a lot, I'm very interested in what your typical workflow looks like.
- speedplane 7y ago> As someone who has to use Excel a lot, I'm very interested in what your typical workflow looks like. It's pretty standard Excel stuff: summing columns, averaging, functions combining multiple columns, pivot tables. The significant difference is that I routinely deal with 20k+ rows, and occasionally 100k+ rows. For simple stuff, standard Excel starts breaking down with 10k+ rows, and it even struggles with 1k rows with more complicated manipulation.
- e12e 7y agoI'm not sure - but if you're in the MS world, you might want to just import that csv file into PowerBI?