3 ms·

I used to do a fair amount of ad hoc analysis with large (more than Excel can handle is large enough) csv files. Sqlite3 was my saviour. It was the work of mi

I used to do a fair amount of ad hoc analysis with large (more than Excel can handle is large enough) csv files. Sqlite3 was my saviour. It was the work of minutes to have the data in a db and be chopping out the bits I needed.

speedplane 7y ago
> I used to do a fair amount of ad hoc analysis with large (more than Excel can handle is large enough) csv files. Sqlite3 was my saviour. This was probably the right decision 5 years ago, but I'm not sure Sqllite is the proper solution for "ad hoc" analysis now. Excel now has plugins to handle very large datasources. And of course, the company leading this charge here is Tableau, with many others doing something similar. Sqllite can still do all the manipulation that these others can, but the others can also be used by non developers and come with pretty graphs and pictures. For large CSV analysis on an ad hoc basis, SQLlite doesn't stack up well to the competition.
- snthpy 7y ago
  As someone who has to use Excel a lot, I'm very interested in what your typical workflow looks like.
  - speedplane 7y ago
    > As someone who has to use Excel a lot, I'm very interested in what your typical workflow looks like. It's pretty standard Excel stuff: summing columns, averaging, functions combining multiple columns, pivot tables. The significant difference is that I routinely deal with 20k+ rows, and occasionally 100k+ rows. For simple stuff, standard Excel starts breaking down with 10k+ rows, and it even struggles with 1k rows with more complicated manipulation.
- e12e 7y ago
  I'm not sure - but if you're in the MS world, you might want to just import that csv file into PowerBI?
PudgePacket 7y ago
Perhaps you don't need it anymore but for anyone else reading I've worked with some really large csv files that I don't date open in most editors with xsv. https://github.com/BurntSushi/xsv https://github.com/BurntSushi/xsv

speedplane 7y ago

> I used to do a fair amount of ad hoc analysis with large (more than Excel can handle is large enough) csv files. Sqlite3 was my saviour. This was probably the right decision 5 years ago, but I'm not sure Sqllite is the proper solution for "ad hoc" analysis now. Excel now has plugins to handle very large datasources. And of course, the company leading this charge here is Tableau, with many others doing something similar. Sqllite can still do all the manipulation that these others can, but the others can also be used by non developers and come with pretty graphs and pictures. For large CSV analysis on an ad hoc basis, SQLlite doesn't stack up well to the competition.

snthpy 7y ago
As someone who has to use Excel a lot, I'm very interested in what your typical workflow looks like.
- speedplane 7y ago
  > As someone who has to use Excel a lot, I'm very interested in what your typical workflow looks like. It's pretty standard Excel stuff: summing columns, averaging, functions combining multiple columns, pivot tables. The significant difference is that I routinely deal with 20k+ rows, and occasionally 100k+ rows. For simple stuff, standard Excel starts breaking down with 10k+ rows, and it even struggles with 1k rows with more complicated manipulation.
e12e 7y ago
I'm not sure - but if you're in the MS world, you might want to just import that csv file into PowerBI?

PudgePacket 7y ago

Perhaps you don't need it anymore but for anyone else reading I've worked with some really large csv files that I don't date open in most editors with xsv. https://github.com/BurntSushi/xsv https://github.com/BurntSushi/xsv