3 ms·
Here is the thing made me a happier person this year: I realized when CSVs take too long to load, I should not be using them in the first place! I love SQLite
by klhugo 6y ago
Here is the thing made me a happier person this year: I realized when CSVs take too long to load, I should not be using them in the first place!
I love SQLite now <3
- karbarcca 6y agoIt's always a transcendental experience to move data from CSV into any more structured/flexible format!
- klhugo 6y agoAnd fast! I got easily 3 orders of magnitude speedup. I used to load CSVs with pandas and do typical filtering operations... oh boy, some of the stuff would take a hole morning, not to mention the memory usage. I do all that in minutes now.
- benhurmarcel 6y agoSorry if that sounds stupid, but then when you get a csv file you write it to an SQLite file before doing computations?