4 ms·
For interactive use, sqlite queries are snappier than pandas at the ~4gb dataset size. I’ve switched over for some logfile analysis tasks and both memory usage
by CraigJPerry 6y ago
For interactive use, sqlite queries are snappier than pandas at the ~4gb dataset size. I’ve switched over for some logfile analysis tasks and both memory usage and speed are a bit better.
They’re not massive differences but i definitely appreciate the snappiness. Interactive work is just nicer that way.
I could do postgres - the docker version is pretty handy i find but it’s just more faffing about (-v /path/to/wherever:/var/postgres password, cleanup when you’re done etc.)
- iagovar 6y agoIf you're using pandas, try duck db
- chrisweekly 6y agoFor logfile analysis, take a look at https://lnav.org https://lnav.org -- it has SQLite embedded, along with a bunch of helpful utils for ETL on a small scale.
- CraigJPerry 6y agoI'll need to give that a whirl, one feature i like the look of from glancing the docs - https://docs.lnav.org/en/latest/hotkeys.html#chronological-navigation https://docs.lnav.org/en/latest/hotkeys.html#chronological-n... Sure beats regexing my way to the next minute marker or whatever.