2 ms·
Very nice. Having a one-line wrapper for the fast, native ways to do bulk data import for databases is fantastic on its own: http://odo.readthedocs.io/en/latest
by danso 10y ago
Very nice. Having a one-line wrapper for the fast, native ways to do bulk data import for databases is fantastic on its own: http://odo.readthedocs.io/en/latest/perf.html http://odo.readthedocs.io/en/latest/perf.html
I ended up learning how to do this from the various SQL shells. But, that was a bit of a cognitive load, especially when the CSV files were complicated. AFAIK, for example, SQLite won't properly ignore commas in quoted fields, so you have to throw in an extra utility (like csvkit) to change the delimiter before importing into SQLite.
sidenote: through Odo's homepage, I discovered this amazing library for generating network graph diagrams, NetworkX: http://networkx.github.io/ http://networkx.github.io/
- flashman 10y agoNetworkX is very powerful, however for more hands-on network graph diagrams (perhaps more art than science) I rely heavily on http://gephi.org http://gephi.org. (Here's my Gephi map of NSFW subreddits according to their links to each other: http://electronsoup.net/nsfw_subreddits/ http://electronsoup.net/nsfw_subreddits/)
- danso 10y agoNice! Have definitely heard of Gephi but haven't made an effort to use it, out of reluctance to learn a new GUI/system and how I rarely ever try to solve problems that require graph analysis. For that subreddit visualization, how much data prep/wrangling did you do (after making the API requests of course) before you worked with the data in Gephi?
- flashman 10y agoNot too bad. I just had to get it in the format of one line per source-target pair. The data came from /u/uglyasblasphemy in SQL format though apparently he's removed the link. Most of the fun was in arranging nodes with the move tool. With the layout I used (Fruchterman-Reingold) the nodes form a circle and you can pick up clusters and move them where you want them. Useful to make things more meaningful to the eye, if probably less mathematically correct.