5 ms·
Odo: Shapeshifting for your data
- merqurio 10y agoI use odo a lot to take any data source into pandas or blaze and it's been great so far. For my it's the lazy and easy way of moving data around.
- RileyKyeden 10y agoDoes it have to travel to its homeworld to learn the full range of its shapeshifting abilities?
- cholantesh 10y agoYes, and it also has to be electrocuted repeatedly before it runs for the first time.
- oneplane 10y agoYes, but later on, it will be forced to hard-fork because of a merge conflict and lose most of it's functions until it merges again with a young downstream project.
- danso 10y agoVery nice. Having a one-line wrapper for the fast, native ways to do bulk data import for databases is fantastic on its own: http://odo.readthedocs.io/en/latest/perf.html http://odo.readthedocs.io/en/latest/perf.html I ended up learning how to do this from the various SQL shells. But, that was a bit of a cognitive load, especially when the CSV files were complicated. AFAIK, for example, SQLite won't properly ignore commas in quoted fields, so you have to throw in an extra utility (like csvkit) to change the delimiter before importing into SQLite. sidenote: through Odo's homepage, I discovered this amazing library for generating network graph diagrams, NetworkX: http://networkx.github.io/ http://networkx.github.io/
- flashman 10y agoNetworkX is very powerful, however for more hands-on network graph diagrams (perhaps more art than science) I rely heavily on http://gephi.org http://gephi.org. (Here's my Gephi map of NSFW subreddits according to their links to each other: http://electronsoup.net/nsfw_subreddits/ http://electronsoup.net/nsfw_subreddits/)
- danso 10y agoNice! Have definitely heard of Gephi but haven't made an effort to use it, out of reluctance to learn a new GUI/system and how I rarely ever try to solve problems that require graph analysis. For that subreddit visualization, how much data prep/wrangling did you do (after making the API requests of course) before you worked with the data in Gephi?
- flashman 10y agoNot too bad. I just had to get it in the format of one line per source-target pair. The data came from /u/uglyasblasphemy in SQL format though apparently he's removed the link. Most of the fun was in arranging nodes with the move tool. With the layout I used (Fruchterman-Reingold) the nodes form a circle and you can pick up clusters and move them where you want them. Useful to make things more meaningful to the eye, if probably less mathematically correct.
- marchenko 10y agoWhat a great tool. I'm going to bookmark odo's homepage so that I don't have to wait three seasons to find it again.
- skyrw 10y agoBigger nerd? You for making that joke, me for getting it.
- GrinningFool 10y agoStatistically it was likely that at least one person in this audience would get it. Therefore I think that OP wins this nerd throwdown. Full disclosure: I still don't.
- cpr 10y agoThrow us a clue?
- danso 10y agoOdo is the name of a shapeshifter on Star Trek: Deep Space 9, who doesn't know his origins or homeworld until around season 3. I guess the homeworld isn't revisited as a major plot point until season 6? I don't remember...seasons 6 and 7 were honestly kind of a blur for me. http://memory-alpha.wikia.com/wiki/Odo http://memory-alpha.wikia.com/wiki/Odo
- goatlover 10y agoI think prior to season 6, the Klingons and Romulans tried to destroy the Dominion by taking out Odo's homeworld in a sneak attack. Odo's people turned out to be the rulers of a large empire that sought to conquer solids (non-shapeshifters).
- moosingin3space 10y ago
- A_Crazy_Idea 10y agoITT: Never heard of Perl
- kevinwang 10y agoOh my god it looks beautiful
- ElComradio 10y agoAlso today is the 23rd anniversary of the premiere of DS9.
- eximius 10y agoI wish it supported large csv to partitioned parquet. THAT is something I need a good solution for.
- quasiben 10y agoAre you familiar with fastparquet (https://github.com/dask/fastparquet https://github.com/dask/fastparquet) and pyarrow (https://pyarrow.readthedocs.io/en/latest https://pyarrow.readthedocs.io/en/latest) ?
- ColanR 10y agoYour link is malformed. https://github.com/dask/fastparquet https://github.com/dask/fastparquet
- ColanR 10y ago...and its fixed.
- eximius 10y agoNot until you linked me. :) They look like good steps but not quite what I need, for different reasons. I'll have to check when I get back to the office next week.
- nl 10y ago(Py)Spark? I run it locally and do that kind of operation pretty frequently.
- eximius 10y agoOn what scale? Part of the problem is the scale, but another part is that writing partitioned parquets seems poorly documented (I would love corrections, I spent a decent amount of time last week looking for good information)
- 10y ago
- misterdata 10y agoVery nice! I have been working on a similar library in Swift [1] which does this, but also has a nice user interface on Mac and (soon) iOS [2]. Coincidentally also uses a Star Trek themed name :-) [1] http://github.com/pixelspark/warp http://github.com/pixelspark/warp [2] https://warp.one https://warp.one
- 0x54MUR41 10y agoWell done. By the way, I have problem when opening that site. The site has problem with SSL certificate on Mozilla Firefox.
- panic 10y agoNeat! Unless I'm misunderstanding, I think Odo is more general, though, in that you don't need to pass data through any sort of unified Dataset protocol: there's a graph of direct translators between formats.
- makmanalp 10y agoI've loved the idea of Odo since I first saw it, but I've always been wary: the devil is in the details. I'm curious when and how information gets lost during each transfer because of the peculiarities of each format, or how those decisions are made and exposed. Min/max limits, truncation nulls, floating point precision, encodings, picking CHAR vs VARCHAR or string vs categorical, metadata like indices, etc are some of the hard problems behind bulk loading.