3 ms·
This is riddled with problems. a) I routinely load 6GB of financial data into R in less than 45 seconds, and it's complex and hierarchical to boot. Admittedly
by vegabook 12y ago
This is riddled with problems.
a) I routinely load 6GB of financial data into R in less than 45 seconds, and it's complex and hierarchical to boot. Admittedly that's in R's native file format but even a flat CSV takes less than 5 minutes.
b) Why SQL? Has this person considered Pytables or h5py? Here is an excellent file format created by guys who work with large data sets all day long, which is vastly superior to an on-disk relational database for data sets of this size (and much bigger). Moreover, they map directly to Numpy.
c) Why SQL (II)? Why is all the wrangling being done in slow SQL when Pandas has 100x faster in memory querying capability via groupby and query methods?
d) The visualization is terrible. "Other" dominates the dataset and dwarfs any of the more interesting categories and their relationship to time of day. This is a very clear case for when a chart grid would have been preferable, easily constructable in Pandas or of course GG Plot, and surely doable with Plot.ly. We would have been table to compare how different categories have different time-of-day profiles.
e) If we're going to push data visualization tutorials into the big wide world, why not introduce people to quantile plots, useful here to compare the distribution of complaints (literally one line in R), or more simply and very intuitively, we could have superimposed all the hour-by-hour complaint density plots on each other and see how they compare (again, literally a few lines in R as in plot(density(x)); par(new = TRUE); plot(density(y), add = TRUE, col = "xxx"). After having normalised the data of course. This has its analogues in Python. A second, bog-standard, unstacked, bar chart could then have been used for aggregate complaint volume by hour, and dare I say it, even a pie chart would have been better here for comparing aggregate complaint category sizes.
This looks like a pitch for Plotly aimed at casual HTML developers, not serious data scientiests.
- semi-extrinsic 12y agoPandas has native HDF5 read/write. So they could just do a one-time read from SQL, dump to HDF5, and enjoy subsequent fast reads. I've recently used Pandas with HDF5 backing for building a web app with Flask that does complex queries on a small dataset (40k rows, ~ 15 columns of which four are indices). Full stack Python, no SQL anywhere, it's awesome. Stick it in a virtualenv, fire and forget.
- IndianAstronaut 12y ago>The visualization is terrible. That's a feature not a bug. Plotting libraries for Python have always been abysmal. Matplotlib is both ugly and hard to use.
- vegabook 12y agoWow I have to agree with this (even though the post is using plotly). I don't know where the culture difference comes from between the R and Python ecosystems that makes the latter so much worse than the former when it comes to graphics. Python seems to be able to do so many things so well, but then at the final output stage it all falls apart and I find myself back in R. On that point, I even wonder if with all the Julia bruhaha, the actual real competitor in data science might actually be Javascript! Much faster than Python, and a community that truly understands graphics quality (I'm looking primarily at D3 but things like Mathbox.js and even Hicharts are pretty decent). I'm thinking that SVG and increasingly Webgl will incentivize linear algebra skills in the Javascript ecosystem, and those same skills will suddenly be transferrable to data.
- Lofkin 12y agoHave you tried bokeh or vispy? What about seaborn?
- IndianAstronaut 12y agoBesides D3, Google charts, which are also in JS, are also quite elegant.
- apl 12y agoMPL's documentation is a catastrophe, and the API design is indeed abysmal. The plots, however, are only as ugly as you like them to be. Here's a great auxiliary package with much more sensible defaults: https://github.com/mwaskom/seaborn https://github.com/mwaskom/seaborn
- Lofkin 12y agoTry bokeh http://bokeh.pydata.org/en/latest/ http://bokeh.pydata.org/en/latest/