4 ms·
While great to see, this also highlights just how far we are from where data science needs to be. The amount of code necessary to just pull down and reformat d
by mwexler 10y ago
While great to see, this also highlights just how far we are from where data science needs to be. The amount of code necessary to just pull down and reformat data before any real analytic work is done is always staggering; I see it in my work but it's even painful in a fun example like this. I know, 80% of time in prep, 20% in analysis, yadda yadda, but I hope we can turn some of this magical AI into making data transforms easier in the future so we can actually get to the analysis portion faster, along with more accessible "tidy" data to get to the analysis portion faster.
And no, tools like Paxata or Tamr (or http://www.predictiveanalyticstoday.com/data-preparation-tools-and-platforms/ http://www.predictiveanalyticstoday.com/data-preparation-too... etc.) are indeed a good start, but are not really the solution yet.
We'll always have to code our way out of some wacko data situations, but I look forward to (and dream of!) better and better libraries and approaches to getting prep prepped faster.
- minimaxir 10y agoR in general is inefficient with nontabular data manipulation with text data and web scraping, which is why I fall back to Python for those tasks. But even in the context of the article, the code the author posted is extremely confusing and is an odd mix of base and dplyr code.