7 ms·

Have you ever used the tidyverse ecosystem? It's a set of syntacticly-compatible package authored chiefly by one man, which have evolved into a superset of R, i

Have you ever used the tidyverse ecosystem? It's a set of syntacticly-compatible package authored chiefly by one man, which have evolved into a superset of R, if not an outright R2. (R's Scheme lineage makes it very hackable in this way). I've working on python for my current project and constantly longing for R's syntax specifically for cleaning data. It's so much better. Just last night, I wanted to perform an anti-join to find discrepancies between two data sets for debugging One line in R: anti_join(a_tibble, another_tibble, by = c("id_col1", "id_col2")) Witchcraft in Pandas: (from stack overflow): Method 1 # Identify what values are in TableB and not in TableA key_diff = set(TableB.Key).difference(TableA.Key) where_diff = TableB.Key.isin(key_diff) # Slice TableB accordingly and append to TableA TableA.append(TableB[where_diff], ignore_index=True) Method 2: rows = [] for i, row in TableB.iterrows(): if row.Key not in TableA.Key.values: rows.append(row) pd.concat([TableA.T] + rows, axis=1).T I actually fired up R and re-imported the .csv data just for this. Took 15 secs while my colleague was still stuck debugging his own weird for loops.

dwrodri 7y ago
Great example! My first thought is that anti-join could be the basis of a "csv_diff $1 $2" shell function. I have a hunch that there could be a really good follow-up post to this that takes these R hacks to the next level by extending it to work better with pre-structured text (where R really shines) and CSV files as arguments.
isoprophlex 7y ago
Pyspark example: dataframe.join( something_else, how='left_anti' ) pandas is such a shitshow. Every time i use it, im in a world of pain googling the finicky syntax for selecting columns, aggregating, filtering. I never touched R but pandas is so terrible for me. Nowadays it's either raw numpy arrays, plain sql or pyspark...
dmix 7y ago
Nothing beats building stuff in software that just works with a simple small interface while being powerful! May I ask what you use R for? I like learning languages for fun and I've been meaning to do some NBA analytics stuff. I'd love to have a REPL style interface to just do one-off math and analytics or short scripts. I haven't dug into the data science stuff yet but I'm disinterested in Python for some reason (maybe because I used to write Ruby for a living).
- mushufasa 7y ago
  I started using R because I needed a better tool for formal statistical analysis. (Econometrics, didn't want to pay for STATA. Much better packages around variants of linear regressions + panel/time series data than python). Since, I've used it for some random scripts, data visualizations, and financial analysis (Josh Ulrich's packages + tidyquant). R is a thoroughbred at doing data analysis from your laptop. It's bad at living on a server and operating any sort of app. For your use case (NBA analytics) I think it is a great tool and would recommend R. In fact, here are a couple packages/tutorials specifically for doing that: http://asbcllc.com/nbastatR/ http://asbcllc.com/nbastatR/ https://blog.revolutionanalytics.com/2016/09/analyzing-nba-basketball-data-with-r.html https://blog.revolutionanalytics.com/2016/09/analyzing-nba-b...
gammarator 7y ago
That's a numpy one-liner. np.setdiff1d(array_one, array_two)
- tfehring 7y ago
  This is completely different functionality than the parent comment's code. It's only joining on one "column", not two, and it only returns the values in that "column", while the parent's code returns complete rows from the dataframe.
oarabbus_ 7y ago
>Just last night, I wanted to perform an anti-join to find discrepancies between two data sets for debugging Sounds like a SQL use case to me: select * from table_1 except select * from table_2 (one of a half dozen ways you could do this)
just_myles 7y ago
"Took 15 secs while my colleague was still stuck debugging his own weird for loops." This is a fair response.
cuchoi 7y ago
Isn't this the same as an outer join that filters the ones that do not match? In pandas: outer_join = df1.merge(df2, left_on='id_col1', right_on='id_col2', indicator=True, how='outer') outer_join[outer_join['_merge'] == 'right_only']

dwrodri 7y ago

Great example! My first thought is that anti-join could be the basis of a "csv_diff $1 $2" shell function. I have a hunch that there could be a really good follow-up post to this that takes these R hacks to the next level by extending it to work better with pre-structured text (where R really shines) and CSV files as arguments.

isoprophlex 7y ago

Pyspark example: dataframe.join( something_else, how='left_anti' ) pandas is such a shitshow. Every time i use it, im in a world of pain googling the finicky syntax for selecting columns, aggregating, filtering. I never touched R but pandas is so terrible for me. Nowadays it's either raw numpy arrays, plain sql or pyspark...

dmix 7y ago

Nothing beats building stuff in software that just works with a simple small interface while being powerful! May I ask what you use R for? I like learning languages for fun and I've been meaning to do some NBA analytics stuff. I'd love to have a REPL style interface to just do one-off math and analytics or short scripts. I haven't dug into the data science stuff yet but I'm disinterested in Python for some reason (maybe because I used to write Ruby for a living).

mushufasa 7y ago
I started using R because I needed a better tool for formal statistical analysis. (Econometrics, didn't want to pay for STATA. Much better packages around variants of linear regressions + panel/time series data than python). Since, I've used it for some random scripts, data visualizations, and financial analysis (Josh Ulrich's packages + tidyquant). R is a thoroughbred at doing data analysis from your laptop. It's bad at living on a server and operating any sort of app. For your use case (NBA analytics) I think it is a great tool and would recommend R. In fact, here are a couple packages/tutorials specifically for doing that: http://asbcllc.com/nbastatR/ http://asbcllc.com/nbastatR/ https://blog.revolutionanalytics.com/2016/09/analyzing-nba-basketball-data-with-r.html https://blog.revolutionanalytics.com/2016/09/analyzing-nba-b...

gammarator 7y ago

That's a numpy one-liner. np.setdiff1d(array_one, array_two)

tfehring 7y ago
This is completely different functionality than the parent comment's code. It's only joining on one "column", not two, and it only returns the values in that "column", while the parent's code returns complete rows from the dataframe.

oarabbus_ 7y ago

>Just last night, I wanted to perform an anti-join to find discrepancies between two data sets for debugging Sounds like a SQL use case to me: select * from table_1 except select * from table_2 (one of a half dozen ways you could do this)

just_myles 7y ago

"Took 15 secs while my colleague was still stuck debugging his own weird for loops." This is a fair response.

cuchoi 7y ago

Isn't this the same as an outer join that filters the ones that do not match? In pandas: outer_join = df1.merge(df2, left_on='id_col1', right_on='id_col2', indicator=True, how='outer') outer_join[outer_join['_merge'] == 'right_only']