3 ms·

"R is only great if your data is already great." Disagree. R has an incredible ecosystem for parsing, cleaning, and manipulating data. Even ignoring the tidyve

"R is only great if your data is already great." Disagree. R has an incredible ecosystem for parsing, cleaning, and manipulating data. Even ignoring the tidyverse, base R provides more than enough functionality to clean and analyze data--you just need to spend some time learning it. If you use the tidyverse, it's even easier. The only other ecosystem that comes close to R is Julia, which was designed taking many of the best parts of R into consideration.

dwrodri 7y ago
Good point. The original draft included this as a value proposition because I generally find that bold opinions are a good starting point for developing valuable insights. Sometimes, that insight comes from those with dissenting opinions. I've since edited the post to emphasize more "R can be a great CLI tool" and less "cleaning data in R is bad".
wodenokoto 7y ago
How would you go about parsing logs in R? The problem with logs is that lines have different number of columns depending on what that line is logging. What a given line is logging can usually be determined by a regex (e.g. ends with a ip-address, starts with this words, etc., etc) I'm genuinely curious, as this is a use-case where I have a lot of difficulty using python or R. I can see how grep + awk are strong as a preprocessor, as they can scan through a larger than memory file, and select columns from rows that match your criteria.
- tylermw 7y ago
  I was just doing this yesterday--the "base R" way would be to simply use readLines() and grep through the same way you would in the command line. You're right though, grep + awk are super useful and parsing larger-than-memory logs is where they really shine and the shortcomings of R become apparent. However, if your data fits in memory and you have a non-trivial analysis to perform, R is a great choice. I had a file with tabular data interspersed with metadata at random points and it was straightforward to parse and store the data in a custom data structure.
_Wintermute 7y ago
R and the tidyverse are great at cleaning already nicely formatted tabular data. As someone else has already mentioned it quickly turns pretty ugly if you stray away from this.
dagw 7y ago
The only other ecosystem that comes close to R is Julia What is the Julia equivalent of the tidyverse
- Eugeleo 7y ago
  Well... Julia. Of course, the tidyverse is big and I guess there isn’t one to one ration of features between tidyverse and Julia, but mostly it is true. For example, in Julia you can work with the dataframes in a SQL/LINQ/dplyr fashion like this: x_thread = @linq df |> transform(y = 10 * :x) |> where(:a .> 2) |> by(:b, meanX = mean(:x), meanY = mean(:y)) |> orderby(:meanX) |> select(:meanX, :meanY, var = :b (sorry not sure how to format code properly on HN) Theres also a version of ggplot, although I’m not sure if its a rewrite ot just a wrapping around the R version. Edit: Actually, this I believe is a part of the DataFramesMeta.jl package... So the answer would be rather “most of the things are baked into Julia and when they arent, there’s a package for that”.

dwrodri 7y ago

Good point. The original draft included this as a value proposition because I generally find that bold opinions are a good starting point for developing valuable insights. Sometimes, that insight comes from those with dissenting opinions. I've since edited the post to emphasize more "R can be a great CLI tool" and less "cleaning data in R is bad".

wodenokoto 7y ago

How would you go about parsing logs in R? The problem with logs is that lines have different number of columns depending on what that line is logging. What a given line is logging can usually be determined by a regex (e.g. ends with a ip-address, starts with this words, etc., etc) I'm genuinely curious, as this is a use-case where I have a lot of difficulty using python or R. I can see how grep + awk are strong as a preprocessor, as they can scan through a larger than memory file, and select columns from rows that match your criteria.

tylermw 7y ago
I was just doing this yesterday--the "base R" way would be to simply use readLines() and grep through the same way you would in the command line. You're right though, grep + awk are super useful and parsing larger-than-memory logs is where they really shine and the shortcomings of R become apparent. However, if your data fits in memory and you have a non-trivial analysis to perform, R is a great choice. I had a file with tabular data interspersed with metadata at random points and it was straightforward to parse and store the data in a custom data structure.

_Wintermute 7y ago

R and the tidyverse are great at cleaning already nicely formatted tabular data. As someone else has already mentioned it quickly turns pretty ugly if you stray away from this.

dagw 7y ago

The only other ecosystem that comes close to R is Julia What is the Julia equivalent of the tidyverse

Eugeleo 7y ago
Well... Julia. Of course, the tidyverse is big and I guess there isn’t one to one ration of features between tidyverse and Julia, but mostly it is true. For example, in Julia you can work with the dataframes in a SQL/LINQ/dplyr fashion like this: x_thread = @linq df |> transform(y = 10 * :x) |> where(:a .> 2) |> by(:b, meanX = mean(:x), meanY = mean(:y)) |> orderby(:meanX) |> select(:meanX, :meanY, var = :b (sorry not sure how to format code properly on HN) Theres also a version of ggplot, although I’m not sure if its a rewrite ot just a wrapping around the R version. Edit: Actually, this I believe is a part of the DataFramesMeta.jl package... So the answer would be rather “most of the things are baked into Julia and when they arent, there’s a package for that”.