11 ms·

R is a joy if you treat it like Awk

168 points by dwrodri 7y ago

dwrodri 7y ago
OP Here: I’m aware of some CSS issues on mobile and will work to fix them shortly. UPDATE: fixed color palette issue, although it still looks quite bad.
- thebelal 7y ago
  FWIW `scan()` is going to be much faster if you are just reading doubles from standard input than `as.numeric(readLines())`.
dredmorbius 7y ago
The precursor to R, S, was designed as a Unix tool, and pretty much implicitly relied on awk as its data-cleaning preprocessor. For those who come from the world of large enterprise statistical and data reporting tools such as SAS, awk shares an exceedingly strong resemblance to the SAS DATA step, whilst R effectively provides a host of analysis and graphics tools the correspond to numerous other SAS procedure and products. The hacks for pipelining R are cool and useful. Thanks.
bllguo 7y ago
> In my experience, just about anything beats R when it comes to cleaning dirty data. > I used to resent R, it was shoved upon me as a strange tool that promised to replace Python, but failed miserably. well I'm glad the author found a way to make R work for their purposes, but this just reeks of inexperience with R..
- dwrodri 7y ago
  I am definitely inexperienced with R. The intention of this post was to highlight a key point of friction I encountered throughout my introduction to the language. Namely, I was introduced to R (by someone more senior than myself) as "data science in a runtime/IDE", but there quite finite boundaries where the conveniences of R stop and other tools begin. What motivated me to write this post was the lack of discussion of R in this use case. I actually stumbled into using "Rscript -e" one liners while looking to do basic stats in a Linux CLI. That being said, I still stand by my point. Taking data from "out in the wild" (log files, tarballs of images, unstructured text) and making use of it can be frustrating in R because cleaning up edge cases, removing unwanted data, and getting everything into the correct container/type often involved unintuitive chaining of function calls. This is coming from the perspective of someone who worked with Python/awk/sed prior to being exposed to R. If you have good counterarguments, I'd be more than happy to hear them and addresss them in an end note in the post.
  - mushufasa 7y ago
    Have you ever used the tidyverse ecosystem? It's a set of syntacticly-compatible package authored chiefly by one man, which have evolved into a superset of R, if not an outright R2. (R's Scheme lineage makes it very hackable in this way). I've working on python for my current project and constantly longing for R's syntax specifically for cleaning data. It's so much better. Just last night, I wanted to perform an anti-join to find discrepancies between two data sets for debugging One line in R: anti_join(a_tibble, another_tibble, by = c("id_col1", "id_col2")) Witchcraft in Pandas: (from stack overflow): Method 1 # Identify what values are in TableB and not in TableA key_diff = set(TableB.Key).difference(TableA.Key) where_diff = TableB.Key.isin(key_diff) # Slice TableB accordingly and append to TableA TableA.append(TableB[where_diff], ignore_index=True) Method 2: rows = [] for i, row in TableB.iterrows(): if row.Key not in TableA.Key.values: rows.append(row) pd.concat([TableA.T] + rows, axis=1).T I actually fired up R and re-imported the .csv data just for this. Took 15 secs while my colleague was still stuck debugging his own weird for loops.
    - dwrodri 7y ago
      Great example! My first thought is that anti-join could be the basis of a "csv_diff $1 $2" shell function. I have a hunch that there could be a really good follow-up post to this that takes these R hacks to the next level by extending it to work better with pre-structured text (where R really shines) and CSV files as arguments.
    - isoprophlex 7y ago
      Pyspark example: dataframe.join( something_else, how='left_anti' ) pandas is such a shitshow. Every time i use it, im in a world of pain googling the finicky syntax for selecting columns, aggregating, filtering. I never touched R but pandas is so terrible for me. Nowadays it's either raw numpy arrays, plain sql or pyspark...
    - dmix 7y ago
      Nothing beats building stuff in software that just works with a simple small interface while being powerful! May I ask what you use R for? I like learning languages for fun and I've been meaning to do some NBA analytics stuff. I'd love to have a REPL style interface to just do one-off math and analytics or short scripts. I haven't dug into the data science stuff yet but I'm disinterested in Python for some reason (maybe because I used to write Ruby for a living).
wodenokoto 7y ago
I haven't worked with logs, but I do find R a joy to work with in general, especially the tidy verse is a joy to use, but slow and memory hungry on very large datasets. I haven't found a good way around very inconsistently formatted csv files in any language (a row only represents column 3,4 and 6 if it starts with a comma, all other rows have all columns, but are space separated and values may contain comma, etc, etc)
isostatic 7y ago
Something to bypass while you reach for your Perl interpreter?
crispyambulance 7y ago
The regex stuff in stringr, and read_csv from readr are adequate for almost anything if you're talking about ordinary delimited file import. What does awk give you that these don't?
iron0013 7y ago
I use R a lot, and nothing in this post rings even the slightest bit true to me. This guy's use case for R seems to be very unlike almost anyone I've ever heard of. Towards the end of the article, when he's talking about how all he really needs from R is the summary() and boxplot() functions, it really becomes clear that he's never done anything more than dip the very tip of his pinky toe into R. There are lots of valid criticisms of R, but this article doesn't touch on them. It's so off-base that it's the proverbial "not even wrong".
- dwrodri 7y ago
  I appreciate the feedback! There is definitely more that R offers than what I discuss here. In retrospect, I will be more restrained on my opinions when I have little experience in my pocket. That being said, it was absolutely my intention to present CLI that deviate from R's intended use. There is already plenty out there on R's intended use. What's that adage that goes something like, "if you want to get an answer on the Internet, don't pose a question..."?
  - iron0013 7y ago
    I should also say that I didn't mean to sound so curt, and would have chosen my words more carefully if I'd known the author was here in the thread!
    - dwrodri 7y ago
      No offense taken! I'm relatively new to technical writing and the discussion from all of these comments (yours included) has been really helpful for guiding how I write future posts. HN is (in my experience) a lot more cynical and straightforward than other places[1]. It's something I've learned to appreciate and also take it with a grain of salt. 1 = https://news.ycombinator.com/item?id=8863 https://news.ycombinator.com/item?id=8863
      - jrumbut 7y ago
        This is incredibly gracious feedback taking, a lesson in how not to be defensive. I agree with you where some here don't. I think there are often better tools for any single data cleaning task. R's strength is being second best at an enormous range of tasks (and often being first to get new techniques) and packaging that with analysis and visualization.
safgasCVS 7y ago
I’ve recently discovered the pipe command allowing R to consume the result of a terminal command’s output (link: https://youtu.be/RYhwZW6ofbI https://youtu.be/RYhwZW6ofbI). Quite useful for reading in compressed files and it’s made me learn a bit of awk to fix text files on the fly Oh and there’s also Rio if you want to explore injecting R into your command line workflow.
Myrmornis 7y ago
dwodri: Not sure if you're a MacOS user, but iTerm2 distributes a script to display images inline in the terminal. I've forgotten how to get R to output png data directly to stdout, but it could be nice to do that and display the images inline in the terminal https://www.iterm2.com/documentation-images.html https://www.iterm2.com/documentation-images.html
- dwrodri 7y ago
  I do in fact use macOS, but I'm more preferential towards kitty[1]. Sadly, OpenGL is getting deprecated in macOS Catalina so I will probably crawl back to iTerm2 eventually. I'm quite familiar with asciinema[2] as well, which I've considered using for creating animated examples. In general, I like to err on the side of caution, and optimize my website for people with poor/metered connections. Also, I didn't expect this to get this much attention! Mea culpa for not putting more work into the post itself. 1 = https://sw.kovidgoyal.net/kitty/index.html https://sw.kovidgoyal.net/kitty/index.html 2 = https://asciinema.org/ https://asciinema.org/
  - Myrmornis 7y ago
    We might be miscommunicating! Or if not, sorry if I didn't understand what you said in reply. What I was trying to say is: how about extending your blog post with one line to show people how to make the boxplot appear in their terminal as soon as they run the command? (I think at the moment it's going to appear as a PDF file named Rplots.pdf in the current directory, right?)
    - dwrodri 7y ago
      This is correct. A good cross platform tool for rendering images in many modern terminal emulators is imgcat[1]. iTerm2 appears to use this under the hood. 1 = https://github.com/eddieantonio/imgcat https://github.com/eddieantonio/imgcat
- daleroberts 7y ago
  This has been done in the Python ecosystem with matplotlib: https://github.com/daleroberts/itermplot https://github.com/daleroberts/itermplot
rcthompson 7y ago
R has lots of packages available for data cleaning. They just aren't necessarily included in base R. Most of the ones I'm aware of are in the Tidyverse. Even if you're dealing with a hand-written Excel file with multiple tables in the same sheet and various information encoded in text/background colors and such, there's they tidyxl package to help you do that.
tylermw 7y ago
"R is only great if your data is already great." Disagree. R has an incredible ecosystem for parsing, cleaning, and manipulating data. Even ignoring the tidyverse, base R provides more than enough functionality to clean and analyze data--you just need to spend some time learning it. If you use the tidyverse, it's even easier. The only other ecosystem that comes close to R is Julia, which was designed taking many of the best parts of R into consideration.
- dwrodri 7y ago
  Good point. The original draft included this as a value proposition because I generally find that bold opinions are a good starting point for developing valuable insights. Sometimes, that insight comes from those with dissenting opinions. I've since edited the post to emphasize more "R can be a great CLI tool" and less "cleaning data in R is bad".
- wodenokoto 7y ago
  How would you go about parsing logs in R? The problem with logs is that lines have different number of columns depending on what that line is logging. What a given line is logging can usually be determined by a regex (e.g. ends with a ip-address, starts with this words, etc., etc) I'm genuinely curious, as this is a use-case where I have a lot of difficulty using python or R. I can see how grep + awk are strong as a preprocessor, as they can scan through a larger than memory file, and select columns from rows that match your criteria.
  - tylermw 7y ago
    I was just doing this yesterday--the "base R" way would be to simply use readLines() and grep through the same way you would in the command line. You're right though, grep + awk are super useful and parsing larger-than-memory logs is where they really shine and the shortcomings of R become apparent. However, if your data fits in memory and you have a non-trivial analysis to perform, R is a great choice. I had a file with tabular data interspersed with metadata at random points and it was straightforward to parse and store the data in a custom data structure.
- 7y ago
chubot 7y ago
I get what the author is saying -- I use R in shell scripts too. It's really useful and composes well with other shell tools. I also get what the commenters are saying, because R is a useful interactive language too, and it's also pretty good for data cleaning. Although it's significantly slower than Python, which is why I do all cleaning that cuts down the data before loading it into R. As an example, I generate some benchmarks with every release of Oil: https://www.oilshell.org/release/0.7.pre5/benchmarks.wwz/osh-parser/ https://www.oilshell.org/release/0.7.pre5/benchmarks.wwz/osh... and the tables are manipulated with R, but running the benchmarks is done with shell, and creating the HTML is done with Python: https://github.com/oilshell/oil/blob/master/benchmarks/report.R https://github.com/oilshell/oil/blob/master/benchmarks/repor... (and yes this page is meant to be motivation to speed up my principled but slow shell parser) R and tidyverse are the best tools for manipulating tables by far. I wrote an intro here: What Is a Data Frame? (In Python, R, and SQL) http://www.oilshell.org/blog/2018/11/30.html http://www.oilshell.org/blog/2018/11/30.html
larrydag 7y ago
To be honest I don't find Python to be very useful for data cleansing. Full disclosure I use R quite a bit and use Python sparingly. If you want to use R for data analysis I suggest the R package Tidyverse or use a product like OpenRefine. If you just need to run summary statistics with an application then using Python is just fine.
- objektif 7y ago
  What part of it you dont like? I use Python for data manipulation on a daily basis and I find it very useful.
  - psv1 7y ago
    The pandas syntax is far less intuitive than R dataframes, especially if you're also using dplyr.
    - larrydag 7y ago
      Agree. Pandas seems to be a mixture of Python indexing syntax and R dataframes.
- collyw 7y ago
  Perl is way better at that sort of task.
halfeatenpie 7y ago
I use R on a daily level. A significant portion of our climate change adaptation research code and decision planning systems (which are used in various utilities around the world to support decision makers) are built using R. I understand what the author is stating, but I just feel like this is from inexperience with R and ignoring the vast amount of packages available within it that are specifically targeted at data science. There are some valid issues and criticisms of R, but I think this article only focuses on the application of R in a single context. A significant portion of data science is about cleaning the data and an entire suite of packages known as tidyverse solves these problems (for me anyways) while also being very simple and easy to understand. I mean tidyverse supports piping, which is exactly what this article is saying to use. Obviously your mileage may vary, but this post irks me the wrong way.
- throwaway66920 7y ago
  The tidyverse is excellent, and has its own stylistic choices that are arguably quite good. A lot of other random packages also have their own stylistic choices and they are not good. So I suppose I would say that R makes it really easy to write data pipelines well, but also really hard to write them poorly; and doesn’t make it very obvious on what the better choices are.
  - halfeatenpie 7y ago
    I agree. But I'd also argue that it's based on how you're using the tools you're given (or what others have given you). Pretty much the argument of using apply vs for loops. Most of the time, apply is going to be significantly more computationally efficient. However, some people think of the problem in different context than others, and if it works for them then it works.
  - scottlocklin 7y ago
    If only Hadley hadn't picked snake-case, which doesn't work so well out of the box with ESS. Yah, I know there is a way of fixing this with a setq in ESS, but I shouldn't have to.
joker3 7y ago
Rscript can also be used to run R scripts from the command line. Just put >#!/usr/bin/env Rscript on the first line.
yiyus 7y ago
awk '/Recovery time:/{print $2}' output.log | boxplot
oarabbus_ 7y ago
R isn't a joy, but it is powerful. This is the best blog post I've read on the matter: https://www.talyarkoni.org/blog/2012/06/08/r-the-master-troll-of-statistical-languages/comment-page-1/ https://www.talyarkoni.org/blog/2012/06/08/r-the-master-trol...
hkt 7y ago
Awk is one of the few tools that possesses the ability to make me beg for death.
dfgdghdf 7y ago
Lots of people don't realize this, but R is very similar to JavaScript. Shiny apps offer a nice functional web framework too.
- collyw 7y ago
  That gives me even less incentive to try it out.
CreRecombinase 7y ago
If you want to treat R like awk, you should really check out the littler package, a super useful R package which provides an alternative to both Rscript and R CMD BATCH designed for writing one-liners https://github.com/eddelbuettel/littler https://github.com/eddelbuettel/littler
just_myles 7y ago
I used to use awk all the time whenever I wanted to evaluate specific portions of a columnar file (With a short one liner). Python comes close but, awk is much faster. In conjunction with SED, I think that is a little much and imo exceeds what I was doing. If I need to do any replacing or 'cleaning', I would then use something like Python or SQL.
dima55 7y ago
R is really overkill here. Simpler and nicer tools exist if you want to munge data on the command line and/or plot it. For instance: https://github.com/dkogan/feedgnuplot/ https://github.com/dkogan/feedgnuplot/ https://github.com/dkogan/vnlog/ https://github.com/dkogan/vnlog/
- dwrodri 7y ago
  I love CLI tools like these. Great find!
goodside 7y ago
For historical context, R was not initially intended to be a standalone scripting language runnable via POSIX conventions and didn’t gain these features until circa 2010. R is designed to be used interactively, like the S language it was based on, a style later made familiar to students via MATLAB and TI graphing calculators. Before Rscript it was common to hack together shell scripts that cope with the expectation of human TTY input to run R in production, which functioned well as a “you must be this tall to ride” sign for people who might not appreciate how unreliable R code can be compared to a traditional language.
xiaodai 7y ago
"R is a joy if you don't use it." Joke. I am a heavy R user.
alexhutcheson 7y ago
If you're interested in learning how to do data cleaning and restructuring in R, I highly recommend the "Wrangle" chapters of Hadley Wickham's R for Data Science book, which you can read online here: https://r4ds.had.co.nz/wrangle-intro.html https://r4ds.had.co.nz/wrangle-intro.html
jwilber 7y ago
I don’t want to post something inflammatory or condescending, but it’s very clearly that the author does not have experience using R.
arminiusreturns 7y ago
Here is how I like to use R, which can use the authors more introductory methods or full fledged data crunch: Like I do all my other data sources, as code blocks inside an emacs-org notebook. If you are doing data science, you quickly find that it's management and combination of the various particular projects that becomes the most daunting (imho), and your data science notebook becomes the most important part of that organization. In that arena for me it's pretty much either jupyter or emacs org-mode.
mikorym 7y ago
So, this is not about the article. I've recently started using rpy2 as a python wrapper for R functions and libraries and I'm finding that it is not that bad. There is some performance issues, but I am OK to trade that off for the convenience of using "try:" rather than R's "tryCatch". Having tryCatch as a function rather than built into the syntax is unacceptable to me. But there are some libraries in R that don't have elegant alternatives in Python or which you have multiple options or perhaps not the time to unlearn.
enriquto 7y ago
for that simple use case gnuplot is even better than R
gpvos 7y ago
> Half-assing something in Unix can produce great results as long you chose the right half of the ass. That is a great quote!

168 points by dwrodri 7y ago

OP Here: I’m aware of some CSS issues on mobile and will work to fix them shortly. UPDATE: fixed color palette issue, although it still looks quite bad.

thebelal 7y ago
FWIW `scan()` is going to be much faster if you are just reading doubles from standard input than `as.numeric(readLines())`.

dredmorbius 7y ago

The precursor to R, S, was designed as a Unix tool, and pretty much implicitly relied on awk as its data-cleaning preprocessor. For those who come from the world of large enterprise statistical and data reporting tools such as SAS, awk shares an exceedingly strong resemblance to the SAS DATA step, whilst R effectively provides a host of analysis and graphics tools the correspond to numerous other SAS procedure and products. The hacks for pipelining R are cool and useful. Thanks.

bllguo 7y ago

> In my experience, just about anything beats R when it comes to cleaning dirty data. > I used to resent R, it was shoved upon me as a strange tool that promised to replace Python, but failed miserably. well I'm glad the author found a way to make R work for their purposes, but this just reeks of inexperience with R..

dwrodri 7y ago
I am definitely inexperienced with R. The intention of this post was to highlight a key point of friction I encountered throughout my introduction to the language. Namely, I was introduced to R (by someone more senior than myself) as "data science in a runtime/IDE", but there quite finite boundaries where the conveniences of R stop and other tools begin. What motivated me to write this post was the lack of discussion of R in this use case. I actually stumbled into using "Rscript -e" one liners while looking to do basic stats in a Linux CLI. That being said, I still stand by my point. Taking data from "out in the wild" (log files, tarballs of images, unstructured text) and making use of it can be frustrating in R because cleaning up edge cases, removing unwanted data, and getting everything into the correct container/type often involved unintuitive chaining of function calls. This is coming from the perspective of someone who worked with Python/awk/sed prior to being exposed to R. If you have good counterarguments, I'd be more than happy to hear them and addresss them in an end note in the post.
- mushufasa 7y ago
  Have you ever used the tidyverse ecosystem? It's a set of syntacticly-compatible package authored chiefly by one man, which have evolved into a superset of R, if not an outright R2. (R's Scheme lineage makes it very hackable in this way). I've working on python for my current project and constantly longing for R's syntax specifically for cleaning data. It's so much better. Just last night, I wanted to perform an anti-join to find discrepancies between two data sets for debugging One line in R: anti_join(a_tibble, another_tibble, by = c("id_col1", "id_col2")) Witchcraft in Pandas: (from stack overflow): Method 1 # Identify what values are in TableB and not in TableA key_diff = set(TableB.Key).difference(TableA.Key) where_diff = TableB.Key.isin(key_diff) # Slice TableB accordingly and append to TableA TableA.append(TableB[where_diff], ignore_index=True) Method 2: rows = [] for i, row in TableB.iterrows(): if row.Key not in TableA.Key.values: rows.append(row) pd.concat([TableA.T] + rows, axis=1).T I actually fired up R and re-imported the .csv data just for this. Took 15 secs while my colleague was still stuck debugging his own weird for loops.
  - dwrodri 7y ago
    Great example! My first thought is that anti-join could be the basis of a "csv_diff $1 $2" shell function. I have a hunch that there could be a really good follow-up post to this that takes these R hacks to the next level by extending it to work better with pre-structured text (where R really shines) and CSV files as arguments.
  - isoprophlex 7y ago
    Pyspark example: dataframe.join( something_else, how='left_anti' ) pandas is such a shitshow. Every time i use it, im in a world of pain googling the finicky syntax for selecting columns, aggregating, filtering. I never touched R but pandas is so terrible for me. Nowadays it's either raw numpy arrays, plain sql or pyspark...
  - dmix 7y ago
    Nothing beats building stuff in software that just works with a simple small interface while being powerful! May I ask what you use R for? I like learning languages for fun and I've been meaning to do some NBA analytics stuff. I'd love to have a REPL style interface to just do one-off math and analytics or short scripts. I haven't dug into the data science stuff yet but I'm disinterested in Python for some reason (maybe because I used to write Ruby for a living).

wodenokoto 7y ago

I haven't worked with logs, but I do find R a joy to work with in general, especially the tidy verse is a joy to use, but slow and memory hungry on very large datasets. I haven't found a good way around very inconsistently formatted csv files in any language (a row only represents column 3,4 and 6 if it starts with a comma, all other rows have all columns, but are space separated and values may contain comma, etc, etc)

isostatic 7y ago

Something to bypass while you reach for your Perl interpreter?

crispyambulance 7y ago

The regex stuff in stringr, and read_csv from readr are adequate for almost anything if you're talking about ordinary delimited file import. What does awk give you that these don't?

iron0013 7y ago

I use R a lot, and nothing in this post rings even the slightest bit true to me. This guy's use case for R seems to be very unlike almost anyone I've ever heard of. Towards the end of the article, when he's talking about how all he really needs from R is the summary() and boxplot() functions, it really becomes clear that he's never done anything more than dip the very tip of his pinky toe into R. There are lots of valid criticisms of R, but this article doesn't touch on them. It's so off-base that it's the proverbial "not even wrong".

dwrodri 7y ago
I appreciate the feedback! There is definitely more that R offers than what I discuss here. In retrospect, I will be more restrained on my opinions when I have little experience in my pocket. That being said, it was absolutely my intention to present CLI that deviate from R's intended use. There is already plenty out there on R's intended use. What's that adage that goes something like, "if you want to get an answer on the Internet, don't pose a question..."?
- iron0013 7y ago
  I should also say that I didn't mean to sound so curt, and would have chosen my words more carefully if I'd known the author was here in the thread!
  - dwrodri 7y ago
    No offense taken! I'm relatively new to technical writing and the discussion from all of these comments (yours included) has been really helpful for guiding how I write future posts. HN is (in my experience) a lot more cynical and straightforward than other places[1]. It's something I've learned to appreciate and also take it with a grain of salt. 1 = https://news.ycombinator.com/item?id=8863 https://news.ycombinator.com/item?id=8863
    - jrumbut 7y ago
      This is incredibly gracious feedback taking, a lesson in how not to be defensive. I agree with you where some here don't. I think there are often better tools for any single data cleaning task. R's strength is being second best at an enormous range of tasks (and often being first to get new techniques) and packaging that with analysis and visualization.

safgasCVS 7y ago

I’ve recently discovered the pipe command allowing R to consume the result of a terminal command’s output (link: https://youtu.be/RYhwZW6ofbI https://youtu.be/RYhwZW6ofbI). Quite useful for reading in compressed files and it’s made me learn a bit of awk to fix text files on the fly Oh and there’s also Rio if you want to explore injecting R into your command line workflow.

Myrmornis 7y ago

dwodri: Not sure if you're a MacOS user, but iTerm2 distributes a script to display images inline in the terminal. I've forgotten how to get R to output png data directly to stdout, but it could be nice to do that and display the images inline in the terminal https://www.iterm2.com/documentation-images.html https://www.iterm2.com/documentation-images.html

dwrodri 7y ago
I do in fact use macOS, but I'm more preferential towards kitty[1]. Sadly, OpenGL is getting deprecated in macOS Catalina so I will probably crawl back to iTerm2 eventually. I'm quite familiar with asciinema[2] as well, which I've considered using for creating animated examples. In general, I like to err on the side of caution, and optimize my website for people with poor/metered connections. Also, I didn't expect this to get this much attention! Mea culpa for not putting more work into the post itself. 1 = https://sw.kovidgoyal.net/kitty/index.html https://sw.kovidgoyal.net/kitty/index.html 2 = https://asciinema.org/ https://asciinema.org/
- Myrmornis 7y ago
  We might be miscommunicating! Or if not, sorry if I didn't understand what you said in reply. What I was trying to say is: how about extending your blog post with one line to show people how to make the boxplot appear in their terminal as soon as they run the command? (I think at the moment it's going to appear as a PDF file named Rplots.pdf in the current directory, right?)
  - dwrodri 7y ago
    This is correct. A good cross platform tool for rendering images in many modern terminal emulators is imgcat[1]. iTerm2 appears to use this under the hood. 1 = https://github.com/eddieantonio/imgcat https://github.com/eddieantonio/imgcat
daleroberts 7y ago
This has been done in the Python ecosystem with matplotlib: https://github.com/daleroberts/itermplot https://github.com/daleroberts/itermplot

rcthompson 7y ago

R has lots of packages available for data cleaning. They just aren't necessarily included in base R. Most of the ones I'm aware of are in the Tidyverse. Even if you're dealing with a hand-written Excel file with multiple tables in the same sheet and various information encoded in text/background colors and such, there's they tidyxl package to help you do that.

tylermw 7y ago

"R is only great if your data is already great." Disagree. R has an incredible ecosystem for parsing, cleaning, and manipulating data. Even ignoring the tidyverse, base R provides more than enough functionality to clean and analyze data--you just need to spend some time learning it. If you use the tidyverse, it's even easier. The only other ecosystem that comes close to R is Julia, which was designed taking many of the best parts of R into consideration.

dwrodri 7y ago
Good point. The original draft included this as a value proposition because I generally find that bold opinions are a good starting point for developing valuable insights. Sometimes, that insight comes from those with dissenting opinions. I've since edited the post to emphasize more "R can be a great CLI tool" and less "cleaning data in R is bad".
wodenokoto 7y ago
How would you go about parsing logs in R? The problem with logs is that lines have different number of columns depending on what that line is logging. What a given line is logging can usually be determined by a regex (e.g. ends with a ip-address, starts with this words, etc., etc) I'm genuinely curious, as this is a use-case where I have a lot of difficulty using python or R. I can see how grep + awk are strong as a preprocessor, as they can scan through a larger than memory file, and select columns from rows that match your criteria.
- tylermw 7y ago
  I was just doing this yesterday--the "base R" way would be to simply use readLines() and grep through the same way you would in the command line. You're right though, grep + awk are super useful and parsing larger-than-memory logs is where they really shine and the shortcomings of R become apparent. However, if your data fits in memory and you have a non-trivial analysis to perform, R is a great choice. I had a file with tabular data interspersed with metadata at random points and it was straightforward to parse and store the data in a custom data structure.
7y ago

chubot 7y ago

I get what the author is saying -- I use R in shell scripts too. It's really useful and composes well with other shell tools. I also get what the commenters are saying, because R is a useful interactive language too, and it's also pretty good for data cleaning. Although it's significantly slower than Python, which is why I do all cleaning that cuts down the data before loading it into R. As an example, I generate some benchmarks with every release of Oil: https://www.oilshell.org/release/0.7.pre5/benchmarks.wwz/osh-parser/ https://www.oilshell.org/release/0.7.pre5/benchmarks.wwz/osh... and the tables are manipulated with R, but running the benchmarks is done with shell, and creating the HTML is done with Python: https://github.com/oilshell/oil/blob/master/benchmarks/report.R https://github.com/oilshell/oil/blob/master/benchmarks/repor... (and yes this page is meant to be motivation to speed up my principled but slow shell parser) R and tidyverse are the best tools for manipulating tables by far. I wrote an intro here: What Is a Data Frame? (In Python, R, and SQL) http://www.oilshell.org/blog/2018/11/30.html http://www.oilshell.org/blog/2018/11/30.html

larrydag 7y ago

To be honest I don't find Python to be very useful for data cleansing. Full disclosure I use R quite a bit and use Python sparingly. If you want to use R for data analysis I suggest the R package Tidyverse or use a product like OpenRefine. If you just need to run summary statistics with an application then using Python is just fine.

objektif 7y ago
What part of it you dont like? I use Python for data manipulation on a daily basis and I find it very useful.
- psv1 7y ago
  The pandas syntax is far less intuitive than R dataframes, especially if you're also using dplyr.
  - larrydag 7y ago
    Agree. Pandas seems to be a mixture of Python indexing syntax and R dataframes.
collyw 7y ago
Perl is way better at that sort of task.

halfeatenpie 7y ago

I use R on a daily level. A significant portion of our climate change adaptation research code and decision planning systems (which are used in various utilities around the world to support decision makers) are built using R. I understand what the author is stating, but I just feel like this is from inexperience with R and ignoring the vast amount of packages available within it that are specifically targeted at data science. There are some valid issues and criticisms of R, but I think this article only focuses on the application of R in a single context. A significant portion of data science is about cleaning the data and an entire suite of packages known as tidyverse solves these problems (for me anyways) while also being very simple and easy to understand. I mean tidyverse supports piping, which is exactly what this article is saying to use. Obviously your mileage may vary, but this post irks me the wrong way.

throwaway66920 7y ago
The tidyverse is excellent, and has its own stylistic choices that are arguably quite good. A lot of other random packages also have their own stylistic choices and they are not good. So I suppose I would say that R makes it really easy to write data pipelines well, but also really hard to write them poorly; and doesn’t make it very obvious on what the better choices are.
- halfeatenpie 7y ago
  I agree. But I'd also argue that it's based on how you're using the tools you're given (or what others have given you). Pretty much the argument of using apply vs for loops. Most of the time, apply is going to be significantly more computationally efficient. However, some people think of the problem in different context than others, and if it works for them then it works.
- scottlocklin 7y ago
  If only Hadley hadn't picked snake-case, which doesn't work so well out of the box with ESS. Yah, I know there is a way of fixing this with a setq in ESS, but I shouldn't have to.

joker3 7y ago

Rscript can also be used to run R scripts from the command line. Just put >#!/usr/bin/env Rscript on the first line.

yiyus 7y ago

awk '/Recovery time:/{print $2}' output.log | boxplot

oarabbus_ 7y ago

R isn't a joy, but it is powerful. This is the best blog post I've read on the matter: https://www.talyarkoni.org/blog/2012/06/08/r-the-master-troll-of-statistical-languages/comment-page-1/ https://www.talyarkoni.org/blog/2012/06/08/r-the-master-trol...

hkt 7y ago

Awk is one of the few tools that possesses the ability to make me beg for death.

dfgdghdf 7y ago

Lots of people don't realize this, but R is very similar to JavaScript. Shiny apps offer a nice functional web framework too.

collyw 7y ago
That gives me even less incentive to try it out.

CreRecombinase 7y ago

If you want to treat R like awk, you should really check out the littler package, a super useful R package which provides an alternative to both Rscript and R CMD BATCH designed for writing one-liners https://github.com/eddelbuettel/littler https://github.com/eddelbuettel/littler

just_myles 7y ago

I used to use awk all the time whenever I wanted to evaluate specific portions of a columnar file (With a short one liner). Python comes close but, awk is much faster. In conjunction with SED, I think that is a little much and imo exceeds what I was doing. If I need to do any replacing or 'cleaning', I would then use something like Python or SQL.

dima55 7y ago

R is really overkill here. Simpler and nicer tools exist if you want to munge data on the command line and/or plot it. For instance: https://github.com/dkogan/feedgnuplot/ https://github.com/dkogan/feedgnuplot/ https://github.com/dkogan/vnlog/ https://github.com/dkogan/vnlog/

dwrodri 7y ago
I love CLI tools like these. Great find!

goodside 7y ago

For historical context, R was not initially intended to be a standalone scripting language runnable via POSIX conventions and didn’t gain these features until circa 2010. R is designed to be used interactively, like the S language it was based on, a style later made familiar to students via MATLAB and TI graphing calculators. Before Rscript it was common to hack together shell scripts that cope with the expectation of human TTY input to run R in production, which functioned well as a “you must be this tall to ride” sign for people who might not appreciate how unreliable R code can be compared to a traditional language.

xiaodai 7y ago

"R is a joy if you don't use it." Joke. I am a heavy R user.

alexhutcheson 7y ago

If you're interested in learning how to do data cleaning and restructuring in R, I highly recommend the "Wrangle" chapters of Hadley Wickham's R for Data Science book, which you can read online here: https://r4ds.had.co.nz/wrangle-intro.html https://r4ds.had.co.nz/wrangle-intro.html

jwilber 7y ago

I don’t want to post something inflammatory or condescending, but it’s very clearly that the author does not have experience using R.

arminiusreturns 7y ago

Here is how I like to use R, which can use the authors more introductory methods or full fledged data crunch: Like I do all my other data sources, as code blocks inside an emacs-org notebook. If you are doing data science, you quickly find that it's management and combination of the various particular projects that becomes the most daunting (imho), and your data science notebook becomes the most important part of that organization. In that arena for me it's pretty much either jupyter or emacs org-mode.

mikorym 7y ago

So, this is not about the article. I've recently started using rpy2 as a python wrapper for R functions and libraries and I'm finding that it is not that bad. There is some performance issues, but I am OK to trade that off for the convenience of using "try:" rather than R's "tryCatch". Having tryCatch as a function rather than built into the syntax is unacceptable to me. But there are some libraries in R that don't have elegant alternatives in Python or which you have multiple options or perhaps not the time to unlearn.

enriquto 7y ago

for that simple use case gnuplot is even better than R

gpvos 7y ago

> Half-assing something in Unix can produce great results as long you chose the right half of the ass. That is a great quote!