5 ms·

Rvest: Easy web scraping with R

142 points by hadley 12y ago

earino 12y ago
The community and ecosystem around R is rapidly changing and adapting. R has a long and storied history as a niche language for statistics and analysis. Much like those disciplines have entered the mainstream of modern technology enabled businesses, so follows the R ecosystem. Previously laborious tasks are being revamped with new elegant APIs such as Rvest does for scraping (and dplyr for manipulation, and lubridate for date manipulation, etc...) Performance is also historically an R bugaboo, but with changes to R's copy on write semantics and other optimizations in the base language, current benchmarks show it behaving on par with Python and other dynamic languages (if not even slightly better with tools such as dplyr and data.table.) The maggritr package's implementation of a "pipe semantic" (often considered the only truly successful implementation of a 'component architecture') and the adoption of the model for tools such as Rvset are really allowing for the functional, vectorized nature of R to shine through. These are really darned exciting times to be a part of this community!
- mapcar 12y ago
  Does lubridate solve the unintended timezone conversions of POSIXct classes?
  - hadley 12y ago
    Can you provide a bit more detail?
    - mapcar 12y ago
      In the R Help Desk 2004 (http://www.r-project.org/doc/Rnews/Rnews_2004-1.pdf http://www.r-project.org/doc/Rnews/Rnews_2004-1.pdf), Gabor Grothendieck recommends chron over POSIXct classes on account of the time zone conversions which occur when the tz attribute of the latter object is not "GMT". Will this not be a problem with lubridate? Thanks in advance.
      - hadley 12y ago
        I've never found that to be a problem in practice. Do you have an example where it's bitten you in practice? (Also you should use UTC and not GMT)
        mapcar 12y ago
        Hi Hadley, yes for instance > as.chron("1970-01-01")+unclass(as.chron("2001-04-01")) [1] 04/01/01 > as.POSIXct("1970-01-01","EST")+unclass(as.POSIXct("2014-06-01","EST")) [1] "2014-06-01 05:00:00 EST" If there is any conversion necessary it is difficult to get back the original intended time.
        hadley 12y ago
        What does adding two dates together mean?
        mapcar 12y ago
        Isn't this the conventional way of converting variables which have been coerced to their numeric representations back to time/date classes?
- elliott34 12y ago
  Do you have any links to articles on the current benchmarks that you mention? Not being snide just curious to read more.
  - hadley 12y ago
    I'd start here: https://github.com/Rdatatable/data.table/wiki/Benchmarks-:-Grouping https://github.com/Rdatatable/data.table/wiki/Benchmarks-:-G... Performance isn't currently a huge focus for dplyr. In my opinion dplyr is fast enough that the bottleneck becomes mostly cognitive - you spend more time thinking about what you want to do than actually doing it.
minimaxir 12y ago
Will this work with https websites? One of the reasons I learned Python for data scraping was that R in general does not play nice with https (RCurl requires a certificate and even then it's pretty fussy)
- hadley 12y ago
  Yes. It uses httr which wraps RCurl in such a way that everything should just work.
danso 12y ago
I guess I really do skip over who the submitter is when checking out HN links...if this submission had been titled, "Rvest: Easy web scraping R library by Hadley Wickham", I would've immediately been non-skeptical. It looks like rvest intends to be the equivalent of Mechanize, with stateful navigation in the works. Is there an R equivalent to just Beautiful Soup or Nokogiri?
- hadley 12y ago
  What are you looking for? rvest should support all the navigation tools from beautiful soup/nokogiri (unless I've missed something), but currently doesn't have any support for modifying the document (in which case I think your only option is the XML package).
  - danso 12y ago
    No you didn't miss anything...I meant if there were standalone parsers for R...Mechanize uses Nokogiri/Soup as a dependency.
    - hadley 12y ago
      Not that I'm aware of - rvest uses the R package XML which uses the C library libxml.
12423gsd 12y ago
The RStudio guys have really made R a pleasure to use. Thank you guys! The core language is still a confusing mess (I'm still never sure when to use a matrix, a dataframe, a list..), but if you use their tools you can ignore it for the most part. In under 10 lines you can massage data and generate fantastic graphics. A little off topic: but does anyone know what their business model is? Are they going to run out of money and burnout in a year or two?
- Hansi 12y ago
  Business model is sell to enterprise and consulting: http://www.rstudio.com/pricing/ http://www.rstudio.com/pricing/
  - hadley 12y ago
    FWIW we don't do any consulting, although we do a decent amount of training.
- hadley 12y ago
  If you're confused about R's data structures, please read http://adv-r.had.co.nz/Data-structures.html http://adv-r.had.co.nz/Data-structures.html and let me know if it doesn't help. And no, we're not planning on burning out. We currently sell three things: * RStudio Server Pro. An commercial version of the open-source server version that provides stuff that corporate IT wants (e.g. monitoring, more auth options, ...) * Shiny Server Pro. A more flexible version of the open-source shiny server that offers more configurability (e.g. number of R processes per app), and again other stuff that corporate IT wants. * Right to use the RStudio desktop IDE to companies who don't want to use AGPL software
  - deleted 12y ago
    [deleted]
    - hadley 12y ago
      Have a look at http://adv-r.had.co.nz/Subsetting.html#subsetting-operators http://adv-r.had.co.nz/Subsetting.html#subsetting-operators. There was just too much material to fit in one chapter.
deleted 12y ago
[deleted]
hudibras 12y ago
Hadley Wickham writes R packages faster than I can read the documentation on them. A true 10xer.

142 points by hadley 12y ago

The community and ecosystem around R is rapidly changing and adapting. R has a long and storied history as a niche language for statistics and analysis. Much like those disciplines have entered the mainstream of modern technology enabled businesses, so follows the R ecosystem. Previously laborious tasks are being revamped with new elegant APIs such as Rvest does for scraping (and dplyr for manipulation, and lubridate for date manipulation, etc...) Performance is also historically an R bugaboo, but with changes to R's copy on write semantics and other optimizations in the base language, current benchmarks show it behaving on par with Python and other dynamic languages (if not even slightly better with tools such as dplyr and data.table.) The maggritr package's implementation of a "pipe semantic" (often considered the only truly successful implementation of a 'component architecture') and the adoption of the model for tools such as Rvset are really allowing for the functional, vectorized nature of R to shine through. These are really darned exciting times to be a part of this community!

mapcar 12y ago
Does lubridate solve the unintended timezone conversions of POSIXct classes?
- hadley 12y ago
  Can you provide a bit more detail?
  - mapcar 12y ago
    In the R Help Desk 2004 (http://www.r-project.org/doc/Rnews/Rnews_2004-1.pdf http://www.r-project.org/doc/Rnews/Rnews_2004-1.pdf), Gabor Grothendieck recommends chron over POSIXct classes on account of the time zone conversions which occur when the tz attribute of the latter object is not "GMT". Will this not be a problem with lubridate? Thanks in advance.
    - hadley 12y ago
      I've never found that to be a problem in practice. Do you have an example where it's bitten you in practice? (Also you should use UTC and not GMT)
      - mapcar 12y ago
        Hi Hadley, yes for instance > as.chron("1970-01-01")+unclass(as.chron("2001-04-01")) [1] 04/01/01 > as.POSIXct("1970-01-01","EST")+unclass(as.POSIXct("2014-06-01","EST")) [1] "2014-06-01 05:00:00 EST" If there is any conversion necessary it is difficult to get back the original intended time.
        hadley 12y ago
        What does adding two dates together mean?
        mapcar 12y ago
        Isn't this the conventional way of converting variables which have been coerced to their numeric representations back to time/date classes?
elliott34 12y ago
Do you have any links to articles on the current benchmarks that you mention? Not being snide just curious to read more.
- hadley 12y ago
  I'd start here: https://github.com/Rdatatable/data.table/wiki/Benchmarks-:-Grouping https://github.com/Rdatatable/data.table/wiki/Benchmarks-:-G... Performance isn't currently a huge focus for dplyr. In my opinion dplyr is fast enough that the bottleneck becomes mostly cognitive - you spend more time thinking about what you want to do than actually doing it.

minimaxir 12y ago

Will this work with https websites? One of the reasons I learned Python for data scraping was that R in general does not play nice with https (RCurl requires a certificate and even then it's pretty fussy)

hadley 12y ago
Yes. It uses httr which wraps RCurl in such a way that everything should just work.

danso 12y ago

I guess I really do skip over who the submitter is when checking out HN links...if this submission had been titled, "Rvest: Easy web scraping R library by Hadley Wickham", I would've immediately been non-skeptical. It looks like rvest intends to be the equivalent of Mechanize, with stateful navigation in the works. Is there an R equivalent to just Beautiful Soup or Nokogiri?

hadley 12y ago
What are you looking for? rvest should support all the navigation tools from beautiful soup/nokogiri (unless I've missed something), but currently doesn't have any support for modifying the document (in which case I think your only option is the XML package).
- danso 12y ago
  No you didn't miss anything...I meant if there were standalone parsers for R...Mechanize uses Nokogiri/Soup as a dependency.
  - hadley 12y ago
    Not that I'm aware of - rvest uses the R package XML which uses the C library libxml.

12423gsd 12y ago

The RStudio guys have really made R a pleasure to use. Thank you guys! The core language is still a confusing mess (I'm still never sure when to use a matrix, a dataframe, a list..), but if you use their tools you can ignore it for the most part. In under 10 lines you can massage data and generate fantastic graphics. A little off topic: but does anyone know what their business model is? Are they going to run out of money and burnout in a year or two?

Hansi 12y ago
Business model is sell to enterprise and consulting: http://www.rstudio.com/pricing/ http://www.rstudio.com/pricing/
- hadley 12y ago
  FWIW we don't do any consulting, although we do a decent amount of training.
hadley 12y ago
If you're confused about R's data structures, please read http://adv-r.had.co.nz/Data-structures.html http://adv-r.had.co.nz/Data-structures.html and let me know if it doesn't help. And no, we're not planning on burning out. We currently sell three things: * RStudio Server Pro. An commercial version of the open-source server version that provides stuff that corporate IT wants (e.g. monitoring, more auth options, ...) * Shiny Server Pro. A more flexible version of the open-source shiny server that offers more configurability (e.g. number of R processes per app), and again other stuff that corporate IT wants. * Right to use the RStudio desktop IDE to companies who don't want to use AGPL software
- deleted 12y ago
  [deleted]
  - hadley 12y ago
    Have a look at http://adv-r.had.co.nz/Subsetting.html#subsetting-operators http://adv-r.had.co.nz/Subsetting.html#subsetting-operators. There was just too much material to fit in one chapter.

deleted 12y ago

[deleted]

hudibras 12y ago

Hadley Wickham writes R packages faster than I can read the documentation on them. A true 10xer.