6 ms·

This alone is reason to upgrade -> "R now uses a stringsAsFactors = FALSE default, and hence by default no longer converts strings to factors in calls to data.f

by zhdc1 6y ago

This alone is reason to upgrade -> "R now uses a stringsAsFactors = FALSE default, and hence by default no longer converts strings to factors in calls to data.frame() and read.table()."

FiReaNG3L 6y ago
One of the big reasons why I quit R 10 years ago and never looked back - Python wasn't secretely converting anything, AND failing silently when its not the expected type.
- kyllo 6y ago
  R's come a long way in the last decade. The tibble and data.table packages both address this issue. data.table (https://github.com/Rdatatable/data.table https://github.com/Rdatatable/data.table) is the more strongly-typed of the two, by default it fails loudly when it encounters data that doesn't conform to the column type. It's also quite fast--binds to C code that parallelizes with OpenMP. It has very terse and expressive syntax, I find it so much more intuitive and easy to work with than pandas. If you're happy with Python, by all means keep using it. I use both languages. Just suggesting that if you gave up on R that long ago, you might be pleasantly surprised by how much better it's gotten since then.
  - Tarq0n 6y ago
    vctrs [0] is the latest effort by the Rstudio developers to help people write type-stable code. The R standard library has a lot of issues with silently casting types, but the wonderful thing about it being so scheme-like is that many of these things can be evolved through libraries. [0] https://github.com/r-lib/vctrs https://github.com/r-lib/vctrs
- closed 6y ago
  By python, are you also including pandas? Because it is definitely doing a lot of that!
  - Breza 6y ago
    Exactly! TensorFlow is the only Python data analysis package I've used that doesn't automatically convert things in the background. I was helping a friend with STATA the other day, which doesn't automatically convert, and I realized I've gotten so used to that behavior in R and Python.
- lmg643 6y ago
  i stopped using R a number of years ago because it is not useful for very large datasets (in the tens to hundreds of millions) and now use kdb almost exclusively.
  - stewbrew 6y ago
    R has quite a few specialized libs to deal with large datasets (out of memory). Nothing keeps you from hosting the data in a DBMS and using SQL (or dplyr) to pull the data in an appropriate format.
    - Breza 6y ago
      I run a data science department at a corporation and this is exactly how we handle our massive amounts of data. It's rare that we're using a billion+ data points in one model so we use SQL to get the data we need in the format we need and move forward from there in R.
this_is_not_you 6y ago
My thoughts exactly when I saw that bullet point.
amrrs 6y ago
This is one primary reason I started using read_csv from readr than read.csv of base-R. Everytime you teach someone to read a csv using base-R this always came up somewhere in the middle of an analysis because those strings were read as Factors.
- s1t5 6y ago
  But readr gives you a tibble instead of a plain data.frame and that adds a bunch of other headaches.
  - crispyambulance 6y ago
    How so? I can't think of any drawbacks of tibbles vs data.frames?
    - kkoncevicius 6y ago
      Using tibbles outside of tidyverse can be dangerous. df_iris <- iris tb_iris <- tibble(iris) nunique <- function(x, colname) length(unique(x[,colname])) nunique(df_iris, "Species") > 3 nunique(tb_iris, "Species") > 1 Imagine now using some complex function from a repository (i.e. BioConductor) that works on data.frames and passing a tibble to it.
      - notafraudster 6y ago
        Post edit: Preserving the below post, which I think highlights some of the issues in the parent I'm responding to's example, but it turns out he is correct on another issue -- narrowly, in his example, the `[` subset command dispatches differently on tibbles than data.frames, so you can narrowly produce weird behaviour. So to anyone reading, please consider upvoting parent and reading rest of thread Original post follows: Right off the bat, the problem is not "using" tibbles, it's that you've incorrectly constructed one by passing the data through the tibble() constructor rather than using as_tibble(). The tibble constructor -- for pretty good reasons in other circumstances that seem crazy to you here because of your intent -- infers that you want the entire data frame to be a single column inside the tibble, called "iris". It does this because it evaluates the variable name passed to the tibble constructor as both the intended column name and the data to be placed inside the column. This demonstrates nesting, which is one of the great features of tibbles and otherwise used for a bunch of stuff. If you had done `tb_iris <- as_tibble(iris)`, it would have worked fine. `as_tibble()` is the function to convert an existing data structure to a tibble. R is obviously not "type safe" in any way, but you can engage in defensive programming, and one way you can do that is being hyper-aware of the steps you take during type conversions. If you check the documentation for `tibble()`, it tells you explicitly to "Use as_tibble() to turn an existing object into a tibble." Is there a reason you didn't? Imagine this related example: my_string <- "10" numeric(my_string) as.numeric(my_string) Would we conclude that "using the numeric type can be dangerous" because the constructor interpreted the argument different than the conversion helper? Second, I suspect you must be using extremely old versions of things, because on more recent versions, your nunique function would fail, not produce 1. I correctly get "Error: Can't find column `Species` in `.data`." This error message is maybe a little confusing if you don't check the structure `str(tb_iris)` of tb_iris to see what I mentioned above, but is the correct error to output in light of it. You'd also be able to flag this by just checking `colnames(tb_iris)` or `View(tb_iris)` if you're working in RStudio or using the embedded environment pane or really any other way of looking at the data. But your broader point is also false. Once a tibble has been formed, it should work EXACTLY the way a data.frame works because R objects can have multiple classes. The only thing that makes a tibble different than a data.frame is that it has an additional class label. All dispatches that work on data.frame objects work on tibbles because of how multiple classing works in R. This has been a goal since the beginning of tibble. The one exception I'm aware of is external functions that incorrectly check `if(class(obj) == "data.frame")` instead of using `is.data.frame()` or `if("data.frame" %in% class(obj))`. The former is and always has been incorrect because of how multiple dispatch is designed to work in R and should generate an error with multi-classed objects because the if statement evaluates to a vector of logicals instead of a logical. Once way you can tell that tibbles and data frames are identical save the above caveat is to run the following code: df_iris <- iris tb_iris <- as_tibble(iris) identical(df_iris, tb_iris) class(tb_iris) <- "data.frame" identical(df_iris, tb_iris) Note that you are not "downconverting" a tibble into a data.frame in this code (but that would work too) -- you are taking the tibble exactly as is and hacking its class label to look like a data frame. It's identical because a tibble was always a data frame.
kgwgk 6y ago
You could add to your .Rprofile options(stringsAsFactors=FALSE)
- thom 6y ago
  This is likely to break things when you share or publish your code.
  - kgwgk 6y ago
    Upgrading will also break things. And code which depends on defaults will be broken either in new releases or in older releases.
    - s1t5 6y ago
      Upgrading makes the changes explicit, tinkering with your environment variables doesn't.
      - kgwgk 6y ago
        Upgrading is not different than changing environment variables as far as breaking existing code is concerned. You can always run R --no-init-file to be sure that you have the default settings. Now you have to know what default settings the code that you want to run expects.
    - thom 6y ago
      Yes, I mean, we could list things all day that make R a horrible environment for reproducible analysis or production deployment, but it is what it is.
      - kgwgk 6y ago
        You’re right. My defensive reply was unwarranted.
- truculent 6y ago
ggrothendieck 6y ago
The new C++ derived syntax for string literals seems to me to be the top new feature. It will make it possible to support markdown, latex, R code and Windows path literals without munging them first.
- ryndbfsrw 6y ago
  If I understand, does this mean on windows when I've copied a file location I would no longer have to replace backslash in the file path with double back slash or forward slashes? Or am I off-piste?
  - ggrothendieck 6y ago
    Correct. One can write r"(c:\Windows\System32)" . Check out additional examples at the end of the help file: https://github.com/wch/r-source/blob/trunk/src/library/base/man/Quotes.Rd https://github.com/wch/r-source/blob/trunk/src/library/base/...
    - stilisstuk 6y ago
      Awesome
ryndbfsrw 6y ago
I'm so used to using data.table to import data I forgot this was still a thing
topheroo 6y ago
I thought exactly the same thing! Been a long time coming…
minimaxir 6y ago
One of the selling points of tidyverse’s readr was “no stringAsFactors = FALSE necessary.” That’s how annoying it is.
paultopia 6y ago
This makes me want to sing and dance. Ding-dong, the stringsAsFactors witch is dead!

by zhdc1 6y ago

This alone is reason to upgrade -> "R now uses a stringsAsFactors = FALSE default, and hence by default no longer converts strings to factors in calls to data.frame() and read.table()."

FiReaNG3L 6y ago

One of the big reasons why I quit R 10 years ago and never looked back - Python wasn't secretely converting anything, AND failing silently when its not the expected type.

kyllo 6y ago
R's come a long way in the last decade. The tibble and data.table packages both address this issue. data.table (https://github.com/Rdatatable/data.table https://github.com/Rdatatable/data.table) is the more strongly-typed of the two, by default it fails loudly when it encounters data that doesn't conform to the column type. It's also quite fast--binds to C code that parallelizes with OpenMP. It has very terse and expressive syntax, I find it so much more intuitive and easy to work with than pandas. If you're happy with Python, by all means keep using it. I use both languages. Just suggesting that if you gave up on R that long ago, you might be pleasantly surprised by how much better it's gotten since then.
- Tarq0n 6y ago
  vctrs [0] is the latest effort by the Rstudio developers to help people write type-stable code. The R standard library has a lot of issues with silently casting types, but the wonderful thing about it being so scheme-like is that many of these things can be evolved through libraries. [0] https://github.com/r-lib/vctrs https://github.com/r-lib/vctrs
closed 6y ago
By python, are you also including pandas? Because it is definitely doing a lot of that!
- Breza 6y ago
  Exactly! TensorFlow is the only Python data analysis package I've used that doesn't automatically convert things in the background. I was helping a friend with STATA the other day, which doesn't automatically convert, and I realized I've gotten so used to that behavior in R and Python.
lmg643 6y ago
i stopped using R a number of years ago because it is not useful for very large datasets (in the tens to hundreds of millions) and now use kdb almost exclusively.
- stewbrew 6y ago
  R has quite a few specialized libs to deal with large datasets (out of memory). Nothing keeps you from hosting the data in a DBMS and using SQL (or dplyr) to pull the data in an appropriate format.
  - Breza 6y ago
    I run a data science department at a corporation and this is exactly how we handle our massive amounts of data. It's rare that we're using a billion+ data points in one model so we use SQL to get the data we need in the format we need and move forward from there in R.

this_is_not_you 6y ago

My thoughts exactly when I saw that bullet point.

amrrs 6y ago

This is one primary reason I started using read_csv from readr than read.csv of base-R. Everytime you teach someone to read a csv using base-R this always came up somewhere in the middle of an analysis because those strings were read as Factors.

s1t5 6y ago
But readr gives you a tibble instead of a plain data.frame and that adds a bunch of other headaches.
- crispyambulance 6y ago
  How so? I can't think of any drawbacks of tibbles vs data.frames?
  - kkoncevicius 6y ago
    Using tibbles outside of tidyverse can be dangerous. df_iris <- iris tb_iris <- tibble(iris) nunique <- function(x, colname) length(unique(x[,colname])) nunique(df_iris, "Species") > 3 nunique(tb_iris, "Species") > 1 Imagine now using some complex function from a repository (i.e. BioConductor) that works on data.frames and passing a tibble to it.
    - notafraudster 6y ago
      Post edit: Preserving the below post, which I think highlights some of the issues in the parent I'm responding to's example, but it turns out he is correct on another issue -- narrowly, in his example, the `[` subset command dispatches differently on tibbles than data.frames, so you can narrowly produce weird behaviour. So to anyone reading, please consider upvoting parent and reading rest of thread Original post follows: Right off the bat, the problem is not "using" tibbles, it's that you've incorrectly constructed one by passing the data through the tibble() constructor rather than using as_tibble(). The tibble constructor -- for pretty good reasons in other circumstances that seem crazy to you here because of your intent -- infers that you want the entire data frame to be a single column inside the tibble, called "iris". It does this because it evaluates the variable name passed to the tibble constructor as both the intended column name and the data to be placed inside the column. This demonstrates nesting, which is one of the great features of tibbles and otherwise used for a bunch of stuff. If you had done `tb_iris <- as_tibble(iris)`, it would have worked fine. `as_tibble()` is the function to convert an existing data structure to a tibble. R is obviously not "type safe" in any way, but you can engage in defensive programming, and one way you can do that is being hyper-aware of the steps you take during type conversions. If you check the documentation for `tibble()`, it tells you explicitly to "Use as_tibble() to turn an existing object into a tibble." Is there a reason you didn't? Imagine this related example: my_string <- "10" numeric(my_string) as.numeric(my_string) Would we conclude that "using the numeric type can be dangerous" because the constructor interpreted the argument different than the conversion helper? Second, I suspect you must be using extremely old versions of things, because on more recent versions, your nunique function would fail, not produce 1. I correctly get "Error: Can't find column `Species` in `.data`." This error message is maybe a little confusing if you don't check the structure `str(tb_iris)` of tb_iris to see what I mentioned above, but is the correct error to output in light of it. You'd also be able to flag this by just checking `colnames(tb_iris)` or `View(tb_iris)` if you're working in RStudio or using the embedded environment pane or really any other way of looking at the data. But your broader point is also false. Once a tibble has been formed, it should work EXACTLY the way a data.frame works because R objects can have multiple classes. The only thing that makes a tibble different than a data.frame is that it has an additional class label. All dispatches that work on data.frame objects work on tibbles because of how multiple classing works in R. This has been a goal since the beginning of tibble. The one exception I'm aware of is external functions that incorrectly check `if(class(obj) == "data.frame")` instead of using `is.data.frame()` or `if("data.frame" %in% class(obj))`. The former is and always has been incorrect because of how multiple dispatch is designed to work in R and should generate an error with multi-classed objects because the if statement evaluates to a vector of logicals instead of a logical. Once way you can tell that tibbles and data frames are identical save the above caveat is to run the following code: df_iris <- iris tb_iris <- as_tibble(iris) identical(df_iris, tb_iris) class(tb_iris) <- "data.frame" identical(df_iris, tb_iris) Note that you are not "downconverting" a tibble into a data.frame in this code (but that would work too) -- you are taking the tibble exactly as is and hacking its class label to look like a data frame. It's identical because a tibble was always a data frame.

kgwgk 6y ago

You could add to your .Rprofile options(stringsAsFactors=FALSE)

thom 6y ago
This is likely to break things when you share or publish your code.
- kgwgk 6y ago
  Upgrading will also break things. And code which depends on defaults will be broken either in new releases or in older releases.
  - s1t5 6y ago
    Upgrading makes the changes explicit, tinkering with your environment variables doesn't.
    - kgwgk 6y ago
      Upgrading is not different than changing environment variables as far as breaking existing code is concerned. You can always run R --no-init-file to be sure that you have the default settings. Now you have to know what default settings the code that you want to run expects.
  - thom 6y ago
    Yes, I mean, we could list things all day that make R a horrible environment for reproducible analysis or production deployment, but it is what it is.
    - kgwgk 6y ago
      You’re right. My defensive reply was unwarranted.
truculent 6y ago

ggrothendieck 6y ago

The new C++ derived syntax for string literals seems to me to be the top new feature. It will make it possible to support markdown, latex, R code and Windows path literals without munging them first.

ryndbfsrw 6y ago
If I understand, does this mean on windows when I've copied a file location I would no longer have to replace backslash in the file path with double back slash or forward slashes? Or am I off-piste?
- ggrothendieck 6y ago
  Correct. One can write r"(c:\Windows\System32)" . Check out additional examples at the end of the help file: https://github.com/wch/r-source/blob/trunk/src/library/base/man/Quotes.Rd https://github.com/wch/r-source/blob/trunk/src/library/base/...
  - stilisstuk 6y ago
    Awesome

ryndbfsrw 6y ago

I'm so used to using data.table to import data I forgot this was still a thing

topheroo 6y ago

I thought exactly the same thing! Been a long time coming…

minimaxir 6y ago

One of the selling points of tidyverse’s readr was “no stringAsFactors = FALSE necessary.” That’s how annoying it is.

paultopia 6y ago

This makes me want to sing and dance. Ding-dong, the stringsAsFactors witch is dead!