5 ms·

I really like R / Tidyverse, but that requires that you have more memory than your datasize (especially if you are using narrow and long dataframes, such as rec

I really like R / Tidyverse, but that requires that you have more memory than your datasize (especially if you are using narrow and long dataframes, such as recommended by tidyverse) and it also requires you to code, instead of using a UI

stuxnet79 7y ago
Is there really no way of processing dataframes by chunks in the R ecosystem? I'm firmly in the Python camp but I wonder how much I'm missing out on in the R world.
- deadcaribou 7y ago
  There are many ways to process larger than RAM objects in R. The dbplyr package makes it possible to use a local db table as if they are in-memory data frames (https://dbplyr.tidyverse.org/ https://dbplyr.tidyverse.org/). https://diskframe.com/ https://diskframe.com/ is also a pretty great solution that processes data by chunks.
- onearmedscissor 7y ago
  Take a look at data.table https://cran.r-project.org/web/packages/data.table/ https://cran.r-project.org/web/packages/data.table/
- livingmargot 7y ago
  Sorry to butt in, but could you point me to a resource on processing data in chunks using Python?