4 ms·
Here's my favorite way to get factors in your R program without wanting them: load an HDF5 file. Python's excellent h5py module and the official HDF5 tools lik
by jzwinck 12y ago
Here's my favorite way to get factors in your R program without wanting them: load an HDF5 file.
Python's excellent h5py module and the official HDF5 tools like "h5dump" understand that a dataset is boolean if it has the type H5T_ENUM with two values "FALSE" (0) and "TRUE" (1).
But R works a different way: if you save a boolean vector from R to HDF5 (using the rhdf5 package), it will create a dataset of type H5T_STD_I32LE, which takes 4x the storage space. And if you read the boolean vector as understood by other tools, you will get factors. Then, the most amazing thing happens: R uses the string names of the enum ("FALSE" and "TRUE"), but remaps them to factors with 1 and 2. So if you do as.integer(h5read(...)) you will get a vector of ones and twos instead of zeros and ones. And of course R itself treats 1 and 2 as truthy values, so now your data is well and truly corrupted: all the elements are truthy!
That is: as.logical(as.integer(h5read(...))) always produces all TRUE values when the input is boolean written by a program other than R. This is a lovely source of bugs in real software.
- mapcar 12y agoIs rhdf5 the best library? I understand there is ncdf4 and h5r, and rgdal. I understood that R support for HDF5 was limited so I have not used R much for this purpose.
- jzwinck 12y agoh5r has been removed from CRAN "due to licensing problems"[1]. ncdf4 and rgdal use totally different formats. rhdf5 may not be the best library, but it seems to be the only library. I don't think R support for HDF5 is "limited," I think it's just poorly implemented in a few ways (another is that you need a separate package to get 64-bit integer support, which is another problem with R in general). [1] http://cran.r-project.org/web/packages/h5r/index.html http://cran.r-project.org/web/packages/h5r/index.html