4 ms·
> ... 100's of terabytes of biological data. When working with this kind of data, Pandas is often the most preferred tool "biological data" is a bit vague, but
by cottonseed 9y ago
> ... 100's of terabytes of biological data. When working with this kind of data, Pandas is often the most preferred tool
"biological data" is a bit vague, but for the data I know to be that big, sequence and array data, it does not naturally have the structure of a dataframe nor is pandas the tool of choice.
- thewizardofaus 9y agoYeah, I tend to use modified hdf5 tools for data that big.
- sfsylvester 9y agoThis is exactly my go-to-move as well. pandas.read_hdf has beaten out ray.dataframe.read_csv in terms of speed on the few files I've just initially tested now. But I imagine the programmable flexibility csvs have over hdfs (I've never used a Unix command to edit a hdf for example) is why this new approach could get some traction.
- tavert 9y agoTry parquet if your data is tabular, pyarrow and related tools are getting parquet up to a pretty comparable speed to hdf5, with arguably more flexibility and a better multithreading story.
- Radim 9y agoThat opening gave me a pause too. Doesn't Pandas target only structured (tabular) in-core datasets? (Unless you use something like Blaze / Dask / Ray on top.) Does anyone really work with Pandas on "100's of terabytes of biological data"?