3 ms·
We extensively use Apache Arrow to store data files as parquet files on S3. This is a cheap way to store data that doesn't require the query speed of a relation
by oxfordmale 6y ago
We extensively use Apache Arrow to store data files as parquet files on S3. This is a cheap way to store data that doesn't require the query speed of a relational (or non relational database). The main advantage of Arrow is that is a columnar database, and loses no information in transit unlike the nightmare of CSV files.
- lsorber 6y agoHave you benchmarked this against pickling those data files? In our experience, parquet's overhead isn't worth it for smaller data files.
- EdwardDiego 6y agoI'm not surprised, Parquet's columnar encoding and compression won't really kick in significantly for smaller files.
- kylebarron 6y agoBut with pickling you can only read the data in Python.
- alfalfasprout 6y agoI just did some benchmarks and it's pretty similar for small files. The difference would only be noticeable if you're serializing a ton of small files.
- lsorber 6y agoHuh, makes a pretty big difference for us. We were using pandas' built-in to_parquet though, which seems to suffer from some overhead.
- cbsmith 6y agoIf pickling is what is working best for you, it can't be much data.