3 ms·
parquet is optimized for storage and compresses well (=> smaller files) feather is optimized for fast reading
by tosh 9mo ago
parquet is optimized for storage and compresses well (=> smaller files)
feather is optimized for fast reading
- aynyc 9mo agoGiven the cost of storage is getting cheaper, wouldn't most firms want to use feather for analytic performance? But everyone uses parquet.
- vb-8448 9mo agoStorage is cheap but bandwidth no.
- outside1234 9mo agoWhat people have done in the face of cheaper storage is store more data.
- yencabulator 9mo agoYou can, still, gain a lot of performance by doing less I/O.
- farsa 9mo agoStorage getting cheaper did not really reach the cloud providers and for self-hosting it has recently gotten even more expensive due to AI bs.
- benrutter 9mo agoThere's definitely a "everyone uses it because everyone uses it" effect. Feather might be a better fit for sime yse cases, but parquet has fantastic support and is still a pretty good choice for things that feather does. Unless they're really focussed on eaking out every bit of read performance, people often opt for the well supported path instead.
- twic 9mo agoAnd now there's Lance! https://lance.org/ https://lance.org/