3 ms·
Big Data Storage and Transfer Formats
- adammarples 4y agoPretty sure duckdb can read and write parquet, if you were loving duckdb but worried about parquet accessibility
- anotherhue 4y agoPlay “Big Data or Pokémon?” To learn more. https://pixelastic.github.io/pokemonorbigdata/ https://pixelastic.github.io/pokemonorbigdata/
- scrappyjoe 4y agoYou don’t need to shut down the ‘generating process’ to read parquet. With arrow, which is available in many languages, you have the notion of an arrow dataset, which is a directory of arrow files. Just keep adding more files to the directory, and your dataset will grow. Duckdb is able to connect to and read the dataset, so you’re all good.