3 ms·
Using Parquet files is more scalable because the amount of data can be vast. It can be impractical to use a normal RDBMS to hold billions of rows, but is no pro
by unoti 4y ago
Using Parquet files is more scalable because the amount of data can be vast. It can be impractical to use a normal RDBMS to hold billions of rows, but is no problem for a bunch of files sitting in S3 or file storage. It can also be radically cheaper because you're just paying for file storage. The Parquet file format is also designed to be both compressed but also skippable to find the rows you're looking for. Also, Parquet files can be mounted in a filesystem for usage with things like Spark.