3 ms·
Why serve APIs using parquet collections as a data source, as compared to a standard RDBMS? What are the pro's and con's?
by Dowwie 4y ago
Why serve APIs using parquet collections as a data source, as compared to a standard RDBMS? What are the pro's and con's?
- jmartin2683 4y agoWe’re talking about hundreds of gigabytes, pushing towards many terabytes of data with extremely sparse access patterns. Loading into anything (including an rdbms) wouldn’t be cost-effective or practical.
- jmartin2683 4y agoThe big pro is that the only data that ever has to move is that which is actually accessed. It’s extremely cost-effective to produce, store and query statically vs moving it into something else of sufficient scale to handle the entire universe of data. The downside is that you have to cache data, there is a latency penalty on the first request for any new data. One would expect performance to be poor when querying parquet directly using these new lazy dataframe libraries, but it’s actually amazingly fast. Very surprising.
- unoti 4y agoUsing Parquet files is more scalable because the amount of data can be vast. It can be impractical to use a normal RDBMS to hold billions of rows, but is no problem for a bunch of files sitting in S3 or file storage. It can also be radically cheaper because you're just paying for file storage. The Parquet file format is also designed to be both compressed but also skippable to find the rows you're looking for. Also, Parquet files can be mounted in a filesystem for usage with things like Spark.