I think it was a really interesting (and smart) choice to convert to Parquet format. Columnar storage is so much more efficient, and working with data in Parqu

I think it was a really interesting (and smart) choice to convert to Parquet format. Columnar storage is so much more efficient, and working with data in Parquet is pretty fast using the engines they mention (Apache Spark, Impala, Hive, etc.). I actually had been thinking about Parquet as a component of ETL, and if it might be possible to make ETL many times faster by compressing to Parquet format on the source and then transmitting to a destination - especially when you're talking about limited bandwidth situations where you need entire data sets moved around in bulk. This looks really nice for sharing public data sets, but I wish that there was a better public non-profit org running indexes of public data sets.... I guess if something like the semantic web had ever taken off, then the Internet itself would be the index of public data sets, but it seems like that dream is still yet to materialize.