2 ms·
For my home projects I generate parquet (columnar and very well suited for DW like queries) files with pyarrow and use Dremio (Direct SQL on data lake): https:/
by gizmodo59 6y ago
For my home projects I generate parquet (columnar and very well suited for DW like queries) files with pyarrow and use Dremio (Direct SQL on data lake): https://github.com/dremio/dremio-oss https://github.com/dremio/dremio-oss (https://www.dremio.com/on-prem/ https://www.dremio.com/on-prem/) to query them (minio or just local disk or s3) and use Apache Superset for quick charts or dashboards.
- unixhero 6y agoWhat does Parquet do for your problems? I am trying to learn for when I might need it?
- gizmodo59 6y agoThere are several advantages compared to CSV, JSON or other non-columnar formats as far as analytics is concerned which are typically not transactional. https://blog.openbridge.com/how-to-be-a-hero-with-powerful-parquet-google-and-amazon-f2ae0f35ee04 https://blog.openbridge.com/how-to-be-a-hero-with-powerful-p... https://stackoverflow.com/a/36831549 https://stackoverflow.com/a/36831549 And with projects like Apache Iceberg you can also have ACID transactions which will make it easy to update or delete just using SQL. This really opens up the separation of compute and data and you can use any engine you want (spark, drill, hive, impala, athena, dremio, redshift spectrum etc) on top of your files.