3 ms·
> commandline tool allowing me to join data from different places, without needing to set up stuff like presto or spark. On another hand, I never encountered an
by minxomat 7y ago
> commandline tool allowing me to join data from different places, without needing to set up stuff like presto or spark. On another hand, I never encountered any tool which allows me to easily query csv and json data using SQL (which at least in my opinion is fairly ergonomic to use).
This exactly describes Drill (https://drill.apache.org/ https://drill.apache.org/) which can query any data source under the sun (RDBMS, NoSQL, files, clusters, serialized data (JSON, CSV, Parquet...), object storage (S3, OpenStack, ...)) using SQL and JOIN between them with pushdown optimizations. It's got an awesome CLI and can be used in any language via REST or JDBC.
- cube2222 7y agoThanks, didn't know about it! I'll make sure to read up on it!
- dlphn___xyz 7y agoi have a reoccurring use case where i have huge csv files that i need to aggregate / summarize using a pretty old netbook. the only way i can get this done is through spark/scala. would drill be a better option?
- minxomat 7y agoSure. Create a filesystem data source in the Drill config (https://drill.apache.org/docs/file-system-storage-plugin/ https://drill.apache.org/docs/file-system-storage-plugin/) with the directories where your CSVs are in. Then query them however you'd like. Use the CLI if you want to produce another CSV or REST/JDBC if you need the data somewhere else. If the query is complex and long-running you can monitor it's state in the Drill web UI.
- sixdimensional 7y agoYou might need to tweak the Drill JVM memory settings a little bit though if you're doing this on a netbook without much memory!
- nerdponx 7y agoI last tried this program several years ago, and I remembered the set up was somewhat difficult and arcane. It still very really useful, I should give it another look.