3 ms·
i have a reoccurring use case where i have huge csv files that i need to aggregate / summarize using a pretty old netbook. the only way i can get this done is t
by dlphn___xyz 7y ago
i have a reoccurring use case where i have huge csv files that i need to aggregate / summarize using a pretty old netbook. the only way i can get this done is through spark/scala. would drill be a better option?
- minxomat 7y agoSure. Create a filesystem data source in the Drill config (https://drill.apache.org/docs/file-system-storage-plugin/ https://drill.apache.org/docs/file-system-storage-plugin/) with the directories where your CSVs are in. Then query them however you'd like. Use the CLI if you want to produce another CSV or REST/JDBC if you need the data somewhere else. If the query is complex and long-running you can monitor it's state in the Drill web UI.
- sixdimensional 7y agoYou might need to tweak the Drill JVM memory settings a little bit though if you're doing this on a netbook without much memory!