4 ms·
nice idea, but exploring the data is very limited. Would be even better if it had some sort of query language and maybe an API.
by dariosalvi78 5y ago
nice idea, but exploring the data is very limited. Would be even better if it had some sort of query language and maybe an API.
- jillesvangurp 5y agoFlat data tool chains include a lot of common command line tools that you can schedule via github actions. Have a look at e.g. csvkit (allows you to run sql queries against csv files; even supports joining files), jq, and other command line tools that can work on simple csv or json files. Lower level tools like grep, awk, cut, etc. can also work of course. I've done quite a bit of ad hoc data querying over the years where our data was just too tiny to bother with a lot of ceremony or infrastructure. Github actions also allows you to run stuff via docker. So loading some csvs into postgresql (or whatever you need) would be a possibility as well. The main limitation here is free build minutes and size of the data. Beyond a certain size you might want to use something more suitable. But I guess the point is that a lot of data just really isn't even close to big data.