4 ms·
Is ETL really even necessary anymore? Why not just run fast ad hoc queries over the raw data with something like Google BigQuery?
by lafay 11y ago
Is ETL really even necessary anymore? Why not just run fast ad hoc queries over the raw data with something like Google BigQuery?
- web007 11y agoYeah - just Extract it from your MySQL / Mongo / Postgres / logfiles / whatever system it's in right now, Transform it into a CSV or whatever the input needs to be and Load it into BigQuery. Once it's there, you can do whatever you need!
- ams6110 11y agoOn a smaller scale, the "q" utility has been a boon for me in the handling of ad-hoc delimited data files. http://harelba.github.io/q/ http://harelba.github.io/q/ Really one of the best things I've discovered in the past 5 years. Saves so much work compared to doing stuff with sed, awk, and the like.
- khc 11y agowow good find! I've been using my own script https://github.com/kahing/bin/blob/master/avg https://github.com/kahing/bin/blob/master/avg but q seems a lot more flexible and works with more than number types.
- v64 11y agoGetting the raw data into BigQuery or another tool is an ETL problem in itself. You have unstructured log files, you have external APIs such as SalesForce, you have various relational databases, etc etc. Someone has to come in and transform this data into a unified format that can be inserted into BigQuery.