3 ms·
That is a hard question to answer, I think you have to start looking at the big component (the data warehouse) and see what fits best for you. At least what ha
by mattbillenstein 9y ago
That is a hard question to answer, I think you have to start looking at the big component (the data warehouse) and see what fits best for you.
At least what has worked best for me is using BigQuery and building custom tooling around that. Using Airflow, some simple python tools, .json.gz archives, and a mix of streaming data and hourly/daily exports has worked well and not been a huge task to keep operational. Of course, every situation is different; ymmv.
- willnewby 9y agoGotcha. Good to know. Thanks!