3 ms·
Data warehouse: - Amazon Redshift (data sourced from many different backend DBs; e.g. PostgreSQL, MySQL, other Redshift instances etc.) - BigQuery (Some teams
by pantene 7y ago
Data warehouse:
- Amazon Redshift (data sourced from many different backend DBs; e.g. PostgreSQL, MySQL, other Redshift instances etc.)
- BigQuery (Some teams store the data for their projects here. For reporting, they're selectively transferred to Redshift.)
Reports:
- Tableau (extracts of pre-aggregated data from Redshift)
- Looker (connects live to and executes its own queries on Redshift)
Anything that is based on pre-aggregated (rather small, e.g. n_rows < 5mil) data is visualized on Tableau. If users want to work on a self-service tool they use Looker which can connect to (cleaned) raw data optimized for this use case.
ETLs for raw data run daily on Redshift. Reports are also scheduled on Redshift (with Airflow) but the refresh interval is report-specific.