3 ms·
Here's one approach: 1. Document all datasets at their sources of record 2. Setup jobs to dump their data into an S3 bucket (daily, hourly, whatever makes sen
by timwis 8y ago
Here's one approach:
1. Document all datasets at their sources of record
2. Setup jobs to dump their data into an S3 bucket (daily, hourly, whatever makes sense). Use IAM to lock down access if necessary.
3. Setup AWS Athena to give you some analysis capability on those files, while you:
4. Setup jobs to [denormalise/cleanup and] load those files from S3 into postgres RDS/citus data/redshift (alternatively you could denormalise after the load ELT-style using materialised views or [dbt](https://github.com/fishtown-analytics/dbt) https://github.com/fishtown-analytics/dbt))
You'll still need a tool to orchestrate it all. I'm excited about trying [pachyderm](https://www.pachyderm.io/ https://www.pachyderm.io/) for my next project.
- bsagdiyev 8y agore: point #2 -- you can also make sure access is locked down via bucket policy instead with IAM policy as well, keeps it cleaner if/when multiple roles, profiles, etc. need to access it later on. I assume OP meant that but just wanted to give my 0.02 in case it helps.