3 ms·
At a previous gig we wanted to have a regular scrubbed snapshot of our production Heroku database for dev and QA work. We didn't want unscrubbed data to hit a d
by zrail 7y ago
At a previous gig we wanted to have a regular scrubbed snapshot of our production Heroku database for dev and QA work. We didn't want unscrubbed data to hit a developer's laptop, both because it contained PII and because it was huge. The nightly process we ended up with (after a lot of iteration) was:
1. fork the production db and wait for it to sync
2. run a task within the production Heroku app pointing at the new fork which does the PII scrubbing and compacting[1]
3. `pg_dump` the compressed result onto the Heroku dyno running the task above
4. upload the dump as a release on the GitHub page for our main codebase
When a developer wanted a fresh dev database all they had to do was run a local rake task that would grab the db dump from the latest GitHub release, drop their local database, and restore the dump.
[1]: This isn't as easy as it sounds because one of the tables we wanted to compress contained raw financial data, but we wanted a coherent story for dev, so we dropped 95% of the data, keeping more recent stuff, randomized all the amounts, and then recalculated summary tables.
- kylecordes 7y agoIt warms my heart to read of people doing this: development on scrubbed data. It is sadly not unusual, even at companies with resources and knowledge to do better, to hand production data (including ample customer personal data) to every developer.