3 ms·

Someone has to scrub the data, though... Agreed, data governance is important for any company to get right, but someone has to have DB access in order to manag

Someone has to scrub the data, though... Agreed, data governance is important for any company to get right, but someone has to have DB access in order to manage it. When you factor in that so many companies derive revenue from the data they generate, then it gets harder. If you’re Twitter, how can you build the services you need as an architect or data scientist without the actual data?

CydeWeys 7y ago
The scrubbers do not need raw DB access. All they need is a strictly audited web UI. Very, very few people need root access and the ability to see all raw data. For example, at Google, this is a tiny number of senior SREs and that's it. Your average employees should be using audited UIs running through service accounts that have restricted permissions.
- AmericanChopper 7y ago
  I’ve achieved this without so much hassle at a smallish/medium sized company (~60 engineers). A typical engineer had access to anonymized data, only a few had access to query raw data. All their queries were logged with justifications. They also wouldn’t query raw data as a matter of BaU (outside of some very rare situations), it was pretty much only ever done when making changes to the ETL pipelines. The solution we had wasn’t particularly difficult to set up, and actually made life much easier for everybody, because we’d provided everybody with a very easy to use interface for the data they wanted (much better than the old school shelling into a DB to run your arbitrary SQL statements).
  - big_chungus 7y ago
    Any chance you'd be willing to share more about your setup, at least high-level?
    - AmericanChopper 7y ago
      We had a data warehouse where we sent pretty much all the data we had in the organisation, and redash in front of it to allow query access, reporting, etc... Everything in the data warehouse was anonymized, and people only had access to the schemas they needed (though this was defined quite broadly). Anonymization was handled by our ETL pipeline. When we first set it up, the requirements were pretty simple and we just wrote a little java app to do it. This scaled pretty poorly, and the team ended up putting a proper ETL product in there. I can’t remember which one they used, but there’s a lot of perfectly decent products in that space (even some open source).

CydeWeys 7y ago

The scrubbers do not need raw DB access. All they need is a strictly audited web UI. Very, very few people need root access and the ability to see all raw data. For example, at Google, this is a tiny number of senior SREs and that's it. Your average employees should be using audited UIs running through service accounts that have restricted permissions.

AmericanChopper 7y ago
I’ve achieved this without so much hassle at a smallish/medium sized company (~60 engineers). A typical engineer had access to anonymized data, only a few had access to query raw data. All their queries were logged with justifications. They also wouldn’t query raw data as a matter of BaU (outside of some very rare situations), it was pretty much only ever done when making changes to the ETL pipelines. The solution we had wasn’t particularly difficult to set up, and actually made life much easier for everybody, because we’d provided everybody with a very easy to use interface for the data they wanted (much better than the old school shelling into a DB to run your arbitrary SQL statements).
- big_chungus 7y ago
  Any chance you'd be willing to share more about your setup, at least high-level?
  - AmericanChopper 7y ago
    We had a data warehouse where we sent pretty much all the data we had in the organisation, and redash in front of it to allow query access, reporting, etc... Everything in the data warehouse was anonymized, and people only had access to the schemas they needed (though this was defined quite broadly). Anonymization was handled by our ETL pipeline. When we first set it up, the requirements were pretty simple and we just wrote a little java app to do it. This scaled pretty poorly, and the team ended up putting a proper ETL product in there. I can’t remember which one they used, but there’s a lot of perfectly decent products in that space (even some open source).