5 ms·
One of the first thing your DevOps team is going to do as you grow is make sure that employees don't have unrestricted access to the production DB / console acc
by patio11 12y ago
One of the first thing your DevOps team is going to do as you grow is make sure that employees don't have unrestricted access to the production DB / console access / root on the production web tier, since down that path lies madness.
This comes down partly to policy and partly to tech. The policy, disclosed early and often, is that misuse of customer data is an instant firing offense. Google, Facebook, etc have indeed terminated people over this, often literally count-the-minutes after the fact of the misuse became known to other people at the company.
Tech-wise, it's spiritually similar to other security measures. You lock down access on a need-to-have basis, you log the heck out of extraordinary requests for access, and you audit those requests.
e.g. Many companies will eventually develop a Use The Software As User X feature. At some, this requires you to a) be logged in as a privileged employee, then b) click to activate the feature, c) write an explanation why you need access to User #12345's account, and d) checkbox that you have receive #12345's consent for this. (I know some companies that skip D, largely in B2C.) When you hit submit, that logs it to the DB and fires an email to the audits@ email address, which goes out to 5 different people, or pipes "Patrick just logged in as #12345 because [chasing down display bug -- customer reports unescaped HTML in the message window, can't reproduce on staging or with own account]" into your team's HipChat/etc channel.
- GuiA 12y agoThanks, that's super helpful. While we're not big enough yet to have a dedicated dev ops team to implement the first things you've described, the logging approach is totally realistic. I especially like the idea of having a bot that monitors extraordinary admin actions and pipes them into the group chat. Thanks again!
- caw 12y agoFrom a more technical basis (being a devops, and having worked in least privilege environments), the big thing you need is centralized logging, and from there you can do what you need. Whether it's syslog or logstash or something else, if you get the logs in one place you can then filter over them for instances that you'd need to alert via email or chatbot. That works great and all until you realize only "sudo" is logged and not root terminal actions, and even then root could delete any logs of its actions. That's why something immediately shipping off logs is nice. I like "rootsh" (available on SourceForge) for forcing any sudo users to either use "sudo" or "sudo rootsh" to get a root terminal. You're not preventing anyone from doing their jobs, you just have an audit trail. Someone asked me why you need an audit trail unless it's to fire people for doing something wrong -- no it's for root cause and preventing certain operator errors from happening again, and in case of maliciousness from either an employee or someone impersonating an employee. The one other big thing to do is get rid of any shared accounts that can access data. If it's AWS, gen up some keys for each user/application or use IAM roles for the hosts. If it's Linux accounts, separate out the accounts. If you must have a singular account for something, only allow sudo access to switch to them. Going back to the previous paragraph, you'll at least get a log of who switched to the shared account. If you want to chat more about this, my email is in my profile.
- dennisgorelik 12y agoAccess to production environment is not the only issue here. Proper testing in development environment is not very practical without restoring a backup from production environment. So typically, even if developers cannot change production data, they still can read few weeks old copy from production environment. Trimming production data after restoring it to development may improve security a little bit, but such trimming is a project in itself and requires constant maintenance of that trimming code. In addition to that, trimming production backup may introduce new bugs in development and make production bugs not reproducible in development environment.
- patio11 12y agoI would strenuously, strenuously argue against ever, at any stage of the company, at any kind of company, with any data of any significance whatsoever, letting developers initialize dev environments with production backups. That's not a disaster waiting to happen. It's a fractal of disasters -- no matter which part of the idea you examine, it has an infinite number of disasters in it. It is an Ice-9 of disasters: you introduce it to perfectly sensible engineering decisions and they become disasters, too, just by contact with it. ("Do we let our developers work on laptops?" "Sure." Great, now our production database is one car window away from causing catastrophic reputational damage to the company.)
- dennisgorelik 12y agoIn a small team there should be single development database restored from backup. That development database should be hosted on development server, not on developers' laptops. Developers are welcome to VPN into corporate network from their laptops in order to work with database. Any other issues?
- collyw 12y agoThat doesn't make sense. What if people want to change the database schema, and 4 developers are working on it at once?