3 ms·
To use permissions for security, follow the principle of least access. If a user or program doesn't need access to something, don't give it to them. User permis
by falcolas 12y ago
To use permissions for security, follow the principle of least access. If a user or program doesn't need access to something, don't give it to them. User permissions are the first tier of this, Apparmor and Selinux are the next (and correspondingly complex) next tier.
For example, for a web stack which communicates exclusively over the network stack, run your entire stack as individual non-root user. Nginx can run as the nginx user, django/rails/node as a separate user which has no access to the nginx configs. MySQL/PostgreSQL/Mongo as yet another user, which can't access the previous two.
By default, config files should not be owned by the process reading them, they should be part of the group (if they owned them, they could be re-written by that process).
Logging, you have a few options. Syslog is simple, and re-directable, which can help increase security. Writing to normal log files is perfectly acceptable as well, just be sure to write to a program specific directory and set up a logrotate config.
Also, set up monitoring and notification on everything you possibly can. First, know when something has gone wrong before you're notified by your customers. Then figure out how to know something will go wrong before it actually goes wrong (like running out of disk space - this happens more often than you might imagine).
There are plenty of articles on hardening linux, and sysadmin best practices on the web - I've only outlined a few here. When acting in the sysadmin role, be skeptical, paranoid, and value service uptime above everything. With this attitude, you will be more prone to automate server setup, limit user access, and less inclined to just throwing things out there because they're shiny.
The real fun is when you design and develop a way to safely and securely let your fellow developers just throw code over the fence, because then they can do their job in a way that suits them, without compromising your servers.
- dozzie 12y ago> Also, set up monitoring and notification on everything you possibly can. Notifying about everything is a very, very bad idea. You'll drown under notifications that do not matter and you will certainly miss those that do matter. There should be as little notifications as possible.
- smt88 12y agoA good rule of thumb is that you should monitor everything you can (tempered with an understanding of performance requirements) and notify for anything that requires action, even if it's not an emergency. It's easy to categorize your notifications after you've received them.
- falcolas 12y agoI used to believe this. I was aggravated by the pager, and would happily acknowledge problems in Nagios just to make the damned thing shut up. More and more pages would come in, the volume increasing each day as "non-issues" piled up. The third time this behavior caused downtime for my clients, I wised up and took my own advice of "uptime is king". I took the "every alert is actionable" to heart, and took a few moments to realize that the action can also be against the monitoring. Alert for disk at 80% on a 2TB volume? Action: Verify growth in Graphite isn't out of the ordinary, and increase the warning threshold to give you about 3-4 weeks of notice that you might need to get bigger disks. Alert for an excessive number of 404 responses? Browse the nginx logs and identify someone trying to hack your corporate-mandated WordPress install. Verify they aren't making any traction, and add exceptions to your 404's so you don't alert on known (and non-whitelisted) endpoints they're hitting. Alert for memory at 80%? What's consuming it; do I have a memory leak and need to restart something? If all is well, and MySQL is just being greedy, up the alert to 90%. Disk capacity warning at 3am? Put some hours around the warning notification, and add a separate "things are growing out of control" alert which doesn't have hours. API endpoint is not responding again, but it's not the system at fault? Add the API developer to the notifications for that alert and remove yourself for a week or two. After an admittedly harrying week of this, you gain two things. One: an operational understanding of what is going on in your system. Two: an alerting system which is tuned to your use case, and which lets you know when you have real problems. Remember - uptime is king.
- bbrazil 12y agoI'd recommend My Philosophy on Alerting by Rob Ewaschuk - https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa8eMWi8zzAn0YfcApr8Q/edit https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa... as a good approach to take. We've fleshed this out a bit more in the Prometheus best practices at http://prometheus.io/docs/practices/alerting/ http://prometheus.io/docs/practices/alerting/ Taking this approach at our company greatly reduced the alert count and improved responsiveness with no degradation in service.