3 ms·

I take it you have not worked in such an environment before Does AOL count? I built the mail system. It was kind of a big deal. We pushed 4,000 TPS through se

I take it you have not worked in such an environment before Does AOL count? I built the mail system. It was kind of a big deal. We pushed 4,000 TPS through servers less powerful than an iPhone. I also spent time at Akamai, though I admittedly worked on a tiny, isolated test lab (150 servers or so, a fraction of a percent of the "real" server base). I have worked with some amazing people, at all scales, and I have never seen a deployment that managed to completely avoid manual logins. No, you aren't logging into a few hundred hosts manually (well, there was this one shop... but let's not hold that up as exemplary). But you're probably logging into a few to test things out manually before you decide what your automated deployment script will be rolling out. Even when you do deploy a fix, with a large sysadmin team it's good to know WHO exactly ran that one-off deployment script. When I said "auditing", I wasn't even thinking of PCI, or auditing against malicious actions of any kind - just simple troubleshooting forensics. But we have a great, robust way to see exactly who modified a file and when; it's the file system. Why reinvent it? (I just spent the last day trying to figure out who edited a script. Why, "postgres" did.. of course it did.) Should you aim to do these manual config changes in the test lab first? Of course. Do you always succeed? No. Some things only get tested in the big lab. Reality intervenes. I try to plan for it. Mind you, I'm arguing this partly to see if I can be talked out of it. I think I believe it, but I've been wrong about way bigger things. If you've achieved my automation nirvana, and you roll things out to a few thousand servers, never manually, and never wondering who did it, tell me how it works.

moe 14y ago
But we have a great, robust way to see exactly who modified a file and when; it's the file system. Why reinvent it? That's a strange statement given your background. Since when does editing a file change the owner or leave an audit-trail behind, on stock linux? If you're running auditd then that's a different story but I've never seen that used in production for this purpose because it's hard to get right and because there are so many simpler options much better suited to the task (version control, puppet, chef, etc.). Auditing or not, in my experience you inevitably end up in permission-hell when multiple users mess with files that need to be readable by a common gid (sticky bits, umask, something is always off). If you really found a way to make this work at scale, a detailed write-up would be interesting (to me). In my experience that way lies unmitigated madness. I've had to clean up multiple such "sudo-style" deployments in the past.
- jaylevitt 14y ago
  GAH! See, I just slipped up and showed my age. I'm thinking of the Stratus VOS filesystem, which did change the modifier when you edited a file. You're right; UNIX/Linux files have an owner, which doesn't change on modification. Which shoots a big blazing hole in my argument. You can back into who modified what if files were changed in a big batch, and they also created a few new files - but you could also back into that via login times. Since those days, it's less that I've seen the individual-user system work at scale - it's that I've seen the single role account fail, repeatedly. I've managed to avoid permissions hell with a few well-chosen supplemental groups, but maybe I'm too optimistic that what used to work in one environment will work in another. Ask me in six months. :)

moe 14y ago

But we have a great, robust way to see exactly who modified a file and when; it's the file system. Why reinvent it? That's a strange statement given your background. Since when does editing a file change the owner or leave an audit-trail behind, on stock linux? If you're running auditd then that's a different story but I've never seen that used in production for this purpose because it's hard to get right and because there are so many simpler options much better suited to the task (version control, puppet, chef, etc.). Auditing or not, in my experience you inevitably end up in permission-hell when multiple users mess with files that need to be readable by a common gid (sticky bits, umask, something is always off). If you really found a way to make this work at scale, a detailed write-up would be interesting (to me). In my experience that way lies unmitigated madness. I've had to clean up multiple such "sudo-style" deployments in the past.

jaylevitt 14y ago
GAH! See, I just slipped up and showed my age. I'm thinking of the Stratus VOS filesystem, which did change the modifier when you edited a file. You're right; UNIX/Linux files have an owner, which doesn't change on modification. Which shoots a big blazing hole in my argument. You can back into who modified what if files were changed in a big batch, and they also created a few new files - but you could also back into that via login times. Since those days, it's less that I've seen the individual-user system work at scale - it's that I've seen the single role account fail, repeatedly. I've managed to avoid permissions hell with a few well-chosen supplemental groups, but maybe I'm too optimistic that what used to work in one environment will work in another. Ask me in six months. :)