3 ms·

I've never worked at any company with more than 50 employees, and have worked on a team with as few as 5 engineers. Even without an explicit SRE role, someone i

I've never worked at any company with more than 50 employees, and have worked on a team with as few as 5 engineers. Even without an explicit SRE role, someone is doing that work -- keeping the servers (reliably) on the internet and performing, mitigating attacks, etc. You should be able to tag the row with a dirty flag on second (or later) write using rules internal to the DB without incurring a large performance penalty, if you already have any kind of sanity checking. So at a basic technical level, this is bad design or an intentional capability. Similarly, even on the team with 5 engineers, our production DB didn't have a way to directly access it, and tunneling through an API box to get network access would trigger an access warning to the appropriate person, because you shouldn't be accessing prod API servers directly either. (Similarly, spinning up a new instance in a security group to inherit permission.) This isn't detecting advanced exfiltration techniques using information theory... It's basic network topology and server monitoring. And I suspect that reddit has at least a couple ops guys. This really smacks of intentional capability, followed to lazy design.

kaishiro 10y ago
Either you are the outlier or I am - I'm honestly not sure. But having consulted with a dozen or so digital agencies over the past year not a single one had the sort of monitoring that you're suggesting. This doesn't mean that it's a bad idea, quite the opposite, but in my experience the reality of a small to medium sized agency is that if a client isn't paying for it than you're not working on it.
- SomeStupidPoint 10y ago
  It's what I consider the difference between a software developer and a computer-systems engineer to be. I'm likely the outlier, because my early mentors (college + first couple years of career) were mostly aerospace engineers or chemical processing equipment engineers, where safety is critical. Code can kill in both those fields. (They themselves had adapted a lot of systems and failure mode analysis from mech engineers in the same field.) I don't think that the mitigations I pointed out require much technical work though -- most of them can be as simple as a single setting, extra program installed during build (and maybe a couple config settings), or a single cloud monitoring rule (eg, detecting launches). They do, however, require adhering to policy, discipline on the part of staff, and analyzing the full scope of your tech (and how it interacts). I guess my point is that these things often aren't done because of ignorance or haste rather than engineering cost (or legitimate need to be done faster). It's just we have poor standards for what "engineering" is in tech, so we don't cross our 't's and dot our 'i's the way a mechanical engineer is expected to. But that's no reason not to try and raise the bar.
stouset 10y ago
> Even without an explicit SRE role, someone is doing that work -- keeping the servers (reliably) on the internet and performing, mitigating attacks, etc. To be honest, not in any of the several places I've worked. And I'm a security engineer. Lean startups are desperately trying to prove their product; time is spent on reliability, performance, and security almost entirely reactively and not proactively. I'm not saying this is a good thing. Bad security (and reliability) posture early on inflicts multiples of the original mitigation cost for years. And yet, it's rational. Time spent securing systems from your own employees when you're tiny is time you're not building your product and finding your market. It's a perverse game of Russian roulette: there are 99 bullets and one empty chamber. The analogy is going to break down here, but spending time and money removing one bullet (1:98 chance of death) is less effective than adding another empty chamber (2:99 chance of death). Especially when having direct access to the database is insanely useful to fix the problems that moving fast and breaking things inevitably causes, only a tiny fraction of startups are going to enact the kinds of measures you mention.
- SomeStupidPoint 10y ago
  I don't disagree with anything you've said, except that I disagree that it's actually a rational choice based on facts, rather than an apparently rational one based on poor information and training -- again, none of the mitigations I pointed out are really expensive. Let's take prod DB access: Im not saying make it super locked down, Im saying set your DB security group to only be connected to from your API security group, and set an email warning when an instance launches in that group. Doesn't stop your engineers doing it, but it makes it pretty easy to shout across the office "Hey, who is fucking with prod and why?" Takes like 30 seconds to configure on AWS. There are serious questions about why your CEO can launch that instance though, and that sounds like massive policy failure. Again, restricting the CEO IAM from launching prod instances takes 30 seconds. (And all your non-engineer accounts should be IAM restricted!) Ignoring that your probabilities don't work out, Id argue that the situation is really remove 5 bullets or add 1 chamber, and people pick the chamber purely because it's constructive, not out of genuine cost-benefit analysis. "If I had 3 hours to chop a tree, Id sharpen my axe" -- I think startups are too hasty to chop, because that's being productive, right?

kaishiro 10y ago

Either you are the outlier or I am - I'm honestly not sure. But having consulted with a dozen or so digital agencies over the past year not a single one had the sort of monitoring that you're suggesting. This doesn't mean that it's a bad idea, quite the opposite, but in my experience the reality of a small to medium sized agency is that if a client isn't paying for it than you're not working on it.

SomeStupidPoint 10y ago
It's what I consider the difference between a software developer and a computer-systems engineer to be. I'm likely the outlier, because my early mentors (college + first couple years of career) were mostly aerospace engineers or chemical processing equipment engineers, where safety is critical. Code can kill in both those fields. (They themselves had adapted a lot of systems and failure mode analysis from mech engineers in the same field.) I don't think that the mitigations I pointed out require much technical work though -- most of them can be as simple as a single setting, extra program installed during build (and maybe a couple config settings), or a single cloud monitoring rule (eg, detecting launches). They do, however, require adhering to policy, discipline on the part of staff, and analyzing the full scope of your tech (and how it interacts). I guess my point is that these things often aren't done because of ignorance or haste rather than engineering cost (or legitimate need to be done faster). It's just we have poor standards for what "engineering" is in tech, so we don't cross our 't's and dot our 'i's the way a mechanical engineer is expected to. But that's no reason not to try and raise the bar.

stouset 10y ago

> Even without an explicit SRE role, someone is doing that work -- keeping the servers (reliably) on the internet and performing, mitigating attacks, etc. To be honest, not in any of the several places I've worked. And I'm a security engineer. Lean startups are desperately trying to prove their product; time is spent on reliability, performance, and security almost entirely reactively and not proactively. I'm not saying this is a good thing. Bad security (and reliability) posture early on inflicts multiples of the original mitigation cost for years. And yet, it's rational. Time spent securing systems from your own employees when you're tiny is time you're not building your product and finding your market. It's a perverse game of Russian roulette: there are 99 bullets and one empty chamber. The analogy is going to break down here, but spending time and money removing one bullet (1:98 chance of death) is less effective than adding another empty chamber (2:99 chance of death). Especially when having direct access to the database is insanely useful to fix the problems that moving fast and breaking things inevitably causes, only a tiny fraction of startups are going to enact the kinds of measures you mention.

SomeStupidPoint 10y ago
I don't disagree with anything you've said, except that I disagree that it's actually a rational choice based on facts, rather than an apparently rational one based on poor information and training -- again, none of the mitigations I pointed out are really expensive. Let's take prod DB access: Im not saying make it super locked down, Im saying set your DB security group to only be connected to from your API security group, and set an email warning when an instance launches in that group. Doesn't stop your engineers doing it, but it makes it pretty easy to shout across the office "Hey, who is fucking with prod and why?" Takes like 30 seconds to configure on AWS. There are serious questions about why your CEO can launch that instance though, and that sounds like massive policy failure. Again, restricting the CEO IAM from launching prod instances takes 30 seconds. (And all your non-engineer accounts should be IAM restricted!) Ignoring that your probabilities don't work out, Id argue that the situation is really remove 5 bullets or add 1 chamber, and people pick the chamber purely because it's constructive, not out of genuine cost-benefit analysis. "If I had 3 hours to chop a tree, Id sharpen my axe" -- I think startups are too hasty to chop, because that's being productive, right?