3 ms·

The question I've had is why didn't reddit SREs stop the DB edits as an attack. Even using valid creds, the queries should've been anomalies and logging should'

The question I've had is why didn't reddit SREs stop the DB edits as an attack. Even using valid creds, the queries should've been anomalies and logging should've picked up the unusual access pattern. There's three possibilities that come to me: 1. The CEO explicitly told reliability/security staff to ignore his edits. Policy fail of epic proportions. 2. The edits were performed using designed tools and queries, which means reddit had before now intended to make silent edits to user data as part of regular business. 3. Reddit lacks the internal safeguards to know if user data was unexpectedly written to. This option is dangerous to hosting code on reddit, because an attacker could silently, maliciously change examples without reddit ever detecting the damage (or users seeing an edit mark). I expect that we're in case 2, followed by 3 and then 1 (in terms of likelihood), but none of the ways that a large tech company could have this happen are reassuring about P&P at reddit, or make me want to rely on them for hosting a community.

stouset 10y ago
> Reddit SREs You've been working at Google too long. Reddit has on the order of a few dozen employees, not all of them technical. Welcome to how the rest of the Internet works.
- SomeStupidPoint 10y ago
  I've never worked at any company with more than 50 employees, and have worked on a team with as few as 5 engineers. Even without an explicit SRE role, someone is doing that work -- keeping the servers (reliably) on the internet and performing, mitigating attacks, etc. You should be able to tag the row with a dirty flag on second (or later) write using rules internal to the DB without incurring a large performance penalty, if you already have any kind of sanity checking. So at a basic technical level, this is bad design or an intentional capability. Similarly, even on the team with 5 engineers, our production DB didn't have a way to directly access it, and tunneling through an API box to get network access would trigger an access warning to the appropriate person, because you shouldn't be accessing prod API servers directly either. (Similarly, spinning up a new instance in a security group to inherit permission.) This isn't detecting advanced exfiltration techniques using information theory... It's basic network topology and server monitoring. And I suspect that reddit has at least a couple ops guys. This really smacks of intentional capability, followed to lazy design.
  - kaishiro 10y ago
    Either you are the outlier or I am - I'm honestly not sure. But having consulted with a dozen or so digital agencies over the past year not a single one had the sort of monitoring that you're suggesting. This doesn't mean that it's a bad idea, quite the opposite, but in my experience the reality of a small to medium sized agency is that if a client isn't paying for it than you're not working on it.
    - SomeStupidPoint 10y ago
      It's what I consider the difference between a software developer and a computer-systems engineer to be. I'm likely the outlier, because my early mentors (college + first couple years of career) were mostly aerospace engineers or chemical processing equipment engineers, where safety is critical. Code can kill in both those fields. (They themselves had adapted a lot of systems and failure mode analysis from mech engineers in the same field.) I don't think that the mitigations I pointed out require much technical work though -- most of them can be as simple as a single setting, extra program installed during build (and maybe a couple config settings), or a single cloud monitoring rule (eg, detecting launches). They do, however, require adhering to policy, discipline on the part of staff, and analyzing the full scope of your tech (and how it interacts). I guess my point is that these things often aren't done because of ignorance or haste rather than engineering cost (or legitimate need to be done faster). It's just we have poor standards for what "engineering" is in tech, so we don't cross our 't's and dot our 'i's the way a mechanical engineer is expected to. But that's no reason not to try and raise the bar.
  - stouset 10y ago
    > Even without an explicit SRE role, someone is doing that work -- keeping the servers (reliably) on the internet and performing, mitigating attacks, etc. To be honest, not in any of the several places I've worked. And I'm a security engineer. Lean startups are desperately trying to prove their product; time is spent on reliability, performance, and security almost entirely reactively and not proactively. I'm not saying this is a good thing. Bad security (and reliability) posture early on inflicts multiples of the original mitigation cost for years. And yet, it's rational. Time spent securing systems from your own employees when you're tiny is time you're not building your product and finding your market. It's a perverse game of Russian roulette: there are 99 bullets and one empty chamber. The analogy is going to break down here, but spending time and money removing one bullet (1:98 chance of death) is less effective than adding another empty chamber (2:99 chance of death). Especially when having direct access to the database is insanely useful to fix the problems that moving fast and breaking things inevitably causes, only a tiny fraction of startups are going to enact the kinds of measures you mention.

stouset 10y ago

> Reddit SREs You've been working at Google too long. Reddit has on the order of a few dozen employees, not all of them technical. Welcome to how the rest of the Internet works.

SomeStupidPoint 10y ago
I've never worked at any company with more than 50 employees, and have worked on a team with as few as 5 engineers. Even without an explicit SRE role, someone is doing that work -- keeping the servers (reliably) on the internet and performing, mitigating attacks, etc. You should be able to tag the row with a dirty flag on second (or later) write using rules internal to the DB without incurring a large performance penalty, if you already have any kind of sanity checking. So at a basic technical level, this is bad design or an intentional capability. Similarly, even on the team with 5 engineers, our production DB didn't have a way to directly access it, and tunneling through an API box to get network access would trigger an access warning to the appropriate person, because you shouldn't be accessing prod API servers directly either. (Similarly, spinning up a new instance in a security group to inherit permission.) This isn't detecting advanced exfiltration techniques using information theory... It's basic network topology and server monitoring. And I suspect that reddit has at least a couple ops guys. This really smacks of intentional capability, followed to lazy design.
- kaishiro 10y ago
  Either you are the outlier or I am - I'm honestly not sure. But having consulted with a dozen or so digital agencies over the past year not a single one had the sort of monitoring that you're suggesting. This doesn't mean that it's a bad idea, quite the opposite, but in my experience the reality of a small to medium sized agency is that if a client isn't paying for it than you're not working on it.
  - SomeStupidPoint 10y ago
    It's what I consider the difference between a software developer and a computer-systems engineer to be. I'm likely the outlier, because my early mentors (college + first couple years of career) were mostly aerospace engineers or chemical processing equipment engineers, where safety is critical. Code can kill in both those fields. (They themselves had adapted a lot of systems and failure mode analysis from mech engineers in the same field.) I don't think that the mitigations I pointed out require much technical work though -- most of them can be as simple as a single setting, extra program installed during build (and maybe a couple config settings), or a single cloud monitoring rule (eg, detecting launches). They do, however, require adhering to policy, discipline on the part of staff, and analyzing the full scope of your tech (and how it interacts). I guess my point is that these things often aren't done because of ignorance or haste rather than engineering cost (or legitimate need to be done faster). It's just we have poor standards for what "engineering" is in tech, so we don't cross our 't's and dot our 'i's the way a mechanical engineer is expected to. But that's no reason not to try and raise the bar.
- stouset 10y ago
  > Even without an explicit SRE role, someone is doing that work -- keeping the servers (reliably) on the internet and performing, mitigating attacks, etc. To be honest, not in any of the several places I've worked. And I'm a security engineer. Lean startups are desperately trying to prove their product; time is spent on reliability, performance, and security almost entirely reactively and not proactively. I'm not saying this is a good thing. Bad security (and reliability) posture early on inflicts multiples of the original mitigation cost for years. And yet, it's rational. Time spent securing systems from your own employees when you're tiny is time you're not building your product and finding your market. It's a perverse game of Russian roulette: there are 99 bullets and one empty chamber. The analogy is going to break down here, but spending time and money removing one bullet (1:98 chance of death) is less effective than adding another empty chamber (2:99 chance of death). Especially when having direct access to the database is insanely useful to fix the problems that moving fast and breaking things inevitably causes, only a tiny fraction of startups are going to enact the kinds of measures you mention.