3 ms·

The symbolic gesture that the project is disassociating is okay. Deleting the repo is not. >It's just too unreliable of a platform to be official in any regard

The symbolic gesture that the project is disassociating is okay. Deleting the repo is not. >It's just too unreliable of a platform to be official in any regard. The /r/golang subreddit is for devs who want answers and who don't want to be held up by the whims of stackoverflow mods. It isn't rocket science and even if we can't trust the answers, why would anyone edit the answers? If you remember, the reddit CEO only edited the usernames, that too, I am sure it was a prank. If anyone edits the messages which we post on /r/golang, it will lead to the answer being invalid and or something else, the world isn't going to collapse, there won't be congressional hearings just because some dev wasn't able to run gofmt and the answer posted by another random guy on the Internet was changed and can't be "trusted". We want answers on /r/golang and it isn't much about trust. Thank you for no longer proposing to delete it.

SomeStupidPoint 10y ago
The question I've had is why didn't reddit SREs stop the DB edits as an attack. Even using valid creds, the queries should've been anomalies and logging should've picked up the unusual access pattern. There's three possibilities that come to me: 1. The CEO explicitly told reliability/security staff to ignore his edits. Policy fail of epic proportions. 2. The edits were performed using designed tools and queries, which means reddit had before now intended to make silent edits to user data as part of regular business. 3. Reddit lacks the internal safeguards to know if user data was unexpectedly written to. This option is dangerous to hosting code on reddit, because an attacker could silently, maliciously change examples without reddit ever detecting the damage (or users seeing an edit mark). I expect that we're in case 2, followed by 3 and then 1 (in terms of likelihood), but none of the ways that a large tech company could have this happen are reassuring about P&P at reddit, or make me want to rely on them for hosting a community.
- stouset 10y ago
  > Reddit SREs You've been working at Google too long. Reddit has on the order of a few dozen employees, not all of them technical. Welcome to how the rest of the Internet works.
  - SomeStupidPoint 10y ago
    I've never worked at any company with more than 50 employees, and have worked on a team with as few as 5 engineers. Even without an explicit SRE role, someone is doing that work -- keeping the servers (reliably) on the internet and performing, mitigating attacks, etc. You should be able to tag the row with a dirty flag on second (or later) write using rules internal to the DB without incurring a large performance penalty, if you already have any kind of sanity checking. So at a basic technical level, this is bad design or an intentional capability. Similarly, even on the team with 5 engineers, our production DB didn't have a way to directly access it, and tunneling through an API box to get network access would trigger an access warning to the appropriate person, because you shouldn't be accessing prod API servers directly either. (Similarly, spinning up a new instance in a security group to inherit permission.) This isn't detecting advanced exfiltration techniques using information theory... It's basic network topology and server monitoring. And I suspect that reddit has at least a couple ops guys. This really smacks of intentional capability, followed to lazy design.
    - kaishiro 10y ago
      Either you are the outlier or I am - I'm honestly not sure. But having consulted with a dozen or so digital agencies over the past year not a single one had the sort of monitoring that you're suggesting. This doesn't mean that it's a bad idea, quite the opposite, but in my experience the reality of a small to medium sized agency is that if a client isn't paying for it than you're not working on it.
      - SomeStupidPoint 10y ago
        It's what I consider the difference between a software developer and a computer-systems engineer to be. I'm likely the outlier, because my early mentors (college + first couple years of career) were mostly aerospace engineers or chemical processing equipment engineers, where safety is critical. Code can kill in both those fields. (They themselves had adapted a lot of systems and failure mode analysis from mech engineers in the same field.) I don't think that the mitigations I pointed out require much technical work though -- most of them can be as simple as a single setting, extra program installed during build (and maybe a couple config settings), or a single cloud monitoring rule (eg, detecting launches). They do, however, require adhering to policy, discipline on the part of staff, and analyzing the full scope of your tech (and how it interacts). I guess my point is that these things often aren't done because of ignorance or haste rather than engineering cost (or legitimate need to be done faster). It's just we have poor standards for what "engineering" is in tech, so we don't cross our 't's and dot our 'i's the way a mechanical engineer is expected to. But that's no reason not to try and raise the bar.
madeofpalk 10y ago
> If you remember, the reddit CEO only edited the usernames, that too, I am sure it was a prank. He edited the mention of a username in a comment and completely changed the meaning of the comment by changing the target. I'm sure he thought it was a prank, but that shows extremely poor judgement of the CEO of a company that calls itself "the front page of the internet". Just imagine how unacceptable it would be for the Facebook CEO to edit someone's status, or Twitter to edit tweets.
- thewhitetulip 10y ago
  I know, it was terrible. What's more terrible is that he accepted doing that. now _that_ is a lack of judgement. Would the world have noticed that some "fuck you" comments have been changed in /r/TheDonald's subreddit? nopes!

SomeStupidPoint 10y ago

The question I've had is why didn't reddit SREs stop the DB edits as an attack. Even using valid creds, the queries should've been anomalies and logging should've picked up the unusual access pattern. There's three possibilities that come to me: 1. The CEO explicitly told reliability/security staff to ignore his edits. Policy fail of epic proportions. 2. The edits were performed using designed tools and queries, which means reddit had before now intended to make silent edits to user data as part of regular business. 3. Reddit lacks the internal safeguards to know if user data was unexpectedly written to. This option is dangerous to hosting code on reddit, because an attacker could silently, maliciously change examples without reddit ever detecting the damage (or users seeing an edit mark). I expect that we're in case 2, followed by 3 and then 1 (in terms of likelihood), but none of the ways that a large tech company could have this happen are reassuring about P&P at reddit, or make me want to rely on them for hosting a community.

stouset 10y ago
> Reddit SREs You've been working at Google too long. Reddit has on the order of a few dozen employees, not all of them technical. Welcome to how the rest of the Internet works.
- SomeStupidPoint 10y ago
  I've never worked at any company with more than 50 employees, and have worked on a team with as few as 5 engineers. Even without an explicit SRE role, someone is doing that work -- keeping the servers (reliably) on the internet and performing, mitigating attacks, etc. You should be able to tag the row with a dirty flag on second (or later) write using rules internal to the DB without incurring a large performance penalty, if you already have any kind of sanity checking. So at a basic technical level, this is bad design or an intentional capability. Similarly, even on the team with 5 engineers, our production DB didn't have a way to directly access it, and tunneling through an API box to get network access would trigger an access warning to the appropriate person, because you shouldn't be accessing prod API servers directly either. (Similarly, spinning up a new instance in a security group to inherit permission.) This isn't detecting advanced exfiltration techniques using information theory... It's basic network topology and server monitoring. And I suspect that reddit has at least a couple ops guys. This really smacks of intentional capability, followed to lazy design.
  - kaishiro 10y ago
    Either you are the outlier or I am - I'm honestly not sure. But having consulted with a dozen or so digital agencies over the past year not a single one had the sort of monitoring that you're suggesting. This doesn't mean that it's a bad idea, quite the opposite, but in my experience the reality of a small to medium sized agency is that if a client isn't paying for it than you're not working on it.
    - SomeStupidPoint 10y ago
      It's what I consider the difference between a software developer and a computer-systems engineer to be. I'm likely the outlier, because my early mentors (college + first couple years of career) were mostly aerospace engineers or chemical processing equipment engineers, where safety is critical. Code can kill in both those fields. (They themselves had adapted a lot of systems and failure mode analysis from mech engineers in the same field.) I don't think that the mitigations I pointed out require much technical work though -- most of them can be as simple as a single setting, extra program installed during build (and maybe a couple config settings), or a single cloud monitoring rule (eg, detecting launches). They do, however, require adhering to policy, discipline on the part of staff, and analyzing the full scope of your tech (and how it interacts). I guess my point is that these things often aren't done because of ignorance or haste rather than engineering cost (or legitimate need to be done faster). It's just we have poor standards for what "engineering" is in tech, so we don't cross our 't's and dot our 'i's the way a mechanical engineer is expected to. But that's no reason not to try and raise the bar.

madeofpalk 10y ago

> If you remember, the reddit CEO only edited the usernames, that too, I am sure it was a prank. He edited the mention of a username in a comment and completely changed the meaning of the comment by changing the target. I'm sure he thought it was a prank, but that shows extremely poor judgement of the CEO of a company that calls itself "the front page of the internet". Just imagine how unacceptable it would be for the Facebook CEO to edit someone's status, or Twitter to edit tweets.

thewhitetulip 10y ago
I know, it was terrible. What's more terrible is that he accepted doing that. now _that_ is a lack of judgement. Would the world have noticed that some "fuck you" comments have been changed in /r/TheDonald's subreddit? nopes!