10 ms·

Fuck this story, and shame on ArsTechnica for publishing it. We shouldn't call out and shame individuals for making a mistake like this. Any company which even

by 015a 6y ago

Fuck this story, and shame on ArsTechnica for publishing it. We shouldn't call out and shame individuals for making a mistake like this. Any company which even does something like this internally is toxic; the blame lies on the company and the process, not the individual. The flimsy excuse "oh, theys a bawd bawd company filled with icky people" doesn't hold water; we're talking about basic decency here. You want to write a story about the commit? Hold the company responsible? Encourage processes and procedures which would have caught this? That's fine. Blur out the names. Ars isn't a tabloid, but they're acting like one; there are real, serious consequences for shit like this. You didn't set out to create the perfect ammunition for someone within a company arguing against the open sourcing of some project, but here it is. I'm physically disgusted by this reporting.

ratsmack 6y ago
Remember that ArsTechnica does reporting for money. The more sensational, the more clicks. I think National Inquirer and The Sun uses the same tact.
- uniqueid 6y ago
  If I were a journalist who wanted to rake in money it wouldn't occur to me to include dull screenshots of diffs and commit messages. For that matter, it wouldn't occur to me to work for a niche outfit like Ars Technica either. My dentist probably earns more than a tech columnist.
smt88 6y ago
What about when people went after the security head of Equifax following their massive breach that ruined a few lives? Was that unfair?
- spydum 6y ago
  Incredibly unfair, the character assassinations are totally uncalled for. The infosec community has a real problem with tearing each other apart. So much so, that journalists now also think it's okay. * Just a reminder, they were tearing into the CISO for having a liberal arts degree (as if college degrees actually prepared you for anything..). Totally agree here: blame the company and the leadership for not investing in the right processes and tools. People will always make mistakes.
  - ekelsen 6y ago
    Blame the people at the company who didn't make sure the right processes and tools were in place? It seems like while CTO made the error, he was also responsible for making sure he wouldn't be able to make such an error. He failed at doing his job as a CTO and then failed at being a moderately competent entry level software engineer.
  - etchalon 6y ago
    So … we should blame the leadership but not blame the person in leadership who made the mistake? That … doesn't make any sense.
- themacguffinman 6y ago
  I think it's unfair. A single person doesn't bear the whole responsibility of a breach like Equifax's. The breach is just the final symptom of deeper problems with company leadership & policy allowed to fester by ineffective public oversight and bad incentives. The witch hunt will change nothing and absent policy, it will probably happen again because the specific people were never the problem.
  - andreareina 6y ago
    The C-levels are the leadership. The buck stops with them.
    - themacguffinman 6y ago
      A single person is not "the leadership". Blaming just corporate leadership is also unproductive; if incentives and lack of oversight encourage this outcome, punishing and replacing the leadership is just playing musical chairs. Saying "the buck stops with them" is pretty useless unless your goal is simply to declare a target for emotional rage.
      - emodendroket 6y ago
        Well gee, in this framework it seems like there is no way for anyone to ever be negligent or liable for anything they do, no matter how ill-considered.
      - andreareina 6y ago
        Isn't the (nominal) reason the people at the top get paid so much is because they're supposed to take responsibility? If not them, then who? They're supposed to set the incentives and culture so that the people below them in the org chart do the right thing. It's their job to know that their suppliers aren't using sweatshop labor to produce their goods. It's their job to make sure managers don't lean on the rank and file so much as to incentivize fraudulent behavior (but the managers would never outright say to do these things, oh no). "Don't ask, don't tell" isn't an excuse. And if the greater environment is such that "everybody does it", well maybe we need better regulations and/or policies, but that still does not absolve them individually. The people at the top might not be at fault, but they sure as hell are responsible.
- slickrick216 6y ago
  Deeply unfair bordering on sexist and elitist. People mentioning her degree was one of the most nasty things I’ve seen.
  - smt88 6y ago
    So you don't think it's newsworthy that the most sensitive financial info held for every American was supposed to be secured by someone with no security background? People's lives were destroyed by that breach. It wasn't a victimless mistake. The public wanted to know how it happened, and hiring the wrong person is part of how it happened.
dj_mc_merlin 6y ago
I was of the same opinion as you before reading the article.. but really? Genuinely just putting user input in SQL queries? I mean, everybody starts somewhere.. but how the hell do you get to be CTO of anything when nothing goes off in your brain when you're interpolating raw user input into SQL queries? WHAT? How? Everyone slips up, concatenates strings when they shouldn't, whatever.. but this genuinely looks like he didn't know that SQL injections are a thing. That is unacceptable for a technical leader.
- a012 6y ago
  In my book, any C levels shouldn't have direct access to production, and espectially when you become a CTO you stop coding.
  - carmen_sandiego 6y ago
    Well, that surely also depends how big the company is.
  - exclusiv 6y ago
    This mistake looks like he just started coding.
    - im3w1l 6y ago
      The only thing experience will do is decrease the frequency of errors. I mean this is why there is a movement to prepared statements and safer language. None of us is smart enough to sanitize correctly or avoid buffer overflows.
      - exclusiv 6y ago
        > None of us is smart enough to sanitize correctly It's just something that isn't and shouldn't be a concern of most engineers. I don't care about these things in terms of solving them. You entrust the collective wisdom in each area like security for these types of things. Which is exactly why it's hard to believe a former Facebook, veteran CTO would commit some code like this. No legitimate electrical engineer would do the equivalent.
markdown 6y ago
On the contrary, more of this please. Personal accountability is sorely lacking these days, from the heads of Fortune 500 companies getting away with golden parachutes after ruining countless lives in 2008, to shit like in the Ars story. Sure, hold the company responsible, but don't just let the person get away with it so they can run off and do the same at another company. Crooks and arseholes go from company to company causing damage everywhere they go because firing companies won't tell anyone while they let a person go. This seems to be a result of just that culture of silence, because how else can you explain someone becoming CTO of an org like this without know basic stuff about their own area of expertise? When minions make mistake, let the company take the heat. But when the person screwing up is in leadership and therefore directly responsible, it ought to fall on them.
- markdown 6y ago
  Here's another example from today of a PM who moves on after screwing US taxpayers: https://news.ycombinator.com/item?id=26336252 https://news.ycombinator.com/item?id=26336252
kstrauser 6y ago
> You didn't set out to create the perfect ammunition for someone within a company arguing against the open sourcing of some project, but here it is. One important note: Gab is currently running a fork of the Mastodon software, licensed under the AGPL-3.0. It's not their right to decide whether they want to publish their source code.
- sokoloff 6y ago
  It is however their option whether they want to squash, rebase, and/or rename committers when they share the source code as they are obligated to do under AGPL.
- shp0ngle 6y ago
  They do provide the source code. Just not as a git repo.
sokoloff 6y ago
At our company, we would not blur/obfuscate the actors' names for internal communications about technical errors. The key is that humans make errors, errors of commission, errors of omission, errors of misplaced/almost brilliance, and errors of outright apparent stupidity. IMO, you should dive deeply to understand exactly what happened (which includes who did what and why), but you don't use that as ammunition against the individual humans. Pretending that nameless, faceless committers did this thing isn't as useful as "even our CTO can make a stupid error [and not get fired for it]; here's how we seek to eliminate this entire category of error reaching production: <1> <2> <3>"
- themacguffinman 6y ago
  Internal communications are pretty different from a news article. A company can enforce a culture & policy for internally discussing mistakes. Publishing details in a popular outlet like Ars Technica opens up individual employees and the company in general to unproductive and unfair polemic, especially if the company in question is already unpopular for other reasons. It's no surprise that few companies open up their internal communication to public scrutiny.
  - sokoloff 6y ago
    I agree with you, though GP explicitly objected to it in internal comms, which is what I replied about.
lmm 6y ago
> We shouldn't call out and shame individuals for making a mistake like this. For a random line worker, sure. For the CTO, the position is absolutely relevant. You can't have a conversation about what processes or procedures might have caught this without taking that into account; in every small company I've worked for the CTO was either tacitly or explicitly able to bypass any amount of code review etc.. The fact that it's the CTO is informative, educational, and important, and should be reported. Their personal details are irrelevant but hiding them would also be meaningless.
- jdxcode 6y ago
  Yes, and I'd argue this was beyond a simple "mistake", it's borderline negligence for anyone but a very junior engineer: https://archive.vn/oxbck https://archive.vn/oxbck EDIT: that link isn't working, but here is a screenshot: https://imgur.com/4lyElZI https://imgur.com/4lyElZI
  - DaniloDias 6y ago
    I don’t know if we have comparable life experience. I have seen more shitty code than high quality code in my life. It’s hard for me to empathize with anyone who thinks startup code is going to generally meet some decent quality bar. People like Brian acton and Jan Korum exist, but they are the exception. If you always expect people to write good code, you will be fucking disappointed. The only solution that scales is having adequate speed bumps and guard rails that prevent people from pushing code till after it’s been properly tested. And just because it’s in a git log doesn’t mean it gets pushed to production. This article reminds me that journalists only make money when they squeeze your amygdala. Reading content from ars is the equivalent of an intellectual jump scare. Add 127.0.0.1 arstechnica.com to your resolv.conf file for mental health. Fuck off ars
    - deleted 6y ago
      [deleted]
    - mrzimmerman 6y ago
      Right, but the point is that either Gab had normal, reasonable safeguards in place (for the reasons you point out) and their CTO bypassed them creating a very serious security hole, or they didn’t have the safeguards in the first place meaning their internal processes are poorly set up and controlled. What’s important here is that the CTO is ultimately responsible for the failure either way. That’s the point of the article and it’s why the article is valid (even if you don’t like it for some reason): the engineering buck stops at the CTO, especially so if they personally create a bug that exfiltrates all of the company user data, regardless of how they managed to screw it up.
qzw 6y ago
>... the blame lies on the company and the process, not the individual. What about when the individual is responsible for establishing the processes? And when you blame "the company," are you not ultimately referring to those in charge, i.e., the C-level executives? After all, don't these same top executives claim a large share of the reward by arguing that their positions carry more responsibility and risk? So it seems entirely fair that they should expect to be held personally responsible when there are failures under their watch.
justapassenger 6y ago
CTO of a hot company, knowingly exposing user data (only 1 year CS students can claim that they don’t know that you have to sanitize any input into hand crafted SQL queries) totally deserves a public scrutiny.
- 0xy 6y ago
  You're using "knowingly" wrong, unless you can prove intent. You're also totally incorrect, considering how many companies have been hit with SQL injection vulns.
  - justapassenger 6y ago
    If I design a bridge without calculating load rating I’m knowingly, as an engineer, putting public safety at risk. Hand crafted SQL statements without sanitizing input in 2020 are same level of negligence from software engineers - you know you’re putting your system safety at risk.
    - deleted 6y ago
      [deleted]
    - goshx 6y ago
      You'd be surprised by how many software engineers still have no clue about security.
      - Arainach 6y ago
        If you were writing SQL queries and not sanitizing inputs in 2010, shame on you. If you're doing so in 2020, you have no excuse. Even if you never read any tech news, all of the major languages and frameworks make it easy to parameterize inputs and hard to do it wrong. Every SINGLE guide about writing SQL on the internet will give this exact code as an example of something not to do. If you're this ignorant, that kind of flagrant disregard means you don't deserve the title engineer - much less "CTO".
      - 6y ago
just_use_eval 6y ago
I mean ArsTechnica is owned by Condé Nast, which in turn owns Reddit. Reddit is very anti-conservative, and has been this way since at least 2014. It's not far-fetched to think that maybe Condé Nast, the parent company is lead by people with similar or compatible values, which in turn affects their other platforms as well (such as ArsTechnica). Another thing to consider is that Gab is sort of a competitor to Reddit, so why wouldn't they fight it. Reddit investors: "Reddit raised $50 million in a funding round led by Sam Altman and including investors Marc Andreessen, Peter Thiel, Ron Conway, Snoop Dogg, and Jared Leto."[1] - Sam Altman is an open democrat and has donated 250k USD in support of the democratic party.[2] - Peter Thiel is pro-LGBT and pro-Trump.[3] - Snoop Dogg is known for liking (and singing about) weed. I don't know about you, but this explains Reddit's front-page and their overall content policy quite well. There is not as much information about ArsTechnica as there is about Reddit, but if we do some research I think the realities of ArsTechnica's potential biases will reveal themselves as well. [1] https://en.wikipedia.org/wiki/Reddit https://en.wikipedia.org/wiki/Reddit [2] https://en.wikipedia.org/wiki/Sam_Altman https://en.wikipedia.org/wiki/Sam_Altman [3] https://www.nbcnews.com/feature/nbc-out/openly-gay-libertarian-tech-entrepreneur-peter-thiel-speak-rnc-n614191 https://www.nbcnews.com/feature/nbc-out/openly-gay-libertari...
ramraj07 6y ago
Take a step back and consider for a moment that these coders replaced their gitlab repo with a zip file with the password JesusChristIsKingTrumpWonTheElection; sounds more like over grown kids who never got an adult lesson run this site so a bit of name calling at the CTO level is warranted.
deleted 6y ago
[deleted]
rodgerd 6y ago
> We shouldn't call out and shame individuals for making a mistake like this. The first response of the GAB exec team to a bug which has exposed all the private information to being leaked was an expletive and slur-laden attack. I am quite comfortable with the CTO being treated the same way he is happy to treat others.
emodendroket 6y ago
He's literally the CTO. The buck stops with him. If the procedure's a mess, guess whose fault it is?
cyansmoker 6y ago
> Hold the company responsible? Technically, the CTO is an officer of the company. In a very real sense, he is (a piece of) the company. Furthermore, how would one "hold the company responsible?" -- review their security culture? Where do you think that culture comes from?
shalmanese 6y ago
Yeah, we should not blame software engineer Fosco Marotto for introducing a rookie error into the site, we should blame the person in charge of setting up standards and processes so such errors never make it into production, Fosco Marotto!
ggggtez 6y ago
> We shouldn't call out and shame individuals for making a mistake like this. Yes we should. This individual is the company. He's the CTO. "Chief" Technology Officer. The buck has to stop somewhere. How can you possibly suggest with a straight face that he should not be held professionally responsible for the code that: A) He personally wrote B) He is responsible for professionally as the CTO
ReactiveJelly 6y ago
> oh, theys a bawd bawd company wow is that what I sound like
ggggtez 6y ago
You also seemed to miss this part of the article, which is very telling beyond just shaming someone > Besides the commit raising questions about Gab’s process for developing secure code, the social media site is also facing criticism for removing the commits from its website. Critics say the move violates terms of the Affero General Public License, which governs Gab’s reuse of Mastodon, an open source software package for hosting social networking platforms. Gab is likely breaking license agreements by performing this sort of cover up. So yes, this is newsworthy.
- Karunamon 6y ago
  Correct me if I’m wrong, but you don’t need a link to the code to comply with the GPL. Providing it on request is still kosher.
- 015a 6y ago
  Gab is likely not breaking license agreements, as long as they provide it upon request. Which was never tested. That's the problem with reporting like this; it jumps to conclusions to justify outrage, while clearly being written by someone who is overworked, underpaid, underexperienced, and only has a cursory understanding of the systems at play. Of course, most people are underexperienced in domains like this, so they rely on traditionally high-quality reporting sites like Ars to provide accurate, precise, and fair reporting. The wording Ars used is accurate; your interpretation of their wording isn't. That's the responsibility journalists have; what they say matters, a lot, because people interpret what they say in different ways. Ars said that they're "being accused" of breaking license agreements. You said they're "likely" breaking license agreements. That's a world of difference; if for no other reason than because one is correct, but useless, and the other is (probably) wrong.
chovybizzass 6y ago
I got fired from people.ai for having a bug slip into production. I had put it behind a feature flag so only our employees could see. Still got fired.
29athrowaway 6y ago
The CTO of an org should be technically competent. If you are not technically competent, you have no business being a CTO. If a civil engineer makes an egregious mistake and a bridge collapses, it's not toxic to say an egregious mistake was made. And, in an organization, if you are unable to recognize and call out mistakes (in a respectful, constructive manner that is), you're a failed organization. Rookie mistake is certainly not a constructive way to report on this problem, but it is true.
atsushin 6y ago
Normally I'd be inclined to agree with you on this, your bottom-line software engineers shouldn't be outed and publicly shamed. In this case, however, the CTO, a person at the top of the org chart, must be cognizant of these sorts of vulnerabilities especially at a company with a growing userbase and a target on its back (mind you I don't support Gab or anything they stand for, it's just the reality of things). More importantly, you have the CEO actually, really trying to scapegoat the incident on 'mentally ill <t-slur> demon hackers' when its their own incompetence in the top ranks to blame. It's an exception to the rule.
eric4smith 6y ago
You know what's really a shame? People overusing the word "toxic". You used it here a couple times in here so far. I've been on the wrong end of a SQL injection once and I was called out publicly multiple times for it to my endless shame from my customers who bought the software and were hacked. Was that toxic for them to call me out? Heck no. In the 15 years since that time, I made triple-sure to never code something that could lead to a SQL injection again.
- climb_stealth 6y ago
  To be honest that situation you have been in sounds horrible. I would not wish that on anyone. There's this tweet floating on the web which this made me think of. Something to think about I guess. > If you have suffered in life and want other people to suffer as you did because you "turned out fine", you did not in fact turn out fine.
doggydogs94 6y ago
I agree with you regarding the tabloid journalism aspect of the ArsTechnica story.
jpeg_hero 6y ago
ultimate engineer entitlement: we are entitled to make $500k per year and not be accountable for our mistakes.
gfosco 6y ago
Hi HN! Great feedback here, as always. I can talk about the commit, and I can talk about the article. We were getting crushed with scaling from 3M to 30M visits a month, and every day was a 16 hour day. The site was falling over on a daily basis, and the performance of the home feed was atrocious. The APM traces were full of loops. The muting and blocking features were incurring tons of additional queries. As I remember it, one night I decided to write a SQL query to replace it. I believe I thought the values were already being sanitized, but I have no problem agreeing that I should've looked into that further and made sure. I'm not a rails dev, and I'm generally negative towards rails and ActiveRecord. I wrote SQL for many years in the past, and as the folks who went through my stackoverflow saw, I am not unaware of the importance of sanitizing user input. I've written much code that properly sanitized user input, in various languages. It's really easy to imagine that Gab is a large company, considering the breadth of their platform and products, but you'd be shocked and impressed. It's still really early days, like this is only Chapter 2 for us.. we're growing and will keep getting better. This article is interesting to me for other reasons though: 1. there's been no confirmation this commit was related to anything that has happened. and 2. the person who reached out to Ars with the story, a link to the commit and a quote, is a former co-worker with a personal grudge. I can only imagine the glee with which he wrote that email. For the last 2 days we were being extorted for nearly half a million dollars. I received a death threat today, first one ever. I think we're doing something important. Go ahead with your code critiques. Have a great day, Fosco
- marcus_holmes 6y ago
  I've been on the other end of a journalist with a grudge, and it's not something I'd wish on anyone. It's interesting that the technical language in the article is so clearly mangled. Either Ars Technica has no editors that know how to edit (unlikely), or that know the basics of git (again, it seems unlikely). It reads like a journalist taking notes from a dev ("uh-huh, a 'git', right. Oh, no a 'git commit'. really?").
- heyitsjlac 6y ago
  Diddums the people using your site send thousands of death threats on it every day. Try to toughen up, buttercup.
m463 6y ago
Step 1: assign blame. ugh.
postalrat 6y ago
Maybe programmers should stop using the word "engineer" to describe themselves if they don't want to hold responsibility.

by 015a 6y ago

ratsmack 6y ago

Remember that ArsTechnica does reporting for money. The more sensational, the more clicks. I think National Inquirer and The Sun uses the same tact.

uniqueid 6y ago
If I were a journalist who wanted to rake in money it wouldn't occur to me to include dull screenshots of diffs and commit messages. For that matter, it wouldn't occur to me to work for a niche outfit like Ars Technica either. My dentist probably earns more than a tech columnist.

smt88 6y ago

What about when people went after the security head of Equifax following their massive breach that ruined a few lives? Was that unfair?

spydum 6y ago
Incredibly unfair, the character assassinations are totally uncalled for. The infosec community has a real problem with tearing each other apart. So much so, that journalists now also think it's okay. * Just a reminder, they were tearing into the CISO for having a liberal arts degree (as if college degrees actually prepared you for anything..). Totally agree here: blame the company and the leadership for not investing in the right processes and tools. People will always make mistakes.
- ekelsen 6y ago
  Blame the people at the company who didn't make sure the right processes and tools were in place? It seems like while CTO made the error, he was also responsible for making sure he wouldn't be able to make such an error. He failed at doing his job as a CTO and then failed at being a moderately competent entry level software engineer.
- etchalon 6y ago
  So … we should blame the leadership but not blame the person in leadership who made the mistake? That … doesn't make any sense.
themacguffinman 6y ago
I think it's unfair. A single person doesn't bear the whole responsibility of a breach like Equifax's. The breach is just the final symptom of deeper problems with company leadership & policy allowed to fester by ineffective public oversight and bad incentives. The witch hunt will change nothing and absent policy, it will probably happen again because the specific people were never the problem.
- andreareina 6y ago
  The C-levels are the leadership. The buck stops with them.
  - themacguffinman 6y ago
    A single person is not "the leadership". Blaming just corporate leadership is also unproductive; if incentives and lack of oversight encourage this outcome, punishing and replacing the leadership is just playing musical chairs. Saying "the buck stops with them" is pretty useless unless your goal is simply to declare a target for emotional rage.
    - emodendroket 6y ago
      Well gee, in this framework it seems like there is no way for anyone to ever be negligent or liable for anything they do, no matter how ill-considered.
    - andreareina 6y ago
      Isn't the (nominal) reason the people at the top get paid so much is because they're supposed to take responsibility? If not them, then who? They're supposed to set the incentives and culture so that the people below them in the org chart do the right thing. It's their job to know that their suppliers aren't using sweatshop labor to produce their goods. It's their job to make sure managers don't lean on the rank and file so much as to incentivize fraudulent behavior (but the managers would never outright say to do these things, oh no). "Don't ask, don't tell" isn't an excuse. And if the greater environment is such that "everybody does it", well maybe we need better regulations and/or policies, but that still does not absolve them individually. The people at the top might not be at fault, but they sure as hell are responsible.
slickrick216 6y ago
Deeply unfair bordering on sexist and elitist. People mentioning her degree was one of the most nasty things I’ve seen.
- smt88 6y ago
  So you don't think it's newsworthy that the most sensitive financial info held for every American was supposed to be secured by someone with no security background? People's lives were destroyed by that breach. It wasn't a victimless mistake. The public wanted to know how it happened, and hiring the wrong person is part of how it happened.

dj_mc_merlin 6y ago

I was of the same opinion as you before reading the article.. but really? Genuinely just putting user input in SQL queries? I mean, everybody starts somewhere.. but how the hell do you get to be CTO of anything when nothing goes off in your brain when you're interpolating raw user input into SQL queries? WHAT? How? Everyone slips up, concatenates strings when they shouldn't, whatever.. but this genuinely looks like he didn't know that SQL injections are a thing. That is unacceptable for a technical leader.

a012 6y ago
In my book, any C levels shouldn't have direct access to production, and espectially when you become a CTO you stop coding.
- carmen_sandiego 6y ago
  Well, that surely also depends how big the company is.
- exclusiv 6y ago
  This mistake looks like he just started coding.
  - im3w1l 6y ago
    The only thing experience will do is decrease the frequency of errors. I mean this is why there is a movement to prepared statements and safer language. None of us is smart enough to sanitize correctly or avoid buffer overflows.
    - exclusiv 6y ago
      > None of us is smart enough to sanitize correctly It's just something that isn't and shouldn't be a concern of most engineers. I don't care about these things in terms of solving them. You entrust the collective wisdom in each area like security for these types of things. Which is exactly why it's hard to believe a former Facebook, veteran CTO would commit some code like this. No legitimate electrical engineer would do the equivalent.

markdown 6y ago

On the contrary, more of this please. Personal accountability is sorely lacking these days, from the heads of Fortune 500 companies getting away with golden parachutes after ruining countless lives in 2008, to shit like in the Ars story. Sure, hold the company responsible, but don't just let the person get away with it so they can run off and do the same at another company. Crooks and arseholes go from company to company causing damage everywhere they go because firing companies won't tell anyone while they let a person go. This seems to be a result of just that culture of silence, because how else can you explain someone becoming CTO of an org like this without know basic stuff about their own area of expertise? When minions make mistake, let the company take the heat. But when the person screwing up is in leadership and therefore directly responsible, it ought to fall on them.

markdown 6y ago
Here's another example from today of a PM who moves on after screwing US taxpayers: https://news.ycombinator.com/item?id=26336252 https://news.ycombinator.com/item?id=26336252

kstrauser 6y ago

> You didn't set out to create the perfect ammunition for someone within a company arguing against the open sourcing of some project, but here it is. One important note: Gab is currently running a fork of the Mastodon software, licensed under the AGPL-3.0. It's not their right to decide whether they want to publish their source code.

sokoloff 6y ago
It is however their option whether they want to squash, rebase, and/or rename committers when they share the source code as they are obligated to do under AGPL.
shp0ngle 6y ago
They do provide the source code. Just not as a git repo.

sokoloff 6y ago

At our company, we would not blur/obfuscate the actors' names for internal communications about technical errors. The key is that humans make errors, errors of commission, errors of omission, errors of misplaced/almost brilliance, and errors of outright apparent stupidity. IMO, you should dive deeply to understand exactly what happened (which includes who did what and why), but you don't use that as ammunition against the individual humans. Pretending that nameless, faceless committers did this thing isn't as useful as "even our CTO can make a stupid error [and not get fired for it]; here's how we seek to eliminate this entire category of error reaching production: <1> <2> <3>"

themacguffinman 6y ago
Internal communications are pretty different from a news article. A company can enforce a culture & policy for internally discussing mistakes. Publishing details in a popular outlet like Ars Technica opens up individual employees and the company in general to unproductive and unfair polemic, especially if the company in question is already unpopular for other reasons. It's no surprise that few companies open up their internal communication to public scrutiny.
- sokoloff 6y ago
  I agree with you, though GP explicitly objected to it in internal comms, which is what I replied about.

lmm 6y ago

> We shouldn't call out and shame individuals for making a mistake like this. For a random line worker, sure. For the CTO, the position is absolutely relevant. You can't have a conversation about what processes or procedures might have caught this without taking that into account; in every small company I've worked for the CTO was either tacitly or explicitly able to bypass any amount of code review etc.. The fact that it's the CTO is informative, educational, and important, and should be reported. Their personal details are irrelevant but hiding them would also be meaningless.

jdxcode 6y ago
Yes, and I'd argue this was beyond a simple "mistake", it's borderline negligence for anyone but a very junior engineer: https://archive.vn/oxbck https://archive.vn/oxbck EDIT: that link isn't working, but here is a screenshot: https://imgur.com/4lyElZI https://imgur.com/4lyElZI
- DaniloDias 6y ago
  I don’t know if we have comparable life experience. I have seen more shitty code than high quality code in my life. It’s hard for me to empathize with anyone who thinks startup code is going to generally meet some decent quality bar. People like Brian acton and Jan Korum exist, but they are the exception. If you always expect people to write good code, you will be fucking disappointed. The only solution that scales is having adequate speed bumps and guard rails that prevent people from pushing code till after it’s been properly tested. And just because it’s in a git log doesn’t mean it gets pushed to production. This article reminds me that journalists only make money when they squeeze your amygdala. Reading content from ars is the equivalent of an intellectual jump scare. Add 127.0.0.1 arstechnica.com to your resolv.conf file for mental health. Fuck off ars
  - deleted 6y ago
    [deleted]
  - mrzimmerman 6y ago
    Right, but the point is that either Gab had normal, reasonable safeguards in place (for the reasons you point out) and their CTO bypassed them creating a very serious security hole, or they didn’t have the safeguards in the first place meaning their internal processes are poorly set up and controlled. What’s important here is that the CTO is ultimately responsible for the failure either way. That’s the point of the article and it’s why the article is valid (even if you don’t like it for some reason): the engineering buck stops at the CTO, especially so if they personally create a bug that exfiltrates all of the company user data, regardless of how they managed to screw it up.

qzw 6y ago

>... the blame lies on the company and the process, not the individual. What about when the individual is responsible for establishing the processes? And when you blame "the company," are you not ultimately referring to those in charge, i.e., the C-level executives? After all, don't these same top executives claim a large share of the reward by arguing that their positions carry more responsibility and risk? So it seems entirely fair that they should expect to be held personally responsible when there are failures under their watch.

justapassenger 6y ago

CTO of a hot company, knowingly exposing user data (only 1 year CS students can claim that they don’t know that you have to sanitize any input into hand crafted SQL queries) totally deserves a public scrutiny.

0xy 6y ago
You're using "knowingly" wrong, unless you can prove intent. You're also totally incorrect, considering how many companies have been hit with SQL injection vulns.
- justapassenger 6y ago
  If I design a bridge without calculating load rating I’m knowingly, as an engineer, putting public safety at risk. Hand crafted SQL statements without sanitizing input in 2020 are same level of negligence from software engineers - you know you’re putting your system safety at risk.
  - deleted 6y ago
    [deleted]
  - goshx 6y ago
    You'd be surprised by how many software engineers still have no clue about security.
    - Arainach 6y ago
      If you were writing SQL queries and not sanitizing inputs in 2010, shame on you. If you're doing so in 2020, you have no excuse. Even if you never read any tech news, all of the major languages and frameworks make it easy to parameterize inputs and hard to do it wrong. Every SINGLE guide about writing SQL on the internet will give this exact code as an example of something not to do. If you're this ignorant, that kind of flagrant disregard means you don't deserve the title engineer - much less "CTO".
    - 6y ago

just_use_eval 6y ago

I mean ArsTechnica is owned by Condé Nast, which in turn owns Reddit. Reddit is very anti-conservative, and has been this way since at least 2014. It's not far-fetched to think that maybe Condé Nast, the parent company is lead by people with similar or compatible values, which in turn affects their other platforms as well (such as ArsTechnica). Another thing to consider is that Gab is sort of a competitor to Reddit, so why wouldn't they fight it. Reddit investors: "Reddit raised $50 million in a funding round led by Sam Altman and including investors Marc Andreessen, Peter Thiel, Ron Conway, Snoop Dogg, and Jared Leto."[1] - Sam Altman is an open democrat and has donated 250k USD in support of the democratic party.[2] - Peter Thiel is pro-LGBT and pro-Trump.[3] - Snoop Dogg is known for liking (and singing about) weed. I don't know about you, but this explains Reddit's front-page and their overall content policy quite well. There is not as much information about ArsTechnica as there is about Reddit, but if we do some research I think the realities of ArsTechnica's potential biases will reveal themselves as well. [1] https://en.wikipedia.org/wiki/Reddit https://en.wikipedia.org/wiki/Reddit [2] https://en.wikipedia.org/wiki/Sam_Altman https://en.wikipedia.org/wiki/Sam_Altman [3] https://www.nbcnews.com/feature/nbc-out/openly-gay-libertarian-tech-entrepreneur-peter-thiel-speak-rnc-n614191 https://www.nbcnews.com/feature/nbc-out/openly-gay-libertari...

ramraj07 6y ago

Take a step back and consider for a moment that these coders replaced their gitlab repo with a zip file with the password JesusChristIsKingTrumpWonTheElection; sounds more like over grown kids who never got an adult lesson run this site so a bit of name calling at the CTO level is warranted.

deleted 6y ago

[deleted]

rodgerd 6y ago

> We shouldn't call out and shame individuals for making a mistake like this. The first response of the GAB exec team to a bug which has exposed all the private information to being leaked was an expletive and slur-laden attack. I am quite comfortable with the CTO being treated the same way he is happy to treat others.

emodendroket 6y ago

He's literally the CTO. The buck stops with him. If the procedure's a mess, guess whose fault it is?

cyansmoker 6y ago

> Hold the company responsible? Technically, the CTO is an officer of the company. In a very real sense, he is (a piece of) the company. Furthermore, how would one "hold the company responsible?" -- review their security culture? Where do you think that culture comes from?

shalmanese 6y ago

Yeah, we should not blame software engineer Fosco Marotto for introducing a rookie error into the site, we should blame the person in charge of setting up standards and processes so such errors never make it into production, Fosco Marotto!

ggggtez 6y ago

> We shouldn't call out and shame individuals for making a mistake like this. Yes we should. This individual is the company. He's the CTO. "Chief" Technology Officer. The buck has to stop somewhere. How can you possibly suggest with a straight face that he should not be held professionally responsible for the code that: A) He personally wrote B) He is responsible for professionally as the CTO

ReactiveJelly 6y ago

> oh, theys a bawd bawd company wow is that what I sound like

ggggtez 6y ago

You also seemed to miss this part of the article, which is very telling beyond just shaming someone > Besides the commit raising questions about Gab’s process for developing secure code, the social media site is also facing criticism for removing the commits from its website. Critics say the move violates terms of the Affero General Public License, which governs Gab’s reuse of Mastodon, an open source software package for hosting social networking platforms. Gab is likely breaking license agreements by performing this sort of cover up. So yes, this is newsworthy.

Karunamon 6y ago
Correct me if I’m wrong, but you don’t need a link to the code to comply with the GPL. Providing it on request is still kosher.
015a 6y ago
Gab is likely not breaking license agreements, as long as they provide it upon request. Which was never tested. That's the problem with reporting like this; it jumps to conclusions to justify outrage, while clearly being written by someone who is overworked, underpaid, underexperienced, and only has a cursory understanding of the systems at play. Of course, most people are underexperienced in domains like this, so they rely on traditionally high-quality reporting sites like Ars to provide accurate, precise, and fair reporting. The wording Ars used is accurate; your interpretation of their wording isn't. That's the responsibility journalists have; what they say matters, a lot, because people interpret what they say in different ways. Ars said that they're "being accused" of breaking license agreements. You said they're "likely" breaking license agreements. That's a world of difference; if for no other reason than because one is correct, but useless, and the other is (probably) wrong.

chovybizzass 6y ago

I got fired from people.ai for having a bug slip into production. I had put it behind a feature flag so only our employees could see. Still got fired.

29athrowaway 6y ago

The CTO of an org should be technically competent. If you are not technically competent, you have no business being a CTO. If a civil engineer makes an egregious mistake and a bridge collapses, it's not toxic to say an egregious mistake was made. And, in an organization, if you are unable to recognize and call out mistakes (in a respectful, constructive manner that is), you're a failed organization. Rookie mistake is certainly not a constructive way to report on this problem, but it is true.

atsushin 6y ago

Normally I'd be inclined to agree with you on this, your bottom-line software engineers shouldn't be outed and publicly shamed. In this case, however, the CTO, a person at the top of the org chart, must be cognizant of these sorts of vulnerabilities especially at a company with a growing userbase and a target on its back (mind you I don't support Gab or anything they stand for, it's just the reality of things). More importantly, you have the CEO actually, really trying to scapegoat the incident on 'mentally ill <t-slur> demon hackers' when its their own incompetence in the top ranks to blame. It's an exception to the rule.

eric4smith 6y ago

You know what's really a shame? People overusing the word "toxic". You used it here a couple times in here so far. I've been on the wrong end of a SQL injection once and I was called out publicly multiple times for it to my endless shame from my customers who bought the software and were hacked. Was that toxic for them to call me out? Heck no. In the 15 years since that time, I made triple-sure to never code something that could lead to a SQL injection again.

climb_stealth 6y ago
To be honest that situation you have been in sounds horrible. I would not wish that on anyone. There's this tweet floating on the web which this made me think of. Something to think about I guess. > If you have suffered in life and want other people to suffer as you did because you "turned out fine", you did not in fact turn out fine.

doggydogs94 6y ago

I agree with you regarding the tabloid journalism aspect of the ArsTechnica story.

jpeg_hero 6y ago

ultimate engineer entitlement: we are entitled to make $500k per year and not be accountable for our mistakes.

gfosco 6y ago

Hi HN! Great feedback here, as always. I can talk about the commit, and I can talk about the article. We were getting crushed with scaling from 3M to 30M visits a month, and every day was a 16 hour day. The site was falling over on a daily basis, and the performance of the home feed was atrocious. The APM traces were full of loops. The muting and blocking features were incurring tons of additional queries. As I remember it, one night I decided to write a SQL query to replace it. I believe I thought the values were already being sanitized, but I have no problem agreeing that I should've looked into that further and made sure. I'm not a rails dev, and I'm generally negative towards rails and ActiveRecord. I wrote SQL for many years in the past, and as the folks who went through my stackoverflow saw, I am not unaware of the importance of sanitizing user input. I've written much code that properly sanitized user input, in various languages. It's really easy to imagine that Gab is a large company, considering the breadth of their platform and products, but you'd be shocked and impressed. It's still really early days, like this is only Chapter 2 for us.. we're growing and will keep getting better. This article is interesting to me for other reasons though: 1. there's been no confirmation this commit was related to anything that has happened. and 2. the person who reached out to Ars with the story, a link to the commit and a quote, is a former co-worker with a personal grudge. I can only imagine the glee with which he wrote that email. For the last 2 days we were being extorted for nearly half a million dollars. I received a death threat today, first one ever. I think we're doing something important. Go ahead with your code critiques. Have a great day, Fosco

marcus_holmes 6y ago
I've been on the other end of a journalist with a grudge, and it's not something I'd wish on anyone. It's interesting that the technical language in the article is so clearly mangled. Either Ars Technica has no editors that know how to edit (unlikely), or that know the basics of git (again, it seems unlikely). It reads like a journalist taking notes from a dev ("uh-huh, a 'git', right. Oh, no a 'git commit'. really?").
heyitsjlac 6y ago
Diddums the people using your site send thousands of death threats on it every day. Try to toughen up, buttercup.

m463 6y ago

Step 1: assign blame. ugh.

postalrat 6y ago

Maybe programmers should stop using the word "engineer" to describe themselves if they don't want to hold responsibility.