8 ms·

Mudge's redacted Twitter whistleblower disclosure [pdf]

122 points by vngzs 4y ago

lkjlkj3t4 4y ago
ratsmack 4y ago
I read most of the document, and even if a fraction of it is true, Twitter will have a lot of explaining to do in front of the SEC. It sounds like it would be easy to find criminal intent.
rvz 4y ago
Twitter is in dire hot water over this and are going to have a nice conversation with Elon, Jack, etc in court with the SEC, and the FTC watching them over this potential case of fraud with this disclosure. Did Twitter really think that continuous denial, lies and more PR deception was going to be that easy to get away with? Not this time.
- bsder 4y ago
  > Did Twitter really think that continuous denial, lies and more PR deception was going to be that easy to get away with? If Elon hadn't just happened to tender an offer, want to back out, and put them under a court spotlight ... yeah, they probably did. And, to be fair, unless someone goes to jail, then ... yeah, they'll just get away with it as the cost of doing business.
johng 4y ago
From my reading it looks like Agrawal might deserve some jail time... this is egregious? Thoughts?
- mandeepj 4y ago
  What are the chances - Vijaya, Parag, Dorsey, few more of their execs and board members - are in a WAR ROOM call on daily basis?
- danielmarkbruce 4y ago
  For what exactly?
- ryan_lane 4y ago
  Most of the complaints in the whistleblower docs aren't about anything illegal. There's some places where they show that twitter's public statements, after a security incident, around their security posture were misleading. That's potentially illegal, but even that could be a grey area. There's some places where they show twitter didn't comply with creating and enforcing an SLDC, but if you've ever worked on an SLDC, you know it's a joke that's there for compliance checkbox reasons. Also, this is certainly a grey area because most likely to actually comply with this, you only need to have a document generated, training for how it should be used, and very basic processes in your git workflows. I'd be surprised if they hadn't done enough to legally comply with this. The statements around "employees with production access" are vague. What _kind_ of production access do they have? If you're an engineer, you have production access, because your code can be deployed there. If you're in ops/support, you have production access, because you need to access user data. Hard to properly judge the seriousness of this without knowing more details. For me, the lack of audit access seems to be the biggest legal issue, especially if they were lying to regulators about their progress. It's not that the accusations aren't bad. They paint a picture of an org that has serious security issues, seriously management and organizational health/culture issues, flailing executives, and an incompetent board. But, it really seems like most of it simply isn't illegal.
  - nrmitchi 4y ago
    I'll agree with most of these statements, but would object to "they show twitter didn't comply with creating and enforcing...". I don't think it's clear that they have shown anything. This is simply a complaint making statements without any real form of evidence. As you say in your final statement, this is just painting a picture of an organization. It's definitely not fair to take everything stated here at face value with all of the context which is clearly missing.
    - ryan_lane 4y ago
      It says in the document that there is an SLDC defined, but that the company isn't fully complying with it.
- fakethenews2022 4y ago
  Giving the Indian government access to your data when they allowed the Indian government to specify an individual to hire and then not have internal controls on what information that person has access to. Sounds very similar to how the Saudi Arabian government spied on Twitter users using a plant but in this case knowingly supported by Twitter executives.
bushbaba 4y ago
I really hope that's not true. Ooph. "Twitter data centers were fragile, and Twitter lacked plans and processes to “cold boot.” That meant that if all the centers went offline simultaneously, even briefly, Twitter was unsure if they could bring the service back up. Downtime estimates ranged from weeks of round-the-clock work, to permanent irreparable failure"
- chickin 4y ago
  How many large tech companies do you think can do that? Each service wants to benefit from the use of other services. Ensuring there aren't dependency loops in this complex graph sounds tricky.
  - danielmarkbruce 4y ago
    Google wasn't 100% certain they could cold boot around 2015. Not sure if it's still true. It's crazy difficult to figure out if you can at some point.
    - yuhong 4y ago
      Part of the point of SSD based search engines is they are much easier to cold boot as well.
  - ryan_lane 4y ago
    You have to have enough people to "be that asshole" that tells people in their design docs that they are creating a dependency loop and need to design things to avoid it. I did that for quite a while at Lyft, and I'm hoping that people continued doing that after I stopped. My guess, though, is that I and others have missed things and that even somewhere that was careful to avoid it, there's still some places that could make cold boots difficult. The best way to avoid this problem is to have a secondary datacenter that is hot, and to never let both fail. Of course, this is what twitter is doing. Designing a DR plan where you can lose multiple primary/secondary datacenters is hard enough that basically no one does it. This is something that really doesn't belong in the whistleblower complaints because it wasn't under his responsibilities, twitter wasn't lying to anyone about having properly implemented DR plans, and twitter's DR plan isn't out of the ordinary across the industry.
maxbond 4y ago
This document feels out of place to me. It's addressed to various regulators, and at a surface level it has the form of an affidavit or other legal document. But the language is far too familiar to be writing for that audience - they refer to Mudge consistently as Mudge rather than, say, "Mr. Zatko"; they make liberal use of superlatives; they add emphasis using bold text. This is kind of like writing a cover letter where you explain why the job would be great for you rather than why you would be great for the job, it's going to sound wrong to your intended audience, and it will degrade your credibility in their eyes (rightly or wrongly). This isn't the style you would adopt if you wanted such austere organizations as the SEC, DOJ, and they even mention _Congress_ to take you seriously. This leads me to believe that this is a document for public consumption adopting the aesthetics of a letter of concern sent to regulators, and that this document is being submitted to the court of public opinion. I don't doubt that Twitter executives are borderline fraudulent and may have crossed the line into outright fraud - I'd be unsurprised to learn that about any group of executives at any large company. But this document has more the feeling of propaganda than a serious appeal to regulators.
- deleted 4y ago
  [deleted]
- peteradio 4y ago
  Is there some other legitimate example affidavit which to compare? Is SEC bound by a non-superlative clause? Perhaps the drafting lawyers demanded mandate of flair.
  - maxbond 4y ago
    To clarify this isn't an affidavit, I just meant that there is clearly a stylistic choice here to have a certain form. Here's an arbitrary example of a criminal complaint (just because it was the first court document I found): https://www.gwern.net/docs/darknet-markets/2013-ulbricht-complaint.pdf https://www.gwern.net/docs/darknet-markets/2013-ulbricht-com... The way things are presented in sections with numbered subsections, the way some things in the beginning appear to be redacted, are clearly meant to convey the authority of an official document submitted to a legal body. I'm sure this doesn't violate any sort of law and that you are permitted to write a letter like this and submit it to regulators. What I'm suggesting is that the stylistic choices they make are indicative of the audience they are writing for, and that it is different from their stated intentions. And that suggests to me that this document has an agenda and is willing to accomplish it through deceptive means (those being, lying about the purpose and audience of the document to enhance it's perceived authority by giving the general public the impression they are the fly on the wall of an official proceeding rather than reading a document for their own consumption that may have no bearing on the outcome of this dispute), which is my personal standard for calling something propaganda.
majormajor 4y ago
Paragraph 15 is amusing in that it undermines their weird attempt to connect to current affairs without much basis - it comes right out and says "executives are incentivized to avoid counting spam bots as mDAU" yet the greater thrust of the section is that Musk's dispute of the 5% number - a number based on "mDAU" - is generally "correct." That seems extremely twisted: Musk is claiming Twitter claimed something they didn't, Mudge's claim here is that Twitter execs are highly incentivized to be honest about that number they actually claim. In terms of "total" bot accounts... it's a free service with open signup on the fucking internet. You aren't gonna crack down on that effectively without draconian measures that few people really want. Twitter "intentionally prioritized" growing the base of users they were confident in showing ads to since that's the core of their business? Yawn. Company focuses on what actually makes them money, seems responsible of them to their stakeholders! If the bot "problem" got bad enough that they couldn't monetize, they'd be incentived to fix it; they shouldn't necessarily spend millions on trying to fix it just because people complain. There are bots on Twitter, news at 11, this supports the idea that Twitter wasn't lying in their financial reporting. So why are they leading with this BS part? The other sections are much more interesting.
- danielmarkbruce 4y ago
  The first 20 pages are basically garbage - as you say, Twitter made it clear what they were measuring and communicated it. He's basically saying "the execs didn't prioritize what I thought was important! It's illegal!"
- nrmitchi 4y ago
  When this story first started, there were obviously comments of "this seems timed to support Musk's argument", and counter points of "this would have been in progress long before Musk's offer and has nothing to do with it". There being an entire section titled "Lying about Bots to Elon Musk" makes this entire document seem flimsy at best. It's entire possible to make whistleblower compaints about security issues at Twitter without trying to testify to an impending lawsuit.
- random314 4y ago
  The fact that the document began with 20 pages of word twisting to pretend that Parag lied to Elon, while no such thing happened is really bizarre. I thought the whistle blowing was about state actors being granted 100% access to user data. Instead it dedicates the first 20 pages whinging about a non issue. I lost any motivation to read the docs any further. He also complains about data center ops quality at Twitter. While Mudge pretended this is extremely unusual, I have never worked at any fortune 500 that would meet his criteria of fault tolerance, and I have worked at several. With original architects disappearing and employee churn this is impossible. And why does he think this needs to be whistle blown? He sounds really immature. Is there any thing of interest beyond the elon musk drama?
dang 4y ago
Recent and related: --- edit: one thing I regret is not changing the cnn.com URL to the original WaPo reports a few days ago, since they were obviously much better - so I'll add them here: https://www.washingtonpost.com/technology/interactive/2022/twitter-whistleblower-sec-spam/ https://www.washingtonpost.com/technology/interactive/2022/t... https://www.washingtonpost.com/technology/2022/08/23/peiter-mudge-zatko-twitter-whistleblower/ https://www.washingtonpost.com/technology/2022/08/23/peiter-... --- Twitter CEO Parag Agrawal on whistleblower story - https://news.ycombinator.com/item?id=32565019 https://news.ycombinator.com/item?id=32565019 - Aug 2022 (82 comments) Twitter’s former security chief says company lied about bots and safety - https://news.ycombinator.com/item?id=32564630 https://news.ycombinator.com/item?id=32564630 - Aug 2022 (2 comments) Ex-Twitter exec blows the whistle, alleging reckless cybersecurity policies - https://news.ycombinator.com/item?id=32562815 https://news.ycombinator.com/item?id=32562815 - Aug 2022 (597 comments) Not so recent, but related: Twitter shakes up its security team - https://news.ycombinator.com/item?id=30026171 https://news.ycombinator.com/item?id=30026171 - Jan 2022 (110 comments) Twitter names famed hacker 'Mudge' as head of security - https://news.ycombinator.com/item?id=25115754 https://news.ycombinator.com/item?id=25115754 - Nov 2020 (172 comments)
- perihelions 4y ago
  Also related (thread on the departure of Mudge from Twitter) https://news.ycombinator.com/item?id=30026171 https://news.ycombinator.com/item?id=30026171 ("Twitter shakes up its security team")
  - dang 4y ago
    Thanks! I've added that one.
- choppaface 4y ago
  Relevant: dang is a YC employee and not a neutral aggregator. For a balanced perspective, it behooves the reader to consider sources that do not have financial conflicts of interest in the industry being covered.
  - 4y ago
Rapzid 4y ago
The implications of what is being alleged are.. Hard to wrap ones head around. Fascinating to ponder. We all assume stuff like this is going on, but here it is; no conspiracy theory. At a minimum it's worth spending brain cycles considering the impact on "national security". Arguing over the blame and legal implications could be interesting in the context of dispassionate technical legal analysis, but most of that discourse seems little more than a couch for people to signal their biases for the parties involved.
danielmarkbruce 4y ago
Has anyone worked with him? This report makes him sound immature, unprofessional, and out of his depth with respect to legal matters. Is that an unfair characterization?
- void-star 4y ago
  I know Peter Zatko (mudge) professionaly (having spoken at computer security conferences and worked at more than one company that also worked with him), and personally (to an extent, hung out and talked at aforementioned security conferences, talked on irc And email in the good old days of #hack and bugtraq, and have several friends in common, was extremely impressed talking to him about how to disclose a bug I and the l0pht had independently discovered back in the day), and reputationally (the guy demanded, rightfully, a ton of respect amongst peers back then and now). I have a very hard time with this characterization.
  - plugin-baby 4y ago
    Demanded or commanded?
  - calgaryeng 4y ago
    More (content inside the brackets than) out!
    - imron 4y ago
      But at least the most important part is right (the brackets are correctly balanced.
  - danielmarkbruce 4y ago
    Fair enough, good insight
- mrex 4y ago
  It seems extremely unfair. Rather than focusing on the content, you're criticizing tone. It comes off as a structured tactic, given that famous image of Mudge with unkempt hair in a suit sitting before Congress, intended to draw attention away from his words and onto his rough edges. It's an ad hominem attack of a particularly cheap variety. What sounds immature, unprofessional, and out of depth is Agrawal's mass e-mail tarnishing Zatko's reputation, a tactic that your post seems to repeat from a different angle. I'd rather have the theory of general relativity written in crayon, than the most beautiful calligraphic illustration of nonsense. Wouldn't you?
deleted 4y ago
[deleted]
blitzar 4y ago
> 32. Unfortunately, as detailed in the rest of this disclosure, Agrawal's misrepresentations about spam bots are just the tip of the iceberg. If the smoking gun is the bot section, then there is nothing here at all.
a_puppy 4y ago
Wow, this is a wild read. Some of the most shocking parts: - Lack of development and testing environments; engineers build, deploy, and test code directly on the production environment - >50% of employees having access to the live production environment and sensitive user data (getting _worse_ over time) - Lack of logging of what people did with their production environment access - 30% of employees' systems had disabled software updates - Twitter "has never held proper licenses to the data sets and/or software" they used for some ML models - "The majority of the systems in the data centers were running out of date software no longer supported by vendors" - Misleading the board (e.g. trumpeting "we have endpoint monitoring software on 92% of employee systems!" but neglecting to mention that endpoint monitoring software reported 30% of employees' systems had disabled software updates) - Misleading the FTC (e.g. implying that data was deleted when users closed their accounts, when in fact it was not)
Aissen 4y ago
Interesting tidbits to HN users, who like to complain about new Twitter accounts being disabled and required to add a phone number: this is called "ROPO" internally, and multiple executives actually want that to be disabled. But Mudge asked for research that proved it was one of the most efficient anti-spam measure they have at their disposal.
TheDesolate0 4y ago
SilverBirch 4y ago
I think a lot of this is... bad. But not bad bad. It's bad in the sense of "This company is rubbish and Mudge clearly fought with Agrawal" but not bad in the sense of "This company is clearly doing illegal things". For example, Mudge claiming he was ordered not to present a report to the board. He might not be happy about that, but that's perfectly fine, it's not down to him to chose what the CEO decides should be presented to the board, you can have perfectly reasonable disagreements about what's appropriate and at the end of the day the call is down to the CEO. The fact that there are other cases where board members intervene to tell Mudge this reinforces that. Or the claim that the CEO instructed Mudge to send the board documents they both knew were misleading. This is an explosive claim, but it seems highly unlikely that Mudge can prove Agrawal knew the documents to be false, and misleading is impossible to know, because he can't know the context they were presented in. I think it's just highly unlikely that Twitter's CEO is so incompetent that he's just moustache twirling and lying to everyone, it seems highly likely this will come down to Agrawal having a different opinion or interpretation of the facts. What really undermines the claim is when we get to this section: >Agrawal’s tweet was a lie. In fact, Agrawal knows very well that Twitter executives are not incentivized to accurately “detect” or report total spam bots on the platform. Mudge is massively over-reaching here. At best, an argument can be made that at some point there are some perverse incentives where allowing spam bots could inflate numbers to make the company look successful. But even if that argument were convincing, which it isn't, Agrawal clearly doesn't believe it. It's trivial to make Agrawal's argument here. That's why this looks like big claims, but unsupported claims. Because where it's clear that people can reasonably believe what they say but disagree with each other, Mudge claims one side must be lying. What could easily presented as openness, honesty and transparency about the challenges the company faces("Mudge asked the Head of Site... what the underlying spam bot numbers were. Their response was "We don't really know.”) - Mudge basslessly claims this is essentially proves they were acting in bad faith. This all looks designed to be explosive on first sight, but not actually correct in the detail.
- peteradio 4y ago
  > it's not down to him to chose what the CEO decides should be presented to the board, Isn't it? Does the CEO get to carte blanche decide to hide security holes from the board?

122 points by vngzs 4y ago

I read most of the document, and even if a fraction of it is true, Twitter will have a lot of explaining to do in front of the SEC. It sounds like it would be easy to find criminal intent.

rvz 4y ago

Twitter is in dire hot water over this and are going to have a nice conversation with Elon, Jack, etc in court with the SEC, and the FTC watching them over this potential case of fraud with this disclosure. Did Twitter really think that continuous denial, lies and more PR deception was going to be that easy to get away with? Not this time.

bsder 4y ago
> Did Twitter really think that continuous denial, lies and more PR deception was going to be that easy to get away with? If Elon hadn't just happened to tender an offer, want to back out, and put them under a court spotlight ... yeah, they probably did. And, to be fair, unless someone goes to jail, then ... yeah, they'll just get away with it as the cost of doing business.

johng 4y ago

From my reading it looks like Agrawal might deserve some jail time... this is egregious? Thoughts?

mandeepj 4y ago
What are the chances - Vijaya, Parag, Dorsey, few more of their execs and board members - are in a WAR ROOM call on daily basis?
danielmarkbruce 4y ago
For what exactly?
ryan_lane 4y ago
Most of the complaints in the whistleblower docs aren't about anything illegal. There's some places where they show that twitter's public statements, after a security incident, around their security posture were misleading. That's potentially illegal, but even that could be a grey area. There's some places where they show twitter didn't comply with creating and enforcing an SLDC, but if you've ever worked on an SLDC, you know it's a joke that's there for compliance checkbox reasons. Also, this is certainly a grey area because most likely to actually comply with this, you only need to have a document generated, training for how it should be used, and very basic processes in your git workflows. I'd be surprised if they hadn't done enough to legally comply with this. The statements around "employees with production access" are vague. What _kind_ of production access do they have? If you're an engineer, you have production access, because your code can be deployed there. If you're in ops/support, you have production access, because you need to access user data. Hard to properly judge the seriousness of this without knowing more details. For me, the lack of audit access seems to be the biggest legal issue, especially if they were lying to regulators about their progress. It's not that the accusations aren't bad. They paint a picture of an org that has serious security issues, seriously management and organizational health/culture issues, flailing executives, and an incompetent board. But, it really seems like most of it simply isn't illegal.
- nrmitchi 4y ago
  I'll agree with most of these statements, but would object to "they show twitter didn't comply with creating and enforcing...". I don't think it's clear that they have shown anything. This is simply a complaint making statements without any real form of evidence. As you say in your final statement, this is just painting a picture of an organization. It's definitely not fair to take everything stated here at face value with all of the context which is clearly missing.
  - ryan_lane 4y ago
    It says in the document that there is an SLDC defined, but that the company isn't fully complying with it.
fakethenews2022 4y ago
Giving the Indian government access to your data when they allowed the Indian government to specify an individual to hire and then not have internal controls on what information that person has access to. Sounds very similar to how the Saudi Arabian government spied on Twitter users using a plant but in this case knowingly supported by Twitter executives.

bushbaba 4y ago

I really hope that's not true. Ooph. "Twitter data centers were fragile, and Twitter lacked plans and processes to “cold boot.” That meant that if all the centers went offline simultaneously, even briefly, Twitter was unsure if they could bring the service back up. Downtime estimates ranged from weeks of round-the-clock work, to permanent irreparable failure"

chickin 4y ago
How many large tech companies do you think can do that? Each service wants to benefit from the use of other services. Ensuring there aren't dependency loops in this complex graph sounds tricky.
- danielmarkbruce 4y ago
  Google wasn't 100% certain they could cold boot around 2015. Not sure if it's still true. It's crazy difficult to figure out if you can at some point.
  - yuhong 4y ago
    Part of the point of SSD based search engines is they are much easier to cold boot as well.
- ryan_lane 4y ago
  You have to have enough people to "be that asshole" that tells people in their design docs that they are creating a dependency loop and need to design things to avoid it. I did that for quite a while at Lyft, and I'm hoping that people continued doing that after I stopped. My guess, though, is that I and others have missed things and that even somewhere that was careful to avoid it, there's still some places that could make cold boots difficult. The best way to avoid this problem is to have a secondary datacenter that is hot, and to never let both fail. Of course, this is what twitter is doing. Designing a DR plan where you can lose multiple primary/secondary datacenters is hard enough that basically no one does it. This is something that really doesn't belong in the whistleblower complaints because it wasn't under his responsibilities, twitter wasn't lying to anyone about having properly implemented DR plans, and twitter's DR plan isn't out of the ordinary across the industry.

maxbond 4y ago

This document feels out of place to me. It's addressed to various regulators, and at a surface level it has the form of an affidavit or other legal document. But the language is far too familiar to be writing for that audience - they refer to Mudge consistently as Mudge rather than, say, "Mr. Zatko"; they make liberal use of superlatives; they add emphasis using bold text. This is kind of like writing a cover letter where you explain why the job would be great for you rather than why you would be great for the job, it's going to sound wrong to your intended audience, and it will degrade your credibility in their eyes (rightly or wrongly). This isn't the style you would adopt if you wanted such austere organizations as the SEC, DOJ, and they even mention _Congress_ to take you seriously. This leads me to believe that this is a document for public consumption adopting the aesthetics of a letter of concern sent to regulators, and that this document is being submitted to the court of public opinion. I don't doubt that Twitter executives are borderline fraudulent and may have crossed the line into outright fraud - I'd be unsurprised to learn that about any group of executives at any large company. But this document has more the feeling of propaganda than a serious appeal to regulators.

deleted 4y ago
[deleted]
peteradio 4y ago
Is there some other legitimate example affidavit which to compare? Is SEC bound by a non-superlative clause? Perhaps the drafting lawyers demanded mandate of flair.
- maxbond 4y ago
  To clarify this isn't an affidavit, I just meant that there is clearly a stylistic choice here to have a certain form. Here's an arbitrary example of a criminal complaint (just because it was the first court document I found): https://www.gwern.net/docs/darknet-markets/2013-ulbricht-complaint.pdf https://www.gwern.net/docs/darknet-markets/2013-ulbricht-com... The way things are presented in sections with numbered subsections, the way some things in the beginning appear to be redacted, are clearly meant to convey the authority of an official document submitted to a legal body. I'm sure this doesn't violate any sort of law and that you are permitted to write a letter like this and submit it to regulators. What I'm suggesting is that the stylistic choices they make are indicative of the audience they are writing for, and that it is different from their stated intentions. And that suggests to me that this document has an agenda and is willing to accomplish it through deceptive means (those being, lying about the purpose and audience of the document to enhance it's perceived authority by giving the general public the impression they are the fly on the wall of an official proceeding rather than reading a document for their own consumption that may have no bearing on the outcome of this dispute), which is my personal standard for calling something propaganda.

majormajor 4y ago

Paragraph 15 is amusing in that it undermines their weird attempt to connect to current affairs without much basis - it comes right out and says "executives are incentivized to avoid counting spam bots as mDAU" yet the greater thrust of the section is that Musk's dispute of the 5% number - a number based on "mDAU" - is generally "correct." That seems extremely twisted: Musk is claiming Twitter claimed something they didn't, Mudge's claim here is that Twitter execs are highly incentivized to be honest about that number they actually claim. In terms of "total" bot accounts... it's a free service with open signup on the fucking internet. You aren't gonna crack down on that effectively without draconian measures that few people really want. Twitter "intentionally prioritized" growing the base of users they were confident in showing ads to since that's the core of their business? Yawn. Company focuses on what actually makes them money, seems responsible of them to their stakeholders! If the bot "problem" got bad enough that they couldn't monetize, they'd be incentived to fix it; they shouldn't necessarily spend millions on trying to fix it just because people complain. There are bots on Twitter, news at 11, this supports the idea that Twitter wasn't lying in their financial reporting. So why are they leading with this BS part? The other sections are much more interesting.

danielmarkbruce 4y ago
The first 20 pages are basically garbage - as you say, Twitter made it clear what they were measuring and communicated it. He's basically saying "the execs didn't prioritize what I thought was important! It's illegal!"
nrmitchi 4y ago
When this story first started, there were obviously comments of "this seems timed to support Musk's argument", and counter points of "this would have been in progress long before Musk's offer and has nothing to do with it". There being an entire section titled "Lying about Bots to Elon Musk" makes this entire document seem flimsy at best. It's entire possible to make whistleblower compaints about security issues at Twitter without trying to testify to an impending lawsuit.
random314 4y ago
The fact that the document began with 20 pages of word twisting to pretend that Parag lied to Elon, while no such thing happened is really bizarre. I thought the whistle blowing was about state actors being granted 100% access to user data. Instead it dedicates the first 20 pages whinging about a non issue. I lost any motivation to read the docs any further. He also complains about data center ops quality at Twitter. While Mudge pretended this is extremely unusual, I have never worked at any fortune 500 that would meet his criteria of fault tolerance, and I have worked at several. With original architects disappearing and employee churn this is impossible. And why does he think this needs to be whistle blown? He sounds really immature. Is there any thing of interest beyond the elon musk drama?

dang 4y ago

Recent and related: --- edit: one thing I regret is not changing the cnn.com URL to the original WaPo reports a few days ago, since they were obviously much better - so I'll add them here: https://www.washingtonpost.com/technology/interactive/2022/twitter-whistleblower-sec-spam/ https://www.washingtonpost.com/technology/interactive/2022/t... https://www.washingtonpost.com/technology/2022/08/23/peiter-mudge-zatko-twitter-whistleblower/ https://www.washingtonpost.com/technology/2022/08/23/peiter-... --- Twitter CEO Parag Agrawal on whistleblower story - https://news.ycombinator.com/item?id=32565019 https://news.ycombinator.com/item?id=32565019 - Aug 2022 (82 comments) Twitter’s former security chief says company lied about bots and safety - https://news.ycombinator.com/item?id=32564630 https://news.ycombinator.com/item?id=32564630 - Aug 2022 (2 comments) Ex-Twitter exec blows the whistle, alleging reckless cybersecurity policies - https://news.ycombinator.com/item?id=32562815 https://news.ycombinator.com/item?id=32562815 - Aug 2022 (597 comments) Not so recent, but related: Twitter shakes up its security team - https://news.ycombinator.com/item?id=30026171 https://news.ycombinator.com/item?id=30026171 - Jan 2022 (110 comments) Twitter names famed hacker 'Mudge' as head of security - https://news.ycombinator.com/item?id=25115754 https://news.ycombinator.com/item?id=25115754 - Nov 2020 (172 comments)

perihelions 4y ago
Also related (thread on the departure of Mudge from Twitter) https://news.ycombinator.com/item?id=30026171 https://news.ycombinator.com/item?id=30026171 ("Twitter shakes up its security team")
- dang 4y ago
  Thanks! I've added that one.
choppaface 4y ago
Relevant: dang is a YC employee and not a neutral aggregator. For a balanced perspective, it behooves the reader to consider sources that do not have financial conflicts of interest in the industry being covered.
- 4y ago

Rapzid 4y ago

The implications of what is being alleged are.. Hard to wrap ones head around. Fascinating to ponder. We all assume stuff like this is going on, but here it is; no conspiracy theory. At a minimum it's worth spending brain cycles considering the impact on "national security". Arguing over the blame and legal implications could be interesting in the context of dispassionate technical legal analysis, but most of that discourse seems little more than a couch for people to signal their biases for the parties involved.

danielmarkbruce 4y ago

Has anyone worked with him? This report makes him sound immature, unprofessional, and out of his depth with respect to legal matters. Is that an unfair characterization?

void-star 4y ago
I know Peter Zatko (mudge) professionaly (having spoken at computer security conferences and worked at more than one company that also worked with him), and personally (to an extent, hung out and talked at aforementioned security conferences, talked on irc And email in the good old days of #hack and bugtraq, and have several friends in common, was extremely impressed talking to him about how to disclose a bug I and the l0pht had independently discovered back in the day), and reputationally (the guy demanded, rightfully, a ton of respect amongst peers back then and now). I have a very hard time with this characterization.
- plugin-baby 4y ago
  Demanded or commanded?
- calgaryeng 4y ago
  More (content inside the brackets than) out!
  - imron 4y ago
    But at least the most important part is right (the brackets are correctly balanced.
- danielmarkbruce 4y ago
  Fair enough, good insight
mrex 4y ago
It seems extremely unfair. Rather than focusing on the content, you're criticizing tone. It comes off as a structured tactic, given that famous image of Mudge with unkempt hair in a suit sitting before Congress, intended to draw attention away from his words and onto his rough edges. It's an ad hominem attack of a particularly cheap variety. What sounds immature, unprofessional, and out of depth is Agrawal's mass e-mail tarnishing Zatko's reputation, a tactic that your post seems to repeat from a different angle. I'd rather have the theory of general relativity written in crayon, than the most beautiful calligraphic illustration of nonsense. Wouldn't you?

deleted 4y ago

[deleted]

blitzar 4y ago

> 32. Unfortunately, as detailed in the rest of this disclosure, Agrawal's misrepresentations about spam bots are just the tip of the iceberg. If the smoking gun is the bot section, then there is nothing here at all.

a_puppy 4y ago

Wow, this is a wild read. Some of the most shocking parts: - Lack of development and testing environments; engineers build, deploy, and test code directly on the production environment - >50% of employees having access to the live production environment and sensitive user data (getting _worse_ over time) - Lack of logging of what people did with their production environment access - 30% of employees' systems had disabled software updates - Twitter "has never held proper licenses to the data sets and/or software" they used for some ML models - "The majority of the systems in the data centers were running out of date software no longer supported by vendors" - Misleading the board (e.g. trumpeting "we have endpoint monitoring software on 92% of employee systems!" but neglecting to mention that endpoint monitoring software reported 30% of employees' systems had disabled software updates) - Misleading the FTC (e.g. implying that data was deleted when users closed their accounts, when in fact it was not)

Aissen 4y ago

Interesting tidbits to HN users, who like to complain about new Twitter accounts being disabled and required to add a phone number: this is called "ROPO" internally, and multiple executives actually want that to be disabled. But Mudge asked for research that proved it was one of the most efficient anti-spam measure they have at their disposal.

TheDesolate0 4y ago

SilverBirch 4y ago

I think a lot of this is... bad. But not bad bad. It's bad in the sense of "This company is rubbish and Mudge clearly fought with Agrawal" but not bad in the sense of "This company is clearly doing illegal things". For example, Mudge claiming he was ordered not to present a report to the board. He might not be happy about that, but that's perfectly fine, it's not down to him to chose what the CEO decides should be presented to the board, you can have perfectly reasonable disagreements about what's appropriate and at the end of the day the call is down to the CEO. The fact that there are other cases where board members intervene to tell Mudge this reinforces that. Or the claim that the CEO instructed Mudge to send the board documents they both knew were misleading. This is an explosive claim, but it seems highly unlikely that Mudge can prove Agrawal knew the documents to be false, and misleading is impossible to know, because he can't know the context they were presented in. I think it's just highly unlikely that Twitter's CEO is so incompetent that he's just moustache twirling and lying to everyone, it seems highly likely this will come down to Agrawal having a different opinion or interpretation of the facts. What really undermines the claim is when we get to this section: >Agrawal’s tweet was a lie. In fact, Agrawal knows very well that Twitter executives are not incentivized to accurately “detect” or report total spam bots on the platform. Mudge is massively over-reaching here. At best, an argument can be made that at some point there are some perverse incentives where allowing spam bots could inflate numbers to make the company look successful. But even if that argument were convincing, which it isn't, Agrawal clearly doesn't believe it. It's trivial to make Agrawal's argument here. That's why this looks like big claims, but unsupported claims. Because where it's clear that people can reasonably believe what they say but disagree with each other, Mudge claims one side must be lying. What could easily presented as openness, honesty and transparency about the challenges the company faces("Mudge asked the Head of Site... what the underlying spam bot numbers were. Their response was "We don't really know.”) - Mudge basslessly claims this is essentially proves they were acting in bad faith. This all looks designed to be explosive on first sight, but not actually correct in the detail.

peteradio 4y ago
> it's not down to him to chose what the CEO decides should be presented to the board, Isn't it? Does the CEO get to carte blanche decide to hide security holes from the board?