16 ms·

XCheck at Meta: Why it exists and how it works

337 points by nindalf 4y ago

1letterunixname 4y ago
I'm wondering if it's XCheck that locked-out my later account when it decided by zombie account (I don't have the email address anymore) was a "duplicate". The check condition it fired on was visiting the 2FA settings page. Some nice wf on-call fixed it from an oops task and I don't touch that page anymore. :)
zug_zug 4y ago
- MonkeyMalarky 4y ago
  Well now my interest is piqued 1000%!
  - navigate8310 4y ago
    10th Oct: If BJP's Amit Malviya Reports Your Post, Instagram Will Take it Down – No Questions Asked https://thewire.in/tech/amit-malviya-instagram-meta-xcheck https://thewire.in/tech/amit-malviya-instagram-meta-xcheck 11th Oct: 'How the Hell Did Document Leak?' – Meta Internal Mail Belies 'Fabricated' Charge Against The Wire https://thewire.in/tech/meta-xcheck-internal-email-watchlist https://thewire.in/tech/meta-xcheck-internal-email-watchlist 15th Oct: Meta Said Damaging Internal Email is ‘Fake’, URL 'Not in Use', Here's Evidence They're Wrong https://thewire.in/tech/meta-andy-stone-email-xcheck https://thewire.in/tech/meta-andy-stone-email-xcheck Basically, ruling party in India is exploiting XCheck program to curb any dissent.
    - SanjayMehta 4y ago
      Yeah right. Looks like The Wire got fooled by their source OR is lying. https://twitter.com/alexstamos/status/1581407731159748608 https://twitter.com/alexstamos/status/1581407731159748608
    - nindalf 4y ago
      Like I mention, all of the evidence The Wire has used here is likely fabricated. Not by the Wire themselves, but by the source they’re relying on.
      - webmobdev 4y ago
        The source has been verified by The Wire to be a Meta employee / contract worker. We reiterate the faith we have in our sources, whose identities and positions in Meta are known to us. Our reporters have had a productive relationship with them for some time already, prior to the Instagram story. Meta’s suggestion that sources who don’t know each other have teamed up to “hoax” The Wire is ludicrous. Source: https://thewire.in/media/statement-by-the-wire-on-metas-responses-so-far https://thewire.in/media/statement-by-the-wire-on-metas-resp... The emails have been verified to be authentic by The Wire and 2 independent experts, one of whom works at Microsoft. Further proof of the "instagram URL" has been shared. Even if we assume that the "instagram" workplace portal is the "weakest" evidence and a "fake" (as you and others claim), than Meta should be easily able to identify the person who created the "fake" instance and populated it with all the "fake" tickets. This is what Alex Stamos too tweeted: If The Wire is wrong, then Meta has all the evidence they need. While you could create a whole fake Workplace, the easier move is to just create a free trial instance, meaning those fake notes are sitting in Meta's databases along with the metadata of whomever created them. I think, at this point, it's on Meta to write-up a detailed response with whatever technical evidence they have. This will not go away just by ignoring it. The fact is that The Wire have an obligation to protect their source and so are limited in the evidence they can share publicly (unless taken to court). So this begs the question that if Meta and you and other FB employees) are so sure of the evidence being fabricated, why do you think The Wire hasn't been sued yet to take down the "fake" articles? (Note that you do not even need to identify who faked the evidence, to sue them, as long as you can prove that it is fake, which should be easy to do so according to all the claims made by Meta, you and other FB employees).
      - csande17 4y ago
        If that's the case, it seems like someone should be informing Facebook that their DKIM signing keys have been compromised. The impact of that would be much worse than just a forged .eml file sent to a journalist.
    - KrishnaShripad 4y ago
- _micheee 4y ago
  Why?
  - nvarsj 4y ago
    This kind of internal tooling and workflow is almost always under NDA.
  - bagels 4y ago
    If they worked at Facebook/Meta they violated their nda and put themself in legal peril.
    - lopkeny12ko 4y ago
      Have you never seen https://engineering.fb.com/ https://engineering.fb.com/? Engineers there blog about their tech tools all the time. "Legal peril" sounds like a bit of a stretch.
      - bagels 4y ago
        Those blog posts likely go through legal, privacy and marketing review. If you think that Facebook wouldn't enforce an NDA, especially on something sensitive like this, I think you are incorrect.
      - wrigby 4y ago
        These posts are all thoroughly reviewed by comms and legal teams. In onboarding, it’s thoroughly communicated that you need to go through the proper channels to publicly publish anything with technical details.
    - nindalf 4y ago
      I think I’ll be fine. My reasoning is two fold - I haven’t shared anything that could be exploited by anyone. And second, Meta and others in the industry try to share information about how their integrity efforts work so we can learn from each other.
      - 4y ago
- tjpnz 4y ago
  https://archive.ph/AK0zi https://archive.ph/AK0zi
- crmd 4y ago
  He doesn’t work there anymore[0] Also, nobody likes a tattle tale. [0] https://uk.linkedin.com/in/krishna-sundarram-17a76954 https://uk.linkedin.com/in/krishna-sundarram-17a76954
- nindalf 4y ago
  Thanks for the email. But I think it should be fine. I haven’t said anything that could be exploited by bad actors. As for competitors, there are no competitors in integrity. The entire industry goes to great lengths to share knowledge on what works and what doesn’t. We all win when we combat abuse well.
  - hitekker 4y ago
    You should be fine. The GP seemed to have forgotten they're not posting on FB"s internal workplace.
    - MertsA 4y ago
      Internal comms are notoriously open, this kind of post would be a welcome contribution for explaining XCheck to a general audience outside of the integrity org. I don't know what kind of confidentiality agreement they signed but mine would have technically prohibited discussing this. Not only that, it's not time limited in nature, it applies "at all times during the term of my Relationship with the Company and thereafter". I wouldn't expect Meta to actually do anything about this blog post but I wouldn't have felt comfortable posting it.
- doliveira 4y ago
  Do you at least get extra money for snitching on a fellow worker like that?
otterley 4y ago
> a system that exists and mostly works is preferred to a hypothetical perfect system that is never built. At any sufficiently large scale - especially with a product you're not paying for - this is the best you're likely to get. Once you fully understand and embrace this, your stress levels about imperfection tend to go way down.
- trasz 4y ago
  - bagels 4y ago
    What is an example of "this" being done properly?
  - pilgrimfff 4y ago
    > this has been done properly numerous times in the past I'm genuinely curious if you have any examples of this. I'm not aware of any modern product at scale that doesn't suffer from these same issues.
    - djohnston 4y ago
      When people say that I imagine them thinking about a web forum with 200 monthly visitors.
    - trasz 4y ago
      Every newspaper and pretty much any other medium over the past century or so. That’s what editors used to do. Scaling is not the problem - profits scale with the number of users, the cost of moderation would too, in the worst case. The problem is company’s unwillingness to pay anything at all. Their profits didn’t came from offering a good product, but from discovering a new way to offload the costs of their operation while keeping the income. In this case - by pretending to be a part telco, part newspaper (which brings income), but without taking their main responsibilities (which cost money).
smrtinsert 4y ago
itake 4y ago
I wish they would explain what xcheck is and how it works before the talk about the problems with it. This reads like a blog post to other Integrity FB engineers that already know that system.
- nindalf 4y ago
  It’s explained later on but I could be clearer. XCheck is a system that supports tagging specific accounts with tags that exempt that account from certain integrity enforcement. In the post I give an example of a tag that used to exempt accounts from the fake account checkpoint. Applying these tags is usually a manual process and usually carefully vetted.
  - webmobdev 4y ago
    You yourself claim that you are only aware of Xcheck "features" since 2021, As of June 2021 no XCheck tag I knew of allowed a user to take down content at will. Are you also claiming that there have been no changes, and no new features, added to XCheck since June 2021? Are you aware of the new social media regulations recently passed by the current Modi government of India? (To be credible, you need to honestly answer these questions and address it in your article too, as otherwise you are basing your assumptions on incomplete information, thereby misleading yourself and others.) To clarify, one of the requirement of this regulation is: Under the new Intermediaries Rules, intermediaries must *complete* the takedown process under Section 79(3) of the IT Act, within 36 hours. Source: https://internetfreedom.in/intermediaries-rules-2021/ https://internetfreedom.in/intermediaries-rules-2021/ (a more brief summary of the regulation is here - https://techwireasia.com/2021/06/indias-new-social-media-rules-explained/ https://techwireasia.com/2021/06/indias-new-social-media-rul...) Very simply put, large social media platforms are legally required to respond and delete content when requested by government representatives. Under such a scenario, can you confidently say that XCheck hasn't been extended with new features, like a tag that allows a user to immediately flag posts that must be compulsorily taken down when the particular user reports it? It seems a very obvious extension when you consider the political demands that are placed on Meta from around the world, and in India in particular.
  - 4y ago
SanjayMehta 4y ago
- iudqnolq 4y ago
  > who hates India, the current government and probably himself as well ... As someone who dislikes both Meta and The Wire, this is a source of great entertainment. As an American it's fascinating how the tone of HN completely changes whenever the merits of cancel culture, the Internet Archive, or the current government of India comes up. It's an odd list.
  - KrishnaShripad 4y ago
    Because, believe it or not, plenty of Indians are satisfied with the current Government in India. We have had shit Governments for more than 70 years. So don't be surprised if Indians don't support Western media narratives on Indian Government (which is mostly fabricated/fake).
    - webartisan 4y ago
      I think you should speak for yourself instead of for all Indians. You believing or not believing something has little effect on facts. Bullying everyone into believing your unquestioned love for the present day government is rather disenginious.
      - KrishnaShripad 4y ago
        throwaway384629 4y ago
        You are mostly sharing your opinion. I am not saying that you are wrong, but stating your opinion has much less weightage than stating the facts where your claims are supported by some data/references.
DisjointedHunt 4y ago
[flagged]
localhost 4y ago
This is an excellent piece which really does a good job at explaining the nuance that exists in this challenging problem domain. Thank you for writing this. It's a great piece to point to whenever the next scandal gets "exposed" by click-seeking folks on the internet.
- uoaei 4y ago
  Are you saying that this article excuses data breaches?
  - ironrabbit 4y ago
    The "scandal" that prompted this article is referenced with sources under the heading "Why XCheck is in the news". Neither the article nor the comment you're replying to says anything about data breaches.
- sneak 4y ago
  It's good that it keeps getting exposed that a single organization without accountability decides what people are or are not allowed to publish on a platform to the people who explicitly subscribed to their publications. The fact that moderation is a hard problem is orthogonal. The fact remains that there are different rules for different people, both for flagging and for content. The fact that there is more nuance than "two sets of rules" doesn't really change the fundamental problem that private companies now get to arbitrarily police publication by normal people. I have the distinct impression that in 100 years the default mode is going to be "if it's legal you can post it, if it's not you can't, and if you have a problem with the boundary, take it up with the lawmaking authority and not us". We will look back at this time at the emergence of any-to-any publishing like the dawn of radio or TV or newspaper where a few rich guys got to decide what could or could not be published.
  - CursedUrn 4y ago
    What makes you think something as unique and hard-fought as the First Amendment will be applied to the Internet when all trends point in the other direction, both on and offline?
yardstick 4y ago
> That works well for about two weeks until users figure out how to exploit this. They form groups that agree to coordinate to report content. Now reports become lower signal than before, but still somewhat useful. We use the reports, but try to limit exploitation. What about human employees reviewing users reports, especially randomly picked samples from those that resulted in successful takedowns. If the content was found to be sound under the rules, penalise the reporters. Suspend their ability to make reports (probably in the form of silently ignoring them). Ie some human moderation should still be done at this scale, but using a combo of random sampling and system-flagged-suspicious.
- xani_ 4y ago
  I remember Dota2 (video game) did this, people fake reporting had less reports and they counted for less, while people that were consistently reported players that got punished got more reporting power. Only one report per match "counted" too so a group of player couldn't gang on reports on one player. Of course that could be abused as well,but you'd have to make a group of people that first got the good rating then reported same people and that would be significantly harder
  - WorkerBee28474 4y ago
    Well that just sounds like Bayes's theorum with extra steps
  - qualudeheart 4y ago
    So that’s why I was never banned from Dota even though I sucked at it.
- coredog64 4y ago
  This is Slashdot's meta-moderation system from 20(?) years ago? Logged in users would randomly be selected to check moderation, and accounts that abused their mod points were flagged. Granted, you probably couldn't do that today as you could see meta-moderation brigades out in the wild. Employees should be better, although there's still the chance that an employee might use their position to punish opinions they disagree with.
timzaman 4y ago
How is it people are allowed to blog about company internal implementation details and numbers barely a year after they leave?
- deleted 4y ago
  [deleted]
- tyingq 4y ago
  Seems unsurprising to me. If you, as a company, want ironclad confidentiality from people after they leave, that wouldn't be free or automatic. None of what was shared seems to qualify as a trade secret, which would have some protection.
  - extr 4y ago
    I've never worked somewhere that didn't have me sign some kind of broad NDA-esque clause saying I wouldn't share proprietary or non-public info for a certain period after release (imo, completely reasonably). This blog post is literally nothing BUT non-public proprietary info. Like the parent I'm really surprised the blog author feels comfortable sharing all this. Even if it ended up being technically legal, I have to imagine this kind of thing is extremely frowned upon.
- Kiro 4y ago
  Why wouldn't they?
  - extr 4y ago
    Have you ever read your employment contracts? I've never worked somewhere that would allow something like this to be shared so soon after departure.
    - alliao 4y ago
      meta being so big they're almost part of public institution just privately held; and public scrutiny is less frowned upon especially if many believes they do not contribute to the betterment of the republic
numair 4y ago
I think that the key takeaway from the entire Wire vs Meta fiasco is that there is a lot of absolutely weird Spy vs. Spy behavior going in the Indian political and media industries. I was initially extremely sympathetic to the story presented by The Wire, because it's quite believable that Meta/FB would go to extreme lengths to try to distance themselves from such a situation, but the facts ... just ... don't add up. As Alex Stamos has noted, there is little debate regarding collusion between Meta's "government relations" people and policy groups, so that's not really much of a scandal -- it's not like Meta will deny that Modi was treated like a divine being when he visited their campus, so the idea that his people can easily call in a favor to crush a social media post won't really surprise anyone. The bulletproof evidence with DKIM authentication and a video of a logged-in admin instance doesn't look so bulletproof after all, based on the credible reports from those who know how Meta's admin tooling actually looks and functions, and those with other DKIM authenticated emails from the fb.com domain. So, the question is, what's the agenda here? Why would someone go through all of this effort, to create a scandal out of something that is not very far from the truth? What's the point of this entire thing? Maybe this is like Nick Denton and Gawker losing their entire business over the stupidest sex tape story ever; or maybe this is part of something else that requires domain knowledge regarding Indian politics and media to understand (do page views monetize so well that this mini-scandal is going to be super-profitable to The Wire? Highly doubtful, right? What could they possibly get out of this?). The whole thing is just really weird. It's also a major distraction from the very real problems that Meta doesn't even try to hide. Meta's relationship with the governments in various countries -- including the United States -- is way too close for comfort, and absolutely toxic on multiple levels. If this story turns out to be fake news, it'll do a lot to help the company deny, deflect and discredit the next real scandal. I think this is what the wacky conspiracy theorists call "4D chess," but I don't think that's what's happening here. The true story behind all of this is bound to be very strange, and very stupid.
- DisjointedHunt 4y ago
  I'd speculate this is part of an "Opposition research" move by a political party that might benefit. The extreme lengths that these people seem to have gone to is shocking. Whatever it is, FB has the evidence on their servers as Alex Stamos points out since they created a Workspace instance.
wiml 4y ago
This is a mildly interesting post only because of how it comes up to, but avoids ever talking about, the central flaw in the system it describes — they're conflating several things without ever really acknowledging that they're different. Theres "integrity", which is a Meta-internal term; verification of ID, which is a far deeper rabbit hole than they're going to go into; abuse; and undesirable behavior. They're all different things.
- remus 4y ago
  > They're all different things. Different but related, I would say. For example abuse of a service and undesirable behaviour on a service could be thought of as different points on a spectrum.
SilverBirch 4y ago
It makes sense from a technical perspective why you'd build a system like XCheck. The problem is that once you build XCheck you actually need to say to your C-suite and PR department "Actually no, you can't say we treat everyone equally, we're manually going in and exempting certain people from scrutiny, and that's entirely within our discretion." The problem isn't technical, it's legal - you can't build a system that operates one way and then publicly misrepresent it.
- yeasurebut 4y ago
- xani_ 4y ago
  As facebook shows you can, and get away with it.
- pjc50 4y ago
  Legally, their TOS probably allows them to be completely opaque and arbitrary. And any system which involves reporting from the public will need to build a "this account gets lots of false reports of category X, to save time assume they're all false" system. As well as "all the reports made by this user are bogus".
- etchalon 4y ago
  There are no laws which say a company has to be transparent and accurate when discussing its moderation systems.
  - lmm 4y ago
    Given that people pay to advertise on facebook, any false statements they make about their moderation systems via the internet almost certainly constitute wire fraud.
    - pavlov 4y ago
      You make that claim with a high level of confidence, but have you actually read the contract in place between Meta and the advertiser?
oars 4y ago
I would love to work at a FAANG in the future to gain exposure to these interesting problems and systems at huge scale.
- philjohn 4y ago
  They're interesting problems for sure; it's equal parts fun, technically challenging, and then you get to throw in having to quickly react to adversarial responses.
1letterunixname 4y ago
So I work at Meta and just got MetaMorphized s/@fb/@meta/g. When I started, my personal account was banned twice on the condition of adjusting 2FA. It was sorted out. I had a past account that I could no longer access as email domains changed, so it looked like I was trying to maintain duplicate accounts. Why it triggered on adjusting 2FA didn't make sense. Then, it wanted me to verify my identity in a way that created a Catch-22 of wanting a login from itself or to a device that didn't have a valid login session. I set the options for better security and left it, rolled all of my personal passwords, eliminated public data, canceled unnecessary social media accounts, and enrolled in security features like Google's Advanced Protection Program. I'm hesitant to make changes risking getting locked-out again. I recently spoke with someone in Readiness who works with hiring human verifier contractors around the world. A primary issue is scale. You would need to hire the entire world's population to moderate the content being produced. Still there will be bias and misinterpretation of sarcasm, and varying standards of acceptability and decorum. AI is a force-multiplier to an extent, but it takes human judgement to rectify mistakes and data to identify brigades, scammers, terrorists, and political manipulators seeking to exploit an imperfect system. Sadly, the humans with the best judgement typically have better career options that they couldn't be paid to do social media moderation. It can be made better, but the ultimate realization is with even great care, good intentions, and attempts at making things sensible and fair, there are always going to be mistakes. It's trying to minimize mistakes and not enable genocides, election sentiment manipulation, or product scams. It's doable to minimize mistakes, but it take persistent vigilance, wisdom about human nature, and creative solutions to deter and prevent harm while avoiding harming innocent persons. Mistakes are bad and disappointing and it feels bad when they happen. In case anyone were wondering, the security is the inverse of Twitter's. Everything is logged and access requires a business purpose for a limited time, narrow scope, and approval to get that access. Almost no one has access to production data. PII is taken very seriously. There are no laptops with copies of user data. All laptops are encrypted, just in case, and for general principles. Password complexity requirements are insane. I can see my work/personal FBID user object in the graph, but as soon as I try to prod any links to other users, big warnings appear. There's an army of insane genius security researchers and practitioners who create and deploy defense-in-depth tools for broad and specific solutions to prod, corp, and endpoints that reduce our risks to being compromised, data being exfil'd, and security "oops"es from happening. Work users who transit through certain "hostile" countries lose some security credentials and access. I'm actually wondering why laptops aren't spot-checked for malware implants and hardware/firmware modifications. I would assume employees with critical access who travel internationally with their work laptops and phones are prime targets. PS: I wonder if people would pay $X / month (say $199) to have a high-signal social media service that requires a level of "vouching" invitation, names with faces profiles (not visible to the wider internet), sensible/proportional mediation and civilized feedback, politically-neutral, and free speech-loving to increase the sense of community and reduce the potential of anonymous bad actors. 37signals/basecamp accumulated research that showed that smaller communities with faces and real profile names lead to nicer interactions. I don't recall the source, but communities that are defended in terms of politeness and boundaries tend to endure while undefended communities drive away users and tend to disperse.
civilized 4y ago
> Maybe after 5 or 10 or 100 reports, the system will automatically make the reported content invisible. This explains why so much "moderation" on sites like Twitter seems to lack any rhyme or reason.
josephcsible 4y ago
> Normally such content would be taken down after sufficient number of reports were received. I can't be the only one who sees a problem with this. No content should ever be taken down automatically just because a bunch of random people report it.
- throwawaylinux 4y ago
  > No content should ever be taken down automatically just because a bunch of random people report it. Serious question, why not?
  - fragmede 4y ago
    Because you have no idea if those reports are at all genuine, or if the reporters met up elsewhere (online or off) in order to brigade and mass-report said content, with the intention of getting it taken down despite breaking no rules. Sometimes, the coordination isn't even necessary, it just needs to be the right target posting something online. (Eg more than a few people have gone and reported every post by a politician you like/dislike for hate speech and inciting violence.)
    - ummonk 4y ago
      That’s why politicians tend to get X-Check.
    - throwawaylinux 4y ago
      The article well explains this downside of user reports, so I don't see what this comment adds. It does not answer my question. The article also describes problems of not acting on them, so the conclusion requires more than just finding a negative.
      - fragmede 4y ago
        I'm sorry you're unable to understand my answer. Let me try saying the same thing as the article another time, and maybe you'll be able to understand the answer? Your question was Why should no content should ever be taken down automatically just because a bunch of random people report it. It's because the random people reporting it can't be trusted to be acting honestly. Without a human in the loop, the automated system becomes a tool for cyberbullies, and harms the very users you intend to protect. The foregone conclusion, thus, is that a fully-automated system will do more harm for the user-base than good.
asah 4y ago
"The number of reports number in the millions per day. That's too many for human moderators" Isn't Reddit the counterexample? A dozen of us moderated a controversial sub with 2M users, no problem.
- HiJon89 4y ago
  With 2 million users, I imagine it would be on the order of thousands of posts per day, and maybe hundreds of reports?
- sophiebits 4y ago
  So for a few billion users, you need tens of thousands of moderators, assuming that ratio holds?
  - smolder 4y ago
    This is a very efficient moderation force. I'd be happy to get away with 10s of thousands per billion. That's still under 100 per million. Under 1 per 10,000.
- Spivak 4y ago
  2M “subscribed” users to a subreddit is about 10% or 200,000 daily viewers with about 1% of those commenting and 0.1% posting so 2,000 comments and 200 posts per day. Some subs will be more or less but that’s the magnitudes we’re talking about. The equivalent situation assuming mods review every comment would be a subreddit with 1 billion users. If mods are only reviewing reports which is some small percentage of comments then adjust accordingly.
  - fullsend 4y ago
    Very well put, it's not even close to the same ballpark. It would also be like moderating a 1 billion user subreddit where nearly every single post is off topic and comes from different users, whereas in subreddits a few super users often produce a huge percent of the content.
  - 4y ago
whiddershins 4y ago
Sounds like binary flags that should be weights.
Gumbercules 4y ago
Zenbit_UX 4y ago
If I were designing a system like this, I'd likely implement a reputation system for reporters (users who report, not journalists). When you report content for the first time your reputation score is 0.5, if the content is of a celebrity or public figure, your reputation will invisibly drop, enough of these and your score will be so low as to not be a useful flag when evaluating whether content is objectionable. On the other side, if the content is reviewed as violating ToS, your rep will increase. With a system like this the fake reports and brigading will quickly lose its power, eventually all users who abuse reports will be ranked so low as to not matter. A threshold of rep score can be set before the content is auto removed. Maybe something like 25pts in order for it to be removed, and that's after adding together all the rep of the reporting users.
- koyote 4y ago
  I always thought that this is how it already worked. That being said, the above can also be abused by creating a bunch of high rep accounts or opening up a black market of high rep accounts for sale. It's definitely a higher bar though.
- DannyBee 4y ago
  In a system like this they will play both sides. They do, in fact. Set up accounts that produce content that violates TOS and accounts that report it. The reporting accounts will have their rep increased for correctly reporting tos violations. Then they use the high rep accounts to brigade and fake report. Or sell them. They are patient, so even if you make reputation go up slowly, they will just build a pipeline of account harvesting.
nonrandomstring 4y ago
The corruption in tech, perhaps inevitable or intrinsic, looks more and more like the forces that tore the Soviet Union and then Russia apart. 1) A "revolution" and promises of new fairness 2) A Nomenklatura 3) The rise of unassailable systems and faith in them 4) Widespread failure and instability of those systems 5) Attempts to stabilise by handing more power to privateers 6) Abuse and corruption run wild amongst new oligarchs

337 points by nindalf 4y ago

I'm wondering if it's XCheck that locked-out my later account when it decided by zombie account (I don't have the email address anymore) was a "duplicate". The check condition it fired on was visiting the 2FA settings page. Some nice wf on-call fixed it from an oops task and I don't touch that page anymore. :)

zug_zug 4y ago

MonkeyMalarky 4y ago
Well now my interest is piqued 1000%!
- navigate8310 4y ago
  10th Oct: If BJP's Amit Malviya Reports Your Post, Instagram Will Take it Down – No Questions Asked https://thewire.in/tech/amit-malviya-instagram-meta-xcheck https://thewire.in/tech/amit-malviya-instagram-meta-xcheck 11th Oct: 'How the Hell Did Document Leak?' – Meta Internal Mail Belies 'Fabricated' Charge Against The Wire https://thewire.in/tech/meta-xcheck-internal-email-watchlist https://thewire.in/tech/meta-xcheck-internal-email-watchlist 15th Oct: Meta Said Damaging Internal Email is ‘Fake’, URL 'Not in Use', Here's Evidence They're Wrong https://thewire.in/tech/meta-andy-stone-email-xcheck https://thewire.in/tech/meta-andy-stone-email-xcheck Basically, ruling party in India is exploiting XCheck program to curb any dissent.
  - SanjayMehta 4y ago
    Yeah right. Looks like The Wire got fooled by their source OR is lying. https://twitter.com/alexstamos/status/1581407731159748608 https://twitter.com/alexstamos/status/1581407731159748608
  - nindalf 4y ago
    Like I mention, all of the evidence The Wire has used here is likely fabricated. Not by the Wire themselves, but by the source they’re relying on.
    - webmobdev 4y ago
      The source has been verified by The Wire to be a Meta employee / contract worker. We reiterate the faith we have in our sources, whose identities and positions in Meta are known to us. Our reporters have had a productive relationship with them for some time already, prior to the Instagram story. Meta’s suggestion that sources who don’t know each other have teamed up to “hoax” The Wire is ludicrous. Source: https://thewire.in/media/statement-by-the-wire-on-metas-responses-so-far https://thewire.in/media/statement-by-the-wire-on-metas-resp... The emails have been verified to be authentic by The Wire and 2 independent experts, one of whom works at Microsoft. Further proof of the "instagram URL" has been shared. Even if we assume that the "instagram" workplace portal is the "weakest" evidence and a "fake" (as you and others claim), than Meta should be easily able to identify the person who created the "fake" instance and populated it with all the "fake" tickets. This is what Alex Stamos too tweeted: If The Wire is wrong, then Meta has all the evidence they need. While you could create a whole fake Workplace, the easier move is to just create a free trial instance, meaning those fake notes are sitting in Meta's databases along with the metadata of whomever created them. I think, at this point, it's on Meta to write-up a detailed response with whatever technical evidence they have. This will not go away just by ignoring it. The fact is that The Wire have an obligation to protect their source and so are limited in the evidence they can share publicly (unless taken to court). So this begs the question that if Meta and you and other FB employees) are so sure of the evidence being fabricated, why do you think The Wire hasn't been sued yet to take down the "fake" articles? (Note that you do not even need to identify who faked the evidence, to sue them, as long as you can prove that it is fake, which should be easy to do so according to all the claims made by Meta, you and other FB employees).
    - csande17 4y ago
      If that's the case, it seems like someone should be informing Facebook that their DKIM signing keys have been compromised. The impact of that would be much worse than just a forged .eml file sent to a journalist.
  - KrishnaShripad 4y ago
_micheee 4y ago
Why?
- nvarsj 4y ago
  This kind of internal tooling and workflow is almost always under NDA.
- bagels 4y ago
  If they worked at Facebook/Meta they violated their nda and put themself in legal peril.
  - lopkeny12ko 4y ago
    Have you never seen https://engineering.fb.com/ https://engineering.fb.com/? Engineers there blog about their tech tools all the time. "Legal peril" sounds like a bit of a stretch.
    - bagels 4y ago
      Those blog posts likely go through legal, privacy and marketing review. If you think that Facebook wouldn't enforce an NDA, especially on something sensitive like this, I think you are incorrect.
    - wrigby 4y ago
      These posts are all thoroughly reviewed by comms and legal teams. In onboarding, it’s thoroughly communicated that you need to go through the proper channels to publicly publish anything with technical details.
  - nindalf 4y ago
    I think I’ll be fine. My reasoning is two fold - I haven’t shared anything that could be exploited by anyone. And second, Meta and others in the industry try to share information about how their integrity efforts work so we can learn from each other.
    - 4y ago
tjpnz 4y ago
https://archive.ph/AK0zi https://archive.ph/AK0zi
crmd 4y ago
He doesn’t work there anymore[0] Also, nobody likes a tattle tale. [0] https://uk.linkedin.com/in/krishna-sundarram-17a76954 https://uk.linkedin.com/in/krishna-sundarram-17a76954
nindalf 4y ago
Thanks for the email. But I think it should be fine. I haven’t said anything that could be exploited by bad actors. As for competitors, there are no competitors in integrity. The entire industry goes to great lengths to share knowledge on what works and what doesn’t. We all win when we combat abuse well.
- hitekker 4y ago
  You should be fine. The GP seemed to have forgotten they're not posting on FB"s internal workplace.
  - MertsA 4y ago
    Internal comms are notoriously open, this kind of post would be a welcome contribution for explaining XCheck to a general audience outside of the integrity org. I don't know what kind of confidentiality agreement they signed but mine would have technically prohibited discussing this. Not only that, it's not time limited in nature, it applies "at all times during the term of my Relationship with the Company and thereafter". I wouldn't expect Meta to actually do anything about this blog post but I wouldn't have felt comfortable posting it.
doliveira 4y ago
Do you at least get extra money for snitching on a fellow worker like that?

otterley 4y ago

> a system that exists and mostly works is preferred to a hypothetical perfect system that is never built. At any sufficiently large scale - especially with a product you're not paying for - this is the best you're likely to get. Once you fully understand and embrace this, your stress levels about imperfection tend to go way down.

trasz 4y ago
- bagels 4y ago
  What is an example of "this" being done properly?
- pilgrimfff 4y ago
  > this has been done properly numerous times in the past I'm genuinely curious if you have any examples of this. I'm not aware of any modern product at scale that doesn't suffer from these same issues.
  - djohnston 4y ago
    When people say that I imagine them thinking about a web forum with 200 monthly visitors.
  - trasz 4y ago
    Every newspaper and pretty much any other medium over the past century or so. That’s what editors used to do. Scaling is not the problem - profits scale with the number of users, the cost of moderation would too, in the worst case. The problem is company’s unwillingness to pay anything at all. Their profits didn’t came from offering a good product, but from discovering a new way to offload the costs of their operation while keeping the income. In this case - by pretending to be a part telco, part newspaper (which brings income), but without taking their main responsibilities (which cost money).

smrtinsert 4y ago

itake 4y ago

I wish they would explain what xcheck is and how it works before the talk about the problems with it. This reads like a blog post to other Integrity FB engineers that already know that system.

nindalf 4y ago
It’s explained later on but I could be clearer. XCheck is a system that supports tagging specific accounts with tags that exempt that account from certain integrity enforcement. In the post I give an example of a tag that used to exempt accounts from the fake account checkpoint. Applying these tags is usually a manual process and usually carefully vetted.
- webmobdev 4y ago
  You yourself claim that you are only aware of Xcheck "features" since 2021, As of June 2021 no XCheck tag I knew of allowed a user to take down content at will. Are you also claiming that there have been no changes, and no new features, added to XCheck since June 2021? Are you aware of the new social media regulations recently passed by the current Modi government of India? (To be credible, you need to honestly answer these questions and address it in your article too, as otherwise you are basing your assumptions on incomplete information, thereby misleading yourself and others.) To clarify, one of the requirement of this regulation is: Under the new Intermediaries Rules, intermediaries must *complete* the takedown process under Section 79(3) of the IT Act, within 36 hours. Source: https://internetfreedom.in/intermediaries-rules-2021/ https://internetfreedom.in/intermediaries-rules-2021/ (a more brief summary of the regulation is here - https://techwireasia.com/2021/06/indias-new-social-media-rules-explained/ https://techwireasia.com/2021/06/indias-new-social-media-rul...) Very simply put, large social media platforms are legally required to respond and delete content when requested by government representatives. Under such a scenario, can you confidently say that XCheck hasn't been extended with new features, like a tag that allows a user to immediately flag posts that must be compulsorily taken down when the particular user reports it? It seems a very obvious extension when you consider the political demands that are placed on Meta from around the world, and in India in particular.
- 4y ago

SanjayMehta 4y ago

iudqnolq 4y ago
> who hates India, the current government and probably himself as well ... As someone who dislikes both Meta and The Wire, this is a source of great entertainment. As an American it's fascinating how the tone of HN completely changes whenever the merits of cancel culture, the Internet Archive, or the current government of India comes up. It's an odd list.
- KrishnaShripad 4y ago
  Because, believe it or not, plenty of Indians are satisfied with the current Government in India. We have had shit Governments for more than 70 years. So don't be surprised if Indians don't support Western media narratives on Indian Government (which is mostly fabricated/fake).
  - webartisan 4y ago
    I think you should speak for yourself instead of for all Indians. You believing or not believing something has little effect on facts. Bullying everyone into believing your unquestioned love for the present day government is rather disenginious.
    - KrishnaShripad 4y ago
      - throwaway384629 4y ago
        You are mostly sharing your opinion. I am not saying that you are wrong, but stating your opinion has much less weightage than stating the facts where your claims are supported by some data/references.

DisjointedHunt 4y ago

[flagged]

localhost 4y ago

This is an excellent piece which really does a good job at explaining the nuance that exists in this challenging problem domain. Thank you for writing this. It's a great piece to point to whenever the next scandal gets "exposed" by click-seeking folks on the internet.

uoaei 4y ago
Are you saying that this article excuses data breaches?
- ironrabbit 4y ago
  The "scandal" that prompted this article is referenced with sources under the heading "Why XCheck is in the news". Neither the article nor the comment you're replying to says anything about data breaches.
sneak 4y ago
It's good that it keeps getting exposed that a single organization without accountability decides what people are or are not allowed to publish on a platform to the people who explicitly subscribed to their publications. The fact that moderation is a hard problem is orthogonal. The fact remains that there are different rules for different people, both for flagging and for content. The fact that there is more nuance than "two sets of rules" doesn't really change the fundamental problem that private companies now get to arbitrarily police publication by normal people. I have the distinct impression that in 100 years the default mode is going to be "if it's legal you can post it, if it's not you can't, and if you have a problem with the boundary, take it up with the lawmaking authority and not us". We will look back at this time at the emergence of any-to-any publishing like the dawn of radio or TV or newspaper where a few rich guys got to decide what could or could not be published.
- CursedUrn 4y ago
  What makes you think something as unique and hard-fought as the First Amendment will be applied to the Internet when all trends point in the other direction, both on and offline?

yardstick 4y ago

> That works well for about two weeks until users figure out how to exploit this. They form groups that agree to coordinate to report content. Now reports become lower signal than before, but still somewhat useful. We use the reports, but try to limit exploitation. What about human employees reviewing users reports, especially randomly picked samples from those that resulted in successful takedowns. If the content was found to be sound under the rules, penalise the reporters. Suspend their ability to make reports (probably in the form of silently ignoring them). Ie some human moderation should still be done at this scale, but using a combo of random sampling and system-flagged-suspicious.

xani_ 4y ago
I remember Dota2 (video game) did this, people fake reporting had less reports and they counted for less, while people that were consistently reported players that got punished got more reporting power. Only one report per match "counted" too so a group of player couldn't gang on reports on one player. Of course that could be abused as well,but you'd have to make a group of people that first got the good rating then reported same people and that would be significantly harder
- WorkerBee28474 4y ago
  Well that just sounds like Bayes's theorum with extra steps
- qualudeheart 4y ago
  So that’s why I was never banned from Dota even though I sucked at it.
coredog64 4y ago
This is Slashdot's meta-moderation system from 20(?) years ago? Logged in users would randomly be selected to check moderation, and accounts that abused their mod points were flagged. Granted, you probably couldn't do that today as you could see meta-moderation brigades out in the wild. Employees should be better, although there's still the chance that an employee might use their position to punish opinions they disagree with.

timzaman 4y ago

How is it people are allowed to blog about company internal implementation details and numbers barely a year after they leave?

deleted 4y ago
[deleted]
tyingq 4y ago
Seems unsurprising to me. If you, as a company, want ironclad confidentiality from people after they leave, that wouldn't be free or automatic. None of what was shared seems to qualify as a trade secret, which would have some protection.
- extr 4y ago
  I've never worked somewhere that didn't have me sign some kind of broad NDA-esque clause saying I wouldn't share proprietary or non-public info for a certain period after release (imo, completely reasonably). This blog post is literally nothing BUT non-public proprietary info. Like the parent I'm really surprised the blog author feels comfortable sharing all this. Even if it ended up being technically legal, I have to imagine this kind of thing is extremely frowned upon.
Kiro 4y ago
Why wouldn't they?
- extr 4y ago
  Have you ever read your employment contracts? I've never worked somewhere that would allow something like this to be shared so soon after departure.
  - alliao 4y ago
    meta being so big they're almost part of public institution just privately held; and public scrutiny is less frowned upon especially if many believes they do not contribute to the betterment of the republic

numair 4y ago

I think that the key takeaway from the entire Wire vs Meta fiasco is that there is a lot of absolutely weird Spy vs. Spy behavior going in the Indian political and media industries. I was initially extremely sympathetic to the story presented by The Wire, because it's quite believable that Meta/FB would go to extreme lengths to try to distance themselves from such a situation, but the facts ... just ... don't add up. As Alex Stamos has noted, there is little debate regarding collusion between Meta's "government relations" people and policy groups, so that's not really much of a scandal -- it's not like Meta will deny that Modi was treated like a divine being when he visited their campus, so the idea that his people can easily call in a favor to crush a social media post won't really surprise anyone. The bulletproof evidence with DKIM authentication and a video of a logged-in admin instance doesn't look so bulletproof after all, based on the credible reports from those who know how Meta's admin tooling actually looks and functions, and those with other DKIM authenticated emails from the fb.com domain. So, the question is, what's the agenda here? Why would someone go through all of this effort, to create a scandal out of something that is not very far from the truth? What's the point of this entire thing? Maybe this is like Nick Denton and Gawker losing their entire business over the stupidest sex tape story ever; or maybe this is part of something else that requires domain knowledge regarding Indian politics and media to understand (do page views monetize so well that this mini-scandal is going to be super-profitable to The Wire? Highly doubtful, right? What could they possibly get out of this?). The whole thing is just really weird. It's also a major distraction from the very real problems that Meta doesn't even try to hide. Meta's relationship with the governments in various countries -- including the United States -- is way too close for comfort, and absolutely toxic on multiple levels. If this story turns out to be fake news, it'll do a lot to help the company deny, deflect and discredit the next real scandal. I think this is what the wacky conspiracy theorists call "4D chess," but I don't think that's what's happening here. The true story behind all of this is bound to be very strange, and very stupid.

DisjointedHunt 4y ago
I'd speculate this is part of an "Opposition research" move by a political party that might benefit. The extreme lengths that these people seem to have gone to is shocking. Whatever it is, FB has the evidence on their servers as Alex Stamos points out since they created a Workspace instance.

wiml 4y ago

This is a mildly interesting post only because of how it comes up to, but avoids ever talking about, the central flaw in the system it describes — they're conflating several things without ever really acknowledging that they're different. Theres "integrity", which is a Meta-internal term; verification of ID, which is a far deeper rabbit hole than they're going to go into; abuse; and undesirable behavior. They're all different things.

remus 4y ago
> They're all different things. Different but related, I would say. For example abuse of a service and undesirable behaviour on a service could be thought of as different points on a spectrum.

SilverBirch 4y ago

It makes sense from a technical perspective why you'd build a system like XCheck. The problem is that once you build XCheck you actually need to say to your C-suite and PR department "Actually no, you can't say we treat everyone equally, we're manually going in and exempting certain people from scrutiny, and that's entirely within our discretion." The problem isn't technical, it's legal - you can't build a system that operates one way and then publicly misrepresent it.

yeasurebut 4y ago
xani_ 4y ago
As facebook shows you can, and get away with it.
pjc50 4y ago
Legally, their TOS probably allows them to be completely opaque and arbitrary. And any system which involves reporting from the public will need to build a "this account gets lots of false reports of category X, to save time assume they're all false" system. As well as "all the reports made by this user are bogus".
etchalon 4y ago
There are no laws which say a company has to be transparent and accurate when discussing its moderation systems.
- lmm 4y ago
  Given that people pay to advertise on facebook, any false statements they make about their moderation systems via the internet almost certainly constitute wire fraud.
  - pavlov 4y ago
    You make that claim with a high level of confidence, but have you actually read the contract in place between Meta and the advertiser?

oars 4y ago

I would love to work at a FAANG in the future to gain exposure to these interesting problems and systems at huge scale.

philjohn 4y ago
They're interesting problems for sure; it's equal parts fun, technically challenging, and then you get to throw in having to quickly react to adversarial responses.

1letterunixname 4y ago

So I work at Meta and just got MetaMorphized s/@fb/@meta/g. When I started, my personal account was banned twice on the condition of adjusting 2FA. It was sorted out. I had a past account that I could no longer access as email domains changed, so it looked like I was trying to maintain duplicate accounts. Why it triggered on adjusting 2FA didn't make sense. Then, it wanted me to verify my identity in a way that created a Catch-22 of wanting a login from itself or to a device that didn't have a valid login session. I set the options for better security and left it, rolled all of my personal passwords, eliminated public data, canceled unnecessary social media accounts, and enrolled in security features like Google's Advanced Protection Program. I'm hesitant to make changes risking getting locked-out again. I recently spoke with someone in Readiness who works with hiring human verifier contractors around the world. A primary issue is scale. You would need to hire the entire world's population to moderate the content being produced. Still there will be bias and misinterpretation of sarcasm, and varying standards of acceptability and decorum. AI is a force-multiplier to an extent, but it takes human judgement to rectify mistakes and data to identify brigades, scammers, terrorists, and political manipulators seeking to exploit an imperfect system. Sadly, the humans with the best judgement typically have better career options that they couldn't be paid to do social media moderation. It can be made better, but the ultimate realization is with even great care, good intentions, and attempts at making things sensible and fair, there are always going to be mistakes. It's trying to minimize mistakes and not enable genocides, election sentiment manipulation, or product scams. It's doable to minimize mistakes, but it take persistent vigilance, wisdom about human nature, and creative solutions to deter and prevent harm while avoiding harming innocent persons. Mistakes are bad and disappointing and it feels bad when they happen. In case anyone were wondering, the security is the inverse of Twitter's. Everything is logged and access requires a business purpose for a limited time, narrow scope, and approval to get that access. Almost no one has access to production data. PII is taken very seriously. There are no laptops with copies of user data. All laptops are encrypted, just in case, and for general principles. Password complexity requirements are insane. I can see my work/personal FBID user object in the graph, but as soon as I try to prod any links to other users, big warnings appear. There's an army of insane genius security researchers and practitioners who create and deploy defense-in-depth tools for broad and specific solutions to prod, corp, and endpoints that reduce our risks to being compromised, data being exfil'd, and security "oops"es from happening. Work users who transit through certain "hostile" countries lose some security credentials and access. I'm actually wondering why laptops aren't spot-checked for malware implants and hardware/firmware modifications. I would assume employees with critical access who travel internationally with their work laptops and phones are prime targets. PS: I wonder if people would pay $X / month (say $199) to have a high-signal social media service that requires a level of "vouching" invitation, names with faces profiles (not visible to the wider internet), sensible/proportional mediation and civilized feedback, politically-neutral, and free speech-loving to increase the sense of community and reduce the potential of anonymous bad actors. 37signals/basecamp accumulated research that showed that smaller communities with faces and real profile names lead to nicer interactions. I don't recall the source, but communities that are defended in terms of politeness and boundaries tend to endure while undefended communities drive away users and tend to disperse.

civilized 4y ago

> Maybe after 5 or 10 or 100 reports, the system will automatically make the reported content invisible. This explains why so much "moderation" on sites like Twitter seems to lack any rhyme or reason.

josephcsible 4y ago

> Normally such content would be taken down after sufficient number of reports were received. I can't be the only one who sees a problem with this. No content should ever be taken down automatically just because a bunch of random people report it.

throwawaylinux 4y ago
> No content should ever be taken down automatically just because a bunch of random people report it. Serious question, why not?
- fragmede 4y ago
  Because you have no idea if those reports are at all genuine, or if the reporters met up elsewhere (online or off) in order to brigade and mass-report said content, with the intention of getting it taken down despite breaking no rules. Sometimes, the coordination isn't even necessary, it just needs to be the right target posting something online. (Eg more than a few people have gone and reported every post by a politician you like/dislike for hate speech and inciting violence.)
  - ummonk 4y ago
    That’s why politicians tend to get X-Check.
  - throwawaylinux 4y ago
    The article well explains this downside of user reports, so I don't see what this comment adds. It does not answer my question. The article also describes problems of not acting on them, so the conclusion requires more than just finding a negative.
    - fragmede 4y ago
      I'm sorry you're unable to understand my answer. Let me try saying the same thing as the article another time, and maybe you'll be able to understand the answer? Your question was Why should no content should ever be taken down automatically just because a bunch of random people report it. It's because the random people reporting it can't be trusted to be acting honestly. Without a human in the loop, the automated system becomes a tool for cyberbullies, and harms the very users you intend to protect. The foregone conclusion, thus, is that a fully-automated system will do more harm for the user-base than good.

asah 4y ago

"The number of reports number in the millions per day. That's too many for human moderators" Isn't Reddit the counterexample? A dozen of us moderated a controversial sub with 2M users, no problem.

HiJon89 4y ago
With 2 million users, I imagine it would be on the order of thousands of posts per day, and maybe hundreds of reports?
sophiebits 4y ago
So for a few billion users, you need tens of thousands of moderators, assuming that ratio holds?
- smolder 4y ago
  This is a very efficient moderation force. I'd be happy to get away with 10s of thousands per billion. That's still under 100 per million. Under 1 per 10,000.
Spivak 4y ago
2M “subscribed” users to a subreddit is about 10% or 200,000 daily viewers with about 1% of those commenting and 0.1% posting so 2,000 comments and 200 posts per day. Some subs will be more or less but that’s the magnitudes we’re talking about. The equivalent situation assuming mods review every comment would be a subreddit with 1 billion users. If mods are only reviewing reports which is some small percentage of comments then adjust accordingly.
- fullsend 4y ago
  Very well put, it's not even close to the same ballpark. It would also be like moderating a 1 billion user subreddit where nearly every single post is off topic and comes from different users, whereas in subreddits a few super users often produce a huge percent of the content.
- 4y ago

whiddershins 4y ago

Sounds like binary flags that should be weights.

Gumbercules 4y ago

Zenbit_UX 4y ago

If I were designing a system like this, I'd likely implement a reputation system for reporters (users who report, not journalists). When you report content for the first time your reputation score is 0.5, if the content is of a celebrity or public figure, your reputation will invisibly drop, enough of these and your score will be so low as to not be a useful flag when evaluating whether content is objectionable. On the other side, if the content is reviewed as violating ToS, your rep will increase. With a system like this the fake reports and brigading will quickly lose its power, eventually all users who abuse reports will be ranked so low as to not matter. A threshold of rep score can be set before the content is auto removed. Maybe something like 25pts in order for it to be removed, and that's after adding together all the rep of the reporting users.

koyote 4y ago
I always thought that this is how it already worked. That being said, the above can also be abused by creating a bunch of high rep accounts or opening up a black market of high rep accounts for sale. It's definitely a higher bar though.
DannyBee 4y ago
In a system like this they will play both sides. They do, in fact. Set up accounts that produce content that violates TOS and accounts that report it. The reporting accounts will have their rep increased for correctly reporting tos violations. Then they use the high rep accounts to brigade and fake report. Or sell them. They are patient, so even if you make reputation go up slowly, they will just build a pipeline of account harvesting.

nonrandomstring 4y ago

The corruption in tech, perhaps inevitable or intrinsic, looks more and more like the forces that tore the Soviet Union and then Russia apart. 1) A "revolution" and promises of new fairness 2) A Nomenklatura 3) The rise of unassailable systems and faith in them 4) Widespread failure and instability of those systems 5) Attempts to stabilise by handing more power to privateers 6) Abuse and corruption run wild amongst new oligarchs