16 ms·

Huh, never thought I’d see XCheck in a news article. I used to work at Facebook and spotted abuse of this system by bad actors and partly fixed it. It’s still n

Huh, never thought I’d see XCheck in a news article. I used to work at Facebook and spotted abuse of this system by bad actors and partly fixed it. It’s still not perfect but it’s better than it used to be. I think I might have agreed with the author of this article before working in Integrity for a few years. But with time I learned that any system that’s meant to work for millions of users will have some edge cases that need to be papered over. Especially when it’s not a system owned and operated by a handful of people. Here’s an example - as far as I know it’s not possible for Mark Zuckerberg to log in to Facebook on a new device. The system that prevents malicious log in attempts sees so many attempts on his account that it disallows any attempt now. There’s no plans to fix it for him specifically because it works reasonably well for hundreds of millions of other users whose accounts are safeguarded from being compromised. His inconvenience is an edge case. With XCheck specifically what would happen is that some team working closely on a specific problem in integrity might find a sub population of users being wrongly persecuted by systems built by other teams located in other time zones. They would use XCheck as a means to prevent these users from being penalised by the other systems. It worked reasonably well, but there’s always room for improvement. I can confirm some of what the article says though. The process for adding shields wasn’t policed internally very well in the past. Like I mentioned, this was being exploited by abusive accounts - if an account was able to verify its identity it would get a “Shielded-ID-Verified” tag applied to it. ID verification was considered to be a strong signal of authenticity. So teams that weren’t related to applying the tag would see the tag and assume the account was authentic. And as I investigated this more I realised no one really “owned” the tag or policed who could apply it and under what circumstances. I closed this particular loop hole. In later years the XCheck system started being actively maintained by a dedicated team that cared. They looked into problems like these and made it better.

annadane 5y ago
The problem seems to be though, that while the company may have tools to detect abuse, if they're choosing selectively when to enforce things it defeats the entire point Edit: downvotes from shills
- nindalf 5y ago
  That wasn’t my experience over several years. Whenever we found a new vector of abuse or detected systems misfiring we would take it seriously.
  - some_hacker33 5y ago
    are you from the UK or india?
gkop 5y ago
Your analogy relating to Zuckerberg is absurdly tone deaf.
- nindalf 5y ago
  I think it’s valid because it’s an example of a system not working for a small minority while still working well for others. And more pertinently, there’s no plans to fix it for him specifically just because he’s the CEO. It’s better to spend time making the account compromise system better for the vast majority of users instead.
  - smsm42 5y ago
    The problem is not account compromise. Nobody is complaining about inability to compromise Zuckerberg's account or it being to hard to register a new device or anything like that. The issue is question is the two-tier (or maybe multiple-tier) system of rules that secretly exists inside Facebook while the public materials falsely claim all users are guided by the same rules.
  - elmomle 5y ago
    Or perhaps the underlying business/product model is inherently flawed in a way that's bad for society, all patches have proven woefully insufficient in mitigating that, and Facebook have been intentionally concealing this.
  - IncRnd 5y ago
    > I think it’s valid because it’s an example of a system not working for a small minority while still working well for others. That's not what the article mentioned, which is why people are saying you don't seem to understand. The article mentioned there is a system, which does work for a small minority of people, not that it doesn't work for them. It's as if you unintentionally have blinders on and can't see the issue.
  - SamBam 5y ago
    It is tone-deaf. You seem to be regaling it as an inconvenience or problem that Zuckerburg has. It's not an inconvenience, it's a personal bodyguard that he alone has. If he changes devices, he has a team to allow him to log into it. How is that relevant? The rest of us don't get that. If someone hacks our account and changes our password, the majority of us have little hope of really getting FB's attention to help us fix it. The point about popular accounts getting 10000x the number of abuse reports is much more relevant.
otterley 5y ago
You haven’t really touched on the main problem discussed in the article, which is that to Facebook, there are special users - mainly celebrities and politicians - who get to play by different rules than the rest of us. Social media was supposed to help level the playing field of society, not exacerbate its inequalities.
- nindalf 5y ago
  I did touch on that problem. Like I pointed out Zuckerberg can’t log in on new devices anymore. That’s because of the thousands of attempts per second to log into his account. Those attempts happen because he’s a celebrity. His experience is objectively different because of who he is. It’s the same with Neymar. How many times do you think his profile is reported for any number of violations by people who don’t like him? If an ordinary persons account got 100 reports a minute it would be taken down. Neymar’s won’t be. I don’t know how every Integrity system could be modified to make an exception for any of these classes of accounts or how to codify it in a way that would seem “fair”. If you have an idea for a better way, you should share it.
  - otterley 5y ago
    I concede that some accounts probably deserve that specific type of protection. However, it doesn’t explain the other kinds of protections these people have, including exemption from content-based rules. Those are the issues of real concern.
    - nindalf 5y ago
      The fact that the content couldn’t be taken down was a bug. I feel like you, the author and most critics of Big Tech discount the possibility of bugs existing.
      - otterley 5y ago
        I’m sure you’ve read the old fable of “The Boy Who Cried Wolf.” Facebook has made voluntary decisions at the highest levels, reported over the course of the past decade, to shield certain people from its content rules that apply to the general public. So if it was a bug this time (and I have no reason to believe that it wasn’t), I’m sure you can understand people’s skepticism about it. Moreover, I assume this bug was reported internally, probably pretty quickly. How long did it take to get fixed? If the fix wasn’t prioritized and corrected within, say, a day (along with a regression test to ensure it never happens again!), then that would be pretty damning of the company’s culture and priorities as well.
smsm42 5y ago
I think people that work on this feature mean well - or at least they think that they mean well. But as a result, we have a two-tier system where the peasants have one set of rules and the nobility has an entirely different one. It may have started as a hack to correct the obvious inadequacies of the moderation system, but it grew into something much more sinister and alien to the spirit of free speech, and is ripe for capture by ideologically driven partisans (which, in my opinion, has already happened). And once it did, the care that people implementing and maintaining the unjust system have for it isn't exactly a consolation for anybody who encounters it.
- q1w2 5y ago
  You have to remember that high profile accounts get 10000x the number of abuse-reports than a normal account - nearly all bogus. The normal automated moderation functions simply do not work. Many users will simply mash the "report abuse" button if they see a politician they don't like, or a sports player for an opposing team. If the normal rules applied identically to everyone, all high profile accounts would simply be inactive in perpetuity. Maybe a better system would penalize reporters that are found to have reported content that do NOT violent content policies?
  - otterley 5y ago
    Nobody is complaining that Facebook has exceptions for automatically suspending accounts just because people are misreporting them as abusive. The issue is, and has always been, exceptions to the content-based rules which are supposed to apply equally to everyone.
    - nindalf 5y ago
      So if hundreds of people reported the content posted by a celebrity, or if a classifier misfired on the content posted by a celebrity, there shouldn’t be any protection for the content?
      - otterley 5y ago
dundermuffl1n 5y ago
Most of your response treats the service and its flaws as an engineering problem, whereas the ramifications in the real world aren't something Facebook gets to absolve itself from. They need to own the problem completely. If they can't solve the issue through engineering, it is their responsibility to hire hundreds of thousands of moderators.
lmilcin 5y ago
I disagree. Let's take your example of Mark Z. What makes you think that this is unique case? What about people that suddenly come to fame, like viral video subjects? A simple solution is to disallow logging in from new devices and the attempt being silently dropped so you are not bothered, unless you do some magic like generate one time key to complete the procedure on the new device. I could think of a lot of people that would find it useful. Or allow setting up 2FA token (other than mobile) correctly. Instead what FB does is make it impossible to secure your account because they insist whatever you want you should always be able to recover your password with your phone number. Years ago when I was still using it (I had reason) I tried to secure it with my Yubico. Unfortunately, it wasn't possible to configure FB to not allow you to log in on a new device without the key. I understand how the discussion probably went: "Let's make it so that we can score some marketing points but let's not really make it requirement because we will be flooded with requests from people who do not understand they will never be able to log in if they loose the token." But that's exactly what I want. I have a small fleet of these so it is not possible for me to loose them all but unfortunately most sites that purport to allowing 2FA can't do it well because they either don't allow configuring multiple tokens or if they do, they don't allow really lock your account so it is not possible to log in the next time without the token.
- Ansil849 5y ago
  > unfortunately most sites that purport to allowing 2FA can't do it well because they either don't allow configuring multiple tokens or if they do, they don't allow really lock your account so it is not possible to log in the next time without the token. This is a great point. AFAIK, Google is the only service which allows you to set mandatory U2F login requirements. Does any other service offer this functionality?
  - manquer 5y ago
    Many enterprise apps you can force it or depend on authentication from Google or Azure kind of SSO providers, who have this feature. Consumer apps try hard not to do hard security unless they are forced to, usually for cost reasons. Security measures like these create a ton of administrative tickets - check any sysad ticket queue a good chunk is password reset/recovery, in enterprise apps the org sysads are paid to handle this. In consumer apps, the app company has to manage it, also it lot harder to verify identity of a random user than company employees making it harder to do this kind of support. A good jarring example is AWS, the amazon.com and AWS did (does?) share authentication stack so some basic 2FA functionality like backup codes is not there for AWS . Google is better at this because they have for long time also focused on SSO service as a product. Many companies use Google AD /SSO workspace/suite because third party apps support google login out-of the box free[1], maybe charge for AzureAD/SAML2 and likely not support others at all without customization costs. [1]It is standard because SMB/mid market companies are more likely to use Google for productivity than Azure/o365 as it is easier to manage albeit with lesser features. Third party apps don't want to expend support time on smaller customers if they can avoid it
bo1024 5y ago
Thanks a lot for posting these details and dealing with the critical replies. I think that with your background and investment in improving these problems, it will be hard for you to understand the perspective many people have that Facebook is fundamentally rotten at this point. These conflicts arise from FB's core business model. It calls up a torrent of hate speech and misinformation with the right hand while trying to clumsily moderate with the left. You can hire whole teams to prevent singed fingers or protect certain possessions, but the point of a fire is to burn. If there are no good solutions while maintaining FB's core approach and business model, then it would be better for the world if it were extinguished.
- cbsmith 5y ago
  Note that moderation is not part of the business plan. FB was pretty much dragged kicking and screaming into that function.
  - jamal-kumar 5y ago
    I recall that their approach to that was outsourcing it because I read that article that was going into how some of the employees working for the company doing the moderation were losing their minds by being the mental sewage treatment facility of social media
    - cbsmith 5y ago
      Yeah... that's pretty late in the narrative. Go back farther. The original plan was no moderation.
- helen___keller 5y ago
  > It calls up a torrent of hate speech and misinformation with the right hand while trying to clumsily moderate with the left. Not a Facebook employee (or supporter for that matter), but I'm curious if you consider this an issue of Facebook or of social media in general. Not saying it's OK for FB because everyone does it, but you generally see the same dynamic of the "torrent of hate speech and misinformation" on Twitter, on Reddit, on Youtube even (personal experience: I have a family member that was radicalized by misinformation on the internet. It was all on Youtube, she had never even used Facebook). I've noticed that people go a lot harder on Facebook than on other tech companies. I think Facebook's reputation is well deserved, but I do think that reputation should be shared with really all social media in general.
josefresco 5y ago
> Huh, never thought I’d see XCheck in a news article. Is everyone at Facebook this naive? You didn't think a system that creates a secret tier of VIP accounts where the rules (and laws) don't apply while publicly claiming the opposite would end up ... in the news?!?!
- anigbrowl 5y ago
  One of the nice things about transparency is not having to engage in performative theatrics at a later date.
cheath 5y ago
One of my favorite things about HN is seeing people come out of the woodwork to raise their hand and say that they worked on a system and give their insight. Thanks for sharing this perspective.
rwmj 5y ago
So when Mark Zuckerberg buys a new phone or whatever, what happens?
- 908B64B197 5y ago
  My bet is he emails the auth team and gives as much info as he can about the device. Then they create auth rules as if a regular login had happened. I also assume if he's on the internal network it might be easier to just manually allow auth attemps from a few internal IPs for a few seconds.
  - massysett 5y ago
    My bet is that he emails someone on his staff one day and says “I want the new iPhone” and the next day he has one and he can do whatever he needs to on it. This whole idea of “Zuck can’t log in from a new device” is laughable. That’s like saying Biden doesn’t have the keys to Air Force One. He doesn’t need them.
schwank 5y ago
This system also made it impossible for me to ever log in again. It had been a few years since I used FB but some friends tagged me at an event, so I figured what the heck. I was presented with a system I had never configured, which asked me to contact people I don't know to get them to vouch for me. At the same time my FB profile was blackholed, and my wife and long time actual friends can't even see that I exist anymore. Just some person that astroturfed my name with no content (I have a globally unique name). So I no longer exist from FB perspective, which made both my decision to not use FB as well as never use any FB products like Occulus much easier.
tlear 5y ago
You are surprised that after blatantly lying for years the truth came out? I mean unless you are the 5th directorate of KGB.. but even then shit like this always comes out.
sub7 5y ago
"I was only following orders"
iammisc 5y ago
That's all well and great, but your comment as an insider directly implicates Facebook CEO of perjury by lying to congress. During a hearing he claimed all users were treated equally. This is clearly not the case. Perjury before congress can result in jail time and I hope he's made an example of.

annadane 5y ago

The problem seems to be though, that while the company may have tools to detect abuse, if they're choosing selectively when to enforce things it defeats the entire point Edit: downvotes from shills

nindalf 5y ago
That wasn’t my experience over several years. Whenever we found a new vector of abuse or detected systems misfiring we would take it seriously.
- some_hacker33 5y ago
  are you from the UK or india?

gkop 5y ago

Your analogy relating to Zuckerberg is absurdly tone deaf.

nindalf 5y ago
I think it’s valid because it’s an example of a system not working for a small minority while still working well for others. And more pertinently, there’s no plans to fix it for him specifically just because he’s the CEO. It’s better to spend time making the account compromise system better for the vast majority of users instead.
- smsm42 5y ago
  The problem is not account compromise. Nobody is complaining about inability to compromise Zuckerberg's account or it being to hard to register a new device or anything like that. The issue is question is the two-tier (or maybe multiple-tier) system of rules that secretly exists inside Facebook while the public materials falsely claim all users are guided by the same rules.
- elmomle 5y ago
  Or perhaps the underlying business/product model is inherently flawed in a way that's bad for society, all patches have proven woefully insufficient in mitigating that, and Facebook have been intentionally concealing this.
- IncRnd 5y ago
  > I think it’s valid because it’s an example of a system not working for a small minority while still working well for others. That's not what the article mentioned, which is why people are saying you don't seem to understand. The article mentioned there is a system, which does work for a small minority of people, not that it doesn't work for them. It's as if you unintentionally have blinders on and can't see the issue.
- SamBam 5y ago
  It is tone-deaf. You seem to be regaling it as an inconvenience or problem that Zuckerburg has. It's not an inconvenience, it's a personal bodyguard that he alone has. If he changes devices, he has a team to allow him to log into it. How is that relevant? The rest of us don't get that. If someone hacks our account and changes our password, the majority of us have little hope of really getting FB's attention to help us fix it. The point about popular accounts getting 10000x the number of abuse reports is much more relevant.

otterley 5y ago

You haven’t really touched on the main problem discussed in the article, which is that to Facebook, there are special users - mainly celebrities and politicians - who get to play by different rules than the rest of us. Social media was supposed to help level the playing field of society, not exacerbate its inequalities.

nindalf 5y ago
I did touch on that problem. Like I pointed out Zuckerberg can’t log in on new devices anymore. That’s because of the thousands of attempts per second to log into his account. Those attempts happen because he’s a celebrity. His experience is objectively different because of who he is. It’s the same with Neymar. How many times do you think his profile is reported for any number of violations by people who don’t like him? If an ordinary persons account got 100 reports a minute it would be taken down. Neymar’s won’t be. I don’t know how every Integrity system could be modified to make an exception for any of these classes of accounts or how to codify it in a way that would seem “fair”. If you have an idea for a better way, you should share it.
- otterley 5y ago
  I concede that some accounts probably deserve that specific type of protection. However, it doesn’t explain the other kinds of protections these people have, including exemption from content-based rules. Those are the issues of real concern.
  - nindalf 5y ago
    The fact that the content couldn’t be taken down was a bug. I feel like you, the author and most critics of Big Tech discount the possibility of bugs existing.
    - otterley 5y ago
      I’m sure you’ve read the old fable of “The Boy Who Cried Wolf.” Facebook has made voluntary decisions at the highest levels, reported over the course of the past decade, to shield certain people from its content rules that apply to the general public. So if it was a bug this time (and I have no reason to believe that it wasn’t), I’m sure you can understand people’s skepticism about it. Moreover, I assume this bug was reported internally, probably pretty quickly. How long did it take to get fixed? If the fix wasn’t prioritized and corrected within, say, a day (along with a regression test to ensure it never happens again!), then that would be pretty damning of the company’s culture and priorities as well.

smsm42 5y ago

I think people that work on this feature mean well - or at least they think that they mean well. But as a result, we have a two-tier system where the peasants have one set of rules and the nobility has an entirely different one. It may have started as a hack to correct the obvious inadequacies of the moderation system, but it grew into something much more sinister and alien to the spirit of free speech, and is ripe for capture by ideologically driven partisans (which, in my opinion, has already happened). And once it did, the care that people implementing and maintaining the unjust system have for it isn't exactly a consolation for anybody who encounters it.

q1w2 5y ago
You have to remember that high profile accounts get 10000x the number of abuse-reports than a normal account - nearly all bogus. The normal automated moderation functions simply do not work. Many users will simply mash the "report abuse" button if they see a politician they don't like, or a sports player for an opposing team. If the normal rules applied identically to everyone, all high profile accounts would simply be inactive in perpetuity. Maybe a better system would penalize reporters that are found to have reported content that do NOT violent content policies?
- otterley 5y ago
  Nobody is complaining that Facebook has exceptions for automatically suspending accounts just because people are misreporting them as abusive. The issue is, and has always been, exceptions to the content-based rules which are supposed to apply equally to everyone.
  - nindalf 5y ago
    So if hundreds of people reported the content posted by a celebrity, or if a classifier misfired on the content posted by a celebrity, there shouldn’t be any protection for the content?
    - otterley 5y ago

dundermuffl1n 5y ago

Most of your response treats the service and its flaws as an engineering problem, whereas the ramifications in the real world aren't something Facebook gets to absolve itself from. They need to own the problem completely. If they can't solve the issue through engineering, it is their responsibility to hire hundreds of thousands of moderators.

lmilcin 5y ago

I disagree. Let's take your example of Mark Z. What makes you think that this is unique case? What about people that suddenly come to fame, like viral video subjects? A simple solution is to disallow logging in from new devices and the attempt being silently dropped so you are not bothered, unless you do some magic like generate one time key to complete the procedure on the new device. I could think of a lot of people that would find it useful. Or allow setting up 2FA token (other than mobile) correctly. Instead what FB does is make it impossible to secure your account because they insist whatever you want you should always be able to recover your password with your phone number. Years ago when I was still using it (I had reason) I tried to secure it with my Yubico. Unfortunately, it wasn't possible to configure FB to not allow you to log in on a new device without the key. I understand how the discussion probably went: "Let's make it so that we can score some marketing points but let's not really make it requirement because we will be flooded with requests from people who do not understand they will never be able to log in if they loose the token." But that's exactly what I want. I have a small fleet of these so it is not possible for me to loose them all but unfortunately most sites that purport to allowing 2FA can't do it well because they either don't allow configuring multiple tokens or if they do, they don't allow really lock your account so it is not possible to log in the next time without the token.

Ansil849 5y ago
> unfortunately most sites that purport to allowing 2FA can't do it well because they either don't allow configuring multiple tokens or if they do, they don't allow really lock your account so it is not possible to log in the next time without the token. This is a great point. AFAIK, Google is the only service which allows you to set mandatory U2F login requirements. Does any other service offer this functionality?
- manquer 5y ago
  Many enterprise apps you can force it or depend on authentication from Google or Azure kind of SSO providers, who have this feature. Consumer apps try hard not to do hard security unless they are forced to, usually for cost reasons. Security measures like these create a ton of administrative tickets - check any sysad ticket queue a good chunk is password reset/recovery, in enterprise apps the org sysads are paid to handle this. In consumer apps, the app company has to manage it, also it lot harder to verify identity of a random user than company employees making it harder to do this kind of support. A good jarring example is AWS, the amazon.com and AWS did (does?) share authentication stack so some basic 2FA functionality like backup codes is not there for AWS . Google is better at this because they have for long time also focused on SSO service as a product. Many companies use Google AD /SSO workspace/suite because third party apps support google login out-of the box free[1], maybe charge for AzureAD/SAML2 and likely not support others at all without customization costs. [1]It is standard because SMB/mid market companies are more likely to use Google for productivity than Azure/o365 as it is easier to manage albeit with lesser features. Third party apps don't want to expend support time on smaller customers if they can avoid it

bo1024 5y ago

Thanks a lot for posting these details and dealing with the critical replies. I think that with your background and investment in improving these problems, it will be hard for you to understand the perspective many people have that Facebook is fundamentally rotten at this point. These conflicts arise from FB's core business model. It calls up a torrent of hate speech and misinformation with the right hand while trying to clumsily moderate with the left. You can hire whole teams to prevent singed fingers or protect certain possessions, but the point of a fire is to burn. If there are no good solutions while maintaining FB's core approach and business model, then it would be better for the world if it were extinguished.

cbsmith 5y ago
Note that moderation is not part of the business plan. FB was pretty much dragged kicking and screaming into that function.
- jamal-kumar 5y ago
  I recall that their approach to that was outsourcing it because I read that article that was going into how some of the employees working for the company doing the moderation were losing their minds by being the mental sewage treatment facility of social media
  - cbsmith 5y ago
    Yeah... that's pretty late in the narrative. Go back farther. The original plan was no moderation.
helen___keller 5y ago
> It calls up a torrent of hate speech and misinformation with the right hand while trying to clumsily moderate with the left. Not a Facebook employee (or supporter for that matter), but I'm curious if you consider this an issue of Facebook or of social media in general. Not saying it's OK for FB because everyone does it, but you generally see the same dynamic of the "torrent of hate speech and misinformation" on Twitter, on Reddit, on Youtube even (personal experience: I have a family member that was radicalized by misinformation on the internet. It was all on Youtube, she had never even used Facebook). I've noticed that people go a lot harder on Facebook than on other tech companies. I think Facebook's reputation is well deserved, but I do think that reputation should be shared with really all social media in general.

josefresco 5y ago

> Huh, never thought I’d see XCheck in a news article. Is everyone at Facebook this naive? You didn't think a system that creates a secret tier of VIP accounts where the rules (and laws) don't apply while publicly claiming the opposite would end up ... in the news?!?!

anigbrowl 5y ago
One of the nice things about transparency is not having to engage in performative theatrics at a later date.

cheath 5y ago

One of my favorite things about HN is seeing people come out of the woodwork to raise their hand and say that they worked on a system and give their insight. Thanks for sharing this perspective.

rwmj 5y ago

So when Mark Zuckerberg buys a new phone or whatever, what happens?

908B64B197 5y ago
My bet is he emails the auth team and gives as much info as he can about the device. Then they create auth rules as if a regular login had happened. I also assume if he's on the internal network it might be easier to just manually allow auth attemps from a few internal IPs for a few seconds.
- massysett 5y ago
  My bet is that he emails someone on his staff one day and says “I want the new iPhone” and the next day he has one and he can do whatever he needs to on it. This whole idea of “Zuck can’t log in from a new device” is laughable. That’s like saying Biden doesn’t have the keys to Air Force One. He doesn’t need them.

schwank 5y ago

This system also made it impossible for me to ever log in again. It had been a few years since I used FB but some friends tagged me at an event, so I figured what the heck. I was presented with a system I had never configured, which asked me to contact people I don't know to get them to vouch for me. At the same time my FB profile was blackholed, and my wife and long time actual friends can't even see that I exist anymore. Just some person that astroturfed my name with no content (I have a globally unique name). So I no longer exist from FB perspective, which made both my decision to not use FB as well as never use any FB products like Occulus much easier.

tlear 5y ago

You are surprised that after blatantly lying for years the truth came out? I mean unless you are the 5th directorate of KGB.. but even then shit like this always comes out.

sub7 5y ago

"I was only following orders"

iammisc 5y ago

That's all well and great, but your comment as an insider directly implicates Facebook CEO of perjury by lying to congress. During a hearing he claimed all users were treated equally. This is clearly not the case. Perjury before congress can result in jail time and I hope he's made an example of.