5 ms·

Excerpts from the press call transcript [1] by Guy Rosen explaining what lead to this breach being possible: > The first bug was that, when using the View As f

Excerpts from the press call transcript [1] by Guy Rosen explaining what lead to this breach being possible: > The first bug was that, when using the View As function to look at your profile as another person would, the video uploader shouldn’t have actually shown up at all. But in a very specific case, on certain types of posts that are encouraging people to post happy birthday greetings, it did show up. > The second bug was that this video uploader incorrectly used the single signon functionally, and it generated an access token that had the permissions of the Facebook mobile app. And that’s not the way the single sign-on functionality is intended to be used. > The third bug was that, when the video uploader showed up as part of View As -- which it wouldn’t do were it not for that first bug -- and it generated an access token which is -- again, wouldn’t do, except for that second bug -- it generated the access token, not for you as the viewer, but for the user that you are looking up. > It’s the combination of those three bugs that became a vulnerability. Now, this was discovered by attackers. Those attackers then, in order to run this attack, needed not just to find this vulnerability, but they needed to get an access token and then to pivot on that access token to other accounts and then look up other users in order to get further access tokens. This is the vulnerability that, yesterday, on Thursday, we fixed that, and we’re resetting all of those access tokens to protect security of people’s accounts so that those access tokens that may have been taken are not usable anymore. This is what is also causing people to be logged out of Facebook to protect their accounts. [1] https://fbnewsroomus.files.wordpress.com/2018/09/9-28-press-call-transcript.pdf https://fbnewsroomus.files.wordpress.com/2018/09/9-28-press-...

kidsnow 8y ago
Just as Microsoft developed Patch Tuesday, Facebook should have Forced Logoff Friday
- labster 8y ago
  Every day on Facebook should be forced logoff day. Or, at least, incognito mode day.
  - slimsag 8y ago
    'We only use incognito mode for security. Since it's annoying to login constantly, everyone has dedicated machines that are powered 24/7 so you never have to shut down those incognito tabs.' =)
  - tialaramex 8y ago
    There's a Firefox Add-on named "Facebook Container". Once you install that Facebook lives in a little box, Facebook cookies, Facebook whatever else, all trapped in the little box. No effort needed, if you click your Facebook bookmark, or follow a link or whatever, the browser goes "Oh, this is Facebook" and traps it inside the box with the rest of Facebook without any extra steps from the user. There's a cute blue "Facebook" icon added to the URL bar so you can see it's working. (I mean, or, stop using Facebook, but for many that isn't a reasonable option)
- mkagenius 8y ago
  "That won't scale" - Sandberg
romed 8y ago
That doesn't just seem like a few unlucky coincidences. That seems like a fundamentally unsound design. Why should it even be theoretically possible for a request under the authority of one user to create a token with the authority of another user?
- flgr 8y ago
  Notably, they previously had issues were "View as" allowed you to view notifications and messages of the user you were viewing as. If they'd done a proper post mortem and corrected the fundamental issue, and made sure it wouldn't have re-occured, this should not have happened.
  - bigiain 8y ago
    Instead they moved fast and broke things.
    - nautilus12 8y ago
      Its more important for you to move fast and break things and make us money than to move slow and do things the right way. The life of an engineer...Do it now! why did you do it that way!? Now we are screwed??
      - pyman 8y ago
        Facebook’s php developers like to move fast and break things. Bad design choices, monkey patching, breaking things on production, it’s all part of Facebook’s “engineering” principles.
- calc_exe 8y ago
  Yeah, sounds like https://en.wikipedia.org/wiki/Confused_deputy_problem https://en.wikipedia.org/wiki/Confused_deputy_problem to me.
- everdev 8y ago
  Not the root cause, but I'm guessing a microservice architecture made it more possible. It sounds like both the token generating service and the video upload service have bugs.
partycoder 8y ago
The "View as" feature has been the source of many security vulnerabilities. There was a time where you could read other peoples' chats using this feature.
- groestl 8y ago
  When designing such a system, the immediate failure mode is obvious: at some point, someone will read data not meant for them. As every feature on FB needs to take "View as" into account when handling their own permissions, a lot of developers on FB's payroll get a chance to f'up. We are all humans, so the probability of this happening is very high. The impact (for the users) is also high, given that it's automated and concerns every user on FB equally. When dealing with a very probable, high impact risk in a software project, considerable additional effort is warranted to mitigate that risk: in this case maybe taint checking and additional implementations of the same feature in different programming paradigms, to ensure the system is fail-stop. But in contrast to airlines and railways, the interests of FB and their users are not aligned. For Facebook, this risk is not (or was not deemed to be of) high impact, so we did't get any of this.
- WikipediasBad 8y ago
  Any link to this type of vulnerability? Sounds like a juicy read.
- Raphael 8y ago
  It seems to warrant checking both the permissions of the true user and the view-as user. If either does not have permission, then the action should fail. Of course, lacking the middleware for this forces you to choose one or the other and hope you remember to check the remaining user in numerous pathways.
gboudrias 8y ago
> The second bug was that this video uploader incorrectly used the single signon functionally, and it generated an access token that had the permissions of the Facebook mobile app. And that’s not the way the single sign-on functionality is intended to be used. Is it just me or does this sound like an terrible idea in the first place? Guess we can't know for sure, but why would anything unrelated to authentication generate access tokens?
- rblatz 8y ago
  Technical debt, multiple systems using multiple old authentication routines getting slowly upgraded to new auth methods. And no one taking the time to fully understand the ramifications. And honestly it seems like that was the right choice for the teams responsible. They all made tons of money delivered features and now years later a bug is found.
  - sizzle 8y ago
    Would you feel the same way if this vulnerability was for, say, a major banking website?
    - fendy3002 8y ago
      Ha, major banking website won't do any improvement. Can't have enhancement vulnerability if there are no enhancement we smart.
    - marvin 8y ago
      I work for a major (by Norwegian standards) bank. This level of authentication integration trickery wouldn't be attempted by us. Mainly because we try hard to avoid serious technical debt (due to timeline/delivery pressure) in our security infrastructure. We occasionally take such shortcuts in places that are not mission-critical, but they are always considered carefully as the tradeoff that they are. I believe that we are considerably better at technology development than most of the banks in the US. That said, I've heard stories of similar bugs in the industry. The difference was that they were more shallow in the effort to reproduce; deep enough to get through QA but discovered quickly in production. But honestly, Facebook has more resources to spend on security than any online bank. Banking security should be defense-in-depth: Strong first layer security, serious monitoring of suspicious activity & openness for reports by users, a certain level of manual approval of irrevocable transfers, a certain revocability of transfers that are able to be automatically processed, transfer size limits to deny one breach to have huge consequences. And finally, a credible economic and legal system that ensures only a tiny minority of people want to rob a bank because there are much better options for making money, and banking regulations that leave the responsibility for security vulnerabilities squarely with the bank's shareholders. Anyone can be owned with enough effort, so it's not just about creating software that's as secure as you can make it. You need to have sound policies as well.
apatters 8y ago
If you're more interested in tech discussion or maybe some subcultures, and less interested in food photos/anecdotes about babies, just join http://mastodon.social/ http://mastodon.social/ already. Set your preferences to show posts of your native language only, start poking around the timelines, and follow people who post something interesting. Follow, boost, reply, it only takes a few days before you have plenty of interesting content in your feed. There's zero chance on Mastodon that you'll get caught up in a gigantic data breach like this. Probably less chance you get caught up in any kind of breach -- it's too obscure to be a target, plus the code is open source so many eyes on it, etc. And you'll enjoy these guaranteed benefits, as well: - No longer subject to the most sophisticated data vacuuming adtech in the world - If you get bored/annoyed you can just take a break from Mastodon because it doesn't own your life the way Facebook tries to
- cheeze 8y ago
  > Probably less chance you get caught up in any kind of breach -- it's too obscure to be a target, plus the code is open source so many eyes on it, etc. Security through obscurity... Open source != secure. I can guarantee that a hell of a lot more folks with a lot of security expertise have combed through the fb codebase than Mastodon.
  - brodsky 8y ago
    i agree, but "guarantee" is a strong word. on the flip side, a hell of a lot more folks _without_ much security expertise have _contributed_ to the fb codebase.
  - apatters 8y ago
    >Security through obscurity... ...is not a solution by itself but is a perfectly valid part of a defense in depth strategy, for example running SSH on a port other than the default is a common and good practice. > I can guarantee that a hell of a lot more folks with a lot of security expertise have combed through the fb codebase than Mastodon. This is the same argument Microsoft always made in defense of Windows security back in the XP era. "We hire the best experts in the world so Windows must be fantastically secure." And Windows security turned out to be a train wreck. Now in Microsoft's defense it has improved considerably over the years, but Windows desktops still get owned far more often than Linux desktops do, for a reason that would probably apply to Mastodon today as well: not that many people use it, so it is not nearly as common a target for exploits. I don't think I deserved downvotes for making these points btw, that button is way overused on HN.
sorokod 8y ago
How likely is it that the three bug combination could be discoverd without access to source code ?
- obituary_latte 8y ago
  Very likely. It happens all of the time. If you read through some cve’s or other bug reports or post mortem’s, you’ll be surprised just how complex attacks can be.
  - sorokod 8y ago
    I suppose that the likelihood estimate would need to take into account the number of people who have (or had) access to the sources. Obviously the alternatives are not mutually exclusive.

kidsnow 8y ago

Just as Microsoft developed Patch Tuesday, Facebook should have Forced Logoff Friday

labster 8y ago
Every day on Facebook should be forced logoff day. Or, at least, incognito mode day.
- slimsag 8y ago
  'We only use incognito mode for security. Since it's annoying to login constantly, everyone has dedicated machines that are powered 24/7 so you never have to shut down those incognito tabs.' =)
- tialaramex 8y ago
  There's a Firefox Add-on named "Facebook Container". Once you install that Facebook lives in a little box, Facebook cookies, Facebook whatever else, all trapped in the little box. No effort needed, if you click your Facebook bookmark, or follow a link or whatever, the browser goes "Oh, this is Facebook" and traps it inside the box with the rest of Facebook without any extra steps from the user. There's a cute blue "Facebook" icon added to the URL bar so you can see it's working. (I mean, or, stop using Facebook, but for many that isn't a reasonable option)
mkagenius 8y ago
"That won't scale" - Sandberg

romed 8y ago

That doesn't just seem like a few unlucky coincidences. That seems like a fundamentally unsound design. Why should it even be theoretically possible for a request under the authority of one user to create a token with the authority of another user?

flgr 8y ago
Notably, they previously had issues were "View as" allowed you to view notifications and messages of the user you were viewing as. If they'd done a proper post mortem and corrected the fundamental issue, and made sure it wouldn't have re-occured, this should not have happened.
- bigiain 8y ago
  Instead they moved fast and broke things.
  - nautilus12 8y ago
    Its more important for you to move fast and break things and make us money than to move slow and do things the right way. The life of an engineer...Do it now! why did you do it that way!? Now we are screwed??
    - pyman 8y ago
      Facebook’s php developers like to move fast and break things. Bad design choices, monkey patching, breaking things on production, it’s all part of Facebook’s “engineering” principles.
calc_exe 8y ago
Yeah, sounds like https://en.wikipedia.org/wiki/Confused_deputy_problem https://en.wikipedia.org/wiki/Confused_deputy_problem to me.
everdev 8y ago
Not the root cause, but I'm guessing a microservice architecture made it more possible. It sounds like both the token generating service and the video upload service have bugs.

partycoder 8y ago

The "View as" feature has been the source of many security vulnerabilities. There was a time where you could read other peoples' chats using this feature.

groestl 8y ago
When designing such a system, the immediate failure mode is obvious: at some point, someone will read data not meant for them. As every feature on FB needs to take "View as" into account when handling their own permissions, a lot of developers on FB's payroll get a chance to f'up. We are all humans, so the probability of this happening is very high. The impact (for the users) is also high, given that it's automated and concerns every user on FB equally. When dealing with a very probable, high impact risk in a software project, considerable additional effort is warranted to mitigate that risk: in this case maybe taint checking and additional implementations of the same feature in different programming paradigms, to ensure the system is fail-stop. But in contrast to airlines and railways, the interests of FB and their users are not aligned. For Facebook, this risk is not (or was not deemed to be of) high impact, so we did't get any of this.
WikipediasBad 8y ago
Any link to this type of vulnerability? Sounds like a juicy read.
Raphael 8y ago
It seems to warrant checking both the permissions of the true user and the view-as user. If either does not have permission, then the action should fail. Of course, lacking the middleware for this forces you to choose one or the other and hope you remember to check the remaining user in numerous pathways.

gboudrias 8y ago

> The second bug was that this video uploader incorrectly used the single signon functionally, and it generated an access token that had the permissions of the Facebook mobile app. And that’s not the way the single sign-on functionality is intended to be used. Is it just me or does this sound like an terrible idea in the first place? Guess we can't know for sure, but why would anything unrelated to authentication generate access tokens?

rblatz 8y ago
Technical debt, multiple systems using multiple old authentication routines getting slowly upgraded to new auth methods. And no one taking the time to fully understand the ramifications. And honestly it seems like that was the right choice for the teams responsible. They all made tons of money delivered features and now years later a bug is found.
- sizzle 8y ago
  Would you feel the same way if this vulnerability was for, say, a major banking website?
  - fendy3002 8y ago
    Ha, major banking website won't do any improvement. Can't have enhancement vulnerability if there are no enhancement we smart.
  - marvin 8y ago
    I work for a major (by Norwegian standards) bank. This level of authentication integration trickery wouldn't be attempted by us. Mainly because we try hard to avoid serious technical debt (due to timeline/delivery pressure) in our security infrastructure. We occasionally take such shortcuts in places that are not mission-critical, but they are always considered carefully as the tradeoff that they are. I believe that we are considerably better at technology development than most of the banks in the US. That said, I've heard stories of similar bugs in the industry. The difference was that they were more shallow in the effort to reproduce; deep enough to get through QA but discovered quickly in production. But honestly, Facebook has more resources to spend on security than any online bank. Banking security should be defense-in-depth: Strong first layer security, serious monitoring of suspicious activity & openness for reports by users, a certain level of manual approval of irrevocable transfers, a certain revocability of transfers that are able to be automatically processed, transfer size limits to deny one breach to have huge consequences. And finally, a credible economic and legal system that ensures only a tiny minority of people want to rob a bank because there are much better options for making money, and banking regulations that leave the responsibility for security vulnerabilities squarely with the bank's shareholders. Anyone can be owned with enough effort, so it's not just about creating software that's as secure as you can make it. You need to have sound policies as well.

apatters 8y ago

If you're more interested in tech discussion or maybe some subcultures, and less interested in food photos/anecdotes about babies, just join http://mastodon.social/ http://mastodon.social/ already. Set your preferences to show posts of your native language only, start poking around the timelines, and follow people who post something interesting. Follow, boost, reply, it only takes a few days before you have plenty of interesting content in your feed. There's zero chance on Mastodon that you'll get caught up in a gigantic data breach like this. Probably less chance you get caught up in any kind of breach -- it's too obscure to be a target, plus the code is open source so many eyes on it, etc. And you'll enjoy these guaranteed benefits, as well: - No longer subject to the most sophisticated data vacuuming adtech in the world - If you get bored/annoyed you can just take a break from Mastodon because it doesn't own your life the way Facebook tries to

cheeze 8y ago
> Probably less chance you get caught up in any kind of breach -- it's too obscure to be a target, plus the code is open source so many eyes on it, etc. Security through obscurity... Open source != secure. I can guarantee that a hell of a lot more folks with a lot of security expertise have combed through the fb codebase than Mastodon.
- brodsky 8y ago
  i agree, but "guarantee" is a strong word. on the flip side, a hell of a lot more folks _without_ much security expertise have _contributed_ to the fb codebase.
- apatters 8y ago
  >Security through obscurity... ...is not a solution by itself but is a perfectly valid part of a defense in depth strategy, for example running SSH on a port other than the default is a common and good practice. > I can guarantee that a hell of a lot more folks with a lot of security expertise have combed through the fb codebase than Mastodon. This is the same argument Microsoft always made in defense of Windows security back in the XP era. "We hire the best experts in the world so Windows must be fantastically secure." And Windows security turned out to be a train wreck. Now in Microsoft's defense it has improved considerably over the years, but Windows desktops still get owned far more often than Linux desktops do, for a reason that would probably apply to Mastodon today as well: not that many people use it, so it is not nearly as common a target for exploits. I don't think I deserved downvotes for making these points btw, that button is way overused on HN.

sorokod 8y ago

How likely is it that the three bug combination could be discoverd without access to source code ?

obituary_latte 8y ago
Very likely. It happens all of the time. If you read through some cve’s or other bug reports or post mortem’s, you’ll be surprised just how complex attacks can be.
- sorokod 8y ago
  I suppose that the likelihood estimate would need to take into account the number of people who have (or had) access to the sources. Obviously the alternatives are not mutually exclusive.