3 ms·

What it's like to be a Facebook engineer in this context: It's your oncall week, and all of a sudden you get an urgent task that you must resolve in the next fe

by bagels 2y ago

What it's like to be a Facebook engineer in this context: It's your oncall week, and all of a sudden you get an urgent task that you must resolve in the next few days: Figure out how to replace a crypto algorithm in some component your code depends on or else we tell your manager that you're not playing ball.

sharpy 2y ago
Never worked at facebook, but typically at a large companies, things like this are sent with instructions on how to go about doing that, and most often just making sure your dependencies are updated so next run of automated build/deploy will pick up the change.
pdpi 2y ago
When I was there, the experience here would've been more like "you see some comms on Workplace saying this thing went live". For a small number of people, it would look like "It's your oncall week, and you unexpectedly receive a large diff stack from a complete stranger". Most engineers at Meta won't touch anything related to C++ during their tenure there, and you can safely assume all this stuff would be abstracted away from them. If you are on a C++ project, odds are that your experience here would be "I'm on-call and got pinged by a random person saying they were submitting a diff stack to implement this feature". Except if your project was sufficiently large, you'd probably have known about this ages ago so the diff wouldn't actually be unexpected, nor the submitter a complete stranger.
- yodsanklai 2y ago
  > you'd probably have known about this ages but nobody paid attention and it falls on oncall's plate
- bagels 2y ago
  Sure, sometimes I got a bunch of automod diffs. Sometimes it was: fix this now. It's impossible to know everything that every infra team is cooking up.
deleted 2y ago
[deleted]
jamra 2y ago
You’d be calling libraries that are vetted by the security team. Engineering at Meta is a lot like coloring with crayons. It’s very limiting but the infra does a lot for you.
- loeg 2y ago
  This is true, but like, sometimes you're using them inappropriately. Or you've got a limited exception for performance reasons, and your replacement solution is, uh, unsound.
  - Mistletoe 2y ago
    Do you ever eat the crayons?
yodsanklai 2y ago
triage and redirect to L7+ colleague
loeg 2y ago
In my experience with the cryptographic security space at Facebook, what happens is the security team reaches out to you and starts a conversation. You collaboratively establish the severity, acceptable mitigation options, and appropriate timeline. There might be a little bikeshedding over various degrees of ideal option (it's more fun for the crypto guys that way). They might volunteer to do the work. You review the diffs, they're good, it lands. They're smart people and the team knows how to collaborate with random infra/product teams.
paxys 2y ago
How it really works - security team files a task to replace a crypto algorithm in your team's code and gives you a SLA deadline of 6 months. 6 months later, no one has touched it, so they escalate. Your manager says "we are short staffed" and requests an extension. They give you another 3 months and write a detailed guide on what lines to change. 3 months later everyone they originally spoke to has left the team, and the new people don't have any context. They try and figure it out but the security person has also left the team. No one remembers what the task was originally created for. A year later the service that was using the old algorithm is itself deprecated.
- bwock 2y ago
  You forgot the multiple rounds of taskcreeper closing the task due to inactivity

by bagels 2y ago

sharpy 2y ago

Never worked at facebook, but typically at a large companies, things like this are sent with instructions on how to go about doing that, and most often just making sure your dependencies are updated so next run of automated build/deploy will pick up the change.

pdpi 2y ago

When I was there, the experience here would've been more like "you see some comms on Workplace saying this thing went live". For a small number of people, it would look like "It's your oncall week, and you unexpectedly receive a large diff stack from a complete stranger". Most engineers at Meta won't touch anything related to C++ during their tenure there, and you can safely assume all this stuff would be abstracted away from them. If you are on a C++ project, odds are that your experience here would be "I'm on-call and got pinged by a random person saying they were submitting a diff stack to implement this feature". Except if your project was sufficiently large, you'd probably have known about this ages ago so the diff wouldn't actually be unexpected, nor the submitter a complete stranger.

yodsanklai 2y ago
> you'd probably have known about this ages but nobody paid attention and it falls on oncall's plate
bagels 2y ago
Sure, sometimes I got a bunch of automod diffs. Sometimes it was: fix this now. It's impossible to know everything that every infra team is cooking up.

deleted 2y ago

[deleted]

jamra 2y ago

You’d be calling libraries that are vetted by the security team. Engineering at Meta is a lot like coloring with crayons. It’s very limiting but the infra does a lot for you.

loeg 2y ago
This is true, but like, sometimes you're using them inappropriately. Or you've got a limited exception for performance reasons, and your replacement solution is, uh, unsound.
- Mistletoe 2y ago
  Do you ever eat the crayons?

yodsanklai 2y ago

triage and redirect to L7+ colleague

loeg 2y ago

In my experience with the cryptographic security space at Facebook, what happens is the security team reaches out to you and starts a conversation. You collaboratively establish the severity, acceptable mitigation options, and appropriate timeline. There might be a little bikeshedding over various degrees of ideal option (it's more fun for the crypto guys that way). They might volunteer to do the work. You review the diffs, they're good, it lands. They're smart people and the team knows how to collaborate with random infra/product teams.

paxys 2y ago

How it really works - security team files a task to replace a crypto algorithm in your team's code and gives you a SLA deadline of 6 months. 6 months later, no one has touched it, so they escalate. Your manager says "we are short staffed" and requests an extension. They give you another 3 months and write a detailed guide on what lines to change. 3 months later everyone they originally spoke to has left the team, and the new people don't have any context. They try and figure it out but the security person has also left the team. No one remembers what the task was originally created for. A year later the service that was using the old algorithm is itself deprecated.

bwock 2y ago
You forgot the multiple rounds of taskcreeper closing the task due to inactivity