8 ms·

This is pretty common actually At Facebook too there was no staging environment. Engineers had their dev VM and then after PR review things just went into prod

This is pretty common actually At Facebook too there was no staging environment. Engineers had their dev VM and then after PR review things just went into prod That said features and bug fixes were often times gated by feature flags and rolled out slowly to understand the product/perf impact better This is how we do it at my current team too…for all the same reasons that OP states

abhishekjha 5y ago
That would be controlling a lot of feature flags given how many can be switched on at once. How do you control them?
- sillysaurusx 5y ago
  flag = true More seriously, at my old company they just never got removed. So it wasn’t really about control. You just forgot about the ones that didn’t matter after awhile. If that sounds horrible, that’s probably the correct reaction. But it’s also common. Namespacing helps too. It’s easier to forget a bunch of flags when they all start with foofeature-.
  - withinboredom 5y ago
    I’ve seen those old flags come in handy once. Someone accidentally deleted a production database (typo) and we needed to stop all writes to restore from a backup. For most of it, it was just turning off the original feature flag, even though the feature was several years old.
  - skybrian 5y ago
    It can become a code maintenance issue, though, when you revisit the code. You need to maintain both paths when you never know if they are being used. Also, where flags interact, you can get a combinatorial explosion of cases to consider.
  - mdoms 5y ago
    At a previous workplace we managed flags with Launch Darkly. We asked developers not to create flags in LD directly but used Jira web hooks to generate flags from any Jira issues of type Feature Flag. This issue type had a workflow that ensured you couldn't close off an epic without having rolled out and then removed every feature flag. Flags should not significantly outlast their 100% rollout.
  - harunurhan 5y ago
    > the ones that didn’t matter after awhile. Ideally you have metrics for all flags and their values, so you can easily tell if one becomes redundant and safe to remove entirely after a while. I've also seen making it a requirement to remove a flag after N days, the feature is completely rolled out.
- clintonb 5y ago
  I work at a different company. Typically feature flags are short-lived (on the order of days or weeks), and only control one feature. When I deploy, I only care about my one feature flag because that is the only thing gating the new functionality being deployed. There may be other feature flags, owned by other teams, but it's rare to have flags that cross team/service boundaries in a fashion that they need to be coordinated for rollout.
- rb2k_ 5y ago
  It's 7 years old by now, but there's some literature: https://research.facebook.com/publications/holistic-configuration-management-at-facebook/ https://research.facebook.com/publications/holistic-configur... You can see that there's a common backend ("configerator") that a lot of other systems ("sitevars", "gatekeeper", ...) build on top of. Just imagine that these systems have been further developed over the last decade :) In general, there's 'configuration change at runtime' systems that the deployed code usually has access to and that can switch things on and off in very short time (or slowly roll it out). Most of these are coupled with a variety of health checks.
- alasdair_ 5y ago
  You have automated tools that yell at you to clean up feature flags and you force people to include sensible expiration dates at part of your PR process. Flags past the date result in increased yelling. If your team has too much crap in the codebase eventually someone politely tells you to clean it up. You also have tooling that measures how many times a flag was encountered vs. how many times it actually triggered etc. Once it looks like it's at 100% of traffic, again you have automations that tell people to clean up their crap.
kgeist 5y ago
>That said features and bug fixes were often times gated by feature flags Sorry for maybe a silly question, but how do feature flags work with migrations? If your migrations run automatically on deploy, then feature flags can't prevent badly tested migrations from corrupting the DB, locking tables and other sorts of regressions. If you run your migrations manually each time, then there's a chance that someone enables a feature toggle without running the required migrations, which can result in all sorts of downtime. Another concern I have is that if a feature toggle isn't enabled in production for a long time (for us, several days is already a long time due to a tight release schedule) new changes to the codebase by another team can conflict with the disabled feature and, since it's disabled, you probably won't know there's a problem until it's too late?
- drewcoo 5y ago
  > how do feature flags work with migrations? The idea is to have migrations that are backward compatible so that the current version of your code can use the db and so can the new version. Part of the reason people started breaking up monoliths is that continuous deployment with a db-backed monolith can be brittle. And making it work well requires a whole bunch of brain power that could go into things like making the product better for customers. > another concern Avoiding "feature flag hell" is a valid concern. It has to be managed. The big problem with conflict is underlying tightly coupled code, though. That should be fixed. Note this is also solved by breaking up monoliths. > tight release schedule If a release in this sense is something product-led, then feature flags almost create an API boundary (a good thing!) between product and dev. Product can determine when their release (meaning set of feature flags to be flipped) is ready and ideally toggle themselves instead of roping devs into release management roles.
  - cmeacham98 5y ago
    I think their question was more "if I wrote a migration that accidentally drops the users table, how does your system prevent that from running on production"? That's a pretty extreme case, but the tldr is how are you testing migrations if you don't have a staging environment.
Gigachad 5y ago
This is how my current place does it. The only issue we are having is library / dependency updates have a tendency to work perfectly fine locally and then fail in production due to either some minor difference in environment or scale. It's a problem to the point that we have 5 year old ruby gems which have no listed breaking changes because no one is brave enough to bump them. I had a go at it and caused a major production incident because the datadog gem decided to kill Kubernetes with too many processes.
- ehnto 5y ago
  Do you have a replicate of the production environment codified somehow, like into a VM? It's rarely perfect, but I usually try and develop locally on the same stack I deploy to which can help with the environment differences. It's also why I think it's smart to rebuild the environment on deploy if it makes sense for your pipeline, so that you wipe any minor differences that have been accruing over time. Working on a long running product, you quickly find yourself with disparities building up, and they're not codified, so they're essentially unknown to the team until they cause an issue.
  - Gigachad 5y ago
    Yes, there is a docker compose file which is usually sufficient. The issue in this case was the problem only shows up if there is sufficient load on the background workers.
aprdm 5y ago
The book Software Engineering at Google or something akin to that mentions the same kind of thing.
- pedrosorio 5y ago
  This one: "Site Reliability Engineering: How Google Runs Production Systems"?
  - aprdm 5y ago
    https://www.oreilly.com/library/view/software-engineering-at/9781492082781/ https://www.oreilly.com/library/view/software-engineering-at...
Rapzid 5y ago
Facebook can completely break the user experience for 4.3 million different users each day and each user would only experience one breakage per year. This is pretty common, but not because most employing it have 1.6bn users and 10k engineers; essentially enough scale to throw bodies at problems.
- lupire 5y ago
  Why would a mistake only affect 4M users and not 400M?
  - Rapzid 5y ago
    No guarantees, but a feature flag or canary deploy can significantly increase likelihood of impacting a targeted subset of users.
  - SgtBastard 5y ago
    Look at it the other way around - you could have a different outage every single day and as long as that outage only impacted 4.3m users and they were different users each day, it would look like a once-a-year event to the average user. They’re saying there’s a lot of leeway to break things (in a small way) at scale.
otterley 5y ago
Was this true for the systems that related to revenue and ad sales as well? While I can believe that a lot of code at Facebook goes into production without first going through a staging environment, I would be extremely surprised if the same were true for their ads systems or anything that dealt with payment flows.
- zdragnar 5y ago
  I don't know about Facebook, but at other companies without similar, each git branch gets deployed to its own subdomain, so manual testing etc. can happen prior to a merge. Dangerous changes are feature flagged or gated as much as possible to allow prod feedback after merge before enabling the changes for everyone.
- alasdair_ 5y ago
  I worked on ad systems at Facebook, and yes it's (approximately) true for those as well. The thing to realize that "in production" almost never means "rolled out to 100% of users from 0%". Instead you'd do very slow rollouts to, say, 1% of people in Hungary (or whatever) and use a ton of automatic measurements over time as well as lots and lots of tests to validate that things were working as expected before rolling out a little more, then a little more after that. By the time the code is actually being hit by the majority of users, it's often been run billions of times already.
alex_young 5y ago
What about third party integrations? Don’t you need some non-production environment to test them in until both parties are satisfied with the integration and it’s impact on users?
ehnto 5y ago
I think it's a difference in relationship with your users. For software I'm currently working on, we require verification that the changes we made were correct before they make it out to production, so there's a requirement for a staging environment they can access. That's a software > business relationship, where there's a known good outcome. This was also true in the ecommerce agency environments I worked in, the business owners want the opportunity to verify and correct things before they go out to production. If it were a product > user relationship, where you're the product owner and you are trying to improve your product without an explicit request from your users, I can see how no staging environment makes sense. You have no responsibility of proof of correctness to your users, what you put out is what they get, and breakages can be handled as fixes after the fact.
ikiris 5y ago
Facebook also managed to lock themselves out of their own network for how many hours again? :P
hinkley 5y ago
Problem I always seem to run into is that these optional features always seem to be added at a rate that's a bit higher than the rate at which the flags are retired. It doesn't take much of a multiplier for the number to become untenable pretty quickly.
reissbaker 5y ago
It's common, but it's more like "this is common at young companies where the cost of maintaining staging can't pay for itself in improved productivity, because there aren't enough engineers for a 5% productivity improvement to be worth hiring multiple engineers for." I'm sure that at FB at one point there wasn't a staging env. Today at FB there are multiple layers of staging, checked automatically as part of the deployment process. I'm ex-FB as well, and we definitely used staging environments every single day as part of the ordinary deploy pipeline. You probably worked there when it was younger, and smaller, and the tooling was less advanced. Large tech companies have advanced dev tooling; eventually, the cost of paying people to make the tools is paid for in productivity gained per engineer, with large enough eng team sizes.
fareesh 5y ago
I am leading a small team working on a social audio product. We follow the same process. The vast majority of our content is live audio conversations and so we need live/production data. If our stakeholders have to test the product it means they have to join those conversations, or setup live conversations in a parallel universe. Feature flags in production are the simplest way forward, but they carry a fair amount of risk. This is offset by automated tests wherever possible.
LtWorf_ 5y ago
Also facebook chats were known to take from days to infinity to deliver messages, so it's not like anyone really expected anything from facebook.

abhishekjha 5y ago

That would be controlling a lot of feature flags given how many can be switched on at once. How do you control them?

sillysaurusx 5y ago
flag = true More seriously, at my old company they just never got removed. So it wasn’t really about control. You just forgot about the ones that didn’t matter after awhile. If that sounds horrible, that’s probably the correct reaction. But it’s also common. Namespacing helps too. It’s easier to forget a bunch of flags when they all start with foofeature-.
- withinboredom 5y ago
  I’ve seen those old flags come in handy once. Someone accidentally deleted a production database (typo) and we needed to stop all writes to restore from a backup. For most of it, it was just turning off the original feature flag, even though the feature was several years old.
- skybrian 5y ago
  It can become a code maintenance issue, though, when you revisit the code. You need to maintain both paths when you never know if they are being used. Also, where flags interact, you can get a combinatorial explosion of cases to consider.
- mdoms 5y ago
  At a previous workplace we managed flags with Launch Darkly. We asked developers not to create flags in LD directly but used Jira web hooks to generate flags from any Jira issues of type Feature Flag. This issue type had a workflow that ensured you couldn't close off an epic without having rolled out and then removed every feature flag. Flags should not significantly outlast their 100% rollout.
- harunurhan 5y ago
  > the ones that didn’t matter after awhile. Ideally you have metrics for all flags and their values, so you can easily tell if one becomes redundant and safe to remove entirely after a while. I've also seen making it a requirement to remove a flag after N days, the feature is completely rolled out.
clintonb 5y ago
I work at a different company. Typically feature flags are short-lived (on the order of days or weeks), and only control one feature. When I deploy, I only care about my one feature flag because that is the only thing gating the new functionality being deployed. There may be other feature flags, owned by other teams, but it's rare to have flags that cross team/service boundaries in a fashion that they need to be coordinated for rollout.
rb2k_ 5y ago
It's 7 years old by now, but there's some literature: https://research.facebook.com/publications/holistic-configuration-management-at-facebook/ https://research.facebook.com/publications/holistic-configur... You can see that there's a common backend ("configerator") that a lot of other systems ("sitevars", "gatekeeper", ...) build on top of. Just imagine that these systems have been further developed over the last decade :) In general, there's 'configuration change at runtime' systems that the deployed code usually has access to and that can switch things on and off in very short time (or slowly roll it out). Most of these are coupled with a variety of health checks.
alasdair_ 5y ago
You have automated tools that yell at you to clean up feature flags and you force people to include sensible expiration dates at part of your PR process. Flags past the date result in increased yelling. If your team has too much crap in the codebase eventually someone politely tells you to clean it up. You also have tooling that measures how many times a flag was encountered vs. how many times it actually triggered etc. Once it looks like it's at 100% of traffic, again you have automations that tell people to clean up their crap.

kgeist 5y ago

>That said features and bug fixes were often times gated by feature flags Sorry for maybe a silly question, but how do feature flags work with migrations? If your migrations run automatically on deploy, then feature flags can't prevent badly tested migrations from corrupting the DB, locking tables and other sorts of regressions. If you run your migrations manually each time, then there's a chance that someone enables a feature toggle without running the required migrations, which can result in all sorts of downtime. Another concern I have is that if a feature toggle isn't enabled in production for a long time (for us, several days is already a long time due to a tight release schedule) new changes to the codebase by another team can conflict with the disabled feature and, since it's disabled, you probably won't know there's a problem until it's too late?

drewcoo 5y ago
> how do feature flags work with migrations? The idea is to have migrations that are backward compatible so that the current version of your code can use the db and so can the new version. Part of the reason people started breaking up monoliths is that continuous deployment with a db-backed monolith can be brittle. And making it work well requires a whole bunch of brain power that could go into things like making the product better for customers. > another concern Avoiding "feature flag hell" is a valid concern. It has to be managed. The big problem with conflict is underlying tightly coupled code, though. That should be fixed. Note this is also solved by breaking up monoliths. > tight release schedule If a release in this sense is something product-led, then feature flags almost create an API boundary (a good thing!) between product and dev. Product can determine when their release (meaning set of feature flags to be flipped) is ready and ideally toggle themselves instead of roping devs into release management roles.
- cmeacham98 5y ago
  I think their question was more "if I wrote a migration that accidentally drops the users table, how does your system prevent that from running on production"? That's a pretty extreme case, but the tldr is how are you testing migrations if you don't have a staging environment.

Gigachad 5y ago

This is how my current place does it. The only issue we are having is library / dependency updates have a tendency to work perfectly fine locally and then fail in production due to either some minor difference in environment or scale. It's a problem to the point that we have 5 year old ruby gems which have no listed breaking changes because no one is brave enough to bump them. I had a go at it and caused a major production incident because the datadog gem decided to kill Kubernetes with too many processes.

ehnto 5y ago
Do you have a replicate of the production environment codified somehow, like into a VM? It's rarely perfect, but I usually try and develop locally on the same stack I deploy to which can help with the environment differences. It's also why I think it's smart to rebuild the environment on deploy if it makes sense for your pipeline, so that you wipe any minor differences that have been accruing over time. Working on a long running product, you quickly find yourself with disparities building up, and they're not codified, so they're essentially unknown to the team until they cause an issue.
- Gigachad 5y ago
  Yes, there is a docker compose file which is usually sufficient. The issue in this case was the problem only shows up if there is sufficient load on the background workers.

aprdm 5y ago

The book Software Engineering at Google or something akin to that mentions the same kind of thing.

pedrosorio 5y ago
This one: "Site Reliability Engineering: How Google Runs Production Systems"?
- aprdm 5y ago
  https://www.oreilly.com/library/view/software-engineering-at/9781492082781/ https://www.oreilly.com/library/view/software-engineering-at...

Rapzid 5y ago

Facebook can completely break the user experience for 4.3 million different users each day and each user would only experience one breakage per year. This is pretty common, but not because most employing it have 1.6bn users and 10k engineers; essentially enough scale to throw bodies at problems.

lupire 5y ago
Why would a mistake only affect 4M users and not 400M?
- Rapzid 5y ago
  No guarantees, but a feature flag or canary deploy can significantly increase likelihood of impacting a targeted subset of users.
- SgtBastard 5y ago
  Look at it the other way around - you could have a different outage every single day and as long as that outage only impacted 4.3m users and they were different users each day, it would look like a once-a-year event to the average user. They’re saying there’s a lot of leeway to break things (in a small way) at scale.

otterley 5y ago

Was this true for the systems that related to revenue and ad sales as well? While I can believe that a lot of code at Facebook goes into production without first going through a staging environment, I would be extremely surprised if the same were true for their ads systems or anything that dealt with payment flows.

zdragnar 5y ago
I don't know about Facebook, but at other companies without similar, each git branch gets deployed to its own subdomain, so manual testing etc. can happen prior to a merge. Dangerous changes are feature flagged or gated as much as possible to allow prod feedback after merge before enabling the changes for everyone.
alasdair_ 5y ago
I worked on ad systems at Facebook, and yes it's (approximately) true for those as well. The thing to realize that "in production" almost never means "rolled out to 100% of users from 0%". Instead you'd do very slow rollouts to, say, 1% of people in Hungary (or whatever) and use a ton of automatic measurements over time as well as lots and lots of tests to validate that things were working as expected before rolling out a little more, then a little more after that. By the time the code is actually being hit by the majority of users, it's often been run billions of times already.

alex_young 5y ago

What about third party integrations? Don’t you need some non-production environment to test them in until both parties are satisfied with the integration and it’s impact on users?

ehnto 5y ago

I think it's a difference in relationship with your users. For software I'm currently working on, we require verification that the changes we made were correct before they make it out to production, so there's a requirement for a staging environment they can access. That's a software > business relationship, where there's a known good outcome. This was also true in the ecommerce agency environments I worked in, the business owners want the opportunity to verify and correct things before they go out to production. If it were a product > user relationship, where you're the product owner and you are trying to improve your product without an explicit request from your users, I can see how no staging environment makes sense. You have no responsibility of proof of correctness to your users, what you put out is what they get, and breakages can be handled as fixes after the fact.

ikiris 5y ago

Facebook also managed to lock themselves out of their own network for how many hours again? :P

hinkley 5y ago

Problem I always seem to run into is that these optional features always seem to be added at a rate that's a bit higher than the rate at which the flags are retired. It doesn't take much of a multiplier for the number to become untenable pretty quickly.

reissbaker 5y ago

It's common, but it's more like "this is common at young companies where the cost of maintaining staging can't pay for itself in improved productivity, because there aren't enough engineers for a 5% productivity improvement to be worth hiring multiple engineers for." I'm sure that at FB at one point there wasn't a staging env. Today at FB there are multiple layers of staging, checked automatically as part of the deployment process. I'm ex-FB as well, and we definitely used staging environments every single day as part of the ordinary deploy pipeline. You probably worked there when it was younger, and smaller, and the tooling was less advanced. Large tech companies have advanced dev tooling; eventually, the cost of paying people to make the tools is paid for in productivity gained per engineer, with large enough eng team sizes.

fareesh 5y ago

I am leading a small team working on a social audio product. We follow the same process. The vast majority of our content is live audio conversations and so we need live/production data. If our stakeholders have to test the product it means they have to join those conversations, or setup live conversations in a parallel universe. Feature flags in production are the simplest way forward, but they carry a fair amount of risk. This is offset by automated tests wherever possible.

LtWorf_ 5y ago

Also facebook chats were known to take from days to infinity to deliver messages, so it's not like anyone really expected anything from facebook.