6 ms·

Coherence is obviously trying to sell you their product here but I'm not sure preview environments and staging environments are opposed. They way I've set thin

Coherence is obviously trying to sell you their product here but I'm not sure preview environments and staging environments are opposed. They way I've set things up at my current company is this: When a developer creates a pull request an ephemeral (preview) environment is generated. When that code is merged it automatically gets deployed first to our staging environment and then immediately to production. Staging gets a sanitized copy of production's data every night so both the code and the data are close mirrors of production and that's what developers (and those preview environments) integrate with when they're calling services other than their own. If you were to get rid of staging would you stand up your entire backend every time you made a preview environment? Does that include replicating databases or would you just use seed data that may or may not resemble what's in production? It seems like the solution they're pitching works great if you have a small team and only a few services/sites.

andrewmutz 4y ago
> If you were to get rid of staging would you stand up your entire backend every time you made a preview environment? Does that include replicating databases or would you just use seed data that may or may not resemble what's in production? What I've seen done is that the staging environment is skipped, but instead concentric releases are performed on the customers. So released code goes first to a small group of users, and then widens from there if everything looks good.
- oasisbob 4y ago
  This technique doesn't seem especially relevant for many kinds of backend work.
- scubbo 4y ago
  > What I've seen done is that the staging environment is skipped, but instead concentric releases are performed on the customers. So released code goes first to a small group of users, and then widens from there if everything looks good. If you really want to invest in extensive testing, you can do both! Hell, _and_ you can have a load testing environment as well.
  - andrewmutz 4y ago
    Yeah, we had tooling to load a snapshot of customer data (with PII removed) into an ephemeral environment for when developers needed data that was more authentic. This worked for performance related work or just for features that needed a complex data model to be useful (ML apps, etc)
- klooney 4y ago
  Schema changes are hard to roll out that way though.
  - andrewmutz 4y ago
    We were a B2B product whose multitenancy model was schema-per-customer, so it was no problem for us. Agree that it would be harder to implement with customers sharing a schema, but you can still have different release pods
hangonhn 4y ago
Yeah. And it's also not that impressive. We've had what they call "preview environments" for a very long time and at my previous companies too. With GitHub and AWS/Cloud it's trivially easy to do. And like you, we have both "preview environments" and staging and for the exact reason you've highlight. I'm rather skeptical of the company behind this product now since they don't seem to understand the difference and really overselling something that's quite common.
- phendrenad2 4y ago
  Yes, this is one of those things that was easier before kubernetes. Before: Your developers have a shared Linux box that serves the preview environment. Your ruby app has a tiny loader in front of it that loads the app code from a specific directory based on subdomain. After: You need to route subdomains to different kubernetes containers, and handle deploying new containers when the developer opens a PR (because it's much more work than just copying files onto a server), and you need to handle destroying old containers (because it's much more work than just deleting files on a server). Not to mention, usually kubernetes is managed by "the ops" who don't know anything about the app. Much more difficult to interface with them when it's not "oh one server, and the developers figure it out, and if they royally fuck it up we'll rescue them".
  - shamiln 4y ago
    It’s really not that difficult to automate it over helm or something. All you need is a registry, and most cloud services provide a registry, even GitHub has one. Routing domains to a container shouldn’t be significantly different to doing it in production.
    - tracker1 4y ago
      For that matter, you generally will have a single app firewall or ingress server that reverse-proxies to your application anyway.
      - 4y ago
colpabar 4y ago
How long do your PRs stay open? Could you explain a bit more about what happens between opening the PR and merging it, like automated/manual testing? At my company we do something similar, except we trigger production deployments off tags, not merged PRs. Merging goes to staging only.
- CSMastermind 4y ago
  It really depends on the PR. Most do get merged fairly quickly (same day) but some where there's more discussion can stay open for several days or even a week. All of our codebases have some level of automated testing that runs when the PR is created. That could be unit tests of a single function or Playwright tests which exercise an UI flow. What those tests are dependent on the type of software and the team building it. The preview environment links that are generated when the PR is open are given to QA, Design, and Product to take a look at. Not every one of those roles review every PR, it really depends on the task. But if their feedback is required, they'll give it at this stage. Probably worth mentioning at this point that we have separate repos for each web application and service. It's service oriented but not microservices, services at our company encapsulate a relatively large domain and it's similar with web applications, right now each web application has its own subdomain. So PR gets merged, the pipeline will deploy that code to staging. At this point the service owners have the option to define a set of integration tests, not every repo has them but it is an option to run them. If those tests fail we do a rollback. If those tests pass (or if they didn't have any defined) then a second deployment to production is triggered. All of our deploys are blue-green, so no downtime except for the occasional blocking database migration in which we have to schedule downtime. User facing features on the web applications are all required to be released first behind a feature flag. For those there will be several releases behind a feature flag, then we'll flip the flag first in staging. We'll have Product and QA do a complete run through and make sure they're okay with everything, then the "real" release happen when we flip the feature flag in production (no deployment needed). Every night we clone the production database, sanitize it (swapping out things like names, phone numbers, etc.) then point staging to that clone. Then we run an end to end regression test that goes through all of the happy path flows for all of our apps.
ArjenM 4y ago
>a sanitized copy Quite the importance over a constant wave of changing code on development in certain projects. Staging seems like a more code locked stable area for all previews that can be polished via quality practices and keen eyes. Now it seems a little fleeting to me as well to have the shock of previews enter into this instead. Let the paint dry adequately before taking it outside.
candiddevmike 4y ago
How do you sanitize data between prd and stg?
- champagnepapi 4y ago
  You can use something like this https://www.replibyte.com/docs/introduction/ https://www.replibyte.com/docs/introduction/
- jcraft 4y ago
  My company can anonymize production data while preserving the utility of the data. You get privacy-safe data that looks just like production data with the same distributions. It's real data. The typical workflow is to anonymize production data, snapshot it, and make it available to preview and testing environments as replicas. It's pretty fast.
talove 4y ago
The reason every complex application I've worked on has had a staging environment is because you do need to test production deploys in an environment that mirrors production dataset and infrastructure. Especially with data migrations, distributed databases. That is prohibitively expensive and not feasible to run in n+1 envs.
- jcraft 4y ago
  Big if, but if you can use database containers it's relatively low cost to spin them up in a namespace, load data from a database snapshot, run the migration, and then tear down the container or even the entire namespace.
sander1095 4y ago
Setting up preview environments automatically (in kubernetes or azure) with sanitized production data is something that always sounds amazing to me, but I have always had difficulty to come up with ideas to implement it. Every company/team uses different tools and ways to implement this, so there isn't a one-size-fits-all solution. Do you have any pointers?

andrewmutz 4y ago

> If you were to get rid of staging would you stand up your entire backend every time you made a preview environment? Does that include replicating databases or would you just use seed data that may or may not resemble what's in production? What I've seen done is that the staging environment is skipped, but instead concentric releases are performed on the customers. So released code goes first to a small group of users, and then widens from there if everything looks good.

oasisbob 4y ago
This technique doesn't seem especially relevant for many kinds of backend work.
scubbo 4y ago
> What I've seen done is that the staging environment is skipped, but instead concentric releases are performed on the customers. So released code goes first to a small group of users, and then widens from there if everything looks good. If you really want to invest in extensive testing, you can do both! Hell, _and_ you can have a load testing environment as well.
- andrewmutz 4y ago
  Yeah, we had tooling to load a snapshot of customer data (with PII removed) into an ephemeral environment for when developers needed data that was more authentic. This worked for performance related work or just for features that needed a complex data model to be useful (ML apps, etc)
klooney 4y ago
Schema changes are hard to roll out that way though.
- andrewmutz 4y ago
  We were a B2B product whose multitenancy model was schema-per-customer, so it was no problem for us. Agree that it would be harder to implement with customers sharing a schema, but you can still have different release pods

hangonhn 4y ago

Yeah. And it's also not that impressive. We've had what they call "preview environments" for a very long time and at my previous companies too. With GitHub and AWS/Cloud it's trivially easy to do. And like you, we have both "preview environments" and staging and for the exact reason you've highlight. I'm rather skeptical of the company behind this product now since they don't seem to understand the difference and really overselling something that's quite common.

phendrenad2 4y ago
Yes, this is one of those things that was easier before kubernetes. Before: Your developers have a shared Linux box that serves the preview environment. Your ruby app has a tiny loader in front of it that loads the app code from a specific directory based on subdomain. After: You need to route subdomains to different kubernetes containers, and handle deploying new containers when the developer opens a PR (because it's much more work than just copying files onto a server), and you need to handle destroying old containers (because it's much more work than just deleting files on a server). Not to mention, usually kubernetes is managed by "the ops" who don't know anything about the app. Much more difficult to interface with them when it's not "oh one server, and the developers figure it out, and if they royally fuck it up we'll rescue them".
- shamiln 4y ago
  It’s really not that difficult to automate it over helm or something. All you need is a registry, and most cloud services provide a registry, even GitHub has one. Routing domains to a container shouldn’t be significantly different to doing it in production.
  - tracker1 4y ago
    For that matter, you generally will have a single app firewall or ingress server that reverse-proxies to your application anyway.
    - 4y ago

colpabar 4y ago

How long do your PRs stay open? Could you explain a bit more about what happens between opening the PR and merging it, like automated/manual testing? At my company we do something similar, except we trigger production deployments off tags, not merged PRs. Merging goes to staging only.

CSMastermind 4y ago
It really depends on the PR. Most do get merged fairly quickly (same day) but some where there's more discussion can stay open for several days or even a week. All of our codebases have some level of automated testing that runs when the PR is created. That could be unit tests of a single function or Playwright tests which exercise an UI flow. What those tests are dependent on the type of software and the team building it. The preview environment links that are generated when the PR is open are given to QA, Design, and Product to take a look at. Not every one of those roles review every PR, it really depends on the task. But if their feedback is required, they'll give it at this stage. Probably worth mentioning at this point that we have separate repos for each web application and service. It's service oriented but not microservices, services at our company encapsulate a relatively large domain and it's similar with web applications, right now each web application has its own subdomain. So PR gets merged, the pipeline will deploy that code to staging. At this point the service owners have the option to define a set of integration tests, not every repo has them but it is an option to run them. If those tests fail we do a rollback. If those tests pass (or if they didn't have any defined) then a second deployment to production is triggered. All of our deploys are blue-green, so no downtime except for the occasional blocking database migration in which we have to schedule downtime. User facing features on the web applications are all required to be released first behind a feature flag. For those there will be several releases behind a feature flag, then we'll flip the flag first in staging. We'll have Product and QA do a complete run through and make sure they're okay with everything, then the "real" release happen when we flip the feature flag in production (no deployment needed). Every night we clone the production database, sanitize it (swapping out things like names, phone numbers, etc.) then point staging to that clone. Then we run an end to end regression test that goes through all of the happy path flows for all of our apps.

ArjenM 4y ago

>a sanitized copy Quite the importance over a constant wave of changing code on development in certain projects. Staging seems like a more code locked stable area for all previews that can be polished via quality practices and keen eyes. Now it seems a little fleeting to me as well to have the shock of previews enter into this instead. Let the paint dry adequately before taking it outside.

candiddevmike 4y ago

How do you sanitize data between prd and stg?

champagnepapi 4y ago
You can use something like this https://www.replibyte.com/docs/introduction/ https://www.replibyte.com/docs/introduction/
jcraft 4y ago
My company can anonymize production data while preserving the utility of the data. You get privacy-safe data that looks just like production data with the same distributions. It's real data. The typical workflow is to anonymize production data, snapshot it, and make it available to preview and testing environments as replicas. It's pretty fast.

talove 4y ago

The reason every complex application I've worked on has had a staging environment is because you do need to test production deploys in an environment that mirrors production dataset and infrastructure. Especially with data migrations, distributed databases. That is prohibitively expensive and not feasible to run in n+1 envs.

jcraft 4y ago
Big if, but if you can use database containers it's relatively low cost to spin them up in a namespace, load data from a database snapshot, run the migration, and then tear down the container or even the entire namespace.

sander1095 4y ago

Setting up preview environments automatically (in kubernetes or azure) with sanitized production data is something that always sounds amazing to me, but I have always had difficulty to come up with ideas to implement it. Every company/team uses different tools and ways to implement this, so there isn't a one-size-fits-all solution. Do you have any pointers?