5 ms·

This article has some very weird trade-offs. They can't spin up test environments quickly, so they have windows when they cannot merge code due to release timi

This article has some very weird trade-offs. They can't spin up test environments quickly, so they have windows when they cannot merge code due to release timing. They can't maintain parity of their staging environments with prod, so they forswear staging environments. These seem like infrastructure problems that aren't addressing the same problem as the staging environment eo ipso. They're not arguing that testing or staging environments are bad, they're just saying their organization couldn't manage to get them working. If they didn't hit those roadblocks in managing their staging environments, presumably they would be using them.

_3u10 5y ago
Having staging always encourages this. It’s really difficult to replicate prod in any non trivial way that exceeds what can be created on a workstation. Eg. Even if you buy the same hardware you can’t replicate production load anyway because it’s not being used by 5 million people concurrently. Your cache access patterns aren’t the same, etc. It’s far better to have a fast path to prod than a staging environment in my opinion.
- nickelpro 5y ago
  Perhaps we have different ideas about what a staging environment is for. I wouldn't expect a staging environment to give accurate performance numbers for a change, the only solution to that is instrumenting the production environment.
- saq7 5y ago
  I think it's too much to expect staging to match the load and access patterns of your prod system. I find staging to be very useful. In various teams I have been a part of, I have seen the following productive use cases for staging 1. Extended development environment - If you use a micro-services or serverless architecture, it becomes really useful to do end-to-end tests of your code on staging. Docker helps locally, but unless you have a $4,000 laptop, the dev experience becomes very poor. 2. User acceptance testing - Generally performed by QAs, PMs or some other businessy folks. This becomes very important for teams that serve a small number of customer who write big checks. 3. Legacy enterprise teams - Very large corporations in which software does not drive revenue directly, but high quality software drives a competitive advantage. Insurance companies are an example. These folks have a much lower tolerance for shipping software that doesn't work exactly right for customers.
  - _3u10 5y ago
    dev workstations should cost at least $4000. Like how much productivity is being wasted because their machine is slow. $4000 workstations are cheap compared to staging.
    - freedomben 5y ago
      when I worked for big corp, the reason we were told in engineering for getting $1,000 laptops was that it wasn't fair to accounting, HR, etc for us to have better machines. In the past people from these departments complained quite a bit. The official reason (which was BS) was "to simplify IT's job by only having to support one model"
      - butlerm 5y ago
        Who cares what is "fair"? A decision like that should be based on an elementary productivity calculation. If not the inmates have taken over the asylum.
    - chupasaurus 5y ago
      Multiply that by the number of employees who need it ... staging is cheaper.
      - _3u10 5y ago
        Multiply that by the lost hours waiting for compilation, etc and whether you have staging or not doesn’t matter.
        eertami 5y ago
        Why would I build on a local machine vs running the build on a server in a datacenter? Per your own arguments, server grade hardware is going to compile much faster than any local workstation.
        5y ago
rileymat2 5y ago
Depending on your tech, staging environments can be very expensive, SQL Server Enterprise licenses at 13k for 2 cores. https://www.microsoft.com/en-us/sql-server/sql-server-2019-pricing https://www.microsoft.com/en-us/sql-server/sql-server-2019-p...
- coder543 5y ago
  If you're choosing to pay large sums of money for SQL Server instead of the open source alternatives, you should also factor in the large sums of money to have good development/staging environments too. All the more reason to just use Postgres or MySQL. EDIT: as someone else hinted at, it does look like the free Developer version of SQL Server is fully featured and licensed for use in any non-prod environment, which seems reasonable.
  - rileymat2 5y ago
    Sure different planning 20 years ago would have made a big difference. Or the will/resources to transition. I am just saying that this scenario exists.
- jiggawatts 5y ago
  Non-prod is free.
  - deleted 5y ago
    [deleted]
  - rileymat2 5y ago
    The developer version, yes. But I have not seen the AWS amis for the developer version: https://aws.amazon.com/about-aws/whats-new/2021/10/amazon-ec2-microsoft-sql-server-windows-ami/ https://aws.amazon.com/about-aws/whats-new/2021/10/amazon-ec... You can't install the enterprise non-prod for free. (But the developer version is supposed to have all the features)
    - booi 5y ago
repler 5y ago
Exactly. “Staging never matches Prod” - well why is that? Make it so!!
- drewcoo 5y ago
  I have never ever even heard of a place where that was possible. The easiest way to make that scenario happen is take do whatever testing you'd have done in staging and do it in prod. Problem solved.
  - saq7 5y ago
    I am curios, why do you think it's impossible? I think we can establish that the database is the biggest culprit in making this difficult. As an independent developer, I have seen several teams that either back sync the prod db into the staging db OR capture known edge cases through diligent use of fixtures. I am not trying to counter your point necessarily, but just trying to understand your POV. Very possible that, in my limited experience, I haven't come across all the problems around this domain.
    - lamontcg 5y ago
      The variety of requests and load in prod never matches production along with all the messiness and jitter you get from requests coming from across the planet and not just from your own LAN. And you'll probably never build it out to the same scale as production and have half your capex dedicated to it, so you'll miss issues which depend on your own internal scaling factors. There's a certain amount of "best practices" effort you can go through in order to make your preprod environments sufficiently prod like but scaled down, with real data in their databases, running all the correct services, you can have a load testing environment where you hit one front end with a replay of real load taking from prod logs to look for perf regressions, etc. But ultimately time is better spent using feature flags and one box tests in prod rather than going down the rabbit hole of trying to simulate packet-level network failures in your preprod environment to try to make it look as prodlike as possible (although if you're writing your own distributed database you should probably be doing that kind of fault injection, but then you probably work somewhere FAANG scale, or you've made a potentially fatal NIH/DIY mistake).
deeteecee 5y ago
> They're not arguing that testing or staging environments are bad, they're just saying their organization couldn't manage to get them working. That is exactly what I got from reading this article. Their staging process was poorly set up and they simply abandoned ship. Additionally, I was getting poor software culture vibes.
- resonious 5y ago
  Indeed, I found the "We only merge code that is ready to go live" part odd. It seems unrelated to the presence of absence of a staging environment. Where I work, we use staging and also only merge code that is ready to go live. Similarly, "Poor ownership of changes" and "People mistakenly let process replace accountability" just don't seem staging-related to me. I've been in environments where people throw code over the fence straight into production.
  - lbriner 5y ago
    The way I read that is, if you have a staging environment, it is someone else's job to test things before they deploy (Product Manager etc.) which allows them to merge it and forget it. I agree if it is the dev's job then it wouldn't make sense. The other issue that is fair is the potential lag between dev and production if you have a gate at staging. This way, a developer is likely to move onto something else instead of watching their baby swim into production with all the errors that could cause!
  - relaxing 5y ago
    Why would you merge code that wasn’t ready to go live?
- sverhagen 5y ago
  I actually empathize with the statements in the article about the challenges of managing a staging environment. For a lot of systems I worked on, short of just copying customer data into the staging environment (which, depending on your industry or the contracts with your customers may be a big no-no, and the more-distributed your system, the harder that also was to do) it was extremely hard to populate the staging environment with representative data. Or representative state in third-party systems. (Not to say that testing on a developer's machine wouldn't have these same problems, of course, which I also find the article glosses over.)
maxbond 5y ago
The parity part is especially confusing to me, since they go on to test locally. So staging isn't close enough to production, but local is...?
- sverhagen 5y ago
  I don't represent the original author. But the way I read that is: the staging environment is said to be a production clone, but it turned out it wasn't, so let's not pretend it is (and fool ourselves) and instead embrace a different test strategy altogether. Thinking of how I test things locally and how I write my own unit and integration tests, I guess that it means doing very isolated, functional tests. An investment in which, to some teams, may be more valuable than throwing it into staging and hoping that the state you found your staging environment in is going to surface some insight about your code change that isn't covered by your test suite.
  - maxbond 5y ago
    I agree that that is what they are trying to do, but they don't appear to be testing with all that different of a strategy; something has been removed, but it doesn't seem like anything has taken it's place. Staging is useful because you cannot predict how your changes will impact the entire application, or how it will interact with configuration. This is precisely where isolated tests fall short. Differences between production and staging are real, but differences between production and local are much more profound. Staging is an imperfect strategy, but this approach doesn't appear to be sound. It seems to shrug it's shoulders and settle for something even worse. I'm baffled, frankly.

_3u10 5y ago

Having staging always encourages this. It’s really difficult to replicate prod in any non trivial way that exceeds what can be created on a workstation. Eg. Even if you buy the same hardware you can’t replicate production load anyway because it’s not being used by 5 million people concurrently. Your cache access patterns aren’t the same, etc. It’s far better to have a fast path to prod than a staging environment in my opinion.

nickelpro 5y ago
Perhaps we have different ideas about what a staging environment is for. I wouldn't expect a staging environment to give accurate performance numbers for a change, the only solution to that is instrumenting the production environment.
saq7 5y ago
I think it's too much to expect staging to match the load and access patterns of your prod system. I find staging to be very useful. In various teams I have been a part of, I have seen the following productive use cases for staging 1. Extended development environment - If you use a micro-services or serverless architecture, it becomes really useful to do end-to-end tests of your code on staging. Docker helps locally, but unless you have a $4,000 laptop, the dev experience becomes very poor. 2. User acceptance testing - Generally performed by QAs, PMs or some other businessy folks. This becomes very important for teams that serve a small number of customer who write big checks. 3. Legacy enterprise teams - Very large corporations in which software does not drive revenue directly, but high quality software drives a competitive advantage. Insurance companies are an example. These folks have a much lower tolerance for shipping software that doesn't work exactly right for customers.
- _3u10 5y ago
  dev workstations should cost at least $4000. Like how much productivity is being wasted because their machine is slow. $4000 workstations are cheap compared to staging.
  - freedomben 5y ago
    when I worked for big corp, the reason we were told in engineering for getting $1,000 laptops was that it wasn't fair to accounting, HR, etc for us to have better machines. In the past people from these departments complained quite a bit. The official reason (which was BS) was "to simplify IT's job by only having to support one model"
    - butlerm 5y ago
      Who cares what is "fair"? A decision like that should be based on an elementary productivity calculation. If not the inmates have taken over the asylum.
  - chupasaurus 5y ago
    Multiply that by the number of employees who need it ... staging is cheaper.
    - _3u10 5y ago
      Multiply that by the lost hours waiting for compilation, etc and whether you have staging or not doesn’t matter.
      - eertami 5y ago
        Why would I build on a local machine vs running the build on a server in a datacenter? Per your own arguments, server grade hardware is going to compile much faster than any local workstation.
        5y ago

rileymat2 5y ago

Depending on your tech, staging environments can be very expensive, SQL Server Enterprise licenses at 13k for 2 cores. https://www.microsoft.com/en-us/sql-server/sql-server-2019-pricing https://www.microsoft.com/en-us/sql-server/sql-server-2019-p...

coder543 5y ago
If you're choosing to pay large sums of money for SQL Server instead of the open source alternatives, you should also factor in the large sums of money to have good development/staging environments too. All the more reason to just use Postgres or MySQL. EDIT: as someone else hinted at, it does look like the free Developer version of SQL Server is fully featured and licensed for use in any non-prod environment, which seems reasonable.
- rileymat2 5y ago
  Sure different planning 20 years ago would have made a big difference. Or the will/resources to transition. I am just saying that this scenario exists.
jiggawatts 5y ago
Non-prod is free.
- deleted 5y ago
  [deleted]
- rileymat2 5y ago
  The developer version, yes. But I have not seen the AWS amis for the developer version: https://aws.amazon.com/about-aws/whats-new/2021/10/amazon-ec2-microsoft-sql-server-windows-ami/ https://aws.amazon.com/about-aws/whats-new/2021/10/amazon-ec... You can't install the enterprise non-prod for free. (But the developer version is supposed to have all the features)
  - booi 5y ago

repler 5y ago

Exactly. “Staging never matches Prod” - well why is that? Make it so!!

drewcoo 5y ago
I have never ever even heard of a place where that was possible. The easiest way to make that scenario happen is take do whatever testing you'd have done in staging and do it in prod. Problem solved.
- saq7 5y ago
  I am curios, why do you think it's impossible? I think we can establish that the database is the biggest culprit in making this difficult. As an independent developer, I have seen several teams that either back sync the prod db into the staging db OR capture known edge cases through diligent use of fixtures. I am not trying to counter your point necessarily, but just trying to understand your POV. Very possible that, in my limited experience, I haven't come across all the problems around this domain.
  - lamontcg 5y ago
    The variety of requests and load in prod never matches production along with all the messiness and jitter you get from requests coming from across the planet and not just from your own LAN. And you'll probably never build it out to the same scale as production and have half your capex dedicated to it, so you'll miss issues which depend on your own internal scaling factors. There's a certain amount of "best practices" effort you can go through in order to make your preprod environments sufficiently prod like but scaled down, with real data in their databases, running all the correct services, you can have a load testing environment where you hit one front end with a replay of real load taking from prod logs to look for perf regressions, etc. But ultimately time is better spent using feature flags and one box tests in prod rather than going down the rabbit hole of trying to simulate packet-level network failures in your preprod environment to try to make it look as prodlike as possible (although if you're writing your own distributed database you should probably be doing that kind of fault injection, but then you probably work somewhere FAANG scale, or you've made a potentially fatal NIH/DIY mistake).

deeteecee 5y ago

> They're not arguing that testing or staging environments are bad, they're just saying their organization couldn't manage to get them working. That is exactly what I got from reading this article. Their staging process was poorly set up and they simply abandoned ship. Additionally, I was getting poor software culture vibes.

resonious 5y ago
Indeed, I found the "We only merge code that is ready to go live" part odd. It seems unrelated to the presence of absence of a staging environment. Where I work, we use staging and also only merge code that is ready to go live. Similarly, "Poor ownership of changes" and "People mistakenly let process replace accountability" just don't seem staging-related to me. I've been in environments where people throw code over the fence straight into production.
- lbriner 5y ago
  The way I read that is, if you have a staging environment, it is someone else's job to test things before they deploy (Product Manager etc.) which allows them to merge it and forget it. I agree if it is the dev's job then it wouldn't make sense. The other issue that is fair is the potential lag between dev and production if you have a gate at staging. This way, a developer is likely to move onto something else instead of watching their baby swim into production with all the errors that could cause!
- relaxing 5y ago
  Why would you merge code that wasn’t ready to go live?
sverhagen 5y ago
I actually empathize with the statements in the article about the challenges of managing a staging environment. For a lot of systems I worked on, short of just copying customer data into the staging environment (which, depending on your industry or the contracts with your customers may be a big no-no, and the more-distributed your system, the harder that also was to do) it was extremely hard to populate the staging environment with representative data. Or representative state in third-party systems. (Not to say that testing on a developer's machine wouldn't have these same problems, of course, which I also find the article glosses over.)

maxbond 5y ago

The parity part is especially confusing to me, since they go on to test locally. So staging isn't close enough to production, but local is...?

sverhagen 5y ago
I don't represent the original author. But the way I read that is: the staging environment is said to be a production clone, but it turned out it wasn't, so let's not pretend it is (and fool ourselves) and instead embrace a different test strategy altogether. Thinking of how I test things locally and how I write my own unit and integration tests, I guess that it means doing very isolated, functional tests. An investment in which, to some teams, may be more valuable than throwing it into staging and hoping that the state you found your staging environment in is going to surface some insight about your code change that isn't covered by your test suite.
- maxbond 5y ago
  I agree that that is what they are trying to do, but they don't appear to be testing with all that different of a strategy; something has been removed, but it doesn't seem like anything has taken it's place. Staging is useful because you cannot predict how your changes will impact the entire application, or how it will interact with configuration. This is precisely where isolated tests fall short. Differences between production and staging are real, but differences between production and local are much more profound. Staging is an imperfect strategy, but this approach doesn't appear to be sound. It seems to shrug it's shoulders and settle for something even worse. I'm baffled, frankly.