3 ms·

FWIW I don't think it is weird at all. Maybe a little short on details of what ready really means for example. While I don't think going completely staging-less

FWIW I don't think it is weird at all. Maybe a little short on details of what ready really means for example. While I don't think going completely staging-less makes a lot of sense, going without a shared staging environment is a good thing. It is absolutely awesome to be able to have your own "staging" environment for testing that is independent of everyone else. With the Cloud this is absolutely possible. Shared staging environments are really bad. Things that should take a day at most turn into a coordination and waiting game of weeks. And as pressure mounts to get things tested and out you might have people trying to deploy parts that "won't affect the other tests" going on at the same time. And then they do and you have no idea if it's your changes or their changes that made the tests fail. And since it's been 2 weeks since the change was made and you finally got time on that environment your devs have already finished working on two or more other changes in the meantime. FWIW we have a similar set up where devs and QA can spin up a complete environment that is almost the exact same as prod and do so independently. They can turn on and off feature flags individually without affecting each other. Since we don't need to wait (except for the few minutes to deploy or a bit longer to create a new env from scratch) any bugs found can be fixed rather quickly as devs at most have started working on another task. The environment can be torn down once finished but probably will just be reused until the end of the day. (while it's almost the same as prod it isn't completely like it for cost reasons meaning less nodes by default and such but honestly for most changes that is completely irrelevant and when it might be relevant it's easy to spin up more nodes temporarily through the exact same means as one would use to handle load spikes in prod).

xorcist 5y ago
> "staging" environment for testing that is independent of everyone else That's not usually what people mean by staging. Staging is a type of pre-production test environment where several different features that are continously developed by different teams can be tested together. Third party integrations such as logistics, ordering and payment systems can also have their integration testing here. > Shared staging environments are really bad That sounds dangerously close to "testing is hard, let's go shopping". That it can be a logistical challenge to test code that touches many parts of a multi stakeholder system does not mean we shouldn't do it. Having to wait weeks to test a feature sounds like the process has broken down, not that the process is unnecessary.
- tharkun__ 5y ago
  Staging is a type of pre-production test environment where several different features that are continously developed by different teams can be tested together Yes, that is exactly what we can do on these environments that any dev or QA has at their fingertips. We have multiple teams that work on several different features/parts of the application that sometimes live in the same services as other teams work on as well or they are smaller, more dedicated services and they're the only team working on those specifically but they still usually work together with other services in the overall system. Some of those have third party integrations. FWIW, in our case usually we are the ones testing integrations with a third party and not the other way around. That it can be a logistical challenge to test code that touches many parts of a multi stakeholder system does not mean we shouldn't do it. As mentioned earlier, these individually deployable environments are fully functional and fully integrated. Now I understand that in some situations it can be hard to do a full integration with a third party, because the third party is not able to able to accommodate the numerous environments that you are able to provide. In those cases compromises might need to be made. That is overall a bad thing though. E.g. to take your "third party ordering system" example again. It would be best, if you could have a separate account/tenant/instance (whatever makes sense in the exact circumstances and nature of system) in said third party system but sometimes that might not be possible and a third party system might need to be shared somehow between all your own staging/dev envinronments. That sounds dangerously close to "testing is hard, let's go shopping". Having to wait weeks to test a feature sounds like the process has broken down, not that the process is unnecessary. That is never what I said. I said that we can create such staging environments that are a fully integrated set of services on our end really easily and that that is awesome to have. The "having to wait weeks on end" is something I have experienced at previous clients/employers and I absolutely agree with you that it's a broken system. I all too well remember the "yes, we can have INT-3 for 2 hours next Tuesday, do you think we can get all our tests done in that time frame? After that they need it for extensive performance testing for a week and INT-1 won't be available until Thursday at 11". And then you gotta answer "Sorry 2 hours is barely enough to do the deploy and re-configuring of the environment because we need to manually restore that special data set for the third party logistics system mock and then adjust the configuration and that alone takes those 2 hours if the issues we had last time are any indication". Sooo much better to click the deploy button (or in the dev case we usually use the command line ;) ) and 10 minutes later you have your code and everything else deployed.
tatersolid 5y ago
How do you get realistic data into these environments? Our production has 10TB of database data and 20M files uploaded by customers. We regularly see devs write queries or nested loops that work fine on their tiny dev system but explode when run against realistic customer data. Right now we go through a 12 hour restore then anonymization pass to keep staging up to date. Very painful. Who has a better way?
- tharkun__ 5y ago
  YMMV as always, i.e. this might be faster/easier to implement for some use cases/companies than others. I would argue that for most changes in most companies it does not matter if you have a full data set equivalent of Prod. Especially since it's an ever growing target (hopefully for you :)). As we can see from your use case, that can create challenges. If we were testing every little change with a complete replica of our Prod environment we'd spend a lot more money on this and would also have to wait way too long for these environments to come up. This might then drive us towards keeping a small set of staging environments running all the time and share them. Now I don't know your specific field, what kind of system you provide, what the cost of failures in Prod would be, what kind of guards you have in your systems against a complete outage like that (e.g. can a bad query running for one customer take out your entire product or will it be an isolated incident to that one customer or a small group of customers?) etc. If so maybe you first want to isolate failures between customers more. FWIW, I also see devs write these kinds of queries and code and we catch most of them in PRs and most people learn from this and next time they don't write this kind of query any longer. But let's say they do and we don't catch them. Can you analyze your data set and figure out a small enough example that is not simply an ever growing anonymized version of actual Prod data that would exhibit the same catastrophic characteristics for most queries these devs might write? I'm pretty sure the kinds of queries that fail on 10TB of data w/ 20M files fail just as spectacularly on 9TB and 19M files? Have you tried if they also fail 'badly enough' on 500GB of data and 1M files? How fast would it be to restore a fixed data set like that? Maybe refresh it every month from Prod if that's needed for some reason or another. A lot of what you might actually want to try out in the end will depend on your exact system set up I would argue and it's hard to give general advice that will definitely fit.

xorcist 5y ago

> "staging" environment for testing that is independent of everyone else That's not usually what people mean by staging. Staging is a type of pre-production test environment where several different features that are continously developed by different teams can be tested together. Third party integrations such as logistics, ordering and payment systems can also have their integration testing here. > Shared staging environments are really bad That sounds dangerously close to "testing is hard, let's go shopping". That it can be a logistical challenge to test code that touches many parts of a multi stakeholder system does not mean we shouldn't do it. Having to wait weeks to test a feature sounds like the process has broken down, not that the process is unnecessary.

tharkun__ 5y ago
Staging is a type of pre-production test environment where several different features that are continously developed by different teams can be tested together Yes, that is exactly what we can do on these environments that any dev or QA has at their fingertips. We have multiple teams that work on several different features/parts of the application that sometimes live in the same services as other teams work on as well or they are smaller, more dedicated services and they're the only team working on those specifically but they still usually work together with other services in the overall system. Some of those have third party integrations. FWIW, in our case usually we are the ones testing integrations with a third party and not the other way around. That it can be a logistical challenge to test code that touches many parts of a multi stakeholder system does not mean we shouldn't do it. As mentioned earlier, these individually deployable environments are fully functional and fully integrated. Now I understand that in some situations it can be hard to do a full integration with a third party, because the third party is not able to able to accommodate the numerous environments that you are able to provide. In those cases compromises might need to be made. That is overall a bad thing though. E.g. to take your "third party ordering system" example again. It would be best, if you could have a separate account/tenant/instance (whatever makes sense in the exact circumstances and nature of system) in said third party system but sometimes that might not be possible and a third party system might need to be shared somehow between all your own staging/dev envinronments. That sounds dangerously close to "testing is hard, let's go shopping". Having to wait weeks to test a feature sounds like the process has broken down, not that the process is unnecessary. That is never what I said. I said that we can create such staging environments that are a fully integrated set of services on our end really easily and that that is awesome to have. The "having to wait weeks on end" is something I have experienced at previous clients/employers and I absolutely agree with you that it's a broken system. I all too well remember the "yes, we can have INT-3 for 2 hours next Tuesday, do you think we can get all our tests done in that time frame? After that they need it for extensive performance testing for a week and INT-1 won't be available until Thursday at 11". And then you gotta answer "Sorry 2 hours is barely enough to do the deploy and re-configuring of the environment because we need to manually restore that special data set for the third party logistics system mock and then adjust the configuration and that alone takes those 2 hours if the issues we had last time are any indication". Sooo much better to click the deploy button (or in the dev case we usually use the command line ;) ) and 10 minutes later you have your code and everything else deployed.

tatersolid 5y ago

How do you get realistic data into these environments? Our production has 10TB of database data and 20M files uploaded by customers. We regularly see devs write queries or nested loops that work fine on their tiny dev system but explode when run against realistic customer data. Right now we go through a 12 hour restore then anonymization pass to keep staging up to date. Very painful. Who has a better way?

tharkun__ 5y ago
YMMV as always, i.e. this might be faster/easier to implement for some use cases/companies than others. I would argue that for most changes in most companies it does not matter if you have a full data set equivalent of Prod. Especially since it's an ever growing target (hopefully for you :)). As we can see from your use case, that can create challenges. If we were testing every little change with a complete replica of our Prod environment we'd spend a lot more money on this and would also have to wait way too long for these environments to come up. This might then drive us towards keeping a small set of staging environments running all the time and share them. Now I don't know your specific field, what kind of system you provide, what the cost of failures in Prod would be, what kind of guards you have in your systems against a complete outage like that (e.g. can a bad query running for one customer take out your entire product or will it be an isolated incident to that one customer or a small group of customers?) etc. If so maybe you first want to isolate failures between customers more. FWIW, I also see devs write these kinds of queries and code and we catch most of them in PRs and most people learn from this and next time they don't write this kind of query any longer. But let's say they do and we don't catch them. Can you analyze your data set and figure out a small enough example that is not simply an ever growing anonymized version of actual Prod data that would exhibit the same catastrophic characteristics for most queries these devs might write? I'm pretty sure the kinds of queries that fail on 10TB of data w/ 20M files fail just as spectacularly on 9TB and 19M files? Have you tried if they also fail 'badly enough' on 500GB of data and 1M files? How fast would it be to restore a fixed data set like that? Maybe refresh it every month from Prod if that's needed for some reason or another. A lot of what you might actually want to try out in the end will depend on your exact system set up I would argue and it's hard to give general advice that will definitely fit.