4 ms·

I set up a staging instance for my previous startup, then never used it. I don't really understand what the value is unless you have dedicated QA staff that run

I set up a staging instance for my previous startup, then never used it. I don't really understand what the value is unless you have dedicated QA staff that run some manual tests against staging. If the tests are automated, they should run in a dev environment. If you don't have QA, just test in prod. This seems to work well for the largest tech companies, who (in)famously don't really use branches and ship directly to prod with feature flags.

paranoidrobot 6y ago
The answer obviously varies wildly for every product and company. Not everything can be feature flagged. Not all tests can be automated. Not all features can be safely tested in production. You can't always roll-back code. Scenarios where I've found staging environments to be invaluable: - Features that make wide-scale changes to the application - Changes that require user acceptance testing. Management/Customers might ask for Feature X, you have designers mock up what it'll look like, you might even build prototypes of it... but until they see it and play with it, they might not fully realise that what they actually asked for isn't what they actually intended. - Integrating changes from multiple developers and seeing how they play together - Testing that your feature flagging actually works - Testing how the system responds to dumping large amounts of data/requests at it - a half-day outage in production might easily exceed the cost of a whole year of running a staging environment. - Testing how the system behaves after data migrations/changes (things where there's no roll-back without losing data) - Testing integrations with external systems. - Detecting poor performance, memory leaks, deadlocks, and other stuff that isn't possible on a dev's machine Testing code on your machine is fine as a smoke test, but your laptop probably has multiple times the capacity of a single node on production, for a single user. So your change that results in 1GB/sec of full table scans on the database might not even be noticed because hey, you have an NVME drive. Your tests also run for maybe a few minutes, but that staging instance might be up for days or hours, and deals with hundreds I've picked up numerous issues because someone (I'm including myself here) committed some code that resulted in Staging going down or behaving poorly. I've even overheard conversations that were along the lines of "Yeah, this feature isn't working right at the moment in testing, but that's because staging is running slow today It's fine, we can push this out to production and it'll be better there." if it wern't for me overhearing that, it'd have resulted in a multi-GB video file being sent for every visitor to the site, costing a fortune in CDN charges and still running like crap because it was a 30 second video.

paranoidrobot 6y ago

The answer obviously varies wildly for every product and company. Not everything can be feature flagged. Not all tests can be automated. Not all features can be safely tested in production. You can't always roll-back code. Scenarios where I've found staging environments to be invaluable: - Features that make wide-scale changes to the application - Changes that require user acceptance testing. Management/Customers might ask for Feature X, you have designers mock up what it'll look like, you might even build prototypes of it... but until they see it and play with it, they might not fully realise that what they actually asked for isn't what they actually intended. - Integrating changes from multiple developers and seeing how they play together - Testing that your feature flagging actually works - Testing how the system responds to dumping large amounts of data/requests at it - a half-day outage in production might easily exceed the cost of a whole year of running a staging environment. - Testing how the system behaves after data migrations/changes (things where there's no roll-back without losing data) - Testing integrations with external systems. - Detecting poor performance, memory leaks, deadlocks, and other stuff that isn't possible on a dev's machine Testing code on your machine is fine as a smoke test, but your laptop probably has multiple times the capacity of a single node on production, for a single user. So your change that results in 1GB/sec of full table scans on the database might not even be noticed because hey, you have an NVME drive. Your tests also run for maybe a few minutes, but that staging instance might be up for days or hours, and deals with hundreds I've picked up numerous issues because someone (I'm including myself here) committed some code that resulted in Staging going down or behaving poorly. I've even overheard conversations that were along the lines of "Yeah, this feature isn't working right at the moment in testing, but that's because staging is running slow today It's fine, we can push this out to production and it'll be better there." if it wern't for me overhearing that, it'd have resulted in a multi-GB video file being sent for every visitor to the site, costing a fortune in CDN charges and still running like crap because it was a 30 second video.