3 ms·

While a book of screw-ups might be amusing, I think it might be more instructive to look at the "old school" ways that screw-ups like this were avoided. I work

While a book of screw-ups might be amusing, I think it might be more instructive to look at the "old school" ways that screw-ups like this were avoided. I worked, back in the dark ages, for a health insurer that had two parallel environments-- one for testing/development, and one for production. None of the developers were allowed anywhere near the production environment. There was one full time employee, a former developer, whose primary job was to move code into production-- which he would only do when he received a signed document authorizing the change. Said document included the telephone numbers where the developer responsible for the code would be for the next 24 hours, so that one of the operators could call you in the middle of the night if your code caused any problems to the system. At the time, I used to think that this was ridiculous. After managing a staff of programmers myself, I'm not so sure.

viraptor 16y ago
It depends on the software a bit, but I got a taste of "programmers not allowed on production" environment (in a very small scale). The problem I run into is that you have no idea what's going on on the production environment. Some characteristics are maybe reproducible on the dev site, but actual users will always do something differently. Sometimes not being able to poke the live system in a specific way while it's live, will cost you weeks of guessing in the dev environment. Then again, there is no perfect solution for this, is there? If you tried installing new version with more debugging points, you'd be deploying something unstable over something previously unstable and trying to push more data out which might be problematic in itself. I'm not even going into "rollback to stable didn't work" scenarios :(
- arethuza 16y ago
  Even in fairly conservative financial environments I've seen developers getting read-only access to production application servers and databases to troubleshoot specific problems.
- peterwwillis 16y ago
  In my experience the best solution for a large-scale dynamic site is a combination of read-only access to production and deploy management. Deploy management is the combination of a change management system with a deployment tool in a flexible way. So for example, with the right options, in an emergency the manager of a team of developers could deploy anything to the site immediately without requiring approval from change management. The tools are still right there to revert any change, and of course there's snapshots and daily backups for the most critical data. Except for emergencies, all changes to production would come with an approval from a higher-up with potential code review back-and-forth first. Contact data and reversion capabilities are built in, so everybody knows who did what, how to contact them and how to revert it if they're unavailable. And of course your trending data will tell you when your peak use is and code pushes are typically frozen during that time, minimizing further potential loss. However, besides having read-only access to production, devs should also have two kinds of testing: "development" and "staging". Development is where the bleeding-edge broken stuff lives and code is written. Staging is an identical machine to those in production. Often you'll see test or qa machines which aren't identical to production, usually because changes aren't pushed to them the way they are in production. The staging machine gets all changes pushed to it like any other production machine, except it lives in a LAN that cannot access production or anything else. A method to reproduce incoming requests and sessions from production system to this staging server will give you a pretty good idea what "real traffic" looks like on this box, if you need it.

viraptor 16y ago

It depends on the software a bit, but I got a taste of "programmers not allowed on production" environment (in a very small scale). The problem I run into is that you have no idea what's going on on the production environment. Some characteristics are maybe reproducible on the dev site, but actual users will always do something differently. Sometimes not being able to poke the live system in a specific way while it's live, will cost you weeks of guessing in the dev environment. Then again, there is no perfect solution for this, is there? If you tried installing new version with more debugging points, you'd be deploying something unstable over something previously unstable and trying to push more data out which might be problematic in itself. I'm not even going into "rollback to stable didn't work" scenarios :(

arethuza 16y ago
Even in fairly conservative financial environments I've seen developers getting read-only access to production application servers and databases to troubleshoot specific problems.
peterwwillis 16y ago
In my experience the best solution for a large-scale dynamic site is a combination of read-only access to production and deploy management. Deploy management is the combination of a change management system with a deployment tool in a flexible way. So for example, with the right options, in an emergency the manager of a team of developers could deploy anything to the site immediately without requiring approval from change management. The tools are still right there to revert any change, and of course there's snapshots and daily backups for the most critical data. Except for emergencies, all changes to production would come with an approval from a higher-up with potential code review back-and-forth first. Contact data and reversion capabilities are built in, so everybody knows who did what, how to contact them and how to revert it if they're unavailable. And of course your trending data will tell you when your peak use is and code pushes are typically frozen during that time, minimizing further potential loss. However, besides having read-only access to production, devs should also have two kinds of testing: "development" and "staging". Development is where the bleeding-edge broken stuff lives and code is written. Staging is an identical machine to those in production. Often you'll see test or qa machines which aren't identical to production, usually because changes aren't pushed to them the way they are in production. The staging machine gets all changes pushed to it like any other production machine, except it lives in a LAN that cannot access production or anything else. A method to reproduce incoming requests and sessions from production system to this staging server will give you a pretty good idea what "real traffic" looks like on this box, if you need it.