3 ms·

Realistically, when you run a website big enough that the people pushing the code to servers aren't the people who wrote it, you're going to have a rollback but

Realistically, when you run a website big enough that the people pushing the code to servers aren't the people who wrote it, you're going to have a rollback button. The ops people can't wait to figure out what commit caused the issue, much less wait for a fix. They're going to roll back the servers. Note that a rollback could be useful even in the example in this post. Machines that have hit the bad codepath are corrupted, but machines that haven't hit it yet still have good data. Sometimes you need to stop the bleeding even if there's more work to be done for a fix. > If developers incorrectly believe that their mistakes can be quickly reversed, they will tend to take more foolish risks. That may be true, but by the same token, if developers believe their code can't be rolled back, they're writing code that's broken in a different way.

mcfunley 10y ago
> Realistically, when you run a website big enough that the people pushing the code to servers aren't the people who wrote it, My position would be that you should push your own code even/especially on large websites.
jwatte 10y ago
We have milions of active users, and a large engineering organization, and our engineers push their own code. Every hour, every day. We've internalized writing code that deploys well both going forwards and going back. 99% of all changes don't even need to worry about that, because they are UI/rules/bug changes, not base data schema. We actually don't roll back database schemata; schemata are applied before code that relies on the changes can be written, and thus must be backwards compatible! If you think you can't do it that way, think harder. It's not cheap to build all the necessary infrastructure and keep training all new engineers in the necessary skills, but it can absolutely be done and pay off handsomely. We never run stale code, and we never diverge far from master.

mcfunley 10y ago

> Realistically, when you run a website big enough that the people pushing the code to servers aren't the people who wrote it, My position would be that you should push your own code even/especially on large websites.

jwatte 10y ago

We have milions of active users, and a large engineering organization, and our engineers push their own code. Every hour, every day. We've internalized writing code that deploys well both going forwards and going back. 99% of all changes don't even need to worry about that, because they are UI/rules/bug changes, not base data schema. We actually don't roll back database schemata; schemata are applied before code that relies on the changes can be written, and thus must be backwards compatible! If you think you can't do it that way, think harder. It's not cheap to build all the necessary infrastructure and keep training all new engineers in the necessary skills, but it can absolutely be done and pay off handsomely. We never run stale code, and we never diverge far from master.