4 ms·

Kind of tangential, but: > Then one day, our designer broke the build in the middle of the night. Everyone came in the next day and couldn’t work until they fi

Kind of tangential, but: > Then one day, our designer broke the build in the middle of the night. Everyone came in the next day and couldn’t work until they figured out what had happened. I've heard this [campfire?] story before. A bad commit happens and now no one can do any work. And I don't understand... Is it really that big of a deal to have to revert to an older commit or comment out the broken code until it gets resolved? Sure, I can see it being annoying, but not bringing an entire group of developers to a standstill. What am I missing?

itishappy 2y ago
Stateless code makes for boring programs, but stateful code can be difficult to roll back. As a toy example, imagine a database with columns for `first name` and `last name` and an update that combined them into a single `full name` column. That's going to be tough to revert for users signing up with just their full names.
- 9rx 2y ago
  While that would certainly impact production, it would be a bit strange for development environments to be stateful in the same way. Those not immediately involved in dealing with the stateful issue seemingly should be able to continue with their work away from production systems just fine. Although in this case the build broke because the last committer didn't have authorization to deploy to production. The "fix" was giving him permission to deploy in the future. It is not clear why the developers going about their regular day wouldn't naturally resolve the broken build.
cdchn 2y ago
How does giving them deploy access prevent them from breaking the build, too?
smag 2y ago
I actually saw this happen at Google: no one could pinpoint the cause of the broken build for a while, and then it turned out to be a UX designer who broke the build. He was apologetic, like McFunley's example. But unlike the example, this guy never made another commit.
seadan83 2y ago
It's not a big deal usually to revert a commit. Sometimes nobody has no idea it was a commit done in some other code repo in some other distant team. It's a problem of visibility; it just turns out the cause was a commit somewhere. Figuring that out can be non-obvious.

itishappy 2y ago

Stateless code makes for boring programs, but stateful code can be difficult to roll back. As a toy example, imagine a database with columns for `first name` and `last name` and an update that combined them into a single `full name` column. That's going to be tough to revert for users signing up with just their full names.

9rx 2y ago
While that would certainly impact production, it would be a bit strange for development environments to be stateful in the same way. Those not immediately involved in dealing with the stateful issue seemingly should be able to continue with their work away from production systems just fine. Although in this case the build broke because the last committer didn't have authorization to deploy to production. The "fix" was giving him permission to deploy in the future. It is not clear why the developers going about their regular day wouldn't naturally resolve the broken build.

cdchn 2y ago

How does giving them deploy access prevent them from breaking the build, too?

smag 2y ago

I actually saw this happen at Google: no one could pinpoint the cause of the broken build for a while, and then it turned out to be a UX designer who broke the build. He was apologetic, like McFunley's example. But unlike the example, this guy never made another commit.

seadan83 2y ago

It's not a big deal usually to revert a commit. Sometimes nobody has no idea it was a commit done in some other code repo in some other distant team. It's a problem of visibility; it just turns out the cause was a commit somewhere. Figuring that out can be non-obvious.