3 ms·

A lot of this is (at least aspirationally) table stakes for modern software development, but for those of us who have seen the sausage get made, a) a lot of the

by patio11 14y ago

A lot of this is (at least aspirationally) table stakes for modern software development, but for those of us who have seen the sausage get made, a) a lot of the Well-Known Best Practices (TM) are not actually observed at a lot of companies, even with arbitrarily high levels of sophistication and resources available and b) they really do save metric tonnes of effort at scale. One of the reasons why seemingly trivial, clearly beneficial changes don't get adopted at some companies is that, at certain scales, it's like trying to turn an aircraft carrier. (Any process improvement which takes an engineer one day worth of productivity to adjust to costs Intuit several million dollars. Something which a startup could decide on in a week -- "We should switch to git and then have per-team branches with integration at..." -- is probably an $X00 million project there and if it failed would be catastrophic. You might blow the shipping schedule on Quickbooks, and if that misses tax season, "oh dear.") It's always interesting how the larger Fortune 500 software companies adopt practices associated with teeny-tiny little software firms, and vice versa.

svdad 14y ago
Emphasis on the "aspirationally" -- it really appears not to be how most companies do things, despite overwhelming evidence that doing these things work.
- patio11 14y ago
  Totally agreed. Incidentally, on evidence of working with savvier smaller companies, talking to one man shops, and hanging out with startups, I think the community feels that there is a body of "Everybody does X" practices which are actually quite rare. Examples I'm intimately familiar with include A/b testing, using usage metrics to drive decisions, customer development, lifecycle email marketing, etc etc. Or for more dev focused stuff, unit tests, Selenium ("the best technology that I'll never use"), code reviews as a routine practice, reproducible server setup/deploys, etc.
  - arohner 14y ago
    > A/B testing, using usage metrics to drive decisions, customer development, lifecycle email marketing... unit tests, Selenium [testing], code reviews as a routine practice, reproducible server setup/deploys, etc. That sounds like the Joel test for 2012.
    - skrebbel 14y ago
      That sounds like the Joel test for webapp firms in 2012.
  - alinajaf 14y ago
    > A/b testing, using usage metrics to drive decisions, customer development, lifecycle email marketing Yes, but I've been pleasantly surprised at the clueyness of of more than a few big-name clients recently. > Or for more dev focused stuff, unit tests, Selenium ("the best technology that I'll never use"), code reviews as a routine practice, reproducible server setup/deploys, etc. For deployments at least I've been blown away by the range of configurations I've seen. Everything from "git push heroku master" to "ssh into these three machines, run svn up, restart apache, merge dev couchdb to live, copy paste these 80 sql patches out of a file into the terminal, sacrifice a hamster to Amon Ra and cross your fingers."
- edwinnathaniel 14y ago
  Many companies favor short-term win or Money over Technicality because there aren't many people who have the skill to put the numbers on Excel spreadsheet the amount of money the company can save had we done things "Right" at the first time during the meeting between Business and Tech team (Product Manager, CEO vs Team Lead/Dev Manager). Basically it comes down to quantifying these quality works over developing new features. I had the pleasure to work in a very small tight-knit team where you have to have the bulldog to bark at people when they're slowing down their defense (i.e.: writing unit-tests, making sure build script is up to date). Results: on-time, within-budget, 1 Saturday, 1 Statutory holiday, 2 days OT until 10.30PM (all re-imburse afterwards) for a rather ambitious project to be shipped in 6 months from zero to live in 2 months afterwards. Oh, and we have 1 performance bug after the release (easy one to solve) and 1 requirement bug 3 months after release. The rest (been live for about a year) was as smooth as the baby skin. We rotate the BB phone (on-call) but we never had it buzzed. Our saving graces were: 0) Management Support 1) Continuous Integration 2) Unit-Testing We use these unit-tests as REPL mostly, so we don't have to re-compile and re-deploy the _whole_ app to GlassFish.
  - spuz 14y ago
    We practice 1 and 2 but not with the same smooth as baby skin releases. Could you explain what you mean by 0 and how it helps you?
    - edwinnathaniel 14y ago
      Basically the people above the developers trust our judgment to work on these non-feature tasks: fixing build script (we use Maven but when it first setup, it didn't handle deployment to multi-environment that well), improving the component (we were newly created team, thus the reference of bulldog that bark) by either re-writing it or adding more unit-tests. The management almost never questioned us except one or two situations.
bguthrie 14y ago
I fundamentally agree with your point: changing processes at large software teams is really, really hard, risky, and expensive. But if I may pick a nit about something I think you were describing as a potential process improvement, per-team branches have been tried at scale many times before, both with and without Git. Many smart people I know who've had to clean up subsequent messes believe, counterintuitively, that the larger and more complex you get, the more branches hurt you, because they encourage you to delay integration, and integration is painful. Git merge capability, while excellent, can't save you if two teams independently decide to rename a method that both rely on; that's still a conflict. I've heard of several other large shops relying successfully on toggles with a single mainline. Fowler describes this pain in terms of feature branches rather than team branches, but it's the same idea: http://martinfowler.com/bliki/FeatureBranch.html http://martinfowler.com/bliki/FeatureBranch.html Branching is a poor substitute for genuine modularity.
- patio11 14y ago
  I'll buy that -- I was just picking an example of a process tweak which sounded like a potential improvement that, absent scale, would be accomplished in two days with a project roadmap that would fit in a blog post.
- Tloewald 14y ago
  Nice article btw. The approach to integration recommended at the end seems exactly like something that might make the management of such a large code base easier.
- alinajaf 14y ago
  IME depends on what we're talking about. I've been at workplaces where we have multiple epic branches that diverge for three or four iterations at a time. In this scenario, it doesn't matter what VCS you happen to be using, integration is going to be painful and error-prone. More recently the branch and pull-request into master model has been working really nicely on a per-story basis, as long as people are considerate and mergeback from master before initiating the request. Overall though, I think the only branch that gets deployed anywhere should be master (or trunk if you prefer). Like you say, it's a terrible substitute for a system built of decoupled components.

by patio11 14y ago

svdad 14y ago

Emphasis on the "aspirationally" -- it really appears not to be how most companies do things, despite overwhelming evidence that doing these things work.

patio11 14y ago
Totally agreed. Incidentally, on evidence of working with savvier smaller companies, talking to one man shops, and hanging out with startups, I think the community feels that there is a body of "Everybody does X" practices which are actually quite rare. Examples I'm intimately familiar with include A/b testing, using usage metrics to drive decisions, customer development, lifecycle email marketing, etc etc. Or for more dev focused stuff, unit tests, Selenium ("the best technology that I'll never use"), code reviews as a routine practice, reproducible server setup/deploys, etc.
- arohner 14y ago
  > A/B testing, using usage metrics to drive decisions, customer development, lifecycle email marketing... unit tests, Selenium [testing], code reviews as a routine practice, reproducible server setup/deploys, etc. That sounds like the Joel test for 2012.
  - skrebbel 14y ago
    That sounds like the Joel test for webapp firms in 2012.
- alinajaf 14y ago
  > A/b testing, using usage metrics to drive decisions, customer development, lifecycle email marketing Yes, but I've been pleasantly surprised at the clueyness of of more than a few big-name clients recently. > Or for more dev focused stuff, unit tests, Selenium ("the best technology that I'll never use"), code reviews as a routine practice, reproducible server setup/deploys, etc. For deployments at least I've been blown away by the range of configurations I've seen. Everything from "git push heroku master" to "ssh into these three machines, run svn up, restart apache, merge dev couchdb to live, copy paste these 80 sql patches out of a file into the terminal, sacrifice a hamster to Amon Ra and cross your fingers."
edwinnathaniel 14y ago
Many companies favor short-term win or Money over Technicality because there aren't many people who have the skill to put the numbers on Excel spreadsheet the amount of money the company can save had we done things "Right" at the first time during the meeting between Business and Tech team (Product Manager, CEO vs Team Lead/Dev Manager). Basically it comes down to quantifying these quality works over developing new features. I had the pleasure to work in a very small tight-knit team where you have to have the bulldog to bark at people when they're slowing down their defense (i.e.: writing unit-tests, making sure build script is up to date). Results: on-time, within-budget, 1 Saturday, 1 Statutory holiday, 2 days OT until 10.30PM (all re-imburse afterwards) for a rather ambitious project to be shipped in 6 months from zero to live in 2 months afterwards. Oh, and we have 1 performance bug after the release (easy one to solve) and 1 requirement bug 3 months after release. The rest (been live for about a year) was as smooth as the baby skin. We rotate the BB phone (on-call) but we never had it buzzed. Our saving graces were: 0) Management Support 1) Continuous Integration 2) Unit-Testing We use these unit-tests as REPL mostly, so we don't have to re-compile and re-deploy the _whole_ app to GlassFish.
- spuz 14y ago
  We practice 1 and 2 but not with the same smooth as baby skin releases. Could you explain what you mean by 0 and how it helps you?
  - edwinnathaniel 14y ago
    Basically the people above the developers trust our judgment to work on these non-feature tasks: fixing build script (we use Maven but when it first setup, it didn't handle deployment to multi-environment that well), improving the component (we were newly created team, thus the reference of bulldog that bark) by either re-writing it or adding more unit-tests. The management almost never questioned us except one or two situations.

bguthrie 14y ago

I fundamentally agree with your point: changing processes at large software teams is really, really hard, risky, and expensive. But if I may pick a nit about something I think you were describing as a potential process improvement, per-team branches have been tried at scale many times before, both with and without Git. Many smart people I know who've had to clean up subsequent messes believe, counterintuitively, that the larger and more complex you get, the more branches hurt you, because they encourage you to delay integration, and integration is painful. Git merge capability, while excellent, can't save you if two teams independently decide to rename a method that both rely on; that's still a conflict. I've heard of several other large shops relying successfully on toggles with a single mainline. Fowler describes this pain in terms of feature branches rather than team branches, but it's the same idea: http://martinfowler.com/bliki/FeatureBranch.html http://martinfowler.com/bliki/FeatureBranch.html Branching is a poor substitute for genuine modularity.

patio11 14y ago
I'll buy that -- I was just picking an example of a process tweak which sounded like a potential improvement that, absent scale, would be accomplished in two days with a project roadmap that would fit in a blog post.
Tloewald 14y ago
Nice article btw. The approach to integration recommended at the end seems exactly like something that might make the management of such a large code base easier.
alinajaf 14y ago
IME depends on what we're talking about. I've been at workplaces where we have multiple epic branches that diverge for three or four iterations at a time. In this scenario, it doesn't matter what VCS you happen to be using, integration is going to be painful and error-prone. More recently the branch and pull-request into master model has been working really nicely on a per-story basis, as long as people are considerate and mergeback from master before initiating the request. Overall though, I think the only branch that gets deployed anywhere should be master (or trunk if you prefer). Like you say, it's a terrible substitute for a system built of decoupled components.