7 ms·

Our Development Workflow

89 points by martindale 12y ago

dkarapetyan 12y ago
This is the first time I've ever seen a write up of a development/deployment process that makes any sense. Good on ZenPayroll for having their fundamentals straight.
kyberias 12y ago
I feel it's slightly concerning that the same text contains: "There’s no room for error." and "Someone on the product team will usually play with the feature on our staging environment trying their best to break things." The first sentence seems to imply that they try to maintain the highest quality possible. Yet the second sentence does not really fit that idea. The article pretty much only lists the tools they use. But on a scale from Random PHP site to NASA space shuttle software, where is zenpayroll located?
- adamnemecek 12y ago
  One presumably talks about the production environment, the other about staging environment.
  - superuser2 12y ago
    If you truly have "no room for error" (i.e. you are shipping code to a hundred-billion-dollar space vehicle), you might have: - Very strict coding guidelines - Rigorous code review - An extremely thorough test suite - Multi-stage approval through a QA process involving many pairs of eyes and thorough, formally defined and scientifically rigorous test procedures - Static analysis and possibly even algorithms proved correct in Coq Simply playing with it in staging for a while and determining that everything looks good is NOT the kind of testing that you do when there's "no room for error."
    - adamnemecek 12y ago
      No room for error probably means one thing to people working on a $10^11 space vehicle and something else to webdevs. Does that surprise you?
      - BillinghamJ 12y ago
        As a statement, it is absolute.
        markc 12y ago
        As a statement, it is meaningless.
      - superuser2 12y ago
        Webdevs (particularly on HN) are highly vocal about TDD/automated testing in general and there is a great deal of tooling around testing Rails and Django. Code review is also part of the fairly dated Joel Test and widely regarded on HN as a good idea. These are substantially higher bars than "play with it for a while and see if you can break anything."
    - maninalift 12y ago
      even if software is formally verified, and even assuming the specification is both totally accurate and complete, there is plenty of room for issues at software deployment in a multi-component system such as this
  - freshflowers 12y ago
    Even so, imho on a staging environment everything should be ready to deploy barring either functional acceptance or very, very unforeseen technical issues that didn't show up systems that mimic production just a tad less closely. Staging = "ready to roll", not "still needs testing to see if things break". Things shouldn't break on staging. Of course you can forgo manual QA if your automated testing is so perfect that staging is just a formality. I'm guessing ZenPayroll has that covered, given that they deploy to production "several times a week". So I assume "trying their best to break things" is just a flippant description of final acceptance, not QA testing.
- snowwrestler 12y ago
  To be fair, the "play" sentence comes after a whole bunch of description of spec writing, test coverage, code reviews, and automated test suites including using anonymized real data. Doing a final human "check it out" step doesn't invalidate all that, and could be helpful.
_mikz 12y ago
Nothing special. It is just normal workflow when you use common sense, no? Except the development branch because master is new development branch.
dlss 12y ago
Error 1008 for me.
elwell 12y ago
> The name of this test server is Leeroy, who does all this with the help of Jenkins.
michaelgrafl 12y ago
Point 2 (Branch off development) confuses me. At my workplace we have branches for dev, staging, and production. We're working on feature branches. Now if my coworker merges feature A into dev, and I merge feature B into dev, and feature A is done and ready for release, but feature B needs more work, and my coworker checks out a new feature branch C from dev, he'll be unable to merge into production without merging in my unfinished feature B. We branch off of production so this won't happen. Are we doing it wrong?
- ryanmacg 12y ago
  In theory yes, a feature that requires more work shouldn’t make it’s way into dev really.
  - michaelgrafl 12y ago
    But I don't always know it will require more work.
- rorykoehein 12y ago
  Couldn't you merge the features that are done to your staging branch and from there merge to production? Your staging branch would be the 'release branch' in this model: http://nvie.com/posts/a-successful-git-branching-model/ http://nvie.com/posts/a-successful-git-branching-model/
  - michaelgrafl 12y ago
    Thanks for the link. We merge feature into dev when we want to show a feature to a fellow developer. We merge feature into staging when a feature is ready to be tested by non-developers and to be included in documentation. We merge feature into production when it's ready for all users. Has been working out well so far.
    - IanCal 12y ago
      > We merge feature into dev when we want to show a feature to a fellow developer. Wouldn't you just show them the feature branch?
deleted 12y ago
[deleted]
ktzar 12y ago
Testing after merging with develop is a very, very bad practise. And there're plenty of tools to help you don't do that, just kick a build to run all your tests, which will add a tick or a comment to the branch's PR. And never merge anything without that tick or comment.
- yegor256a 12y ago
  Totally agree. Testing should be done BEFORE merging, like this article explains: http://www.yegor256.com/2014/07/21/read-only-master-branch.html http://www.yegor256.com/2014/07/21/read-only-master-branch.h...
TorKlingberg 12y ago
This looks like a very sensible workflow in general. The one thing that worries me is long lived feature branches. This workflow is fine as long as features never take more than a few weeks. If it takes longer you end up with bugs that have been fixed on the development branch still appearing on various feature branches. Also if you do a lot of refactoring on a feature branch while development goes on in parallel there will be difficult merges. It is especially problematic when work on a feature stops for some reason and an old branch sits around for months. In general code should be merged to the development branch as soon as it is working, tested and reviewed, even if not all the functionality of the feature is done.
- rimantas 12y ago
  Did you miss the part about rebasing?
  - TorKlingberg 12y ago
    It looks like rebasing is only done after the feature is complete. Maybe that was just a simplification.
    - rimantas 12y ago
      Why does it matter? All the fixes will already be there after rebase anyway.
      - pionar 12y ago
        The point is that if you're not continually merging development down into your feature branch, then rebasing becomes overly difficult. Merging can introduce bugs if not done correctly. At my company, we eschewed feature branches about a year ago. We have two branches - development and main, with a branch ("tag" in git speak, we use TFS) created for each release of our product (we're a software company, not a service company). Development happens in the development branch, and is merged up into the main branch after code freeze. This does not preclude us from releasing often. With judicious use of feature toggles, it's simple to release with a feature that's not done. Feature branches would just create a mess for us as there's necessarily many communicating parts between teams, and the number of feature branches required to keep them all in sync would be exponential.
jiraaya 12y ago
1. "Don’t optimize for the short-term" vs "Keep it simple and straightforward". Often the simplest thing may not work in the long run. But its better to err on the side of keeping things simple. 2. Code reviews + off mainline development is a potential disaster in waiting unless the review process is fast. The fastest review process I have seen is automated testing and pairing when someone is working on something that cannot be caught easily by automated tests like synchronization.
- mtdewcmu 12y ago
  >> 1. "Don’t optimize for the short-term" vs "Keep it simple and straightforward". >> Often the simplest thing may not work in the long run. But its better to err on the side of keeping things simple. IMO, the best reason for simplicity is that simple things can be torn out and replaced more easily than complex things. So the simplest thing isn't necessarily right, but you haven't wasted much resources by choosing it, so you've left your options open. You always want to have options.
  - jiraaya 12y ago
    Ah, that's what I meant by "erring on the side of keeping things simple". It is okay if you make decisions that are not optimal in the long run, but are simple since they can be torn apart.
securingsincity 12y ago
Dedicated QA team? Person? Feels like they are missing a crucial piece of non-devs testing and accepting features. Only my opinion but in my experience developers have a different definition of done then non developers so having both QA and even some user acceptance testing goes a long way.
- securingsincity 12y ago
  Also this article is a year old. Can we get a title change? and maybe someone from ZenPayroll can chime in how their process has changed, if it has?
jorgeleo 12y ago
This was an infomercial in the form of a blog... dissapointing
athenot 12y ago
And THIS is how you write a job posting. Explaining the setup and perhaps some of the culture lets potential candidates know right away if that's an environment matching their style. (And it doubles as an interesting article too.) Disclaimer: satisfied ZenPayroll user here.
ricknew 12y ago
No usability testing?
- acveilleux 12y ago
  Presumably this happens at the spec stage. At least it's implied.
eitally 12y ago
Reading things like this make me long for the simplistic development environment of a web startup. Almost none of this would fly in an "Enterprise IT" environment, except in the very rare case where the team is legitimately building (and owning) a product rather than creating business process optimization crutches.

89 points by martindale 12y ago

This is the first time I've ever seen a write up of a development/deployment process that makes any sense. Good on ZenPayroll for having their fundamentals straight.

kyberias 12y ago

I feel it's slightly concerning that the same text contains: "There’s no room for error." and "Someone on the product team will usually play with the feature on our staging environment trying their best to break things." The first sentence seems to imply that they try to maintain the highest quality possible. Yet the second sentence does not really fit that idea. The article pretty much only lists the tools they use. But on a scale from Random PHP site to NASA space shuttle software, where is zenpayroll located?

adamnemecek 12y ago
One presumably talks about the production environment, the other about staging environment.
- superuser2 12y ago
  If you truly have "no room for error" (i.e. you are shipping code to a hundred-billion-dollar space vehicle), you might have: - Very strict coding guidelines - Rigorous code review - An extremely thorough test suite - Multi-stage approval through a QA process involving many pairs of eyes and thorough, formally defined and scientifically rigorous test procedures - Static analysis and possibly even algorithms proved correct in Coq Simply playing with it in staging for a while and determining that everything looks good is NOT the kind of testing that you do when there's "no room for error."
  - adamnemecek 12y ago
    No room for error probably means one thing to people working on a $10^11 space vehicle and something else to webdevs. Does that surprise you?
    - BillinghamJ 12y ago
      As a statement, it is absolute.
      - markc 12y ago
        As a statement, it is meaningless.
    - superuser2 12y ago
      Webdevs (particularly on HN) are highly vocal about TDD/automated testing in general and there is a great deal of tooling around testing Rails and Django. Code review is also part of the fairly dated Joel Test and widely regarded on HN as a good idea. These are substantially higher bars than "play with it for a while and see if you can break anything."
  - maninalift 12y ago
    even if software is formally verified, and even assuming the specification is both totally accurate and complete, there is plenty of room for issues at software deployment in a multi-component system such as this
- freshflowers 12y ago
  Even so, imho on a staging environment everything should be ready to deploy barring either functional acceptance or very, very unforeseen technical issues that didn't show up systems that mimic production just a tad less closely. Staging = "ready to roll", not "still needs testing to see if things break". Things shouldn't break on staging. Of course you can forgo manual QA if your automated testing is so perfect that staging is just a formality. I'm guessing ZenPayroll has that covered, given that they deploy to production "several times a week". So I assume "trying their best to break things" is just a flippant description of final acceptance, not QA testing.
snowwrestler 12y ago
To be fair, the "play" sentence comes after a whole bunch of description of spec writing, test coverage, code reviews, and automated test suites including using anonymized real data. Doing a final human "check it out" step doesn't invalidate all that, and could be helpful.

_mikz 12y ago

Nothing special. It is just normal workflow when you use common sense, no? Except the development branch because master is new development branch.

dlss 12y ago

Error 1008 for me.

elwell 12y ago

> The name of this test server is Leeroy, who does all this with the help of Jenkins.

michaelgrafl 12y ago

Point 2 (Branch off development) confuses me. At my workplace we have branches for dev, staging, and production. We're working on feature branches. Now if my coworker merges feature A into dev, and I merge feature B into dev, and feature A is done and ready for release, but feature B needs more work, and my coworker checks out a new feature branch C from dev, he'll be unable to merge into production without merging in my unfinished feature B. We branch off of production so this won't happen. Are we doing it wrong?

ryanmacg 12y ago
In theory yes, a feature that requires more work shouldn’t make it’s way into dev really.
- michaelgrafl 12y ago
  But I don't always know it will require more work.
rorykoehein 12y ago
Couldn't you merge the features that are done to your staging branch and from there merge to production? Your staging branch would be the 'release branch' in this model: http://nvie.com/posts/a-successful-git-branching-model/ http://nvie.com/posts/a-successful-git-branching-model/
- michaelgrafl 12y ago
  Thanks for the link. We merge feature into dev when we want to show a feature to a fellow developer. We merge feature into staging when a feature is ready to be tested by non-developers and to be included in documentation. We merge feature into production when it's ready for all users. Has been working out well so far.
  - IanCal 12y ago
    > We merge feature into dev when we want to show a feature to a fellow developer. Wouldn't you just show them the feature branch?

deleted 12y ago

[deleted]

ktzar 12y ago

Testing after merging with develop is a very, very bad practise. And there're plenty of tools to help you don't do that, just kick a build to run all your tests, which will add a tick or a comment to the branch's PR. And never merge anything without that tick or comment.

yegor256a 12y ago
Totally agree. Testing should be done BEFORE merging, like this article explains: http://www.yegor256.com/2014/07/21/read-only-master-branch.html http://www.yegor256.com/2014/07/21/read-only-master-branch.h...

TorKlingberg 12y ago

This looks like a very sensible workflow in general. The one thing that worries me is long lived feature branches. This workflow is fine as long as features never take more than a few weeks. If it takes longer you end up with bugs that have been fixed on the development branch still appearing on various feature branches. Also if you do a lot of refactoring on a feature branch while development goes on in parallel there will be difficult merges. It is especially problematic when work on a feature stops for some reason and an old branch sits around for months. In general code should be merged to the development branch as soon as it is working, tested and reviewed, even if not all the functionality of the feature is done.

rimantas 12y ago
Did you miss the part about rebasing?
- TorKlingberg 12y ago
  It looks like rebasing is only done after the feature is complete. Maybe that was just a simplification.
  - rimantas 12y ago
    Why does it matter? All the fixes will already be there after rebase anyway.
    - pionar 12y ago
      The point is that if you're not continually merging development down into your feature branch, then rebasing becomes overly difficult. Merging can introduce bugs if not done correctly. At my company, we eschewed feature branches about a year ago. We have two branches - development and main, with a branch ("tag" in git speak, we use TFS) created for each release of our product (we're a software company, not a service company). Development happens in the development branch, and is merged up into the main branch after code freeze. This does not preclude us from releasing often. With judicious use of feature toggles, it's simple to release with a feature that's not done. Feature branches would just create a mess for us as there's necessarily many communicating parts between teams, and the number of feature branches required to keep them all in sync would be exponential.

jiraaya 12y ago

1. "Don’t optimize for the short-term" vs "Keep it simple and straightforward". Often the simplest thing may not work in the long run. But its better to err on the side of keeping things simple. 2. Code reviews + off mainline development is a potential disaster in waiting unless the review process is fast. The fastest review process I have seen is automated testing and pairing when someone is working on something that cannot be caught easily by automated tests like synchronization.

mtdewcmu 12y ago
>> 1. "Don’t optimize for the short-term" vs "Keep it simple and straightforward". >> Often the simplest thing may not work in the long run. But its better to err on the side of keeping things simple. IMO, the best reason for simplicity is that simple things can be torn out and replaced more easily than complex things. So the simplest thing isn't necessarily right, but you haven't wasted much resources by choosing it, so you've left your options open. You always want to have options.
- jiraaya 12y ago
  Ah, that's what I meant by "erring on the side of keeping things simple". It is okay if you make decisions that are not optimal in the long run, but are simple since they can be torn apart.

securingsincity 12y ago

Dedicated QA team? Person? Feels like they are missing a crucial piece of non-devs testing and accepting features. Only my opinion but in my experience developers have a different definition of done then non developers so having both QA and even some user acceptance testing goes a long way.

securingsincity 12y ago
Also this article is a year old. Can we get a title change? and maybe someone from ZenPayroll can chime in how their process has changed, if it has?

jorgeleo 12y ago

This was an infomercial in the form of a blog... dissapointing

athenot 12y ago

And THIS is how you write a job posting. Explaining the setup and perhaps some of the culture lets potential candidates know right away if that's an environment matching their style. (And it doubles as an interesting article too.) Disclaimer: satisfied ZenPayroll user here.

ricknew 12y ago

No usability testing?

acveilleux 12y ago
Presumably this happens at the spec stage. At least it's implied.

eitally 12y ago

Reading things like this make me long for the simplistic development environment of a web startup. Almost none of this would fly in an "Enterprise IT" environment, except in the very rare case where the team is legitimately building (and owning) a product rather than creating business process optimization crutches.