5 ms·

I agree with most of these, and vehemently disagree with a couple. I think we move pretty damn fast, we average 5 commits per dev per day. I very much agree wi

I agree with most of these, and vehemently disagree with a couple. I think we move pretty damn fast, we average 5 commits per dev per day. I very much agree with break features into many small pull requests. We merge constantly even (especially) if the code path won't be hit yet. Just get it into production. Makes the code review faster, the complexity lower, and lets you focus on one thing at a time. I think no tests makes you slower. We paid a bit of cost to start writing tests, but now we can deploy 10 times a day because we know the mission critical stuff is working. It's more about testing the right stuff. Decide what can break, and leave it untested. What can't break should be covered by tests. We do stuff with money, all that is robustly tested. The UI isn't. Code freezing / weekly releasing is counter to CI/CD. Just make your deployment take less than 10 minutes, and ship code constantly.

brightball 4y ago
Agree with this strongly. Weekly releases reduce efficiency and lead to more complex production issues as you work to unpack which part of the batch of things you just deployed is causing a problem. Use feature flags to decouple Deployment from Release and you’re fine.
- Gigachad 4y ago
  Feature flags are good but I hate how they enable weird customer specific workflows. Imo the mindset should be that all feature flags will eventually be removed when testing is finished. But management sees the capability and says add a flag for customer Y that changes how this page works.
  - brightball 4y ago
    Agreed. IMO the best approach is a flag that can let you enable for specific customers (alpha or internal sample customers), all beta customers and then all customers. And after it’s enabled for everybody, remove the flag.
    - beaker52 4y ago
      It's not Done until the flag is removed.
SkyPuncher 4y ago
> Just make your deployment take less than 10 minutes This is great until it's not. We're starting to hit this tipping point on our code-base. The challenge for us is it'd likely take an engineer a week or two to optimize our test suites. That's enough time for them to ship meaningful features. Instead, we do occasional freezes for important releases. Not sustainable, but it's a good enough bridge.
- Gigachad 4y ago
  It’s usually cheaper to just add another CI runner to run the tests until the CI runners cost more than the dev time to speed it up.
  - deathanatos 4y ago
    The slowness in my tests comes not from having too few runners (we just dynamically spin up a runner for each commit, and it normally takes only a few seconds for that), but from serialization in the CI pipeline. You cannot run tests until the code has been built; no extra runner can change that. I need dev time to make the builds understand their inputs, and then cache more aggressively than we are, or something. (But I lack the dev time.)
    - chrsig 4y ago
      pipeline time is something that I spend a non trivial amount of effort on. caching and promoting images is an absolute must. caching images has it's downsides, too though. it can be a longer time period between when something in the dockerfile breaks and when it's noticed. generally speaking, my pipeline is generally 10-40% build time, 50-60% integration & system test time, 10% deployment time The system tests that're run deploy the entire dependency graph of services that my project depends on, which can take a bit to settle, has a high chance of transient failures relative to other parts of the pipeline, and notably has a fixed test run time. That last bit really raises the lower bound for how long the pipeline as a whole takes -- number of runners doesn't matter. The pipeline generally takes between 25-45 minutes, and runs on each push and is triggered by changes to dependencies to ensure backwards compatibility. I wouldn't do anything to make it take longer than it does now, and it can be frustrating. but the pipeline will catch things that would otherwise slip through the cracks.
osigurdson 4y ago
>> I think no tests makes you slower Bad tests can absolutely make you slower. Good tests can be existential.
- Gigachad 4y ago
  This might be tied in to the language ecosystem of tooling but after many jobs, I’m yet to see bad tests in ruby. Perhaps there are some CPU cycles wasted on the CI for not so useful tests, but it’s easily paid back by the mountains of useful tests.
  - osigurdson 4y ago
    A simple example of a bad test is one enforces an incorrect behaviour (i.e. 1 + 1 == 3). I can't see how a language could improve this situation. Is it possible it is the community as opposed to the language itself?
    - Gigachad 4y ago
      I’m not sure how this really slows you down though. You see the app code is wrong, it breaks a test so you update the test to the new expected result. The benefit is that the tests highlight what changed in the app. So when you update a library and 1+1 changes, it’s an alarm bell to investigate further because you didn’t expect this to happen.
      - osigurdson 4y ago
        This is an example the test/hash antipattern. Here, the tests approach a hash of the source code. When some code changes, the hash no longer matches which fails the test. Great, this is letting us know that something as changed. After examining the changes and decide that they are correct, we update the hash and now the all the tests are green again. Our code base has 100% coverage.
        abracadabra_ 4y ago
beaker52 4y ago
They had me until they said deploy weekly. Deployments shouldn't be an event. Deployments should happen constantly. Get your stuff into production immediately. All the other practices up to that point support that.
- s1k3 4y ago
  You’re not deploying an iOS app daily. At best it’s weekly. It’s really context dependent. Weekly is still an exceedingly short clip.
- packetlost 4y ago
  As someone working on a hardware project... hahahahah We manage daily, but even that is a miracle. If you're building enterprise CRUD app #310979707 or SaaS project #2009879, sure, deploy constantly. But a very significant number of projects simply cannot do that.
- gjulianm 4y ago
  Deployments depend on what you're deploying. A web app, maybe that's ok. A desktop application with automatic updates can be updated daily at most. Other products without automatic updates are probably better served with longer release cycle.
- beaker52 4y ago
  There are several comments about iOS, desktop or hardware environments that may require different release cycles. All of which are completely valid. I see that I had assumed we were talking about web apps. This assumption narrowed my perspective, and was an error in thinking on my part.
gbersac 4y ago
> we average 5 commits per dev per day. Does that mean anything?
- kareemsabri 4y ago
  Obviously I think it does. Why do you think it doesn't?
  - mewse 4y ago
    I'm not the original asker, but evaluating programmers based upon the number of commits per day is a lot like evaluating them based on the number of lines of code written per day -- it's a measure of activity, rather than a measure of productivity, and those often aren't the same thing. (I'm not trying to imply that that's a mistake you've made; it's just something that it's easy to forget as a manager of programmers, and is maybe worth pointing out explicitly; you need a way to measure the team's productivity, but it's very hard to find a good, measurable proxy for 'productivity' which isn't eventually going to end up being gamed by the whole team!)
    - kareemsabri 4y ago
      I think that's fair. But I would say the article is about activity (shipping features fast) and so as a measure of activity commits per day is a decent proxy. Obviously the devil is in the details. I would not judge an individual developer's productivity on commits per day, agree that would be foolish.
anorsich 4y ago
I feel like I didn't give enough context (I've written that essay) to the kind of projects that was used to test what is written. 1. Pre Series A companies, which didn't have strong engineering team from start and now struggle to ship. 2. Idea/Pre Seed companies, which just want to ship fast and raise funds. That kind of founder usually have 'friends & fools' budget. 1. First kind of projects usually don't have enough tests coverage to deploy safely more often. Typically I would set a goal to deploy twice a week in 6-12 months and build a plan of how to achieve that. 2. Usually I advice to not write lots of tests for the first ~6 months to focus on shipping faster. In my experience products often change dramatically during this period. But a) there are exceptions. Some domains (e.x. finance) are critical to write more tests. b) tests infrastructure are always set as part of CI/CD process, so engineers can decide and write them for area of high importance.
P5fRxh5kUvp2th 4y ago
> It's more about testing the right stuff. This is the important point. Anyone talking about code coverage percentages is already making a mistake. percentage of coverage isn't nearly as important as WHAT is covered, and it's not a good proxy for it either.

brightball 4y ago

Agree with this strongly. Weekly releases reduce efficiency and lead to more complex production issues as you work to unpack which part of the batch of things you just deployed is causing a problem. Use feature flags to decouple Deployment from Release and you’re fine.

Gigachad 4y ago
Feature flags are good but I hate how they enable weird customer specific workflows. Imo the mindset should be that all feature flags will eventually be removed when testing is finished. But management sees the capability and says add a flag for customer Y that changes how this page works.
- brightball 4y ago
  Agreed. IMO the best approach is a flag that can let you enable for specific customers (alpha or internal sample customers), all beta customers and then all customers. And after it’s enabled for everybody, remove the flag.
  - beaker52 4y ago
    It's not Done until the flag is removed.

SkyPuncher 4y ago

> Just make your deployment take less than 10 minutes This is great until it's not. We're starting to hit this tipping point on our code-base. The challenge for us is it'd likely take an engineer a week or two to optimize our test suites. That's enough time for them to ship meaningful features. Instead, we do occasional freezes for important releases. Not sustainable, but it's a good enough bridge.

Gigachad 4y ago
It’s usually cheaper to just add another CI runner to run the tests until the CI runners cost more than the dev time to speed it up.
- deathanatos 4y ago
  The slowness in my tests comes not from having too few runners (we just dynamically spin up a runner for each commit, and it normally takes only a few seconds for that), but from serialization in the CI pipeline. You cannot run tests until the code has been built; no extra runner can change that. I need dev time to make the builds understand their inputs, and then cache more aggressively than we are, or something. (But I lack the dev time.)
  - chrsig 4y ago
    pipeline time is something that I spend a non trivial amount of effort on. caching and promoting images is an absolute must. caching images has it's downsides, too though. it can be a longer time period between when something in the dockerfile breaks and when it's noticed. generally speaking, my pipeline is generally 10-40% build time, 50-60% integration & system test time, 10% deployment time The system tests that're run deploy the entire dependency graph of services that my project depends on, which can take a bit to settle, has a high chance of transient failures relative to other parts of the pipeline, and notably has a fixed test run time. That last bit really raises the lower bound for how long the pipeline as a whole takes -- number of runners doesn't matter. The pipeline generally takes between 25-45 minutes, and runs on each push and is triggered by changes to dependencies to ensure backwards compatibility. I wouldn't do anything to make it take longer than it does now, and it can be frustrating. but the pipeline will catch things that would otherwise slip through the cracks.

osigurdson 4y ago

>> I think no tests makes you slower Bad tests can absolutely make you slower. Good tests can be existential.

Gigachad 4y ago
This might be tied in to the language ecosystem of tooling but after many jobs, I’m yet to see bad tests in ruby. Perhaps there are some CPU cycles wasted on the CI for not so useful tests, but it’s easily paid back by the mountains of useful tests.
- osigurdson 4y ago
  A simple example of a bad test is one enforces an incorrect behaviour (i.e. 1 + 1 == 3). I can't see how a language could improve this situation. Is it possible it is the community as opposed to the language itself?
  - Gigachad 4y ago
    I’m not sure how this really slows you down though. You see the app code is wrong, it breaks a test so you update the test to the new expected result. The benefit is that the tests highlight what changed in the app. So when you update a library and 1+1 changes, it’s an alarm bell to investigate further because you didn’t expect this to happen.
    - osigurdson 4y ago
      This is an example the test/hash antipattern. Here, the tests approach a hash of the source code. When some code changes, the hash no longer matches which fails the test. Great, this is letting us know that something as changed. After examining the changes and decide that they are correct, we update the hash and now the all the tests are green again. Our code base has 100% coverage.
      - abracadabra_ 4y ago

beaker52 4y ago

They had me until they said deploy weekly. Deployments shouldn't be an event. Deployments should happen constantly. Get your stuff into production immediately. All the other practices up to that point support that.

s1k3 4y ago
You’re not deploying an iOS app daily. At best it’s weekly. It’s really context dependent. Weekly is still an exceedingly short clip.
packetlost 4y ago
As someone working on a hardware project... hahahahah We manage daily, but even that is a miracle. If you're building enterprise CRUD app #310979707 or SaaS project #2009879, sure, deploy constantly. But a very significant number of projects simply cannot do that.
gjulianm 4y ago
Deployments depend on what you're deploying. A web app, maybe that's ok. A desktop application with automatic updates can be updated daily at most. Other products without automatic updates are probably better served with longer release cycle.
beaker52 4y ago
There are several comments about iOS, desktop or hardware environments that may require different release cycles. All of which are completely valid. I see that I had assumed we were talking about web apps. This assumption narrowed my perspective, and was an error in thinking on my part.

gbersac 4y ago

> we average 5 commits per dev per day. Does that mean anything?

kareemsabri 4y ago
Obviously I think it does. Why do you think it doesn't?
- mewse 4y ago
  I'm not the original asker, but evaluating programmers based upon the number of commits per day is a lot like evaluating them based on the number of lines of code written per day -- it's a measure of activity, rather than a measure of productivity, and those often aren't the same thing. (I'm not trying to imply that that's a mistake you've made; it's just something that it's easy to forget as a manager of programmers, and is maybe worth pointing out explicitly; you need a way to measure the team's productivity, but it's very hard to find a good, measurable proxy for 'productivity' which isn't eventually going to end up being gamed by the whole team!)
  - kareemsabri 4y ago
    I think that's fair. But I would say the article is about activity (shipping features fast) and so as a measure of activity commits per day is a decent proxy. Obviously the devil is in the details. I would not judge an individual developer's productivity on commits per day, agree that would be foolish.

anorsich 4y ago

I feel like I didn't give enough context (I've written that essay) to the kind of projects that was used to test what is written. 1. Pre Series A companies, which didn't have strong engineering team from start and now struggle to ship. 2. Idea/Pre Seed companies, which just want to ship fast and raise funds. That kind of founder usually have 'friends & fools' budget. 1. First kind of projects usually don't have enough tests coverage to deploy safely more often. Typically I would set a goal to deploy twice a week in 6-12 months and build a plan of how to achieve that. 2. Usually I advice to not write lots of tests for the first ~6 months to focus on shipping faster. In my experience products often change dramatically during this period. But a) there are exceptions. Some domains (e.x. finance) are critical to write more tests. b) tests infrastructure are always set as part of CI/CD process, so engineers can decide and write them for area of high importance.

P5fRxh5kUvp2th 4y ago

> It's more about testing the right stuff. This is the important point. Anyone talking about code coverage percentages is already making a mistake. percentage of coverage isn't nearly as important as WHAT is covered, and it's not a good proxy for it either.