5 ms·

Maybe just a personal thing, but going to a github repo and seeing "build failing" and low test coverage icons are usually turn offs for me to continue reading.

mholt 6y ago
Why? It's normal for builds to fail between releases. And (warning: controversial opinion inbound) code coverage is hardly a useful metric -- one can write a single test case that gets 100% coverage, but doesn't test anything; what you want is assertion coverage but I don't know how possible that is.
- sneak 6y ago
  Is it normal for builds to fail on master between releases? I would think a green build CI should be required for merge to master, even if perhaps the tests don’t all pass.
  - mholt 6y ago
    Those badges are also often useless. For example, here's their current failure reason: Bad response status from coveralls: 422 {"message":"service_job_id (716381595) must be unique for Travis Jobs not supplying a Coveralls Repo Token","error":true} Has nothing to do with the actual compilation status. In projects I'm involved with, the vast majority of CI errors were due to stupid things, not actual code problems. For example, last week our tests were failing because the east coast IBM data center that ran the tests was offline due to extended power outages from the weather.
    - jnwatson 6y ago
      Like all metrics, badges are just a model. However, for open source projects, optics matter, and failing badges are a hint that perhaps quality isn't the top priority.
- jnwatson 6y ago
  It strongly depends on the branch style. An older style makes the main/master branch to be the dev branch, allowing temporary deviations from working. If you want a working branch, you have to pull a release branch. The modern trend is that master is sacred and should always pass.
- reggieband 6y ago
  > It's normal for builds to fail between releases. Why justify this with some categorical normative? In the last 7 or 8 years of my development life the master branch on every project I've worked on has always been clean. Breaking changes live in a branch and those branches are becoming shorter and shorter lived (previously lived for months, then weeks, now days). That doesn't necessarily mean master is at all time ready to release (although some hold that extreme position). But it does mean that whenever you want to start work on some new feature you can be sure that branching from master is safe. And rebasing against master at any point should be safe. In fact, almost all repos I've worked with recently explicitly deny merges into master unless a suite of tests pass, including basic builds, unit tests, and static code linting. The very idea that I could ever pull master and get a build failure makes me shudder.
eat_veggies 6y ago
Is 74% low? The 70-80% range feels about average, and decently acceptable for most projects.
- deleted 6y ago
  [deleted]
- eikenberry 6y ago
  74% isn't low, that's right in the sweet spot. I generally shoot for 65-85%. Lower and you miss important stuff, higher and you start testing implementation instead of APIs and the tests become to brittle.

mholt 6y ago

Why? It's normal for builds to fail between releases. And (warning: controversial opinion inbound) code coverage is hardly a useful metric -- one can write a single test case that gets 100% coverage, but doesn't test anything; what you want is assertion coverage but I don't know how possible that is.

sneak 6y ago
Is it normal for builds to fail on master between releases? I would think a green build CI should be required for merge to master, even if perhaps the tests don’t all pass.
- mholt 6y ago
  Those badges are also often useless. For example, here's their current failure reason: Bad response status from coveralls: 422 {"message":"service_job_id (716381595) must be unique for Travis Jobs not supplying a Coveralls Repo Token","error":true} Has nothing to do with the actual compilation status. In projects I'm involved with, the vast majority of CI errors were due to stupid things, not actual code problems. For example, last week our tests were failing because the east coast IBM data center that ran the tests was offline due to extended power outages from the weather.
  - jnwatson 6y ago
    Like all metrics, badges are just a model. However, for open source projects, optics matter, and failing badges are a hint that perhaps quality isn't the top priority.
jnwatson 6y ago
It strongly depends on the branch style. An older style makes the main/master branch to be the dev branch, allowing temporary deviations from working. If you want a working branch, you have to pull a release branch. The modern trend is that master is sacred and should always pass.
reggieband 6y ago
> It's normal for builds to fail between releases. Why justify this with some categorical normative? In the last 7 or 8 years of my development life the master branch on every project I've worked on has always been clean. Breaking changes live in a branch and those branches are becoming shorter and shorter lived (previously lived for months, then weeks, now days). That doesn't necessarily mean master is at all time ready to release (although some hold that extreme position). But it does mean that whenever you want to start work on some new feature you can be sure that branching from master is safe. And rebasing against master at any point should be safe. In fact, almost all repos I've worked with recently explicitly deny merges into master unless a suite of tests pass, including basic builds, unit tests, and static code linting. The very idea that I could ever pull master and get a build failure makes me shudder.

eat_veggies 6y ago

Is 74% low? The 70-80% range feels about average, and decently acceptable for most projects.

deleted 6y ago
[deleted]
eikenberry 6y ago
74% isn't low, that's right in the sweet spot. I generally shoot for 65-85%. Lower and you miss important stuff, higher and you start testing implementation instead of APIs and the tests become to brittle.