4 ms·

At every single company I've ever worked the company issued laptop has been significantly faster than the machines provisioned for CI (i.e. m1 mac vs Github act

At every single company I've ever worked the company issued laptop has been significantly faster than the machines provisioned for CI (i.e. m1 mac vs Github action free tier runners). Consequently I don't usually push code without running the tests locally, it's such a faster feedback loop. I've always wondered if it would be possible to design some proof of work concept where you could hash your filesystem + test artifacts to verify tests passed for a specific change. FWIW in yeas of development I've never had an issue where "it works on my machine" doesn't equate to the exact same result in CI.

mixmastamyk 3y ago
Maybe but one important reason to have it done externally is to confirm it works elsewhere.
- TylerE 3y ago
  One bug that's bitten me a time or two is the rather annoying case-preserving but not case-sensitive MacOS filesystem. This can mean a link works locally, but not when e.g. deployed to a linux server.
- Jackevansevo 3y ago
  It's a valid reason to have stuff run in CI (i.e. consistent environment). But for my line of work I can't think of a single scenario where the architecture / platform has ever caused issues in tests, typically it's caught at build time / when resolving dependencies/packages.
  - mixmastamyk 3y ago
    Forgetting to commit a file is pretty common. Non-existent folder, not updating out-of-tree config, etc.
- nextlevelwizard 3y ago
  Have easy way to run your tests in container. Then you don’t need to worry, unless you are deploying on strange hardware
krallja 3y ago
A few jobs ago, we didn't bother with a separate CI server, for exactly this reason. We just had the output of the local test run appended to our commit messages.
i13e 3y ago
pre-commit more or less does this. When you install locally as a git hook it will only test on changes that are being committed.
- KeepFlying 3y ago
  I've always liked the theory of this but not the implementation. I'm a big fan of squash merges so my branches are a mess of commi that each may not even build let alone pass tests. If I had to run tests each commit it would slow things down significantly for little benefit. I wish it was more nuanced
  - ForkMeOnTinder 3y ago
    I use pre-commit and have a git alias for those wip commits. cmnv = commit --no-verify
- Jackevansevo 3y ago
  But is it smart enough to go and test transient deps of code you've made changes to? I've been impressed by Pants (Python build tooling) which manages this really well https://www.pantsbuild.org/docs/advanced-target-selection#running-over-changed-files-with---changed-since https://www.pantsbuild.org/docs/advanced-target-selection#ru...
anonacct37 3y ago
I've seen pieces of this without the actual proof of work but with cryptographically hashed inputs and commands such that your test can just be a cache hit. https://bazel.build/remote/caching https://bazel.build/remote/caching
adrianN 3y ago
What kind of software do you develop? In my area, running all the tests on my laptop would take days and I frequently have issues where the tests passes locally but not in CI.
- Jackevansevo 3y ago
  Backend webdev in Python, mostly worked for startups. Nothing crazy complex.

mixmastamyk 3y ago

Maybe but one important reason to have it done externally is to confirm it works elsewhere.

TylerE 3y ago
One bug that's bitten me a time or two is the rather annoying case-preserving but not case-sensitive MacOS filesystem. This can mean a link works locally, but not when e.g. deployed to a linux server.
Jackevansevo 3y ago
It's a valid reason to have stuff run in CI (i.e. consistent environment). But for my line of work I can't think of a single scenario where the architecture / platform has ever caused issues in tests, typically it's caught at build time / when resolving dependencies/packages.
- mixmastamyk 3y ago
  Forgetting to commit a file is pretty common. Non-existent folder, not updating out-of-tree config, etc.
nextlevelwizard 3y ago
Have easy way to run your tests in container. Then you don’t need to worry, unless you are deploying on strange hardware

krallja 3y ago

A few jobs ago, we didn't bother with a separate CI server, for exactly this reason. We just had the output of the local test run appended to our commit messages.

i13e 3y ago

pre-commit more or less does this. When you install locally as a git hook it will only test on changes that are being committed.

KeepFlying 3y ago
I've always liked the theory of this but not the implementation. I'm a big fan of squash merges so my branches are a mess of commi that each may not even build let alone pass tests. If I had to run tests each commit it would slow things down significantly for little benefit. I wish it was more nuanced
- ForkMeOnTinder 3y ago
  I use pre-commit and have a git alias for those wip commits. cmnv = commit --no-verify
Jackevansevo 3y ago
But is it smart enough to go and test transient deps of code you've made changes to? I've been impressed by Pants (Python build tooling) which manages this really well https://www.pantsbuild.org/docs/advanced-target-selection#running-over-changed-files-with---changed-since https://www.pantsbuild.org/docs/advanced-target-selection#ru...

anonacct37 3y ago

I've seen pieces of this without the actual proof of work but with cryptographically hashed inputs and commands such that your test can just be a cache hit. https://bazel.build/remote/caching https://bazel.build/remote/caching

adrianN 3y ago

What kind of software do you develop? In my area, running all the tests on my laptop would take days and I frequently have issues where the tests passes locally but not in CI.

Jackevansevo 3y ago
Backend webdev in Python, mostly worked for startups. Nothing crazy complex.