7 ms·

What if it was possible to have a local build system that would run tests etc and provide a cryptographically secure hash proving that the tests passed, the bui

What if it was possible to have a local build system that would run tests etc and provide a cryptographically secure hash proving that the tests passed, the build was good, etc then a “centralized” CI system just verified that before permitting merge/deploy. There’s a lot of hand waving there but it doesn’t seem wildly implausible. Local use of containers or something like nixos could provide sufficient protection against the “works on my machine but not in production” problem.

swagonomixxx 4y ago
Running tests locally is pretty much a no-no for many codebases because they are so large and tests take a long time if they don't run on a beefy test server or in parallel. Common examples of this are multi-million line C++ codebases (e.g proprietary game engines) and monorepos in any language. Running tests on my computer uses up valuable CPU cycles that I can use to work on something else while the CI servers are running my tests. However I can see this working for small to medium sized codebases that really don't need CI for testing.
- CuriouslyC 4y ago
  Or run a partial suite...
  - swagonomixxx 4y ago
    Partial suite is certainly an option. You can build tools that only test code that has changed and whose changes can have cascading effects on other code (i.e dependent code). Do you know if such tools exist, or would I (as in the software engineer working on the codebase) have to build them?
    - hinkley 4y ago
      Partial is the most valuable when you have red tests that are resisting simple fixes. I would definitely encourage running the entire tests every hour or two at the least. We have a project where the team split the tests into chunks and eventually I figured out the reason why is because they had coupling between tests, and running them all together ran into problems. I worry about other people opening that door, because it’s damned hard to close again.
    - origin_path 4y ago
      This is actually the default result you get if you leave the defaults alone in some CI systems! My current company runs CI this way, in fact. A checkout is re-used between builds, and Gradle computes the minimal set of tasks that need to be run based on changed files just as it would locally. TeamCity will set things up this way by default. If you want a clean build on every run, you have to opt-in by checking a box. How well does it work? Well .... it sorta mostly works. Sometimes a build will fail for inexplicable reasons because Gradle/Kotlin incremental builds don't seem fully reliable. You re-run with a clean checkout and the issue goes away. How much time does it save? Also hard to say. Most of the time goes into integration tests that by their nature are invalidated by more or less any change in the codebase. That's not exactly incorrect. Some changes in some modules do avoid hitting the integration tests, though. TC can also do test sharding in the newest versions when you use JUnit. We don't use this yet though.
- hinkley 4y ago
  This is an oversight that lands in circle of “the things I know that are difficult to sell because everyone else is just ignoring it.” Capacity planning applies to tests. As your test count goes up your budget per test goes down. Every CI tool should plot build and test duration over time and only a couple do. There’s a reason most test frameworks can mark slow tests. You need to not only use that but ratchet down over time. Especially when you get new hardware. I haven’t run benchmarks lately but my old rule of thumb developed over many projects and with several people better at testing than I, was a factor of eight for each layer in the testing pyramid. That certainly puts a lot of runtime pressure on the top of the pyramid, but that’s by design. You don’t want people racing to the top, because that’s how you get a cone.
- reillyse 4y ago
  I'm actually working on something to solve the slow test problem. We go down the massively parallel route. We run everything pre git commit by syncing your project directory to our servers and parallelizing the runs. Here is a demo https://brisktest.com/demos https://brisktest.com/demos running against the react test suite.
  - swagonomixxx 4y ago
    Wow, that's really impressive! Do you plan on adding support for Golang?
    - reillyse 4y ago
      It wasn't top of our list but it is definitely something we could support if there was interest (fairly quickly I'd imagine). My personal experience with go tests is that they tend to be very fast but perhaps I just haven't seen a large enough project. Is it something you'd be interested in? If you want to chat more about it, my email is in my profile.
Frost1x 4y ago
I always found it interesting how git was adopted by industry where a large part of its complexity and feature set is built around suopporting fully distributed developer, developers who could lack internet access for months at a time but need to work collaboratively in a team. Then, industry took that distributed approach and strapped project and team management systems with web based front ends like Github, GitLab, etc. around git and created CI/CD pipelines that built ontop of or in parallel to those environments--all of which were centralized. While it's of course still true that your underlying repo instances still give you a lot of flexibility away from those centralized environments, the centralized environments themselves are critical in regular daily workflows, negating a lot of the flexibility git as a VCS adds. In large part, industry could largely use a significantly simpler model than git for VCS since many features are largely moot on a daily basis. Many could probably get away with SVN in large part, for example (although branching isn't nearly as good). I think you raise a great point in that we need to look at how processes have evolved ontop of version control and look at adapting those to a similar model. In some cases it's just not practical because the way infrastructure testing an deployment works but it's the direction I think we should be going. In fact, the first step would be to make issue systems and project management interfaces provided by efforts like GitLab and GitHub available in a local capable, distributed fashion. Clearly communications often require some degree of centralization or at least peer message propagation but there's no reason that information can't be separated from the infrastructure that displays and interacts with it.
- goodpoint 4y ago
  > industry took that distributed approach and strapped project and team management systems with web based front ends like Github Indeed. Software forges centralized what was a Decentralized VCS. For one reason: profit.
  - _se 4y ago
    And of course the many other reasons: convenience, consistency, built-in integration with other development tools...
Retric 4y ago
Containers alone don’t fix the “works on my machine” issue. CI should be isolated to provide proper gate keeping such that any change required for code to function on CI machine will also be automatically loaded into production and other developers machines. In theory you could get something reasonably close to this working locally, but the it’s a serial process so that’s pointless.
- webdog 4y ago
  >will also be automatically loaded into production and other developers machines. I’m assuming you are only submitting this as theoretical, because in the real world (where I’ve directly experienced this workflow) it’s a nightmare: * Someone merges main which deletes working files in your feature branch. Automatic pushing/rebasing of main onto your feature branch creates needless work the moment you try to push upstream when git tries to ascertain state of local main. * countless times I’ve hit bugs where if I try to merge “patch-a” from a common “feat-1” branch (meaning patch was cut from feat, not from main), but then main is updated by the auto-updater, I then have a messy working directory in which main’s new files are treated as unknown orphans and I have to spend time deleting these by hand. I’m all for having my feat branch be up to date at merge time. But making it a rolling target is something git (from my perspective) hits the boundary of what git can reasonably do, and creates more pain than any type of positive DX
  - Retric 4y ago
    This is actual experience from multiple projects. Ideally starting from a blank machine/VM you should be able to run a single script that get’s the latest, builds it locally, runs it successfully, and passes all tests. Git may or may not be part of that process.
    - webdog 4y ago
      I'm going to be annoyingly pedantic here, apologies in advance. To me, what you're describing is an end state where the fetch/build + test is a state in which the developer is done with some task. But in my working reality, testing/building is a non-linear process that is iteratively run after some certain self-determined point-in-time. >Ideally starting from a blank machine/VM you should be able to run a single script that get’s the latest, builds it locally, runs it successfully, and passes all tests. This isn't automatic, and is a practice that is fairly standard today (And one I agree with). You have placed a requirement/line in the sand that says, "Only when I have ideal state should this pipeline run in linear time and output the final result, which are the return codes from tests." Your example has the user initiating the update of the upstream main, not some other process that runs git fetch on the Developer's behalf. Your original comment, as I understand it, is contradictory to this point: >or code to function on CI machine will also be automatically loaded into production and other developers machines. All of us (I think) agree that auto-deployment to production is a desirable goal. But we all (I think) know that broken commits are routinely delivered to production, where "production" represents the sum of all production environments in the world. So while we can have a reasonable assumption that "Production is, or should be deployable all the time," that doesn't mean the state that is represented by Production is safe to run locally in my environment, unless I *specifically* request it. Since git doesn't have file-locking, some other team/PM/developer can decide it's time for <MASSIVE REFACTOR> that blows away my work/branch mistakenly (Or maybe even intentionally, especially if I work in an org that is terrible with communication), creating unnecessary merge conflicts/mental load. This happens in short-lived and long-lived feature branches. In no setup, do I think it's ever safe to take away the developer's agency and let some other process keep my local machine in "sync." There are so many variables to account for that some daemon/service can't be aware of, to allow for automatic updating (and again automatic updating != user running `git fetch`).
ohthehugemanate 4y ago
That's just decentralized CI, aka CI with extra steps. CI itself is just continually testing before continually merging. Doesn't matter where.
czx4f4bd 4y ago
This and the original article feel very much like solutions in search of problems. Beyond the issues with the original article, what's the point in a cryptographically secure hash from an untrusted environment? If someone were really determined to maliciously push broken code for some reason, they could just tamper with whatever generates the hash.
layer8 4y ago
> provide a cryptographically secure hash proving that the tests passed It’s unlikely that verifying such a hash is possible without rerunning the tests, and not at the same time enabling someone to trivially compute that hash without having run the tests in the first place.
giantrobot 4y ago
Local testing doesn't tend to work outside the small scale. Even if you use containers for everything you'll run into edges where a local development machine can't accurately test a production-like environment. Deploying to production without testing in that environment is just tragedy waiting to happen. Distributed signing of artifacts is only effective if you've got fully reproducible builds. If you don't, because almost no one does because it is a huge effort, then all you have is attestation. If a broken/malicious artifact gets deployed the damage gets done and you only know who to blame afterwards. Local tests are useful for not knowingly committing broken code insofar as your tests can determine. Outside of that a full test suite run by a beefy cluster with ready access to assets and network resources is better suited to test for deployment.
oldjavacoder 4y ago
"What if it was possible to have a local build system that would run tests etc and provide a cryptographically secure hash proving that the tests passed" On who's system? Your? Mine? Someone else's? The purpose of CI is consistent continuous integration in a same like manner, not relying on developer A, B or C's systems which may vary greatly.
- Arbortheus 4y ago
  If your tests ran in a sandbox, for instance the Bazel sandbox or a Docker container, and this has the same configuration on CI and on the local machine, then it does not matter if the tests ran in CI or on a developer's machine. This falls apart for things like integration tests, which may be too large/complex/interconnected to work on a local machine, but most of the time this would be more than sufficient.
jayd16 4y ago
What's the point? To reuse hardware for build workers? You can run a local Gitlab ci runner and get what you're saying, no?
b3morales 4y ago
It's not enough to just validate the tests: you would also want a tamper-proof record of exactly what source code was actually built. Probably also the full environment. 99.99% of the time this will be irrelevant trivia, but that one time you need to know, there's no substitute.
hinkley 4y ago
There’s a lovely feature of TeamCity that I had to enforce on one exuberant but doubly sloppy programmer. In trunk based development you can do a conditional commit, where TC runs your code and only pushes it to trunk if the build is green. This allows you to push something before lunch or a meeting for someone who is blocked without coming back to angry faces because you ding-dong-dashed. Breaking trunk is fundamentally a problem of response time. If you’re not in the office you can’t fix a red build in a timely fashion. He did this regularly and got put under house arrest. I started using it on myself so people didn’t have to wait two hours for me to get out of a meeting and fix their api bug.
coredog64 4y ago
A previous employer used a passport analogy. Your code gets a visa (cryptographically signed) to run in a specific environment for a fixed time. You can renew your visa as long as all the assumptions remain true, but once they’re falsified you must create and submit a new build that makes the assumptions true again.
larusso 4y ago
In the Java world gradle can do this with the help of a shared build cache server. There is also Gradle Enterprise but I never looked deeper into it.
justbaker 4y ago
I like how you said using containers, similar to how CI servers tend to be configured.

swagonomixxx 4y ago

Running tests locally is pretty much a no-no for many codebases because they are so large and tests take a long time if they don't run on a beefy test server or in parallel. Common examples of this are multi-million line C++ codebases (e.g proprietary game engines) and monorepos in any language. Running tests on my computer uses up valuable CPU cycles that I can use to work on something else while the CI servers are running my tests. However I can see this working for small to medium sized codebases that really don't need CI for testing.

CuriouslyC 4y ago
Or run a partial suite...
- swagonomixxx 4y ago
  Partial suite is certainly an option. You can build tools that only test code that has changed and whose changes can have cascading effects on other code (i.e dependent code). Do you know if such tools exist, or would I (as in the software engineer working on the codebase) have to build them?
  - hinkley 4y ago
    Partial is the most valuable when you have red tests that are resisting simple fixes. I would definitely encourage running the entire tests every hour or two at the least. We have a project where the team split the tests into chunks and eventually I figured out the reason why is because they had coupling between tests, and running them all together ran into problems. I worry about other people opening that door, because it’s damned hard to close again.
  - origin_path 4y ago
    This is actually the default result you get if you leave the defaults alone in some CI systems! My current company runs CI this way, in fact. A checkout is re-used between builds, and Gradle computes the minimal set of tasks that need to be run based on changed files just as it would locally. TeamCity will set things up this way by default. If you want a clean build on every run, you have to opt-in by checking a box. How well does it work? Well .... it sorta mostly works. Sometimes a build will fail for inexplicable reasons because Gradle/Kotlin incremental builds don't seem fully reliable. You re-run with a clean checkout and the issue goes away. How much time does it save? Also hard to say. Most of the time goes into integration tests that by their nature are invalidated by more or less any change in the codebase. That's not exactly incorrect. Some changes in some modules do avoid hitting the integration tests, though. TC can also do test sharding in the newest versions when you use JUnit. We don't use this yet though.
hinkley 4y ago
This is an oversight that lands in circle of “the things I know that are difficult to sell because everyone else is just ignoring it.” Capacity planning applies to tests. As your test count goes up your budget per test goes down. Every CI tool should plot build and test duration over time and only a couple do. There’s a reason most test frameworks can mark slow tests. You need to not only use that but ratchet down over time. Especially when you get new hardware. I haven’t run benchmarks lately but my old rule of thumb developed over many projects and with several people better at testing than I, was a factor of eight for each layer in the testing pyramid. That certainly puts a lot of runtime pressure on the top of the pyramid, but that’s by design. You don’t want people racing to the top, because that’s how you get a cone.
reillyse 4y ago
I'm actually working on something to solve the slow test problem. We go down the massively parallel route. We run everything pre git commit by syncing your project directory to our servers and parallelizing the runs. Here is a demo https://brisktest.com/demos https://brisktest.com/demos running against the react test suite.
- swagonomixxx 4y ago
  Wow, that's really impressive! Do you plan on adding support for Golang?
  - reillyse 4y ago
    It wasn't top of our list but it is definitely something we could support if there was interest (fairly quickly I'd imagine). My personal experience with go tests is that they tend to be very fast but perhaps I just haven't seen a large enough project. Is it something you'd be interested in? If you want to chat more about it, my email is in my profile.

Frost1x 4y ago

I always found it interesting how git was adopted by industry where a large part of its complexity and feature set is built around suopporting fully distributed developer, developers who could lack internet access for months at a time but need to work collaboratively in a team. Then, industry took that distributed approach and strapped project and team management systems with web based front ends like Github, GitLab, etc. around git and created CI/CD pipelines that built ontop of or in parallel to those environments--all of which were centralized. While it's of course still true that your underlying repo instances still give you a lot of flexibility away from those centralized environments, the centralized environments themselves are critical in regular daily workflows, negating a lot of the flexibility git as a VCS adds. In large part, industry could largely use a significantly simpler model than git for VCS since many features are largely moot on a daily basis. Many could probably get away with SVN in large part, for example (although branching isn't nearly as good). I think you raise a great point in that we need to look at how processes have evolved ontop of version control and look at adapting those to a similar model. In some cases it's just not practical because the way infrastructure testing an deployment works but it's the direction I think we should be going. In fact, the first step would be to make issue systems and project management interfaces provided by efforts like GitLab and GitHub available in a local capable, distributed fashion. Clearly communications often require some degree of centralization or at least peer message propagation but there's no reason that information can't be separated from the infrastructure that displays and interacts with it.

goodpoint 4y ago
> industry took that distributed approach and strapped project and team management systems with web based front ends like Github Indeed. Software forges centralized what was a Decentralized VCS. For one reason: profit.
- _se 4y ago
  And of course the many other reasons: convenience, consistency, built-in integration with other development tools...

Retric 4y ago

Containers alone don’t fix the “works on my machine” issue. CI should be isolated to provide proper gate keeping such that any change required for code to function on CI machine will also be automatically loaded into production and other developers machines. In theory you could get something reasonably close to this working locally, but the it’s a serial process so that’s pointless.

webdog 4y ago
>will also be automatically loaded into production and other developers machines. I’m assuming you are only submitting this as theoretical, because in the real world (where I’ve directly experienced this workflow) it’s a nightmare: * Someone merges main which deletes working files in your feature branch. Automatic pushing/rebasing of main onto your feature branch creates needless work the moment you try to push upstream when git tries to ascertain state of local main. * countless times I’ve hit bugs where if I try to merge “patch-a” from a common “feat-1” branch (meaning patch was cut from feat, not from main), but then main is updated by the auto-updater, I then have a messy working directory in which main’s new files are treated as unknown orphans and I have to spend time deleting these by hand. I’m all for having my feat branch be up to date at merge time. But making it a rolling target is something git (from my perspective) hits the boundary of what git can reasonably do, and creates more pain than any type of positive DX
- Retric 4y ago
  This is actual experience from multiple projects. Ideally starting from a blank machine/VM you should be able to run a single script that get’s the latest, builds it locally, runs it successfully, and passes all tests. Git may or may not be part of that process.
  - webdog 4y ago
    I'm going to be annoyingly pedantic here, apologies in advance. To me, what you're describing is an end state where the fetch/build + test is a state in which the developer is done with some task. But in my working reality, testing/building is a non-linear process that is iteratively run after some certain self-determined point-in-time. >Ideally starting from a blank machine/VM you should be able to run a single script that get’s the latest, builds it locally, runs it successfully, and passes all tests. This isn't automatic, and is a practice that is fairly standard today (And one I agree with). You have placed a requirement/line in the sand that says, "Only when I have ideal state should this pipeline run in linear time and output the final result, which are the return codes from tests." Your example has the user initiating the update of the upstream main, not some other process that runs git fetch on the Developer's behalf. Your original comment, as I understand it, is contradictory to this point: >or code to function on CI machine will also be automatically loaded into production and other developers machines. All of us (I think) agree that auto-deployment to production is a desirable goal. But we all (I think) know that broken commits are routinely delivered to production, where "production" represents the sum of all production environments in the world. So while we can have a reasonable assumption that "Production is, or should be deployable all the time," that doesn't mean the state that is represented by Production is safe to run locally in my environment, unless I *specifically* request it. Since git doesn't have file-locking, some other team/PM/developer can decide it's time for <MASSIVE REFACTOR> that blows away my work/branch mistakenly (Or maybe even intentionally, especially if I work in an org that is terrible with communication), creating unnecessary merge conflicts/mental load. This happens in short-lived and long-lived feature branches. In no setup, do I think it's ever safe to take away the developer's agency and let some other process keep my local machine in "sync." There are so many variables to account for that some daemon/service can't be aware of, to allow for automatic updating (and again automatic updating != user running `git fetch`).

ohthehugemanate 4y ago

That's just decentralized CI, aka CI with extra steps. CI itself is just continually testing before continually merging. Doesn't matter where.

czx4f4bd 4y ago

This and the original article feel very much like solutions in search of problems. Beyond the issues with the original article, what's the point in a cryptographically secure hash from an untrusted environment? If someone were really determined to maliciously push broken code for some reason, they could just tamper with whatever generates the hash.

layer8 4y ago

> provide a cryptographically secure hash proving that the tests passed It’s unlikely that verifying such a hash is possible without rerunning the tests, and not at the same time enabling someone to trivially compute that hash without having run the tests in the first place.

giantrobot 4y ago

Local testing doesn't tend to work outside the small scale. Even if you use containers for everything you'll run into edges where a local development machine can't accurately test a production-like environment. Deploying to production without testing in that environment is just tragedy waiting to happen. Distributed signing of artifacts is only effective if you've got fully reproducible builds. If you don't, because almost no one does because it is a huge effort, then all you have is attestation. If a broken/malicious artifact gets deployed the damage gets done and you only know who to blame afterwards. Local tests are useful for not knowingly committing broken code insofar as your tests can determine. Outside of that a full test suite run by a beefy cluster with ready access to assets and network resources is better suited to test for deployment.

oldjavacoder 4y ago

"What if it was possible to have a local build system that would run tests etc and provide a cryptographically secure hash proving that the tests passed" On who's system? Your? Mine? Someone else's? The purpose of CI is consistent continuous integration in a same like manner, not relying on developer A, B or C's systems which may vary greatly.

Arbortheus 4y ago
If your tests ran in a sandbox, for instance the Bazel sandbox or a Docker container, and this has the same configuration on CI and on the local machine, then it does not matter if the tests ran in CI or on a developer's machine. This falls apart for things like integration tests, which may be too large/complex/interconnected to work on a local machine, but most of the time this would be more than sufficient.

jayd16 4y ago

What's the point? To reuse hardware for build workers? You can run a local Gitlab ci runner and get what you're saying, no?

b3morales 4y ago

It's not enough to just validate the tests: you would also want a tamper-proof record of exactly what source code was actually built. Probably also the full environment. 99.99% of the time this will be irrelevant trivia, but that one time you need to know, there's no substitute.

hinkley 4y ago

There’s a lovely feature of TeamCity that I had to enforce on one exuberant but doubly sloppy programmer. In trunk based development you can do a conditional commit, where TC runs your code and only pushes it to trunk if the build is green. This allows you to push something before lunch or a meeting for someone who is blocked without coming back to angry faces because you ding-dong-dashed. Breaking trunk is fundamentally a problem of response time. If you’re not in the office you can’t fix a red build in a timely fashion. He did this regularly and got put under house arrest. I started using it on myself so people didn’t have to wait two hours for me to get out of a meeting and fix their api bug.

coredog64 4y ago

A previous employer used a passport analogy. Your code gets a visa (cryptographically signed) to run in a specific environment for a fixed time. You can renew your visa as long as all the assumptions remain true, but once they’re falsified you must create and submit a new build that makes the assumptions true again.

larusso 4y ago

In the Java world gradle can do this with the help of a shared build cache server. There is also Gradle Enterprise but I never looked deeper into it.

justbaker 4y ago

I like how you said using containers, similar to how CI servers tend to be configured.