5 ms·

Cute. > No root access > 5 min max build time > Bring-your-own-binaries – Lambda has a limited selection of installed software > 1.5GB max memory > Linux only

Cute. > No root access > 5 min max build time > Bring-your-own-binaries – Lambda has a limited selection of installed software > 1.5GB max memory > Linux only There's a reason Jenkins is still used so widely. It's not because of utilization or all the other things pointed out. When your project gets big enough, managing the CI pipeline turns into a distributed systems problem with distributed queues, locks, error/failure recovery, and all the other headaches that such systems bring. Heck, reporting alone on a test suite with 12k tests is a problem in and of itself.

empath75 10y ago
Yeah, there is a place for this kind of thing though. For a lot of small projects, Jenkins is overkill. Also, a lot of those issues can be worked around by using ecs for the builds.
- toomuchtodo 10y ago
  Or just spend the time setting up the right tool in the beginning, so you don't have to waste time later moving to the right tool?
Rezo 10y ago
A Lambda / "serverless" approach makes a ton of sense for CI system just on the premise alone. Why have x number of build slaves sitting doing nothing 95% of the time? The ideal build system would paralellize on-demand to as many workers as possible within a few seconds of a build triggering (I don't want a build slave, I want 50), and then immediately terminate them, which is pretty much the whole point of Lambda. Jenkins is a huge pain the behind, fragile, the machines always inevitably turns into special snowflakes, configs in git are poorly supported. It's also very expensive in any non-trivial setting and requires full-time babysitters. Cloud CIs still charge quite a lot for any meaningful parallel builds, and there's the security aspect of uploading your code to third-parties, I trust my S3 bucket permissions more than a random CI SaaS. This seems like a great start for a sweet spot between on-prem and SaaS CI.
- batterseapower 10y ago
  Hydra (https://nixos.org/hydra/ https://nixos.org/hydra/) addresses a couple of these concerns as well, specifically the snowflakiness and version controlled config.
  - chriswarbo 10y ago
    In the past week I've set up Hydra (adapting https://github.com/peti/hydra-tutorial https://github.com/peti/hydra-tutorial ). It's quite nice, but there are a few sore points. These mostly seem to be known, so it's just a case of waiting for the features/fixes to land: - No (semi-)stable releases, requiring mixing and matching git revisions of Hydra and Nix until we find a combination which builds. nixpkgs master (i.e. unstable) contains a Hydra package and module, but I think that's been added after the latest stable nixpkgs release (16.03). - As a consequence of the above, we need to build a lot of stuff locally since they're not in the binary caches. - While nixpkgs master contains a NixOS module for Hydra, these modules don't yet work outside NixOS. Hydra can be run on, say, Ubuntu, but it requires manual, imperative configuration of Hydra, Postgres, etc. - Configuration relies on interacting with the Web UI. "Declarative jobsets", which allow jobset specification via Nix, are in master, but I couldn't get any git revision containing that to build. - Build slaves must be running NixOS. As I'm stuck with an Ubuntu host, this forces me to use VMs, which has overhead and makes some architecture choices more difficult. NixOS containers may fix this, but they currently only work on NixOS. - Since I'm forced into VMs, I might as well deploy them with NixOps. The standard NixOS image used by NixOps is only 10GB, and NixOps doesn't support resizing it (there are open pull requests for this). - The Hydra Web UI allows logging in, configuring, etc. I would prefer if there were a Web UI without write access to the DB, i.e. once the jobsets are set up (via a declarative approach or a write-enabled Web UI), there should be a read-only view of the progress, outputs, logs, etc. (new builds would be triggered by git pushes, or by SSHing to the server and running a command). - It would be nice if the DB requirements were a little lighter; e.g. if there were an SQLite option. Using the Postgres NixOS module is easy enough, but it's a bit overly-complicated (e.g. setting up credentials and sharing them between hydra and the DB, one-shot systemd jobs to initialise the DB, etc.); plus, if anything goes wrong I'd have to learn how to use Postgres (I already know far more about MySQL, MariaDB, SQLServer and Oracle than I'd like!).
falsedan 10y ago
> distributed queues, locks, error/failure recovery, […] [test] reporting Jenkins doesn't help with any of those things.
- dkarapetyan 10y ago
  Neither does AWS lambda but at least Jenkins handles the worker management piece quite well.
  - falsedan 10y ago
    Which plugin does that?
    - dkarapetyan 10y ago
      They have an AWS plugin for spinning up instances and dynamically sizing the pool.
      - falsedan 10y ago
        Oh boy, I hope it solves the job draining/triggered downstream builds problem!
adamb 10y ago
In my (limited) experience, treating your CI pipeline like a distributed system is a design smell. It leads to build processes that are difficult to test, fix, and iterate on. When a build system can only be effectively invoked by CI/CD, it starts to pervert developer incentives. People need to check things in before they can be sure they work. They don't bother with tiny fixes because of the inertia. Flaky jobs get a quick rebuild, because reproducing a build failure locally is complex enough that they'd prefer to avoid it if they can. Over time, these add up to a system that grows through accretion, which is the enemy of both agility and understandability. Better is a build process that uses simple, reusable components that work equally well on developers' machines. These tools can be tested, refined, and replaced incrementally, using the same build processes that the rest of your code base does. You can do this without needing coupling your build processes to the specific way(s) that Jenkins (and company) model builds or their configuration.
- falsedan 10y ago
  We decided to think of our CI system as a distributed job execution engine for one project… now that we know better, we have a maze of twisty jobs configs to unwind.
- GauntletWizard 10y ago
  I disagree. Not that I don't think that simple, reusable components are valuable; They are, and any developer should run tests before sending things off to the cloud. But having things work on developers machines is itself a code smell, because developers have access to them. Your whole project should build straight from very fresh boxes, and doing builds on developers machines will never be fresh boxes. It's hard to remove the cult knowledge from a development team. My project just discovered a new unlisted dependency when doing a deploy, because every developer knew about it and installed it on their machine beforehand. It was explicitly listed in the build/devtools dependencies, but not supposed to be a runtime dep. Had the developers run tests on a fresh machine, they'd have run into it. (Of course, the CI team had also installed it on the CI boxes, because they used it for debugging.) For a large class of tests, having the developers run them in the build environment is fine. But you also need to run them in the deploy environment, and to that end developers should hit the CI system. Every test that the CI system runs on each commit should be runnable before the commit. You should have tooling and spare capacity such that the CI system is used to run tests immediately before commit, not right after - That's too late. You should run them whenever a dev sends off for code review. You should run whenever a dev feels like it; If they're in a good spot, run the unit tests, run the CI tests, see what's broken.

empath75 10y ago

Yeah, there is a place for this kind of thing though. For a lot of small projects, Jenkins is overkill. Also, a lot of those issues can be worked around by using ecs for the builds.

toomuchtodo 10y ago
Or just spend the time setting up the right tool in the beginning, so you don't have to waste time later moving to the right tool?

Rezo 10y ago

A Lambda / "serverless" approach makes a ton of sense for CI system just on the premise alone. Why have x number of build slaves sitting doing nothing 95% of the time? The ideal build system would paralellize on-demand to as many workers as possible within a few seconds of a build triggering (I don't want a build slave, I want 50), and then immediately terminate them, which is pretty much the whole point of Lambda. Jenkins is a huge pain the behind, fragile, the machines always inevitably turns into special snowflakes, configs in git are poorly supported. It's also very expensive in any non-trivial setting and requires full-time babysitters. Cloud CIs still charge quite a lot for any meaningful parallel builds, and there's the security aspect of uploading your code to third-parties, I trust my S3 bucket permissions more than a random CI SaaS. This seems like a great start for a sweet spot between on-prem and SaaS CI.

batterseapower 10y ago
Hydra (https://nixos.org/hydra/ https://nixos.org/hydra/) addresses a couple of these concerns as well, specifically the snowflakiness and version controlled config.
- chriswarbo 10y ago
  In the past week I've set up Hydra (adapting https://github.com/peti/hydra-tutorial https://github.com/peti/hydra-tutorial ). It's quite nice, but there are a few sore points. These mostly seem to be known, so it's just a case of waiting for the features/fixes to land: - No (semi-)stable releases, requiring mixing and matching git revisions of Hydra and Nix until we find a combination which builds. nixpkgs master (i.e. unstable) contains a Hydra package and module, but I think that's been added after the latest stable nixpkgs release (16.03). - As a consequence of the above, we need to build a lot of stuff locally since they're not in the binary caches. - While nixpkgs master contains a NixOS module for Hydra, these modules don't yet work outside NixOS. Hydra can be run on, say, Ubuntu, but it requires manual, imperative configuration of Hydra, Postgres, etc. - Configuration relies on interacting with the Web UI. "Declarative jobsets", which allow jobset specification via Nix, are in master, but I couldn't get any git revision containing that to build. - Build slaves must be running NixOS. As I'm stuck with an Ubuntu host, this forces me to use VMs, which has overhead and makes some architecture choices more difficult. NixOS containers may fix this, but they currently only work on NixOS. - Since I'm forced into VMs, I might as well deploy them with NixOps. The standard NixOS image used by NixOps is only 10GB, and NixOps doesn't support resizing it (there are open pull requests for this). - The Hydra Web UI allows logging in, configuring, etc. I would prefer if there were a Web UI without write access to the DB, i.e. once the jobsets are set up (via a declarative approach or a write-enabled Web UI), there should be a read-only view of the progress, outputs, logs, etc. (new builds would be triggered by git pushes, or by SSHing to the server and running a command). - It would be nice if the DB requirements were a little lighter; e.g. if there were an SQLite option. Using the Postgres NixOS module is easy enough, but it's a bit overly-complicated (e.g. setting up credentials and sharing them between hydra and the DB, one-shot systemd jobs to initialise the DB, etc.); plus, if anything goes wrong I'd have to learn how to use Postgres (I already know far more about MySQL, MariaDB, SQLServer and Oracle than I'd like!).

falsedan 10y ago

> distributed queues, locks, error/failure recovery, […] [test] reporting Jenkins doesn't help with any of those things.

dkarapetyan 10y ago
Neither does AWS lambda but at least Jenkins handles the worker management piece quite well.
- falsedan 10y ago
  Which plugin does that?
  - dkarapetyan 10y ago
    They have an AWS plugin for spinning up instances and dynamically sizing the pool.
    - falsedan 10y ago
      Oh boy, I hope it solves the job draining/triggered downstream builds problem!

adamb 10y ago

In my (limited) experience, treating your CI pipeline like a distributed system is a design smell. It leads to build processes that are difficult to test, fix, and iterate on. When a build system can only be effectively invoked by CI/CD, it starts to pervert developer incentives. People need to check things in before they can be sure they work. They don't bother with tiny fixes because of the inertia. Flaky jobs get a quick rebuild, because reproducing a build failure locally is complex enough that they'd prefer to avoid it if they can. Over time, these add up to a system that grows through accretion, which is the enemy of both agility and understandability. Better is a build process that uses simple, reusable components that work equally well on developers' machines. These tools can be tested, refined, and replaced incrementally, using the same build processes that the rest of your code base does. You can do this without needing coupling your build processes to the specific way(s) that Jenkins (and company) model builds or their configuration.

falsedan 10y ago
We decided to think of our CI system as a distributed job execution engine for one project… now that we know better, we have a maze of twisty jobs configs to unwind.
GauntletWizard 10y ago
I disagree. Not that I don't think that simple, reusable components are valuable; They are, and any developer should run tests before sending things off to the cloud. But having things work on developers machines is itself a code smell, because developers have access to them. Your whole project should build straight from very fresh boxes, and doing builds on developers machines will never be fresh boxes. It's hard to remove the cult knowledge from a development team. My project just discovered a new unlisted dependency when doing a deploy, because every developer knew about it and installed it on their machine beforehand. It was explicitly listed in the build/devtools dependencies, but not supposed to be a runtime dep. Had the developers run tests on a fresh machine, they'd have run into it. (Of course, the CI team had also installed it on the CI boxes, because they used it for debugging.) For a large class of tests, having the developers run them in the build environment is fine. But you also need to run them in the deploy environment, and to that end developers should hit the CI system. Every test that the CI system runs on each commit should be runnable before the commit. You should have tooling and spare capacity such that the CI system is used to run tests immediately before commit, not right after - That's too late. You should run them whenever a dev sends off for code review. You should run whenever a dev feels like it; If they're in a good spot, run the unit tests, run the CI tests, see what's broken.