3 ms·

We have a section dedicated on that with specific CI systems called out: https://www.waypointproject.io/docs/automating-execution https://www.waypointproject.io

We have a section dedicated on that with specific CI systems called out: https://www.waypointproject.io/docs/automating-execution https://www.waypointproject.io/docs/automating-execution Generally speaking I like to describe Waypoint in CI similarly to Terraform for infra. Prior to Terraform, all CI environments had this messy step that was like "create/manage infra" and it was mostly homegrown scripts that are cloud-specific. Today, CI systems often have "deploy" which are similarly: home-grown logic that is platform-specific (kubectl for K8S, packer/terraform for VMs, etc.). We believe the "deploy" step in CIs can be replaced with "waypoint up" or "waypoint deploy". The link above shows you how to do this.

solatic 6y ago
I think I see what you're getting at - you're trying to create a standard way to build and deploy applications, in the way that Terraform is a standard way to build and deploy infrastructure, so to speak. However, I think that one of the big differences here is that a lot of the reason why infrastructure scripts are messy is because they have to deal with state that's hidden behind a remote API, and that Terraform took the imperative APIs for providers like AWS and abstracted them behind a declarative configuration model that also kept track of state. So this was a huge value add compared to infrastructure scripts that would often get corner-cases involved with state wrong. I'm not sure how much that's the case with application development though. Dockerfiles are declarative, as are Kubernetes manifests. The state of the environment is less of an issue - if a new version is being built, that's what gets deployed, and it overwrites what previously exists. If there's any state management around application deployment, it's around detecting errors and rolling back if there are any. If Waypoint would move in the direction of standardizing that, so that developers could encode smoke tests for their application that Waypoint is constantly checking, then that would be huge - but it doesn't seem to be on your roadmap?
- mr_toad 6y ago
  > I'm not sure how much that's the case with application development though. Not all apps fit neatly into a container. In the data warehousing space an app could consist of client code, middleware, and database objects, and then there’s probably data in object storage, and ETL code to deal with. There’s code in your workflow/scheduling system to be updated. The CI and test frameworks need to be updated. And it all has to be deployed and tested in a synchronised way, especially if you want it to be reproducible. I’d rather use a purpose built tool for this, than the mashup of tools and scripts that I currently use.
dbingham 6y ago
Hey Mitchell, It's an interesting idea, but as a Devops Lead, I'm not sure this is any where near our biggest pain point. We've been in the midst of a Devops transition from manually managed EC2 and chef towards CI/CD and infrastructure as code for two years. Terraform solves one of our biggest problems - consistently representing infrastructure as code. But Waypoint won't solve the other. In fact, Waypoint doesn't seem to touch on anything that's actually much of a problem. The deployment step isn't the problem - it's everything surrounding it. See, we're not on CI or CD yet. We're working our way towards them from a legacy system. In that intervening time, what we need is a generalized build tool that provides the ability to do CI or CD, but also allows you to just write general build and deployment pipelines that allow you to gradually automate that process and run them manually outside the context of an automatically triggered pipeline. We've been in this stage for two years, and the only tool that actually does this coherently is Jenkins. But Jenkins is an absolute nightmare to work with on many levels. Our only other real alternative would be to cobble together a mess of bash/python/nodejs scripts or use chat ops. There isn't a good solution in this space. Gaia Pipeline, an open source project in development by a small team lead by a Hashicorp engineer, looks like it could be our long wished for Jenkins killer. I'd really love to hear your thoughts on whether Hashicorp might be willing to throw its full weight behind it: https://github.com/gaia-pipeline https://github.com/gaia-pipeline
- aprdm 6y ago
  Totally agreed ! I still haven’t found something other than jenkins that would work for our use case which is very similar to what you stated.
- sofixa 6y ago
  You can check out Drone.io. It's based around plugins, which are basically Docker containers which accept env variables for configuration and a volume with the code (which Drone clones from git). You can easily write custom plugins in any language you want, test them, and run them locally outside of Drone with little configuration.
- eysi 6y ago
  Depending on where you are in the transition, Garden (https://docs.garden.io/basics/how-garden-works https://docs.garden.io/basics/how-garden-works) might be a good fit. It definitely touches on all of the surrounding problems such as managing dependencies, integration tests, and running arbitrary tasks (think DB migrations) You define one part of your system at a time and Garden compiles that into a graph of modules that it knows how to build, test, and deploy. This means you can adopt it piecemeal and use it to run an entire CI pipeline from any given context, e.g. your local machine, VM or actual CI for that matter. Full disclosure, I'm affiliated with project but your comment describes a problem we see from a lot our users so I couldn't help but replying.

solatic 6y ago

I think I see what you're getting at - you're trying to create a standard way to build and deploy applications, in the way that Terraform is a standard way to build and deploy infrastructure, so to speak. However, I think that one of the big differences here is that a lot of the reason why infrastructure scripts are messy is because they have to deal with state that's hidden behind a remote API, and that Terraform took the imperative APIs for providers like AWS and abstracted them behind a declarative configuration model that also kept track of state. So this was a huge value add compared to infrastructure scripts that would often get corner-cases involved with state wrong. I'm not sure how much that's the case with application development though. Dockerfiles are declarative, as are Kubernetes manifests. The state of the environment is less of an issue - if a new version is being built, that's what gets deployed, and it overwrites what previously exists. If there's any state management around application deployment, it's around detecting errors and rolling back if there are any. If Waypoint would move in the direction of standardizing that, so that developers could encode smoke tests for their application that Waypoint is constantly checking, then that would be huge - but it doesn't seem to be on your roadmap?

mr_toad 6y ago
> I'm not sure how much that's the case with application development though. Not all apps fit neatly into a container. In the data warehousing space an app could consist of client code, middleware, and database objects, and then there’s probably data in object storage, and ETL code to deal with. There’s code in your workflow/scheduling system to be updated. The CI and test frameworks need to be updated. And it all has to be deployed and tested in a synchronised way, especially if you want it to be reproducible. I’d rather use a purpose built tool for this, than the mashup of tools and scripts that I currently use.

dbingham 6y ago

Hey Mitchell, It's an interesting idea, but as a Devops Lead, I'm not sure this is any where near our biggest pain point. We've been in the midst of a Devops transition from manually managed EC2 and chef towards CI/CD and infrastructure as code for two years. Terraform solves one of our biggest problems - consistently representing infrastructure as code. But Waypoint won't solve the other. In fact, Waypoint doesn't seem to touch on anything that's actually much of a problem. The deployment step isn't the problem - it's everything surrounding it. See, we're not on CI or CD yet. We're working our way towards them from a legacy system. In that intervening time, what we need is a generalized build tool that provides the ability to do CI or CD, but also allows you to just write general build and deployment pipelines that allow you to gradually automate that process and run them manually outside the context of an automatically triggered pipeline. We've been in this stage for two years, and the only tool that actually does this coherently is Jenkins. But Jenkins is an absolute nightmare to work with on many levels. Our only other real alternative would be to cobble together a mess of bash/python/nodejs scripts or use chat ops. There isn't a good solution in this space. Gaia Pipeline, an open source project in development by a small team lead by a Hashicorp engineer, looks like it could be our long wished for Jenkins killer. I'd really love to hear your thoughts on whether Hashicorp might be willing to throw its full weight behind it: https://github.com/gaia-pipeline https://github.com/gaia-pipeline

aprdm 6y ago
Totally agreed ! I still haven’t found something other than jenkins that would work for our use case which is very similar to what you stated.
sofixa 6y ago
You can check out Drone.io. It's based around plugins, which are basically Docker containers which accept env variables for configuration and a volume with the code (which Drone clones from git). You can easily write custom plugins in any language you want, test them, and run them locally outside of Drone with little configuration.
eysi 6y ago
Depending on where you are in the transition, Garden (https://docs.garden.io/basics/how-garden-works https://docs.garden.io/basics/how-garden-works) might be a good fit. It definitely touches on all of the surrounding problems such as managing dependencies, integration tests, and running arbitrary tasks (think DB migrations) You define one part of your system at a time and Garden compiles that into a graph of modules that it knows how to build, test, and deploy. This means you can adopt it piecemeal and use it to run an entire CI pipeline from any given context, e.g. your local machine, VM or actual CI for that matter. Full disclosure, I'm affiliated with project but your comment describes a problem we see from a lot our users so I couldn't help but replying.