5 ms·

Hello HN! I'm the founder of HashiCorp. Waypoint is our 2nd day HashiConf announcement and I'm excited to share and talk about it! Compared to Boundary, Waypoi

Hello HN! I'm the founder of HashiCorp. Waypoint is our 2nd day HashiConf announcement and I'm excited to share and talk about it! Compared to Boundary, Waypoint is definitely weirder, it's trying to do things differently. I'll be around here to answer any questions. I think the most common question will be what is this and why? I cover that in detail on the intro page here so I recommend checking that out: https://www.waypointproject.io/docs/intro https://www.waypointproject.io/docs/intro Here are some major things Waypoint is trying to do: * Make it easier to just deploy. If you're bringing a Ruby app to Kubernetes for example, Waypoint uses buildpacks and an opinionated approach to deploying that app for you to Kubernetes. You don't need to write Dockerfiles or any YAML. You can `waypoint up` and go. If you have existing workflows already, you can use a plugin that just shells out to `kubectl`. The important thing here is you get a common workflow "waypoint up" and your future apps should be much easier to deploy. * Provide common debugging tools for deployments. Waypoint gives you `waypoint exec`, `waypoint logs`, etc. and these work on any platform Waypoint supports. So while K8S for example may provide similar functionality, you can now get identical behavior across K8S and serverless, or your VMs, etc. * Build a common workflow that we can plugin other tools around. This is similar to Terraform circa 2015. There wasn't a consistent way then to think about "infrastructure management" outside of a single target platform's tools. With Waypoint, we're trying to do something similar but for the application deployment lifecycle. As always, a disclaimer that Waypoint is a 0.1 and there is a lot we want to do! We have an exciting roadmap[1] that includes promotion, config syncing with KV stores, Vault and others, service brokering for databases, etc. And also, lots of jokes about Otto in the comments here. :) I think Waypoint and Otto similarities end at "they both deploy" (and Otto with HEAVY quotes around "deploy"). They're totally different tools, one didn't inspire the other, though we did make some major changes to avoid Waypoint hitting the same pitfalls as Otto. Super excited to share Waypoint today! [1]: https://www.waypointproject.io/docs/roadmap https://www.waypointproject.io/docs/roadmap

solatic 6y ago
Hi Mitchell, Could you please elaborate further on how you see Waypoint integrating with CI systems? It seems like you expect developers using Waypoint to deploy straight into production from their laptops, which is an anti-pattern, particularly for any company larger than a few people. Companies really need to make sure that code is checked into version control and that tests are run before code is deployed into production. How do you see Waypoint being used in companies where there are more structured pipelines to build code?
- mitchellh 6y ago
  We have a section dedicated on that with specific CI systems called out: https://www.waypointproject.io/docs/automating-execution https://www.waypointproject.io/docs/automating-execution Generally speaking I like to describe Waypoint in CI similarly to Terraform for infra. Prior to Terraform, all CI environments had this messy step that was like "create/manage infra" and it was mostly homegrown scripts that are cloud-specific. Today, CI systems often have "deploy" which are similarly: home-grown logic that is platform-specific (kubectl for K8S, packer/terraform for VMs, etc.). We believe the "deploy" step in CIs can be replaced with "waypoint up" or "waypoint deploy". The link above shows you how to do this.
  - solatic 6y ago
    I think I see what you're getting at - you're trying to create a standard way to build and deploy applications, in the way that Terraform is a standard way to build and deploy infrastructure, so to speak. However, I think that one of the big differences here is that a lot of the reason why infrastructure scripts are messy is because they have to deal with state that's hidden behind a remote API, and that Terraform took the imperative APIs for providers like AWS and abstracted them behind a declarative configuration model that also kept track of state. So this was a huge value add compared to infrastructure scripts that would often get corner-cases involved with state wrong. I'm not sure how much that's the case with application development though. Dockerfiles are declarative, as are Kubernetes manifests. The state of the environment is less of an issue - if a new version is being built, that's what gets deployed, and it overwrites what previously exists. If there's any state management around application deployment, it's around detecting errors and rolling back if there are any. If Waypoint would move in the direction of standardizing that, so that developers could encode smoke tests for their application that Waypoint is constantly checking, then that would be huge - but it doesn't seem to be on your roadmap?
    - mr_toad 6y ago
      > I'm not sure how much that's the case with application development though. Not all apps fit neatly into a container. In the data warehousing space an app could consist of client code, middleware, and database objects, and then there’s probably data in object storage, and ETL code to deal with. There’s code in your workflow/scheduling system to be updated. The CI and test frameworks need to be updated. And it all has to be deployed and tested in a synchronised way, especially if you want it to be reproducible. I’d rather use a purpose built tool for this, than the mashup of tools and scripts that I currently use.
veeralpatel979 6y ago
great work! just curious - could you describe your product development timeline for Waypoint at a high level? how you came up with the idea, how you iterated on it, and how you decided what features to include and not include in the initial release
tinco 6y ago
Hi Mitchell! A problem I had with Gitlab Auto CI/CD was that even though it fully abstracted away Kubernetes and Helm, when something went wrong I was suddenly confronted with having to learn and understand all of it. Then when I fully understood it, writing my own simple .gitlab-ci.yml was orders of magnitude easier and simpler than using the big abstracted auto ci/cd system. Waypoint seems to be similarly abstracting away a lot of complicated details. Is there a philosophy about what waypoint expects of its users, especially when deployments break out of the cookie cutter standard situations?
- mitchellh 6y ago
  The level of Waypoint abstraction is the same as Terraform. You are not expected to use Waypoint and not understand the underlying platform. This is similar to Terraform: you still have to know something like AWS well to use Terraform. With Waypoint, we aren't doing any magic, you specifically opt into a Kubernetes plugin for example. If something goes wrong, you'll have to understand Kubernetes. Waypoint isn't trying to replace that, we're just trying to provide a common workflow on top of these platforms (same as Terraform with infra).
- kennyGitLab 6y ago
  Product Leader from GitLab here - In support of what Mitchell said. One lesson we learned from Auto DevOps is that composability and transparency were key. Just like you mentioned tinco - users struggled when things eventually broke - or they wanted to customize beyond the out of the box customizations that were available. That's why we evolved to have composable Auto DevOps[1] and Helm installs via CI/CD pipelines.[2] [1]https://docs.gitlab.com/ee/topics/autodevops/customize.html#using-components-of-auto-devops https://docs.gitlab.com/ee/topics/autodevops/customize.html#... [2]https://docs.gitlab.com/ee/user/clusters/applications.html#install-using-gitlab-cicd-alpha https://docs.gitlab.com/ee/user/clusters/applications.html#i...
  - tinco 6y ago
    Hey, yeah sorry I didn't mention that. We were a pretty early adopter I guess, we've been on it for well over a year now and it's evolved a lot since then. That it's now split up in separate modules makes it much easier to work with. Something that I couldn't find back then was tutorials or blog posts on how to really work with it, not just deploy happy case apps. The product wasn't very popular yet so there was really no community producing that kind of blog post. Also that the Gitlab issues have now merged into one repository has made reporting an issue so much clearer. You guys are constantly improving at such a rapid pace, definitely kudos for that.
laksdjfkasljdf 6y ago
is there a complex build example, say non-trivial java (e.g spring,jetty,etc) or rust?
- jambay 6y ago
  I work at Product Mgmt at HashiCorp. There is a Java Spring example here that uses buildpacks from either Heroku, Cloud Foundry Paketo (VMware Tanzu team that works closely with Spring), and GCP Buildpacks. https://github.com/hashicorp/waypoint-examples/tree/main/docker/java https://github.com/hashicorp/waypoint-examples/tree/main/doc... You don't have to use buildpacks, you could use the Docker plugin with a Dockerfile as in this reactjs example: https://github.com/hashicorp/waypoint-examples/tree/main/docker/reactjs https://github.com/hashicorp/waypoint-examples/tree/main/doc...
  - Touche 6y ago
    I think these are still trivial though in that they don't use any external resources like a database. Are there no examples that do that?
tn890 6y ago
Could you be more transparent with the roadmap for Waypoint and Boundary? I couldn't find a roadmap anywhere. I'd like to know a rough timeline if I am to consider these even for non-prod for the projects that I do. I understand these are both v0.1. I'm just humbly asking for some transparency in regards to the future.
- mitchellh 6y ago
  Here you go: https://www.waypointproject.io/docs/roadmap https://www.waypointproject.io/docs/roadmap And boundary: https://www.boundaryproject.io/docs/roadmap https://www.boundaryproject.io/docs/roadmap
  - tn890 6y ago
    Maybe I'm an idiot but there's no timeline anywhere in those links.
    - jf 6y ago
      That's likely intentional? My employer doesn't give timelines either because of how hard it is to predict software development times.
      - mitchellh 6y ago
        Yes this is intentional. There are a couple reasons: (1) It really depends on how much community feedback we get. If we’re planning feature A but suddenly loads of people want feature B, we’re gonna reprioritize and change timelines. (2) People get really, really, really upset when you don’t hit your timeline. Its sort of a lose/lose for us. So, we use keep them close to heart.
therealmocker 6y ago
Curious if you are familiar with Habitat (https://community.chef.io/products/chef-habitat/ https://community.chef.io/products/chef-habitat/) and how this compares?
MrSaints 6y ago
I'm quite curious about the URL service when deploying the server onto Kubernetes. How is the public `waypoint.run` able to access deployments in a Kubernetes cluster? I know it uses Horizon. But, are the requests somehow proxied through the deployed Waypoint server?
- mitchellh 6y ago
  We talk a bit about how this works here: https://www.waypointproject.io/docs/url https://www.waypointproject.io/docs/url Our entrypoint reaches back out to the URL service. This feature is optional. That’s how it works. On our roadmap page you can see that we are planning various improvements to continue to make this more secure as well.
  - mfer 6y ago
    Is the source for waypoint.run open source? Can we run our own? I've not had a chance to dig for the source and it wasn't obvious when I skimmed the project.
    - mitchellh 6y ago
      Yes it is: https://github.com/hashicorp/horizon https://github.com/hashicorp/horizon and https://github.com/hashicorp/waypoint-hzn https://github.com/hashicorp/waypoint-hzn Not we haven’t tried to make that easier to self-run, but we didn’t purposely make it difficult either. It just wasn’t a priority for an initial launch. We’ll continue to improve this going forward.

solatic 6y ago

Hi Mitchell, Could you please elaborate further on how you see Waypoint integrating with CI systems? It seems like you expect developers using Waypoint to deploy straight into production from their laptops, which is an anti-pattern, particularly for any company larger than a few people. Companies really need to make sure that code is checked into version control and that tests are run before code is deployed into production. How do you see Waypoint being used in companies where there are more structured pipelines to build code?

mitchellh 6y ago
We have a section dedicated on that with specific CI systems called out: https://www.waypointproject.io/docs/automating-execution https://www.waypointproject.io/docs/automating-execution Generally speaking I like to describe Waypoint in CI similarly to Terraform for infra. Prior to Terraform, all CI environments had this messy step that was like "create/manage infra" and it was mostly homegrown scripts that are cloud-specific. Today, CI systems often have "deploy" which are similarly: home-grown logic that is platform-specific (kubectl for K8S, packer/terraform for VMs, etc.). We believe the "deploy" step in CIs can be replaced with "waypoint up" or "waypoint deploy". The link above shows you how to do this.
- solatic 6y ago
  I think I see what you're getting at - you're trying to create a standard way to build and deploy applications, in the way that Terraform is a standard way to build and deploy infrastructure, so to speak. However, I think that one of the big differences here is that a lot of the reason why infrastructure scripts are messy is because they have to deal with state that's hidden behind a remote API, and that Terraform took the imperative APIs for providers like AWS and abstracted them behind a declarative configuration model that also kept track of state. So this was a huge value add compared to infrastructure scripts that would often get corner-cases involved with state wrong. I'm not sure how much that's the case with application development though. Dockerfiles are declarative, as are Kubernetes manifests. The state of the environment is less of an issue - if a new version is being built, that's what gets deployed, and it overwrites what previously exists. If there's any state management around application deployment, it's around detecting errors and rolling back if there are any. If Waypoint would move in the direction of standardizing that, so that developers could encode smoke tests for their application that Waypoint is constantly checking, then that would be huge - but it doesn't seem to be on your roadmap?
  - mr_toad 6y ago
    > I'm not sure how much that's the case with application development though. Not all apps fit neatly into a container. In the data warehousing space an app could consist of client code, middleware, and database objects, and then there’s probably data in object storage, and ETL code to deal with. There’s code in your workflow/scheduling system to be updated. The CI and test frameworks need to be updated. And it all has to be deployed and tested in a synchronised way, especially if you want it to be reproducible. I’d rather use a purpose built tool for this, than the mashup of tools and scripts that I currently use.

veeralpatel979 6y ago

great work! just curious - could you describe your product development timeline for Waypoint at a high level? how you came up with the idea, how you iterated on it, and how you decided what features to include and not include in the initial release

tinco 6y ago

Hi Mitchell! A problem I had with Gitlab Auto CI/CD was that even though it fully abstracted away Kubernetes and Helm, when something went wrong I was suddenly confronted with having to learn and understand all of it. Then when I fully understood it, writing my own simple .gitlab-ci.yml was orders of magnitude easier and simpler than using the big abstracted auto ci/cd system. Waypoint seems to be similarly abstracting away a lot of complicated details. Is there a philosophy about what waypoint expects of its users, especially when deployments break out of the cookie cutter standard situations?

mitchellh 6y ago
The level of Waypoint abstraction is the same as Terraform. You are not expected to use Waypoint and not understand the underlying platform. This is similar to Terraform: you still have to know something like AWS well to use Terraform. With Waypoint, we aren't doing any magic, you specifically opt into a Kubernetes plugin for example. If something goes wrong, you'll have to understand Kubernetes. Waypoint isn't trying to replace that, we're just trying to provide a common workflow on top of these platforms (same as Terraform with infra).
kennyGitLab 6y ago
Product Leader from GitLab here - In support of what Mitchell said. One lesson we learned from Auto DevOps is that composability and transparency were key. Just like you mentioned tinco - users struggled when things eventually broke - or they wanted to customize beyond the out of the box customizations that were available. That's why we evolved to have composable Auto DevOps[1] and Helm installs via CI/CD pipelines.[2] [1]https://docs.gitlab.com/ee/topics/autodevops/customize.html#using-components-of-auto-devops https://docs.gitlab.com/ee/topics/autodevops/customize.html#... [2]https://docs.gitlab.com/ee/user/clusters/applications.html#install-using-gitlab-cicd-alpha https://docs.gitlab.com/ee/user/clusters/applications.html#i...
- tinco 6y ago
  Hey, yeah sorry I didn't mention that. We were a pretty early adopter I guess, we've been on it for well over a year now and it's evolved a lot since then. That it's now split up in separate modules makes it much easier to work with. Something that I couldn't find back then was tutorials or blog posts on how to really work with it, not just deploy happy case apps. The product wasn't very popular yet so there was really no community producing that kind of blog post. Also that the Gitlab issues have now merged into one repository has made reporting an issue so much clearer. You guys are constantly improving at such a rapid pace, definitely kudos for that.

laksdjfkasljdf 6y ago

is there a complex build example, say non-trivial java (e.g spring,jetty,etc) or rust?

jambay 6y ago
I work at Product Mgmt at HashiCorp. There is a Java Spring example here that uses buildpacks from either Heroku, Cloud Foundry Paketo (VMware Tanzu team that works closely with Spring), and GCP Buildpacks. https://github.com/hashicorp/waypoint-examples/tree/main/docker/java https://github.com/hashicorp/waypoint-examples/tree/main/doc... You don't have to use buildpacks, you could use the Docker plugin with a Dockerfile as in this reactjs example: https://github.com/hashicorp/waypoint-examples/tree/main/docker/reactjs https://github.com/hashicorp/waypoint-examples/tree/main/doc...
- Touche 6y ago
  I think these are still trivial though in that they don't use any external resources like a database. Are there no examples that do that?

tn890 6y ago

Could you be more transparent with the roadmap for Waypoint and Boundary? I couldn't find a roadmap anywhere. I'd like to know a rough timeline if I am to consider these even for non-prod for the projects that I do. I understand these are both v0.1. I'm just humbly asking for some transparency in regards to the future.

mitchellh 6y ago
Here you go: https://www.waypointproject.io/docs/roadmap https://www.waypointproject.io/docs/roadmap And boundary: https://www.boundaryproject.io/docs/roadmap https://www.boundaryproject.io/docs/roadmap
- tn890 6y ago
  Maybe I'm an idiot but there's no timeline anywhere in those links.
  - jf 6y ago
    That's likely intentional? My employer doesn't give timelines either because of how hard it is to predict software development times.
    - mitchellh 6y ago
      Yes this is intentional. There are a couple reasons: (1) It really depends on how much community feedback we get. If we’re planning feature A but suddenly loads of people want feature B, we’re gonna reprioritize and change timelines. (2) People get really, really, really upset when you don’t hit your timeline. Its sort of a lose/lose for us. So, we use keep them close to heart.

therealmocker 6y ago

Curious if you are familiar with Habitat (https://community.chef.io/products/chef-habitat/ https://community.chef.io/products/chef-habitat/) and how this compares?

MrSaints 6y ago

I'm quite curious about the URL service when deploying the server onto Kubernetes. How is the public `waypoint.run` able to access deployments in a Kubernetes cluster? I know it uses Horizon. But, are the requests somehow proxied through the deployed Waypoint server?

mitchellh 6y ago
We talk a bit about how this works here: https://www.waypointproject.io/docs/url https://www.waypointproject.io/docs/url Our entrypoint reaches back out to the URL service. This feature is optional. That’s how it works. On our roadmap page you can see that we are planning various improvements to continue to make this more secure as well.
- mfer 6y ago
  Is the source for waypoint.run open source? Can we run our own? I've not had a chance to dig for the source and it wasn't obvious when I skimmed the project.
  - mitchellh 6y ago
    Yes it is: https://github.com/hashicorp/horizon https://github.com/hashicorp/horizon and https://github.com/hashicorp/waypoint-hzn https://github.com/hashicorp/waypoint-hzn Not we haven’t tried to make that easier to self-run, but we didn’t purposely make it difficult either. It just wasn’t a priority for an initial launch. We’ll continue to improve this going forward.