6 ms·

Show HN: Togomak – declarative pipeline orchestrator based on HCL and Terraform

55 points by srevinsaju 3y ago

Togomak is a command line tool that runs pipelines locally and on the cloud using HashiCorp Configuration Language with a Terraform-like architecture. I am a DevOps engineer and I wanted to build something that is CI/CD provider agnostic - something like GNU make, which I could plug into any provider of choice and not have to rewrite anything while retaining its rich features. I love Terraform and its ecosystem, so I guess I borrowed a huge chunk of their design. Togomak makes parts of your pipeline reusable, by making it a Togomak module, just like Terraform modules. It also gives you power over several HCL functions, loop expressions, templating, and customizing a pipeline according to user input. I started this project over a year ago, and over the course, I learned a lot about how Terraform works. Hopefully, some of our other Terraform friends would find this little tool useful! togomak is actually short for "to-go-and-make", and it sounds similar to "tokamak", a fusion device using magnetic fields to contain hot plasma for energy; perhaps it's containers here!

spqr233 3y ago
This is really interesting! I use github actions almost entirely so I wonder if it'd be possible to integrate it with that CI system. Taking a step back, if you want to see this become more widely useable I'd suggest finding ways to integrate to existing workflows.
- srevinsaju 3y ago
  Yes, I have been taking a look at https://github.com/nektos/act https://github.com/nektos/act which allows running github actions workflow locally. I am investigating if I can integrate it directly in togomak, so that users can reuse existing github action in their pipeline. For the most bit, these pipelines run in node docker containers. I'm hoping to add a concept called "operator" where you can specify or use multiple build backends like Google Cloud Build, or GitHub Actions.
treebeard5440 3y ago
This is a great project - is the goal to be more like Gitlab, i.e. no plugins/marketplace or like Jenkins and GitHub Actions?
- srevinsaju 3y ago
  I am looking forward to a mix of both - I am hoping to add a concept called "operator" which would be a go-plugin [1], just like terraform providers, but build backends. So, someone would be able to, say, write a Slack plugin (in Go, or anything over RPC) which sends a message once a build is complete - like Jenkins/GitHub actions, or just scripts that we can reuse like GitLab CI through `modules`. Perhaps a new registry where we can push custom modules and providers (operators in this case), I'm curious to know about if we have any existing implementations we could reuse for the registry. [1]: https://github.com/hashicorp/go-plugin https://github.com/hashicorp/go-plugin
latchkey 3y ago
It is amazing to me that people call Java verbose when you have things like this to declare a simple variable: variable "service" { type = string description = "Name of the service" } HCL isn't a bonus, it is a hinderance. If you're going to build something like this, you might as well just use TypeScript or Python as your backend language. Beyond that, this just looks like yet another task executor... as you say, like make. At which point, you might as well just use a task executor. If make is not your liking, how about https://github.com/casey/just https://github.com/casey/just
- srevinsaju 3y ago
  One primary design goal togomak had from the beginning was concurrency. All tasks run concurrently, unless a `depends_on` argument is mentioned. `just` didn't support that when I was initially building togomak, but there is a feature coming in soon which I am looking forward to: https://github.com/casey/just/pull/1562 https://github.com/casey/just/pull/1562 . While I was building togomak, I read through Dagger [1], Earthly [2], Concourse CI [3], Jest and Make along with the stuff I was already working with - Jenkins, GitHub actions and GitLab CI. Dagger [1] is really great, I like its design - it supports writing pipelines in Python, Typescript, Go and a few more languages. togomak tries to abstract away a lot of it. Such as dependency management (in the case of python, the requirement of a python interpreter, and its package managers, etc). togomak is just a single statically-linked binary. [1]: https://dagger.io/ https://dagger.io/ [2]: https://earthly.dev/ https://earthly.dev/ [3]: https://concourse-ci.org/ https://concourse-ci.org/
  - marcosnils 3y ago
    Hey there! Dagger employee here. Congrats on the launch, it's grateful to see other people trying to collaborate on this space. > Such as dependency management (in the case of python, the requirement of a python interpreter, and its package managers, etc). togomak is just a single statically-linked binary. Not sure if you've been following the latest updates about Dagger, but since the latest 0.9 release we have a preview feature where all the SDK tooling runs within Dagger itself as a container so they're not required to be installed in the host machine anymore. We've also added support for Dagger Modules which allows you to package and distribute pipelines a-la terraform. Our docs are still not updated yet but if you join our Discord server, you can find all the information there. Cheers!
esafak 3y ago
When you say pipeline I think you only refer to CI/CD pipelines, not ETL pipelines.
- debarshri 3y ago
  You could technically use CI/CD pipelines to run ETL jobs. I have seen rundeck or jenkins being used as an ETL orchestrator.
  - esafak 3y ago
    Jenkins sucks even at its own game, forget about ETL.
- jen20 3y ago
  Not sure I understand why this matters? It also doesn't refer to oil pipelines, which is also fine.
  - esafak 3y ago
    Because orchestration also applies to ETL pipelines. Stating the context would have reduced confusion. A better subtitle might be "Declarative CI/CD pipeline orchestrator".
hamandcheese 3y ago
HCL makes this an instant pass for me. Don't get me wrong - I love the idea of a declarative language for config. But HCL is just not pleasant. Actually decent languages for that would be nix, jsonnet, or starlark.
- nikolay 3y ago
  I have to agree. I'm writing Terraform files most of the day, and I don't stop to wonder why we still have to deal with it, and why every editor should support it in 2023 when we have much better alternatives such as Jsonnet, Skylark/Starlark, Cue, Dhall, etc. It's the "Not Invented Here" syndrome in the context of HashiCorp. The only benefit of HCL is HCL v2, which allows you to write expressions without quotes, but this is supported by all alternatives listed above.
  - srevinsaju 3y ago
    HCL has a JSON representation [1], internally, objects behave that way. so it should be possible to write a Jsonnet wrapper around it. Terraform can currently parse json pipelines too. [1]: https://github.com/hashicorp/hcl/blob/main/json/spec.md https://github.com/hashicorp/hcl/blob/main/json/spec.md
    - nikolay 3y ago
      No, it's not the same - you need a preprocessor. I do use JSON with Terraform via CDKTF although git diff is way more unreadable than with HCL. So, JSON is definitely a bad idea, but HCL v2 is not as good as the alternatives.
- snthpy 3y ago
  Also look at nickel which is an evolution of nix. It's my favorite in this space. nickel-lang.org https://github.com/tweag/nickel https://github.com/tweag/nickel
jen20 3y ago
This looks very good - looking forward to giving it a spin at some point. +1 for HCL over YAML - the experience is much nicer.
- srevinsaju 3y ago
  I shot my foot a couple of times using yaml. Thankfully, not with HCL.
thesorrow 3y ago
Is this similar to Dagger[1] ? [1] https://dagger.io https://dagger.io
- srevinsaju 3y ago
  Similar, but declarative. Dagger uses a programming language, Togomak uses a configuration language on steroids.
  - marcosnils 3y ago
    Just to clarify, Dagger is both declarative and imperative. Yes, you can use multiple programming languages which exposes an imperative API, but your pipeline DAG has to be defined using Dagger's SDKs which is has a declarative form.
    - srevinsaju 3y ago
      Would it make sense to say Dagger is to Pulumi [1], as Terraform is to Togomak? [1]: https://www.pulumi.com/ https://www.pulumi.com/

55 points by srevinsaju 3y ago

spqr233 3y ago

This is really interesting! I use github actions almost entirely so I wonder if it'd be possible to integrate it with that CI system. Taking a step back, if you want to see this become more widely useable I'd suggest finding ways to integrate to existing workflows.

srevinsaju 3y ago
Yes, I have been taking a look at https://github.com/nektos/act https://github.com/nektos/act which allows running github actions workflow locally. I am investigating if I can integrate it directly in togomak, so that users can reuse existing github action in their pipeline. For the most bit, these pipelines run in node docker containers. I'm hoping to add a concept called "operator" where you can specify or use multiple build backends like Google Cloud Build, or GitHub Actions.

treebeard5440 3y ago

This is a great project - is the goal to be more like Gitlab, i.e. no plugins/marketplace or like Jenkins and GitHub Actions?

srevinsaju 3y ago
I am looking forward to a mix of both - I am hoping to add a concept called "operator" which would be a go-plugin [1], just like terraform providers, but build backends. So, someone would be able to, say, write a Slack plugin (in Go, or anything over RPC) which sends a message once a build is complete - like Jenkins/GitHub actions, or just scripts that we can reuse like GitLab CI through `modules`. Perhaps a new registry where we can push custom modules and providers (operators in this case), I'm curious to know about if we have any existing implementations we could reuse for the registry. [1]: https://github.com/hashicorp/go-plugin https://github.com/hashicorp/go-plugin

latchkey 3y ago

It is amazing to me that people call Java verbose when you have things like this to declare a simple variable: variable "service" { type = string description = "Name of the service" } HCL isn't a bonus, it is a hinderance. If you're going to build something like this, you might as well just use TypeScript or Python as your backend language. Beyond that, this just looks like yet another task executor... as you say, like make. At which point, you might as well just use a task executor. If make is not your liking, how about https://github.com/casey/just https://github.com/casey/just

srevinsaju 3y ago
One primary design goal togomak had from the beginning was concurrency. All tasks run concurrently, unless a `depends_on` argument is mentioned. `just` didn't support that when I was initially building togomak, but there is a feature coming in soon which I am looking forward to: https://github.com/casey/just/pull/1562 https://github.com/casey/just/pull/1562 . While I was building togomak, I read through Dagger [1], Earthly [2], Concourse CI [3], Jest and Make along with the stuff I was already working with - Jenkins, GitHub actions and GitLab CI. Dagger [1] is really great, I like its design - it supports writing pipelines in Python, Typescript, Go and a few more languages. togomak tries to abstract away a lot of it. Such as dependency management (in the case of python, the requirement of a python interpreter, and its package managers, etc). togomak is just a single statically-linked binary. [1]: https://dagger.io/ https://dagger.io/ [2]: https://earthly.dev/ https://earthly.dev/ [3]: https://concourse-ci.org/ https://concourse-ci.org/
- marcosnils 3y ago
  Hey there! Dagger employee here. Congrats on the launch, it's grateful to see other people trying to collaborate on this space. > Such as dependency management (in the case of python, the requirement of a python interpreter, and its package managers, etc). togomak is just a single statically-linked binary. Not sure if you've been following the latest updates about Dagger, but since the latest 0.9 release we have a preview feature where all the SDK tooling runs within Dagger itself as a container so they're not required to be installed in the host machine anymore. We've also added support for Dagger Modules which allows you to package and distribute pipelines a-la terraform. Our docs are still not updated yet but if you join our Discord server, you can find all the information there. Cheers!

esafak 3y ago

When you say pipeline I think you only refer to CI/CD pipelines, not ETL pipelines.

debarshri 3y ago
You could technically use CI/CD pipelines to run ETL jobs. I have seen rundeck or jenkins being used as an ETL orchestrator.
- esafak 3y ago
  Jenkins sucks even at its own game, forget about ETL.
jen20 3y ago
Not sure I understand why this matters? It also doesn't refer to oil pipelines, which is also fine.
- esafak 3y ago
  Because orchestration also applies to ETL pipelines. Stating the context would have reduced confusion. A better subtitle might be "Declarative CI/CD pipeline orchestrator".

hamandcheese 3y ago

HCL makes this an instant pass for me. Don't get me wrong - I love the idea of a declarative language for config. But HCL is just not pleasant. Actually decent languages for that would be nix, jsonnet, or starlark.

nikolay 3y ago
I have to agree. I'm writing Terraform files most of the day, and I don't stop to wonder why we still have to deal with it, and why every editor should support it in 2023 when we have much better alternatives such as Jsonnet, Skylark/Starlark, Cue, Dhall, etc. It's the "Not Invented Here" syndrome in the context of HashiCorp. The only benefit of HCL is HCL v2, which allows you to write expressions without quotes, but this is supported by all alternatives listed above.
- srevinsaju 3y ago
  HCL has a JSON representation [1], internally, objects behave that way. so it should be possible to write a Jsonnet wrapper around it. Terraform can currently parse json pipelines too. [1]: https://github.com/hashicorp/hcl/blob/main/json/spec.md https://github.com/hashicorp/hcl/blob/main/json/spec.md
  - nikolay 3y ago
    No, it's not the same - you need a preprocessor. I do use JSON with Terraform via CDKTF although git diff is way more unreadable than with HCL. So, JSON is definitely a bad idea, but HCL v2 is not as good as the alternatives.
snthpy 3y ago
Also look at nickel which is an evolution of nix. It's my favorite in this space. nickel-lang.org https://github.com/tweag/nickel https://github.com/tweag/nickel

jen20 3y ago

This looks very good - looking forward to giving it a spin at some point. +1 for HCL over YAML - the experience is much nicer.

srevinsaju 3y ago
I shot my foot a couple of times using yaml. Thankfully, not with HCL.

thesorrow 3y ago

Is this similar to Dagger[1] ? [1] https://dagger.io https://dagger.io

srevinsaju 3y ago
Similar, but declarative. Dagger uses a programming language, Togomak uses a configuration language on steroids.
- marcosnils 3y ago
  Just to clarify, Dagger is both declarative and imperative. Yes, you can use multiple programming languages which exposes an imperative API, but your pipeline DAG has to be defined using Dagger's SDKs which is has a declarative form.
  - srevinsaju 3y ago
    Would it make sense to say Dagger is to Pulumi [1], as Terraform is to Togomak? [1]: https://www.pulumi.com/ https://www.pulumi.com/