39 ms·

Show HN: Gaia – Build powerful pipelines in any programming language

134 points by michelvocks 8y ago

jacques_chester 8y ago
I can't give an unbiased review because I love Concourse. As I see it, the pros are the availability of the type system and testing ecosystem. I don't mind YAML myself, but I could I use something like Enaml[0] and get the same benefits as you see them? [0] https://github.com/enaml-ops/enaml https://github.com/enaml-ops/enaml
- ramoz 8y ago
  any references/comparisons for Concourse?
  - jacques_chester 8y ago
    Could you elaborate on what you're after? The canonical reference is the main website: https://concourse-ci.org/ https://concourse-ci.org/
myWindoonn 8y ago
"Any programming language" but only Go is currently available and it's not clear whether other languages need anything besides gRPC support in order to be eligible. Cool idea, early code, we'll see whether it goes anywhere.
- masukomi 8y ago
  the screenshot has go, java, c++, and nodeJS as choosable languages....
  - myWindoonn 8y ago
    From the first few paragraphs of the README, "Develop pipelines with the help of SDKs (currently only Go) and simply check-in your code into a git repository." If they've added those others in the intervening time, then great!
    - detaro 8y ago
      https://github.com/gaia-pipeline/gaia/issues/15 https://github.com/gaia-pipeline/gaia/issues/15 2 days ago > Sadly, gaia is in alpha phase and we currently only support pipelines written in go.
- michelvocks 8y ago
  Hello myWindoonn. First of all, sorry for the late reply. Actually, gRPC is the only requirement for pipelines. I hope can make this more clear in the future! :-)
empath75 8y ago
I’d prefer to have a dependency graph to a priority system, personally. It’s much easier to reason about and maintain.
- michelvocks 8y ago
  Hey empath75. Sorry for the late reply. We also discussed this (https://github.com/gaia-pipeline/gaia/issues/19 https://github.com/gaia-pipeline/gaia/issues/19) and you are right. We will switch soon to a dependency system instead of a priority system. :-)
smarx007 8y ago
Wait, didn't we have BPMN more than a decade ago? BTW, if "any" programming language necesarily means Go for you, here is one BPMN implementation in Go: https://zeebe.io https://zeebe.io
- brightball 8y ago
  BPMN is the most under appreciated technology in our field IMO.
  - StefanHoutzager 8y ago
    Agreed. And the ideal language to build a backend interpreter for bpmn - I am doing so - is elixir imho (a language to build build massively scalable soft real-time systems with requirements on high availability, see f.e ds.cs.ut.ee/courses/course-files/To303nis%20Pool%20.pdf ).
- zok3102 8y ago
  AFAICS this still requires a Java broker. Only the client is in go.
  - wints 8y ago
    Zeebe is indeed a Java application, but it's possible to use Zeebe without writing any Java. Broker operations can be managed with a CLI (https://github.com/zeebe-io/zbctl https://github.com/zeebe-io/zbctl). Along with the Go client, there are experimental Node, Ruby, and C# clients, but these are community contributions that aren't yet officially supported by the project.
- alexgoldstone 8y ago
  From the Zeebe README: "Zeebe is currently a tech preview and not meant for production use".
  - wints 8y ago
    I'm on the team that's building Zeebe and want to mention that we release new "dev preview" versions on average once per month and are aiming to have a production ready 1.0 by the end of 2018. If anyone has questions or ideas, or wants to build a prototype with Zeebe, it'd be great to hear from you. There are various ways to contact us here: https://zeebe.io/community/ https://zeebe.io/community/
Karunamon 8y ago
Is this basically a CI tool? It looks very similar to Gitlab CI or Concourse or Jenkins Flow. If so, odd to not see that initialism anywhere; "pipeline" is pretty generic.
- rhencke 8y ago
  'Jenkins Flow' is now called 'Jenkins Pipeline'. Gitlab CI also uses pipelines in its terminology (https://docs.gitlab.com/ee/ci/pipelines.html https://docs.gitlab.com/ee/ci/pipelines.html). I think the use is appropriate here.
  - rahimnathwani 8y ago
    Thank you for that link. Gaia's README assumes readers are familiar with what they mean by 'pipeline' but, because I'd never seen the word used in this way, I was scratching my head wondering whether they were talking about stages of data processing or similar.
    - michelvocks 8y ago
      Hey rahimnathwani. Sorry for the late reply. You are absolutely right, this is my fault. I should have explained the term "pipeline" a bit more. I will work on this!
- discordance 8y ago
  Same as VSTSs Pipelines: https://docs.microsoft.com/en-us/vsts/pipelines/ https://docs.microsoft.com/en-us/vsts/pipelines/
jsd1982 8y ago
The language used in the README is extremely odd. What do you mean by pipeline? The term 'priority' is confusing because what you describe is not a classical interpretation of priority. That sounds more like job ordering with a limited DAG form of dependencies (sequential, fan out, fan in). How does data flow from one job to the next and how does it get combined for a fan-in? What if I have two jobs that get fanned in to? More documentation coverage of the basics of what you're talking about would help.
- michelvocks 8y ago
  Hey jsd1982. Sorry for the late reply. You are right, I didn't describe the term "pipeline" explicitly. A pipeline is a compilation of functions which do "something". We used/use "priority" to determine to order of execution of these functions. We already noticed that priority is probably a bad name and we actually need something different (https://github.com/gaia-pipeline/gaia/issues/19 https://github.com/gaia-pipeline/gaia/issues/19). We are currently working hard on the documentation part (https://gaia-pipeline.io https://gaia-pipeline.io). The project is really early. I hope we can improve within the next weeks/month a lot! :-)
stevekemp 8y ago
Looks like there is no way to downloaded the results of any pipeline that has executed. For example if I wanted to write a pipeline that would: * Compile a program. * Run some tests. * Create a Debian binary package. It looks like I'd have to write a final task to upload to a staging repository in the pipeline itself.
- michelvocks 8y ago
  Hey stevekemp. Sorry for the late reply. You are right, Gaia is alpha so this feature is currently missing. I hope we can provide this soon. :-)
meitham 8y ago
It would be nice to see this mature as an open-source alternative to Informatica PowerCenter.
rs86 8y ago
The design on the product screenshot is too complex. Why that fat curvy font on the page title? We need more brutalism
- michelvocks 8y ago
  Hey rs86. Sorry for the late reply. I actually like the design a lot. I think many developers also enjoy a fresh modern design, too :-)
cjhanks 8y ago
This looks nice enough. It may be worth mentioning that the real challenge for creating a generalized pipeline is rarely the control flow. Usually performance is not achieved due to; poor intermediate data locality, poor system utility balancing. Unless pipeline code has the ability to match estimated job utility to device capabilities - it won't be useful in many non trivial cases. Unless there is an automated way to store intermediate assets such that data locality between stages is (at least somewhat) optimal, significant amounts of all time will be spent in process migration.
oldgun 8y ago
Thanks. Looks like something I've been looking for. Interesting idea! Will evaluate that.
- michelvocks 8y ago
  Hey oldgun. Sorry for the late reply. Good to hear that. Let me know if you have any questions! :-)
faizshah 8y ago
There's a pretty interesting apache project called NiFi that's kind of similar, it's for data flow across different softwares. Apparently was developed and is still used at the NSA. https://nifi.apache.org/docs/nifi-docs/html/overview.html https://nifi.apache.org/docs/nifi-docs/html/overview.html
- snthpy 8y ago
  Very interesting. The provenance tracking is a nice feature that I haven't seen a lot of in other systems. Any examples of this being used in the wild?
regnerba 8y ago
Can anyone give an example of what this would be useful for? I don't really understand its use case and the one sort of example in an image, of creating a k8s namespace and deployment, is covered by our CI/CD system. If this is meant to replace that and be part of the CI/CD I don't see anything talking about how to trigger the pipelines. I feel like I am missing a small but critical piece to understand how this should be used and where.
- fuball63 8y ago
  This could be used as an ETL process; one task is to fetch the data, another task to clean and do quality control, and then branching tasks to put the data in its final destination. Really any task that can be broken into concrete and independent steps could be made into a pipeline for scaling and reliability.
- michelvocks 8y ago
  Hey regnerba. Sorry for the late reply. I don't know which CI/CD system you use but please let me explain my current experience. :-) At my company, we often had the requirement to do complex stuff besides the deployments. For example, we wanted to offer a service for developers that they can via "One-Button-Click" deploy our big monolith into Kubernetes from their code-fork. To get this working, we had to use Spinnaker and Jenkins. We had a really poor experience with Spinnaker cause everything needs to be configured manually and there is no way to actually "code" it. Jenkins supports this via Jenkins Pipelines. Those pipelines are actually nice but they force you to write them in Groovy. Gaia solves this problem. Any programming language can be potentially be used to develop such pipelines.
  - danielodio 8y ago
    @michelvocks -- the Armory team would love to hear more about the challenges you had with Spinnaker. (We're commercializing an enterprise version of OSS Spinnaker). Also, we've created a 'pipelines as code' feature called Dinghy that may have helped. And our installer & configurator provides a much smoother install & configure experience. Details at www.Armory.io Hit us up at hello@armory.io if you have more specific feedback (our exec team reads emails to that addy). DROdio

134 points by michelvocks 8y ago

I can't give an unbiased review because I love Concourse. As I see it, the pros are the availability of the type system and testing ecosystem. I don't mind YAML myself, but I could I use something like Enaml[0] and get the same benefits as you see them? [0] https://github.com/enaml-ops/enaml https://github.com/enaml-ops/enaml

ramoz 8y ago
any references/comparisons for Concourse?
- jacques_chester 8y ago
  Could you elaborate on what you're after? The canonical reference is the main website: https://concourse-ci.org/ https://concourse-ci.org/

myWindoonn 8y ago

"Any programming language" but only Go is currently available and it's not clear whether other languages need anything besides gRPC support in order to be eligible. Cool idea, early code, we'll see whether it goes anywhere.

masukomi 8y ago
the screenshot has go, java, c++, and nodeJS as choosable languages....
- myWindoonn 8y ago
  From the first few paragraphs of the README, "Develop pipelines with the help of SDKs (currently only Go) and simply check-in your code into a git repository." If they've added those others in the intervening time, then great!
  - detaro 8y ago
    https://github.com/gaia-pipeline/gaia/issues/15 https://github.com/gaia-pipeline/gaia/issues/15 2 days ago > Sadly, gaia is in alpha phase and we currently only support pipelines written in go.
michelvocks 8y ago
Hello myWindoonn. First of all, sorry for the late reply. Actually, gRPC is the only requirement for pipelines. I hope can make this more clear in the future! :-)

empath75 8y ago

I’d prefer to have a dependency graph to a priority system, personally. It’s much easier to reason about and maintain.

michelvocks 8y ago
Hey empath75. Sorry for the late reply. We also discussed this (https://github.com/gaia-pipeline/gaia/issues/19 https://github.com/gaia-pipeline/gaia/issues/19) and you are right. We will switch soon to a dependency system instead of a priority system. :-)

smarx007 8y ago

Wait, didn't we have BPMN more than a decade ago? BTW, if "any" programming language necesarily means Go for you, here is one BPMN implementation in Go: https://zeebe.io https://zeebe.io

brightball 8y ago
BPMN is the most under appreciated technology in our field IMO.
- StefanHoutzager 8y ago
  Agreed. And the ideal language to build a backend interpreter for bpmn - I am doing so - is elixir imho (a language to build build massively scalable soft real-time systems with requirements on high availability, see f.e ds.cs.ut.ee/courses/course-files/To303nis%20Pool%20.pdf ).
zok3102 8y ago
AFAICS this still requires a Java broker. Only the client is in go.
- wints 8y ago
  Zeebe is indeed a Java application, but it's possible to use Zeebe without writing any Java. Broker operations can be managed with a CLI (https://github.com/zeebe-io/zbctl https://github.com/zeebe-io/zbctl). Along with the Go client, there are experimental Node, Ruby, and C# clients, but these are community contributions that aren't yet officially supported by the project.
alexgoldstone 8y ago
From the Zeebe README: "Zeebe is currently a tech preview and not meant for production use".
- wints 8y ago
  I'm on the team that's building Zeebe and want to mention that we release new "dev preview" versions on average once per month and are aiming to have a production ready 1.0 by the end of 2018. If anyone has questions or ideas, or wants to build a prototype with Zeebe, it'd be great to hear from you. There are various ways to contact us here: https://zeebe.io/community/ https://zeebe.io/community/

Karunamon 8y ago

Is this basically a CI tool? It looks very similar to Gitlab CI or Concourse or Jenkins Flow. If so, odd to not see that initialism anywhere; "pipeline" is pretty generic.

rhencke 8y ago
'Jenkins Flow' is now called 'Jenkins Pipeline'. Gitlab CI also uses pipelines in its terminology (https://docs.gitlab.com/ee/ci/pipelines.html https://docs.gitlab.com/ee/ci/pipelines.html). I think the use is appropriate here.
- rahimnathwani 8y ago
  Thank you for that link. Gaia's README assumes readers are familiar with what they mean by 'pipeline' but, because I'd never seen the word used in this way, I was scratching my head wondering whether they were talking about stages of data processing or similar.
  - michelvocks 8y ago
    Hey rahimnathwani. Sorry for the late reply. You are absolutely right, this is my fault. I should have explained the term "pipeline" a bit more. I will work on this!
discordance 8y ago
Same as VSTSs Pipelines: https://docs.microsoft.com/en-us/vsts/pipelines/ https://docs.microsoft.com/en-us/vsts/pipelines/

jsd1982 8y ago

The language used in the README is extremely odd. What do you mean by pipeline? The term 'priority' is confusing because what you describe is not a classical interpretation of priority. That sounds more like job ordering with a limited DAG form of dependencies (sequential, fan out, fan in). How does data flow from one job to the next and how does it get combined for a fan-in? What if I have two jobs that get fanned in to? More documentation coverage of the basics of what you're talking about would help.

michelvocks 8y ago
Hey jsd1982. Sorry for the late reply. You are right, I didn't describe the term "pipeline" explicitly. A pipeline is a compilation of functions which do "something". We used/use "priority" to determine to order of execution of these functions. We already noticed that priority is probably a bad name and we actually need something different (https://github.com/gaia-pipeline/gaia/issues/19 https://github.com/gaia-pipeline/gaia/issues/19). We are currently working hard on the documentation part (https://gaia-pipeline.io https://gaia-pipeline.io). The project is really early. I hope we can improve within the next weeks/month a lot! :-)

stevekemp 8y ago

Looks like there is no way to downloaded the results of any pipeline that has executed. For example if I wanted to write a pipeline that would: * Compile a program. * Run some tests. * Create a Debian binary package. It looks like I'd have to write a final task to upload to a staging repository in the pipeline itself.

michelvocks 8y ago
Hey stevekemp. Sorry for the late reply. You are right, Gaia is alpha so this feature is currently missing. I hope we can provide this soon. :-)

meitham 8y ago

It would be nice to see this mature as an open-source alternative to Informatica PowerCenter.

rs86 8y ago

The design on the product screenshot is too complex. Why that fat curvy font on the page title? We need more brutalism

michelvocks 8y ago
Hey rs86. Sorry for the late reply. I actually like the design a lot. I think many developers also enjoy a fresh modern design, too :-)

cjhanks 8y ago

This looks nice enough. It may be worth mentioning that the real challenge for creating a generalized pipeline is rarely the control flow. Usually performance is not achieved due to; poor intermediate data locality, poor system utility balancing. Unless pipeline code has the ability to match estimated job utility to device capabilities - it won't be useful in many non trivial cases. Unless there is an automated way to store intermediate assets such that data locality between stages is (at least somewhat) optimal, significant amounts of all time will be spent in process migration.

oldgun 8y ago

Thanks. Looks like something I've been looking for. Interesting idea! Will evaluate that.

michelvocks 8y ago
Hey oldgun. Sorry for the late reply. Good to hear that. Let me know if you have any questions! :-)

faizshah 8y ago

There's a pretty interesting apache project called NiFi that's kind of similar, it's for data flow across different softwares. Apparently was developed and is still used at the NSA. https://nifi.apache.org/docs/nifi-docs/html/overview.html https://nifi.apache.org/docs/nifi-docs/html/overview.html

snthpy 8y ago
Very interesting. The provenance tracking is a nice feature that I haven't seen a lot of in other systems. Any examples of this being used in the wild?

regnerba 8y ago

Can anyone give an example of what this would be useful for? I don't really understand its use case and the one sort of example in an image, of creating a k8s namespace and deployment, is covered by our CI/CD system. If this is meant to replace that and be part of the CI/CD I don't see anything talking about how to trigger the pipelines. I feel like I am missing a small but critical piece to understand how this should be used and where.

fuball63 8y ago
This could be used as an ETL process; one task is to fetch the data, another task to clean and do quality control, and then branching tasks to put the data in its final destination. Really any task that can be broken into concrete and independent steps could be made into a pipeline for scaling and reliability.
michelvocks 8y ago
Hey regnerba. Sorry for the late reply. I don't know which CI/CD system you use but please let me explain my current experience. :-) At my company, we often had the requirement to do complex stuff besides the deployments. For example, we wanted to offer a service for developers that they can via "One-Button-Click" deploy our big monolith into Kubernetes from their code-fork. To get this working, we had to use Spinnaker and Jenkins. We had a really poor experience with Spinnaker cause everything needs to be configured manually and there is no way to actually "code" it. Jenkins supports this via Jenkins Pipelines. Those pipelines are actually nice but they force you to write them in Groovy. Gaia solves this problem. Any programming language can be potentially be used to develop such pipelines.
- danielodio 8y ago
  @michelvocks -- the Armory team would love to hear more about the challenges you had with Spinnaker. (We're commercializing an enterprise version of OSS Spinnaker). Also, we've created a 'pipelines as code' feature called Dinghy that may have helped. And our installer & configurator provides a much smoother install & configure experience. Details at www.Armory.io Hit us up at hello@armory.io if you have more specific feedback (our exec team reads emails to that addy). DROdio