7 ms·

Show HN: Free hosted open-source alternative to Zapier/Airflow

279 points by newcrobuzon 6y ago

newcrobuzon 6y ago
Hi HN peeps! I single-handedly built Titanoboa (a workflow automation for JVM) and now I am experimenting with hosting dockerized Titanoboa instances - so I built a free hosting service. If you want to play with Titanoboa in your browser without installing anything, give it a try. Feel free to check out Titanoboa on Github: https://github.com/mikub/titanoboa https://github.com/mikub/titanoboa Also: This is an early beta so please do let me know if something breaks or you spot a bug. Atm I have load-balancers set up only on West coast and in Europe, so apologies to folks from down under & similar locations, let me know if it's too laggy :)
- cateye 6y ago
  Why do you describe it as "automation for JVM" instead of "automation that runs on JVM"?
  - newcrobuzon 6y ago
    Good point, not a native speaker so I thought this would roughly mean the same. The main message I wanted to convey was that this runs on JVM and multiple JVM languages (java and clojure) could be used.
kevsim 6y ago
Impressive feat of engineering for one person! But is it really an alternative to Zapier? I mean, while Zapier is obviously about automation, it always felt that their big selling point was the sheer number of integrations they provide.
- newcrobuzon 6y ago
  My aim is to make this easier to customize - i.e. develop custom steps on the run and have an instant feedback loop as you develop or customize existing integrations. Target audience might be slightly different ultimately, but we'll see - I would envision this to be more useful in an enterprise environment where you pretty much always have to customize the integrations that were provided out of the box.
newcrobuzon 6y ago
Wow, this has blown up a bit, so I have added few more servers (now two servers on the West coast and two in Europe), but ultimately I think there is a limit cap of 100 Titanoboa instances in each geography in parallel. So if we break that level please don't be mad if you don't get your instance :) Instead give me a star on github and come back later :) Cheers Miro
- MiroF 6y ago
  Wishing you success, fellow Miro.
- newcrobuzon 6y ago
  Ok, I have just reached 100 concurrent containers in Europe, which is max at this stage and I don't think I can change the aws limit on the fly (will check though) so I might start killing the oldest instances. Apologies for inconvenience folks, but I am sure people rarely play with each instance for the whole 3 hours anyway at this stage. Honestly did not expect this kind of "public beta" rollout, still surprised how well the service holds so far under the load.
  - newcrobuzon 6y ago
    Approaching 100 containers cap in US, so if something does not work, dont be mad :)
    - newcrobuzon 6y ago
      If anybody's curious about this "public beta hosting - HN bear hug" performance load test: I just (sequentially) restarted all 4 servers behind my 2 load balancers - seems that by reaching 100 containers in each geography the provisioning threads may have died after many (unsuccessful) retries, so need to restart the thread pools. That is one more error I need to handle (or increasing my limit in AWS).
      - erulabs 6y ago
        Heya - just a note that I build these kinds of PaaS platforms for a living - if you need a hand building a super-massive cluster for lots of users, let me know!
foolinaround 6y ago
is there some recipes or some common scenarios one can 'import' in to check it out?
- newcrobuzon 6y ago
  You can check out https://github.com/mikub/titanoboa-tasklets https://github.com/mikub/titanoboa-tasklets And also https://github.com/mikub/titanoboa-sample-workflows https://github.com/mikub/titanoboa-sample-workflows There are few blog posts I wrote about some sample workflows, e.g.: https://github.com/mikub/titanoboa/wiki/Getting-Started-with-GUI https://github.com/mikub/titanoboa/wiki/Getting-Started-with... https://www.titanoboa.io/using-titanoboa-for-it-automation.html https://www.titanoboa.io/using-titanoboa-for-it-automation.h... https://www.titanoboa.io/using-titanoboa-for-cloud-integration.html https://www.titanoboa.io/using-titanoboa-for-cloud-integrati... https://www.titanoboa.io/java-repl.html https://www.titanoboa.io/java-repl.html
jchook 6y ago
@newcrobuzon you have some kind of comment string output at the top of your html body tag. Looks like something you do not want exposed. Congratulations on reaching public beta and for submitting to HN! Looks very promising!
- newcrobuzon 6y ago
  Thanks! If it is the recapcha string then it is just an old public string for local testing. But I would upvote you 100x or send you a medal or something if I could, this is really super nice of you letting me know!
juliend2 6y ago
Very interesting! What library did you use for the SVG powered diagram tool?
- juliend2 6y ago
  I think this blog post answers my question: https://titanoboa.io/visualizing-workflows.html https://titanoboa.io/visualizing-workflows.html Very cool to see how you use clojurescript for this.
- newcrobuzon 6y ago
  I am using D3. AFAIK nobody has used it for worfklow visualization so far: https://www.titanoboa.io/visualizing-workflows.html https://www.titanoboa.io/visualizing-workflows.html
  - X6S1x6Okd1st 6y ago
    Luigi does some workflow visualization with d3 https://luigi.readthedocs.io/en/stable/index.html https://luigi.readthedocs.io/en/stable/index.html
    - newcrobuzon 6y ago
      Thanks, wasn't aware of them!
      - X6S1x6Okd1st 6y ago
        It's a very code forward one, but I've certainly found it useful and easier to understand than something like airflow.
sixhobbits 6y ago
I've used Zapier a fair amount and I wrote an article about Airflow so I have a fairly good understanding of that too. I would never really consider them alternatives though? To me Zapier is a low/no code tool that offers a bazillion integrations and Airflow is a workflow orchestration tool. So comparing to both of them confuses me and I guess choosing one would give you a more nieche audience but one that you can connect better to as well.
- newcrobuzon 6y ago
  This is a very good point. My current line of thinking is: A workflow is a workflow is a workflow. Titanoboa is built the way that pretty much everything it shuffles is data - and either it can be big (Airflow) or the steps can have side effects (iPaaS/Zapier). The "no code" is achievable since adding more predefined steps is not that hard (there's not that many at the moment though) - see for instance https://github.com/mikub/titanoboa/wiki/Getting-Started-with-GUI https://github.com/mikub/titanoboa/wiki/Getting-Started-with... - you dont code, you just fill in properties, so it is very straightforward, pretty much like Zappier. My aim in the iPaaS space is more at the enterprise level, where the predefined steps won't do anymore and you have to custom develop them anyway - so in that are I think Titanoboa shines since you really can rapidly prototype steps on the fly, a bit like Repl.it and Zapier together... I agree that it might be good to pick one audience, at this stage I am just experimenting if somehow I can market to both and don't mess up and confuse both groups.
  - j45 6y ago
    Keep converging and building. I have been experimenting with combinations of huggin, camunda, airflow and others to try and achieve an integrated workflow/state/process management. There exists a gap in the enterprise space between all these tools.. and it has been further exacerbated by the disruption introduced to many industries by covid. There is a real opportunity for small and medium businesses to be able to access tooling that is beyond what a base line Zap, etc can accomplish. Would be happy to connect offline.
    - 6y ago
bambam24 6y ago
I stop watching it when he said “here I can add any Maven dependency”
- newcrobuzon 6y ago
  Well yes, this is for JVM, so maven ecosystem, sorry. I have been contemplating also doing a clone on node.js (titanoboa is written in clojure so it would just be a move from clojure to clojurescript), but I did hear that the npm ecosystem is a bit uninviting too... What dependencies tools would you like to see?
ForHackernews 6y ago
How does this compare to Prefect? https://medium.com/the-prefect-blog/why-not-airflow-4cfa423299c4 https://medium.com/the-prefect-blog/why-not-airflow-4cfa4232...
- newcrobuzon 6y ago
  To be honest Prefect just got on my radar last week, so I will need to look at it more closely, I was not aware of them till then. Obviously this is for JVM, plus I am not sure how Prefect addresses following two points I mainly focus on: - make rapid prototyping of new steps possible during runtime, without any need to restart/redeploy - focus on distributed processing, where in master-less Titanoboa cluster you can have pretty much any number of nodes
  - ramon 6y ago
    Prefect is like an hour just to put up the kubernetes environment required for it. The way you have it you can just do a simple docker run and start coding this is the comparison and you're coding on the web no need to steup VSCode or nothing it's fast.
j45 6y ago
Being a JVM friendly tool, it would be great to unlock other JVM syntaxes to allow scripting in any lanugage.. Jython for Python, Quercus for PHP, JRuby for Ruby, etc.
- newcrobuzon 6y ago
  yes, I am considering this and it should not be that hard (also Kotlin, Nashorn javascript...), but at this stage Titanoboa is just a one-man-show - so patience, I might get there ;)
  - j45 6y ago
    For sure. Being a one man army has it's perks and challenges. I hope this project has a community form around it. The JVM is more underrated/unknown than it should be in the startup space, tools like what you're making make it far more accessible.
  - ramon 6y ago
    newcrobuzon you're a genius my friend someone should give you an award I loved your idea and how you approached it it's fantastic for developers. I'm a root Java dev and I have to say you did fantastic work to make developers life easier, you didn't take any approach you made your own this is awesome. If you get Javascript working I feel like I'm getting back into AppJet days! This is fantastic stuff. If you show me how I can help get the javascript stuff going I would be glad to help you in this initative since I have adopted NodeJS recently like you said, being a one man army you have to be efficient and JS makes me more efficient.
    - newcrobuzon 6y ago
      Thanks ramon! Appreciate your kind words! Feel free to shoot me an email or reach me on twitter and let's stay in touch!
  - mdaniel 6y ago
    FWIW, I would have to check all the ones listed but if they're sane scripting languages on the jvm then they'll implement the JSR-223 interface allowing one to pick them at runtime without a huge amount of drama: https://jcp.org/aboutJava/communityprocess/final/jsr223/index.html https://jcp.org/aboutJava/communityprocess/final/jsr223/inde...
deleted 6y ago
[deleted]
PascalW 6y ago
Nice work! How does it compare to n8n? [0] n8n is the closest OSS alternative to Zapier I've seen so far. [0] https://n8n.io/ https://n8n.io/
- newcrobuzon 6y ago
  A bit similar, but on JVM. The main focus of Titanoboa is however as follows: - make rapid prototyping of new steps possible during runtime, without any need to restart/redeploy - focus on distributed processing, where in master-less Titanoboa cluster you can have pretty much any number of nodes
- lulzx 6y ago
  It's license (fair code) sucks.
  - newcrobuzon 6y ago
    That is a fair criticism, I am aware of the AGPL license shortcomings. I just picked up the more restrictive license at the beginning - being a sole funder and not working on this full time etc. I simply did not want somebody (e.g. a big company with a big team) grabbing my code along the way and running away with it. Since now Titanoboa got to the shape I envisioned it to be in I am starting to focus more on adoption, so yes I am definitely thinking about switching to less restrictive license since it will probably help. Also at the beginning I was not aware how badly AGPL is perceived (I always thought if it was good for Mongo it could work for me, but I may have been wrong).
    - cstuder 6y ago
      You misunderstood, it's not you who's being criticized. n8n used to write "open source" on their webpage while publishing the product with a proprietary license. A lot of discussions ensued. https://github.com/n8n-io/n8n/issues/40 https://github.com/n8n-io/n8n/issues/40
      - newcrobuzon 6y ago
caymanjim 6y ago
First impression: rename it now while the cost is low. The name is unwieldy.
- everfree 6y ago
  Ti-ta-no-bo-a 5 syllables. Agreed that this is unwieldy to say in normal speech.
  - newcrobuzon 6y ago
    I mostly considered naming to be a distraction from getting the real work done. I also am probably not good with names as my other alternative was Megalodon (4 syllables) - I just wanted to have some megafauna name. Happy to hear what you folks would suggest :)
    - afarviral 6y ago
      "Dinornis" or "Moa" in maori.
      - StavrosK 6y ago
        If you're going with the former, you might as well spell it "deinornis", it looks cooler and is more correct.
    - acidbaseextract 6y ago
      Titano is nice, but may have some trademark issues. Tanobo is easy to say, if a bit generic. Just playing with syllables.
    - ohyeshedid 6y ago
      A fauna focused synonym for canopy?
    - dddw 6y ago
      Gdoc form for suggestions. 1 mail to all users (small enticement, reward for winner, pro account for life or something else really nice). Remove any lewd suggestions. Then pick one you love, or if you can't choose a poll.
itpragmatik 6y ago
How does it compare with Cadence/Temporal?
prabhatjha 6y ago
Nicely done. Couple of recommendations: - Recommend to change your licensing if you want people to try. - I understand why you want to have AGPL but I don't know many successful OSS projects with this license. There is another HN thread on this topic. https://news.ycombinator.com/item?id=1657071 https://news.ycombinator.com/item?id=1657071
WrtCdEvrydy 6y ago
Honestly I really like this, I can't wait to use it, but my concerns are around certain features being behind a paywall, especially HA and Clustering on the self-hosted version. I understand that the cloud hosted version should cost more for these features but without a way to get pricing for the self-hosted HA and clustered version it will be a tough sell for me to deploy it, use it and have to scrap it in the future.
ramon 6y ago
I liked the video demo really nice and I would say this should be called not low-code or no-code but rather aPaaS which is Application Platform as a Service. So the title should be "Host your own aPaaS" instead.
thrownaway954 6y ago
that is the crappiest website i've seen in quite sometime. my first impression was that this is either a scam or a joke. in this day and age, if you are can't afford a decently designed site, how am i ever even going to give your product a chance.
- dang 6y ago
  This comment breaks HN's rules badly. Please don't post like this again, regardless of how someone's website is. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/showhn.html https://news.ycombinator.com/showhn.html
zomglings 6y ago
This looks really well done, newcrobuzon. Very impressed. I especially love your web UI. It makes it very easy to start experimenting with workflows without the overhead of having to set up a local development environment. I am curious how you store secrets (e.g. AWS access key id, secret access key). There isn't a login wall and it's not clear that the values will be protected from you, so I am loathe to put my credentials on there. Great work overall. Wish you the best.
- zomglings 6y ago
  After clicking logout, I had no way of logging back in. :)
  - newcrobuzon 6y ago
    Ooops, good catch, I will fix that :)
- newcrobuzon 6y ago
  Thanks for the kind words! Yes, I would still be careful with passwords in the public beta. It goes via SSL and you get a unique UUID URL so all-in-all it is kinda secured (plus time-to-live of the instance is only 3 hours which limits any time to hack it) but still this is not 100% secure and I would not recommend it for any kind of production use (including any use of passwords you dont want exposed). The free instances are not (yet) password protected (other then the UUID) - this on the other hand is useful if you want to share the envrionment with somebody, just send them the link... This is just a beginning of the public hosting so I will need to think through further improvements that would go into the hosting, security-wise and other aspects as well. My main objective at this stage for the free hosted instances was to give people way to play with Titnaoboa or quickly test something, especially if something is not working in their local environment (say dependencies) and they want quickly re-test in vanilla environment. If you want to add aws secrets or what not I would suggest to just download/install your local version, it is super easy: https://github.com/mikub/titanoboa#installation https://github.com/mikub/titanoboa#installation Or if you need a public IP to integration just grab the docker instance and spin it in your AWS ECS or something.
phonon 6y ago
How does this compare with https://github.com/openintegrationhub/openintegrationhub https://github.com/openintegrationhub/openintegrationhub ?
slotrans 6y ago
This looks very interesting. The visual component in particular seems very well done. I have a deep interest in DAG-structured ETL tooling and had a couple of questions that the documentation didn't seem to address... 1. Can I execute workflows without a server running? Something like... $ java -jar titanoboa.jar MyWorkFlowName arg1 arg2 ... ...and then my workflow executes, as a program, on my machine, until it's done and then exits? Or does every workflow always execute within the context of a running server? 2. Is there any notion of resuming a partially failed workflow? As a point of comparison, Luigi structures its DAG concept using Tasks which create Targets, and invoking a Task whose Target already exists is a no-op, so if you have a big execution graph that gets 80% finished and then dies, you can easily restart it. I find that many competing tools are missing this concept.
- slotrans 6y ago
  Now that it's Saturday I've had time to play with this and I believe I've found the answers to my questions... 1. Based on the Clojure REPL example in the main README, I think the answer is YES, though not exactly the way I had imagined. It seems what you would need to do is write a top-level script (or java "main") that starts a "system", starts "workers", runs your job using that system, then stops the system and exits. A little clunky compared to how Luigi does it, but usable. 2. Best as I can tell the answer is NO. Neither the documented API, nor the implementation of the API in src/clj/titanoboa/handler.clj contain any hint of an ability to operate on a job id, beyond retrieving the result of its execution. Additional commentary: 1. Resuming failed jobs As implied by my question above, the ability to resume a failed job is essential. One of the major reasons to adopt DAG-structured code is parallel execution, and Titanoboa has that. But the OTHER major reason is to allow partially-failed computation to retry/resume without repeating already-completed work. In particular in the ETL space, we often have job graphs composed of hundreds of nodes, with total runtimes measured in hours. If my 100-node job graph fails due to an error in node #78, preventing an additional 15 downstream nodes from running, I don't want want to run all 100 nodes again after I fix the problem. I want to resume executing my graph at #78, and expect only the 16 total affected nodes to execute, since everything else ran correctly the first time (and presumably persisted their outputs). Luigi gets this one right. Airflow sorta tries but it's clunky and you can tell it's not a priority. 2. Flow/Dependency direction When designing a workflow, either in the GUI or as EDN, you tell Titanoboa what jobs are "next". This is intuitive because it comports with our notion of execution flow through a graph of jobs, but it gets things backwards. That is, when we write A->B->C, we are thinking that A will execute, and then B, and then C (perhaps results will be passed from step to step). It is often better though to describe this as A<-B<-C, which reads as C depends on B, B depends on A, and A depends on nothing. Structuring our thinking in this way focuses the mind on what inputs a node requires in order to perform its effect or compute its output, rather than on what operations should follow it in time. Luigi and Airflow both get this one right. 3. Properties The way Titanoboa defines workflow-level "properties", into which job-level properties are merged, and the way properties flow along the path of execution, is very nice. A constant problem with Luigi is how to flow values from one Task to the next without using an excessive number of Parameters. I can't say for sure that Titanoboa's properties construct doesn't have the same problems, without taking the time to actually use it to build a large project, but on the surface it looks good. 4. Logging I noticed that when a step's function returns a map, to be integrated into "properties", that return value is not logged. The message in the log is like "Step [my-cool-step-name] finshed with result []" which is both unhelpful, and not even literally true, as it most certainly did have a result! When a step returns a scalar value, it does get logged. I found this inconsistency frustrating. Also, the stdout/stderr of each step function apparently goes to /dev/null. I find this odd as the placeholder function when you build a new workflow is (println "Hello World!") but if you actually execute that you'll discover that our classic greeting vanishes into the void. This is a major shortcoming. As a point of comparison, one of the biggest value-adds of using Jenkins as a job scheduler is how it automatically captures the output of anything you run, saves it in a durable log file, AND lets you view it in real time. Job orchestration systems that don't match that level of log-friendliness drive me nuts. 5. Versioning The built-in versioning system is great. Two thumbs up. I don't know how it would work if I were writing my jobs in proper Clojure or Java code in their own repo, but I kinda don't care because the value of storing and versioning what I do in the UI is so great. 6. UI -> data I love the way the interactive UI is just there to generate EDN. In a way this mirrors how Jenkins' UI builds its job XML files, but you have to go hunting for those and they're hard to read (because XML). Being able to see what EDN is generated by your actions in the UI, _right there in the UI_, is fantastic. 7. UI issues The UI is great but it has quite a bit of low-hanging-fruit improvements that could be made. - the run job popup forces you to choose a system every time, even if there is only one - being able to draw arrows in the visualization is cool, but I could not figure out how to delete them there. Needs work. - the UI doesn't lay out well on small screens (I'm on an old 13" Air), I had to zoom to 80% just to be able to see the X to delete a property. It would help if the Workflows panel on the left (the least important UI element by far!) could be collapsed (edit: it can be collapsed, but the collapse button is on the other side of the screen which makes no sense) - the box that pops up after starting a job has nothing clickable in it. I have to close it and go to the jobs tab - the jobs tab doesn't refresh when it loads, even if I just started a job, which needlessly adds clicks to the main workflow - the jobs tab has an "archived" sub-tab but no apparent way to actually move a job to the archive Overall, there's a lot of promise here, and it's amazing to me that you built this by yourself. Still, it has a long way to go. I recommend spending some time with Luigi, which I still think is the best general way of DAG-structuring real world workflow code, and with Jenkins which remains far and away the best UI-driven job orchestration system. It seems you're already familiar with Airflow, but I would recommend you treat it mainly as an example of what not to do.

279 points by newcrobuzon 6y ago

Hi HN peeps! I single-handedly built Titanoboa (a workflow automation for JVM) and now I am experimenting with hosting dockerized Titanoboa instances - so I built a free hosting service. If you want to play with Titanoboa in your browser without installing anything, give it a try. Feel free to check out Titanoboa on Github: https://github.com/mikub/titanoboa https://github.com/mikub/titanoboa Also: This is an early beta so please do let me know if something breaks or you spot a bug. Atm I have load-balancers set up only on West coast and in Europe, so apologies to folks from down under & similar locations, let me know if it's too laggy :)

cateye 6y ago
Why do you describe it as "automation for JVM" instead of "automation that runs on JVM"?
- newcrobuzon 6y ago
  Good point, not a native speaker so I thought this would roughly mean the same. The main message I wanted to convey was that this runs on JVM and multiple JVM languages (java and clojure) could be used.

kevsim 6y ago

Impressive feat of engineering for one person! But is it really an alternative to Zapier? I mean, while Zapier is obviously about automation, it always felt that their big selling point was the sheer number of integrations they provide.

newcrobuzon 6y ago
My aim is to make this easier to customize - i.e. develop custom steps on the run and have an instant feedback loop as you develop or customize existing integrations. Target audience might be slightly different ultimately, but we'll see - I would envision this to be more useful in an enterprise environment where you pretty much always have to customize the integrations that were provided out of the box.

newcrobuzon 6y ago

Wow, this has blown up a bit, so I have added few more servers (now two servers on the West coast and two in Europe), but ultimately I think there is a limit cap of 100 Titanoboa instances in each geography in parallel. So if we break that level please don't be mad if you don't get your instance :) Instead give me a star on github and come back later :) Cheers Miro

MiroF 6y ago
Wishing you success, fellow Miro.
newcrobuzon 6y ago
Ok, I have just reached 100 concurrent containers in Europe, which is max at this stage and I don't think I can change the aws limit on the fly (will check though) so I might start killing the oldest instances. Apologies for inconvenience folks, but I am sure people rarely play with each instance for the whole 3 hours anyway at this stage. Honestly did not expect this kind of "public beta" rollout, still surprised how well the service holds so far under the load.
- newcrobuzon 6y ago
  Approaching 100 containers cap in US, so if something does not work, dont be mad :)
  - newcrobuzon 6y ago
    If anybody's curious about this "public beta hosting - HN bear hug" performance load test: I just (sequentially) restarted all 4 servers behind my 2 load balancers - seems that by reaching 100 containers in each geography the provisioning threads may have died after many (unsuccessful) retries, so need to restart the thread pools. That is one more error I need to handle (or increasing my limit in AWS).
    - erulabs 6y ago
      Heya - just a note that I build these kinds of PaaS platforms for a living - if you need a hand building a super-massive cluster for lots of users, let me know!

foolinaround 6y ago

is there some recipes or some common scenarios one can 'import' in to check it out?

newcrobuzon 6y ago
You can check out https://github.com/mikub/titanoboa-tasklets https://github.com/mikub/titanoboa-tasklets And also https://github.com/mikub/titanoboa-sample-workflows https://github.com/mikub/titanoboa-sample-workflows There are few blog posts I wrote about some sample workflows, e.g.: https://github.com/mikub/titanoboa/wiki/Getting-Started-with-GUI https://github.com/mikub/titanoboa/wiki/Getting-Started-with... https://www.titanoboa.io/using-titanoboa-for-it-automation.html https://www.titanoboa.io/using-titanoboa-for-it-automation.h... https://www.titanoboa.io/using-titanoboa-for-cloud-integration.html https://www.titanoboa.io/using-titanoboa-for-cloud-integrati... https://www.titanoboa.io/java-repl.html https://www.titanoboa.io/java-repl.html

jchook 6y ago

@newcrobuzon you have some kind of comment string output at the top of your html body tag. Looks like something you do not want exposed. Congratulations on reaching public beta and for submitting to HN! Looks very promising!

newcrobuzon 6y ago
Thanks! If it is the recapcha string then it is just an old public string for local testing. But I would upvote you 100x or send you a medal or something if I could, this is really super nice of you letting me know!

juliend2 6y ago

Very interesting! What library did you use for the SVG powered diagram tool?

juliend2 6y ago
I think this blog post answers my question: https://titanoboa.io/visualizing-workflows.html https://titanoboa.io/visualizing-workflows.html Very cool to see how you use clojurescript for this.
newcrobuzon 6y ago
I am using D3. AFAIK nobody has used it for worfklow visualization so far: https://www.titanoboa.io/visualizing-workflows.html https://www.titanoboa.io/visualizing-workflows.html
- X6S1x6Okd1st 6y ago
  Luigi does some workflow visualization with d3 https://luigi.readthedocs.io/en/stable/index.html https://luigi.readthedocs.io/en/stable/index.html
  - newcrobuzon 6y ago
    Thanks, wasn't aware of them!
    - X6S1x6Okd1st 6y ago
      It's a very code forward one, but I've certainly found it useful and easier to understand than something like airflow.

sixhobbits 6y ago

I've used Zapier a fair amount and I wrote an article about Airflow so I have a fairly good understanding of that too. I would never really consider them alternatives though? To me Zapier is a low/no code tool that offers a bazillion integrations and Airflow is a workflow orchestration tool. So comparing to both of them confuses me and I guess choosing one would give you a more nieche audience but one that you can connect better to as well.

newcrobuzon 6y ago
This is a very good point. My current line of thinking is: A workflow is a workflow is a workflow. Titanoboa is built the way that pretty much everything it shuffles is data - and either it can be big (Airflow) or the steps can have side effects (iPaaS/Zapier). The "no code" is achievable since adding more predefined steps is not that hard (there's not that many at the moment though) - see for instance https://github.com/mikub/titanoboa/wiki/Getting-Started-with-GUI https://github.com/mikub/titanoboa/wiki/Getting-Started-with... - you dont code, you just fill in properties, so it is very straightforward, pretty much like Zappier. My aim in the iPaaS space is more at the enterprise level, where the predefined steps won't do anymore and you have to custom develop them anyway - so in that are I think Titanoboa shines since you really can rapidly prototype steps on the fly, a bit like Repl.it and Zapier together... I agree that it might be good to pick one audience, at this stage I am just experimenting if somehow I can market to both and don't mess up and confuse both groups.
- j45 6y ago
  Keep converging and building. I have been experimenting with combinations of huggin, camunda, airflow and others to try and achieve an integrated workflow/state/process management. There exists a gap in the enterprise space between all these tools.. and it has been further exacerbated by the disruption introduced to many industries by covid. There is a real opportunity for small and medium businesses to be able to access tooling that is beyond what a base line Zap, etc can accomplish. Would be happy to connect offline.
  - 6y ago

bambam24 6y ago

I stop watching it when he said “here I can add any Maven dependency”

newcrobuzon 6y ago
Well yes, this is for JVM, so maven ecosystem, sorry. I have been contemplating also doing a clone on node.js (titanoboa is written in clojure so it would just be a move from clojure to clojurescript), but I did hear that the npm ecosystem is a bit uninviting too... What dependencies tools would you like to see?

ForHackernews 6y ago

How does this compare to Prefect? https://medium.com/the-prefect-blog/why-not-airflow-4cfa423299c4 https://medium.com/the-prefect-blog/why-not-airflow-4cfa4232...

newcrobuzon 6y ago
To be honest Prefect just got on my radar last week, so I will need to look at it more closely, I was not aware of them till then. Obviously this is for JVM, plus I am not sure how Prefect addresses following two points I mainly focus on: - make rapid prototyping of new steps possible during runtime, without any need to restart/redeploy - focus on distributed processing, where in master-less Titanoboa cluster you can have pretty much any number of nodes
- ramon 6y ago
  Prefect is like an hour just to put up the kubernetes environment required for it. The way you have it you can just do a simple docker run and start coding this is the comparison and you're coding on the web no need to steup VSCode or nothing it's fast.

j45 6y ago

Being a JVM friendly tool, it would be great to unlock other JVM syntaxes to allow scripting in any lanugage.. Jython for Python, Quercus for PHP, JRuby for Ruby, etc.

newcrobuzon 6y ago
yes, I am considering this and it should not be that hard (also Kotlin, Nashorn javascript...), but at this stage Titanoboa is just a one-man-show - so patience, I might get there ;)
- j45 6y ago
  For sure. Being a one man army has it's perks and challenges. I hope this project has a community form around it. The JVM is more underrated/unknown than it should be in the startup space, tools like what you're making make it far more accessible.
- ramon 6y ago
  newcrobuzon you're a genius my friend someone should give you an award I loved your idea and how you approached it it's fantastic for developers. I'm a root Java dev and I have to say you did fantastic work to make developers life easier, you didn't take any approach you made your own this is awesome. If you get Javascript working I feel like I'm getting back into AppJet days! This is fantastic stuff. If you show me how I can help get the javascript stuff going I would be glad to help you in this initative since I have adopted NodeJS recently like you said, being a one man army you have to be efficient and JS makes me more efficient.
  - newcrobuzon 6y ago
    Thanks ramon! Appreciate your kind words! Feel free to shoot me an email or reach me on twitter and let's stay in touch!
- mdaniel 6y ago
  FWIW, I would have to check all the ones listed but if they're sane scripting languages on the jvm then they'll implement the JSR-223 interface allowing one to pick them at runtime without a huge amount of drama: https://jcp.org/aboutJava/communityprocess/final/jsr223/index.html https://jcp.org/aboutJava/communityprocess/final/jsr223/inde...

deleted 6y ago

[deleted]

PascalW 6y ago

Nice work! How does it compare to n8n? [0] n8n is the closest OSS alternative to Zapier I've seen so far. [0] https://n8n.io/ https://n8n.io/

newcrobuzon 6y ago
A bit similar, but on JVM. The main focus of Titanoboa is however as follows: - make rapid prototyping of new steps possible during runtime, without any need to restart/redeploy - focus on distributed processing, where in master-less Titanoboa cluster you can have pretty much any number of nodes
lulzx 6y ago
It's license (fair code) sucks.
- newcrobuzon 6y ago
  That is a fair criticism, I am aware of the AGPL license shortcomings. I just picked up the more restrictive license at the beginning - being a sole funder and not working on this full time etc. I simply did not want somebody (e.g. a big company with a big team) grabbing my code along the way and running away with it. Since now Titanoboa got to the shape I envisioned it to be in I am starting to focus more on adoption, so yes I am definitely thinking about switching to less restrictive license since it will probably help. Also at the beginning I was not aware how badly AGPL is perceived (I always thought if it was good for Mongo it could work for me, but I may have been wrong).
  - cstuder 6y ago
    You misunderstood, it's not you who's being criticized. n8n used to write "open source" on their webpage while publishing the product with a proprietary license. A lot of discussions ensued. https://github.com/n8n-io/n8n/issues/40 https://github.com/n8n-io/n8n/issues/40
    - newcrobuzon 6y ago

caymanjim 6y ago

First impression: rename it now while the cost is low. The name is unwieldy.

everfree 6y ago
Ti-ta-no-bo-a 5 syllables. Agreed that this is unwieldy to say in normal speech.
- newcrobuzon 6y ago
  I mostly considered naming to be a distraction from getting the real work done. I also am probably not good with names as my other alternative was Megalodon (4 syllables) - I just wanted to have some megafauna name. Happy to hear what you folks would suggest :)
  - afarviral 6y ago
    "Dinornis" or "Moa" in maori.
    - StavrosK 6y ago
      If you're going with the former, you might as well spell it "deinornis", it looks cooler and is more correct.
  - acidbaseextract 6y ago
    Titano is nice, but may have some trademark issues. Tanobo is easy to say, if a bit generic. Just playing with syllables.
  - ohyeshedid 6y ago
    A fauna focused synonym for canopy?
  - dddw 6y ago
    Gdoc form for suggestions. 1 mail to all users (small enticement, reward for winner, pro account for life or something else really nice). Remove any lewd suggestions. Then pick one you love, or if you can't choose a poll.

itpragmatik 6y ago

How does it compare with Cadence/Temporal?

prabhatjha 6y ago

Nicely done. Couple of recommendations: - Recommend to change your licensing if you want people to try. - I understand why you want to have AGPL but I don't know many successful OSS projects with this license. There is another HN thread on this topic. https://news.ycombinator.com/item?id=1657071 https://news.ycombinator.com/item?id=1657071

WrtCdEvrydy 6y ago

Honestly I really like this, I can't wait to use it, but my concerns are around certain features being behind a paywall, especially HA and Clustering on the self-hosted version. I understand that the cloud hosted version should cost more for these features but without a way to get pricing for the self-hosted HA and clustered version it will be a tough sell for me to deploy it, use it and have to scrap it in the future.

ramon 6y ago

I liked the video demo really nice and I would say this should be called not low-code or no-code but rather aPaaS which is Application Platform as a Service. So the title should be "Host your own aPaaS" instead.

thrownaway954 6y ago

that is the crappiest website i've seen in quite sometime. my first impression was that this is either a scam or a joke. in this day and age, if you are can't afford a decently designed site, how am i ever even going to give your product a chance.

dang 6y ago
This comment breaks HN's rules badly. Please don't post like this again, regardless of how someone's website is. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/showhn.html https://news.ycombinator.com/showhn.html

zomglings 6y ago

This looks really well done, newcrobuzon. Very impressed. I especially love your web UI. It makes it very easy to start experimenting with workflows without the overhead of having to set up a local development environment. I am curious how you store secrets (e.g. AWS access key id, secret access key). There isn't a login wall and it's not clear that the values will be protected from you, so I am loathe to put my credentials on there. Great work overall. Wish you the best.

zomglings 6y ago
After clicking logout, I had no way of logging back in. :)
- newcrobuzon 6y ago
  Ooops, good catch, I will fix that :)
newcrobuzon 6y ago
Thanks for the kind words! Yes, I would still be careful with passwords in the public beta. It goes via SSL and you get a unique UUID URL so all-in-all it is kinda secured (plus time-to-live of the instance is only 3 hours which limits any time to hack it) but still this is not 100% secure and I would not recommend it for any kind of production use (including any use of passwords you dont want exposed). The free instances are not (yet) password protected (other then the UUID) - this on the other hand is useful if you want to share the envrionment with somebody, just send them the link... This is just a beginning of the public hosting so I will need to think through further improvements that would go into the hosting, security-wise and other aspects as well. My main objective at this stage for the free hosted instances was to give people way to play with Titnaoboa or quickly test something, especially if something is not working in their local environment (say dependencies) and they want quickly re-test in vanilla environment. If you want to add aws secrets or what not I would suggest to just download/install your local version, it is super easy: https://github.com/mikub/titanoboa#installation https://github.com/mikub/titanoboa#installation Or if you need a public IP to integration just grab the docker instance and spin it in your AWS ECS or something.

phonon 6y ago

How does this compare with https://github.com/openintegrationhub/openintegrationhub https://github.com/openintegrationhub/openintegrationhub ?

slotrans 6y ago

This looks very interesting. The visual component in particular seems very well done. I have a deep interest in DAG-structured ETL tooling and had a couple of questions that the documentation didn't seem to address... 1. Can I execute workflows without a server running? Something like... $ java -jar titanoboa.jar MyWorkFlowName arg1 arg2 ... ...and then my workflow executes, as a program, on my machine, until it's done and then exits? Or does every workflow always execute within the context of a running server? 2. Is there any notion of resuming a partially failed workflow? As a point of comparison, Luigi structures its DAG concept using Tasks which create Targets, and invoking a Task whose Target already exists is a no-op, so if you have a big execution graph that gets 80% finished and then dies, you can easily restart it. I find that many competing tools are missing this concept.

slotrans 6y ago
Now that it's Saturday I've had time to play with this and I believe I've found the answers to my questions... 1. Based on the Clojure REPL example in the main README, I think the answer is YES, though not exactly the way I had imagined. It seems what you would need to do is write a top-level script (or java "main") that starts a "system", starts "workers", runs your job using that system, then stops the system and exits. A little clunky compared to how Luigi does it, but usable. 2. Best as I can tell the answer is NO. Neither the documented API, nor the implementation of the API in src/clj/titanoboa/handler.clj contain any hint of an ability to operate on a job id, beyond retrieving the result of its execution. Additional commentary: 1. Resuming failed jobs As implied by my question above, the ability to resume a failed job is essential. One of the major reasons to adopt DAG-structured code is parallel execution, and Titanoboa has that. But the OTHER major reason is to allow partially-failed computation to retry/resume without repeating already-completed work. In particular in the ETL space, we often have job graphs composed of hundreds of nodes, with total runtimes measured in hours. If my 100-node job graph fails due to an error in node #78, preventing an additional 15 downstream nodes from running, I don't want want to run all 100 nodes again after I fix the problem. I want to resume executing my graph at #78, and expect only the 16 total affected nodes to execute, since everything else ran correctly the first time (and presumably persisted their outputs). Luigi gets this one right. Airflow sorta tries but it's clunky and you can tell it's not a priority. 2. Flow/Dependency direction When designing a workflow, either in the GUI or as EDN, you tell Titanoboa what jobs are "next". This is intuitive because it comports with our notion of execution flow through a graph of jobs, but it gets things backwards. That is, when we write A->B->C, we are thinking that A will execute, and then B, and then C (perhaps results will be passed from step to step). It is often better though to describe this as A<-B<-C, which reads as C depends on B, B depends on A, and A depends on nothing. Structuring our thinking in this way focuses the mind on what inputs a node requires in order to perform its effect or compute its output, rather than on what operations should follow it in time. Luigi and Airflow both get this one right. 3. Properties The way Titanoboa defines workflow-level "properties", into which job-level properties are merged, and the way properties flow along the path of execution, is very nice. A constant problem with Luigi is how to flow values from one Task to the next without using an excessive number of Parameters. I can't say for sure that Titanoboa's properties construct doesn't have the same problems, without taking the time to actually use it to build a large project, but on the surface it looks good. 4. Logging I noticed that when a step's function returns a map, to be integrated into "properties", that return value is not logged. The message in the log is like "Step [my-cool-step-name] finshed with result []" which is both unhelpful, and not even literally true, as it most certainly did have a result! When a step returns a scalar value, it does get logged. I found this inconsistency frustrating. Also, the stdout/stderr of each step function apparently goes to /dev/null. I find this odd as the placeholder function when you build a new workflow is (println "Hello World!") but if you actually execute that you'll discover that our classic greeting vanishes into the void. This is a major shortcoming. As a point of comparison, one of the biggest value-adds of using Jenkins as a job scheduler is how it automatically captures the output of anything you run, saves it in a durable log file, AND lets you view it in real time. Job orchestration systems that don't match that level of log-friendliness drive me nuts. 5. Versioning The built-in versioning system is great. Two thumbs up. I don't know how it would work if I were writing my jobs in proper Clojure or Java code in their own repo, but I kinda don't care because the value of storing and versioning what I do in the UI is so great. 6. UI -> data I love the way the interactive UI is just there to generate EDN. In a way this mirrors how Jenkins' UI builds its job XML files, but you have to go hunting for those and they're hard to read (because XML). Being able to see what EDN is generated by your actions in the UI, _right there in the UI_, is fantastic. 7. UI issues The UI is great but it has quite a bit of low-hanging-fruit improvements that could be made. - the run job popup forces you to choose a system every time, even if there is only one - being able to draw arrows in the visualization is cool, but I could not figure out how to delete them there. Needs work. - the UI doesn't lay out well on small screens (I'm on an old 13" Air), I had to zoom to 80% just to be able to see the X to delete a property. It would help if the Workflows panel on the left (the least important UI element by far!) could be collapsed (edit: it can be collapsed, but the collapse button is on the other side of the screen which makes no sense) - the box that pops up after starting a job has nothing clickable in it. I have to close it and go to the jobs tab - the jobs tab doesn't refresh when it loads, even if I just started a job, which needlessly adds clicks to the main workflow - the jobs tab has an "archived" sub-tab but no apparent way to actually move a job to the archive Overall, there's a lot of promise here, and it's amazing to me that you built this by yourself. Still, it has a long way to go. I recommend spending some time with Luigi, which I still think is the best general way of DAG-structuring real world workflow code, and with Jenkins which remains far and away the best UI-driven job orchestration system. It seems you're already familiar with Airflow, but I would recommend you treat it mainly as an example of what not to do.