5 ms·

Show HN: Odin – The Observable and Distributed Workflow System

84 points by ojs 6y ago

samblr 6y ago
Point the submitted link to repo (it pointing to readme)
mottosso 6y ago
Thought this was related to Odin the programming language. https://github.com/odin-lang/Odin https://github.com/odin-lang/Odin
- noarchy 6y ago
  Or Odin the flashing software for Samsung devices. https://samsungodin.com/ https://samsungodin.com/
- ojs 6y ago
  So much software seems to share the name, I am are currently trying to figure out something a little more unique!
  - throwaway894345 6y ago
    A more obscure figure from the Germanic pantheon, perhaps?
    - ojs 6y ago
      Trying to move away from figures of any pantheon now!
  - white-flame 6y ago
    Just start concatenating words. If you like "Odin", but it's too common, just use something like "OdinFlow" which integrates something about its domain. The actual full name doesn't matter that much, and it can still be called "Odin" as shorthand in the docs, but this way the full name can still have both the implications you want and uniqueness. Plus, adding "Flow" now makes it a tiny bit more self-describing than just a proper name.
deleted 6y ago
[deleted]
lordofgibbons 6y ago
Thanks for posting this. I'm curious how this differs from Uber/Cadence, and Temporal (https://www.temporal.io/ https://www.temporal.io/) which is a fork of cadence. They're both also distributed workflow scheduling and state management
- maxmcd 6y ago
  This seems to be more of a distributed cron or general purpose job runner. I believe Temporal/Cadence allow for much more rich composition. Linked actions with Odin seems limited to this mechanism: https://github.com/theycallmemac/odin/blob/master/DOCS.md#link-jobs-together https://github.com/theycallmemac/odin/blob/master/DOCS.md#li...
- dnautics 6y ago
  I got confused because I'm most faimilar with Oban, which is a distributed workflow manager and scheduler written in a different language: https://github.com/sorentwo/oban https://github.com/sorentwo/oban
- ojs 6y ago
  So Odin is actually configurable workflows in four supported runtime at the moment - Python, Go, Node and Bash. Temporal seems to only support Java and Go as far as I can see! Specifically, the observability hooks in Odin would be the differentiating feature. Information is gathered to help infer the internal state of jobs. This means Odin can directly help diagnose where the problems are and get to the root cause of any interruptions. Debugging, but you're one step ahead.
rubio8 6y ago
Nice job on this it looks well featured. Would like to see a Ruby SDK.
- ojs 6y ago
  I'm not a very confident ruby developer but it's definitely going to be something added to the development roadmap!
sumobashriki 6y ago
What role does MongoDB serve? Why would you even use MongoDB instead of Badger or Bolt
- ojs 6y ago
  Mongo serves as a data store for job metadata, which is in turn then access to be displayed on the dashboard. We are currently working on having the database component as something more pluggable, so down the line you will be able to use the DB of your preference :D
- threeseed 6y ago
  Because Badget/Bolt are key/value stores and MongoDB is a document store. I can't imagine how anyone can model even moderately complex use cases on a key/value store. Even with some sort of ORM filtering and manipulation would be very cumbersome.
commotionfever 6y ago
Hallo james
- ojs 6y ago
  Hallo senan
ttymck 6y ago
This looks awesome! Excited to give it a spin. If I wanted to take a stab at a Scala SDK, is that something you would be interested in supporting, or should community/contrib SDKs be managed as separate projects/plugins?
- ojs 6y ago
  If you wanted to give a Scala SDK a go that would be great! We'd certainly be interested in any contributions you're willing to make to the project!
steventhedev 6y ago
Few bits of advice/comments: 1. the schedule syntax. It can seem much easier to work directly with English, but there's a reason most systems use cron-style schedules and it isn't because it's legacy. There's a benefit in having a syntax that is "alien" when you're crossing that human/computer/dev border, as it forces your users to think about how to achieve something. The other half of the reason is because it's a lot easier to validate a strict syntax with a formal grammar than anything even close to natural language. I can basically guarantee you that if someone had come up with a way to accurately parse english recurrence statements, it would have already been adopted by cron in the past (or at least a website like crontab guru[0]. 2. There is no description of the job execution semantics in the distributed sense. Is it at most once, or at least once? This is important if you're designing jobs, so if someone is switching to your system from an existing system with a different semantic it will cause a lot of friction especially if you don't even list it in your documentation. 3. You have two data stores. You're using Raft for backing your key/value store, and mongo for both job metadata and also communicating with the dashboard. This is a bit confusing because in the docs you wrote that metadata goes in a sidecar yaml file. What job metadata are you referring to then? Results? Fundamentally, having multiple data stores makes your projects orders of magnitude more complicated to operate and monitor. My guess is you'll see this as a major pain point in the future when you want to scale a cluster in different ways (more jobs, more frequent jobs, longer histories, more nodes in the cluster). All in all, it looks like a pretty decent distributed cron, but I wouldn't touch your project for my production systems. It's lacking a lot of features we need for getting Legacy (TM) work done[1] like running a specific job on a specific worker because it needs access to special hardware or some filesystem or because our PHB decided that as a policy only one of our servers can run billing code on Tuesdays. [0]: https://crontab.guru/ https://crontab.guru/ [1]: "Legacy code is code that makes money"
- ojs 6y ago
  Replying to each of your points: 1. I completely agree. We have recently acknowledged the power in an "alien" syntax as you put it. In v2.0.0 Odin will also support the classic cron schedule string syntax! 2. You are right, the repo is lacking a detailed description on this. The execution semantics boils down to at most once. 3. The Yaml file just lends some simple schedule and runtime info to the job. We actually would rather this moved into the code in v2.0.0 - that way all workflow info is self contained and no supplementary files are needed. The multiple data store thing is entirely an issue, in v.2.0.0 we are aiming to make more aspects (such as you have described) pluggable, using whatever data stores you specify. That will require a lot of docs but we are up to task. That you so much for the advice comments, I will share them with the development team, feedback is always appreciated :)
saxonww 6y ago
[edited to remove snark] An earlier discussion on why Odin: https://news.ycombinator.com/item?id=23460066 https://news.ycombinator.com/item?id=23460066
- boulos 6y ago
  I think that Coinbase’s Odin (https://github.com/coinbase/odin https://github.com/coinbase/odin) is not the same as this person’s personal project. They’re certainly not the same repo.
  - saxonww 6y ago
    yikes, my mistake.
meritt 6y ago
This looks like a really complex crontab that requires mongoDB of all things.
jitl 6y ago
Does Odin support task queuing like SQS, RabbitMQ, Redis, etc, or is this just for scheduled jobs? The debugging info looks very cool, but I need this for queue jobs more than scheduled jobs. Relatedly, how many jobs/s or queries/s do you expect Odin to support in the suggested 3-leader configuration on typical VMs? What kind of load do you run this in production?
AtlasBarfed 6y ago
I harp on orchestration and workflow so I'm a bit of a cranky old man on this subject. I get you can technically do orchestration and workflow with this, but you can also technically do it with any turing complete language. Orchestration and workflow should involve UI design elements for workflows, integration/interfaces, and visual execution tracking and workflow instance initiating? And yeah, MongoDB, which just got torn apart by aphyr for anything concurrent or distributed.

84 points by ojs 6y ago

Point the submitted link to repo (it pointing to readme)

Thought this was related to Odin the programming language. https://github.com/odin-lang/Odin https://github.com/odin-lang/Odin

noarchy 6y ago
Or Odin the flashing software for Samsung devices. https://samsungodin.com/ https://samsungodin.com/
ojs 6y ago
So much software seems to share the name, I am are currently trying to figure out something a little more unique!
- throwaway894345 6y ago
  A more obscure figure from the Germanic pantheon, perhaps?
  - ojs 6y ago
    Trying to move away from figures of any pantheon now!
- white-flame 6y ago
  Just start concatenating words. If you like "Odin", but it's too common, just use something like "OdinFlow" which integrates something about its domain. The actual full name doesn't matter that much, and it can still be called "Odin" as shorthand in the docs, but this way the full name can still have both the implications you want and uniqueness. Plus, adding "Flow" now makes it a tiny bit more self-describing than just a proper name.

deleted 6y ago

[deleted]

lordofgibbons 6y ago

Thanks for posting this. I'm curious how this differs from Uber/Cadence, and Temporal (https://www.temporal.io/ https://www.temporal.io/) which is a fork of cadence. They're both also distributed workflow scheduling and state management

maxmcd 6y ago
This seems to be more of a distributed cron or general purpose job runner. I believe Temporal/Cadence allow for much more rich composition. Linked actions with Odin seems limited to this mechanism: https://github.com/theycallmemac/odin/blob/master/DOCS.md#link-jobs-together https://github.com/theycallmemac/odin/blob/master/DOCS.md#li...
dnautics 6y ago
I got confused because I'm most faimilar with Oban, which is a distributed workflow manager and scheduler written in a different language: https://github.com/sorentwo/oban https://github.com/sorentwo/oban
ojs 6y ago
So Odin is actually configurable workflows in four supported runtime at the moment - Python, Go, Node and Bash. Temporal seems to only support Java and Go as far as I can see! Specifically, the observability hooks in Odin would be the differentiating feature. Information is gathered to help infer the internal state of jobs. This means Odin can directly help diagnose where the problems are and get to the root cause of any interruptions. Debugging, but you're one step ahead.

rubio8 6y ago

Nice job on this it looks well featured. Would like to see a Ruby SDK.

ojs 6y ago
I'm not a very confident ruby developer but it's definitely going to be something added to the development roadmap!

sumobashriki 6y ago

What role does MongoDB serve? Why would you even use MongoDB instead of Badger or Bolt

ojs 6y ago
Mongo serves as a data store for job metadata, which is in turn then access to be displayed on the dashboard. We are currently working on having the database component as something more pluggable, so down the line you will be able to use the DB of your preference :D
threeseed 6y ago
Because Badget/Bolt are key/value stores and MongoDB is a document store. I can't imagine how anyone can model even moderately complex use cases on a key/value store. Even with some sort of ORM filtering and manipulation would be very cumbersome.

commotionfever 6y ago

Hallo james

ojs 6y ago
Hallo senan

ttymck 6y ago

This looks awesome! Excited to give it a spin. If I wanted to take a stab at a Scala SDK, is that something you would be interested in supporting, or should community/contrib SDKs be managed as separate projects/plugins?

ojs 6y ago
If you wanted to give a Scala SDK a go that would be great! We'd certainly be interested in any contributions you're willing to make to the project!

steventhedev 6y ago

Few bits of advice/comments: 1. the schedule syntax. It can seem much easier to work directly with English, but there's a reason most systems use cron-style schedules and it isn't because it's legacy. There's a benefit in having a syntax that is "alien" when you're crossing that human/computer/dev border, as it forces your users to think about how to achieve something. The other half of the reason is because it's a lot easier to validate a strict syntax with a formal grammar than anything even close to natural language. I can basically guarantee you that if someone had come up with a way to accurately parse english recurrence statements, it would have already been adopted by cron in the past (or at least a website like crontab guru[0]. 2. There is no description of the job execution semantics in the distributed sense. Is it at most once, or at least once? This is important if you're designing jobs, so if someone is switching to your system from an existing system with a different semantic it will cause a lot of friction especially if you don't even list it in your documentation. 3. You have two data stores. You're using Raft for backing your key/value store, and mongo for both job metadata and also communicating with the dashboard. This is a bit confusing because in the docs you wrote that metadata goes in a sidecar yaml file. What job metadata are you referring to then? Results? Fundamentally, having multiple data stores makes your projects orders of magnitude more complicated to operate and monitor. My guess is you'll see this as a major pain point in the future when you want to scale a cluster in different ways (more jobs, more frequent jobs, longer histories, more nodes in the cluster). All in all, it looks like a pretty decent distributed cron, but I wouldn't touch your project for my production systems. It's lacking a lot of features we need for getting Legacy (TM) work done[1] like running a specific job on a specific worker because it needs access to special hardware or some filesystem or because our PHB decided that as a policy only one of our servers can run billing code on Tuesdays. [0]: https://crontab.guru/ https://crontab.guru/ [1]: "Legacy code is code that makes money"

ojs 6y ago
Replying to each of your points: 1. I completely agree. We have recently acknowledged the power in an "alien" syntax as you put it. In v2.0.0 Odin will also support the classic cron schedule string syntax! 2. You are right, the repo is lacking a detailed description on this. The execution semantics boils down to at most once. 3. The Yaml file just lends some simple schedule and runtime info to the job. We actually would rather this moved into the code in v2.0.0 - that way all workflow info is self contained and no supplementary files are needed. The multiple data store thing is entirely an issue, in v.2.0.0 we are aiming to make more aspects (such as you have described) pluggable, using whatever data stores you specify. That will require a lot of docs but we are up to task. That you so much for the advice comments, I will share them with the development team, feedback is always appreciated :)

saxonww 6y ago

[edited to remove snark] An earlier discussion on why Odin: https://news.ycombinator.com/item?id=23460066 https://news.ycombinator.com/item?id=23460066

boulos 6y ago
I think that Coinbase’s Odin (https://github.com/coinbase/odin https://github.com/coinbase/odin) is not the same as this person’s personal project. They’re certainly not the same repo.
- saxonww 6y ago
  yikes, my mistake.

meritt 6y ago

This looks like a really complex crontab that requires mongoDB of all things.

jitl 6y ago

Does Odin support task queuing like SQS, RabbitMQ, Redis, etc, or is this just for scheduled jobs? The debugging info looks very cool, but I need this for queue jobs more than scheduled jobs. Relatedly, how many jobs/s or queries/s do you expect Odin to support in the suggested 3-leader configuration on typical VMs? What kind of load do you run this in production?

AtlasBarfed 6y ago

I harp on orchestration and workflow so I'm a bit of a cranky old man on this subject. I get you can technically do orchestration and workflow with this, but you can also technically do it with any turing complete language. Orchestration and workflow should involve UI design elements for workflows, integration/interfaces, and visual execution tracking and workflow instance initiating? And yeah, MongoDB, which just got torn apart by aphyr for anything concurrent or distributed.