4 ms·

I have this https://github.com/Azure/go-asyncjob https://github.com/Azure/go-asyncjob library as well, with generic strongType connecting each step input/output

• verdverm 2y ago
  CUE has a package (cue/flow) that has a default implementation but also allows you to build custom tasks on top of the engine & evaluator. The nice thing about this setup is that it infers the dependencies between tasks based on the references to inputs/outputs between them. The main drawback of most of these things is that they don't scale beyond a single machine. That's why people choose tools like Airflow. https://pkg.go.dev/cuelang.org/go@v0.10.1/tools/flow https://pkg.go.dev/cuelang.org/go@v0.10.1/tools/flow (cue cmd is the default implementation of this) https://docs.hofstadter.io/getting-started/task-engine/ https://docs.hofstadter.io/getting-started/task-engine/ (custom example)
  • noneback 2y ago
    Agree. lightweight embedded libs like go-taskflow are used in a much different way compared to distributed workflow systems. Coders should be aware of the pros and cons of those repos or systems.
  • xlii 2y ago
    For what is worth I spent couple weeks evaluating Cue and I don’t think it’s mature enough for production use in context of task runner. There are undocumented quirks (and documentation gaps) requiring hours of experimentation, some subtle bugs and ergonomy paper cuts. For solution integration solution has to be approachable, and as much as I have sympathy for Cue it just isn’t one. It feels like one needs PhD in software engineering to use it. This is IMO reason why people go for things like Airflow - popularity and ease of entry. Using Cue in distributed environment isn’t that hard. It can be used directly from Go or wrapped in Erlang/Elixir port.