4 ms·

Workflow.py: A Minimalist Workflow Engine

70 points by dungwiz 14y ago

127001brewer 14y ago
How is this a workflow engine rather than an interpretation of a *nix daemon? I think this is a good implementation of a Python program, but there's - from what I've personally seen - a tremendous opportunity for an affordable, flexible workflow engine (that's not based on Microsoft SharePoint).
natep 14y ago
Isn't this what inotify and related programs (incron, etc) are for? I expected it to at least be based on python-pyinotify or python-inotifyx.
- mdipierro 14y ago
  Eventually it should be rewritten to use pyinotify. For now it was good enough for my needs.
zobzu 14y ago
So someone remade cron+incrond, in python, using more memory, with a more complicated syntax. Okay. I'll pass.
- jdbernard 14y ago
  Yeah, but it seems like one of those tools you write because you want it to work your way, not necessarily because it doesn't already exist. I have a few of those: tools that already exist but I wrote my own for fun or because it fit my workflow better. Also, the Windows version of cron and the like is not as developer-friendly, IMHO. I do not see why this could not work well on both systems.
arocks 14y ago
This is not a workflow engine but rather a rules engine. A workflow engine is like a navigating a directed graph or a flowchart. Executing one node would take you to the same or different node based on some rules. So a rules engine is typically a part of a workflow engine or invoked by it.
- mdipierro 14y ago
  There are two types of workflow engines: sequential and finite state machines. The latter is more general and includes the former. In a finite state machine external event let you move through the nodes of a graph. The present node is the state. In workflow.py the creation/update of a file is the external event. The state is also characterized by the files present in the file system. The change of state is associated to the execution of a program. I agree it is a little different than what people are used to and I am not arguing semantics. It was useful to me. Hope it is useful to others. That is all. I will be happy to take patches to improve functionality.
  - arocks 14y ago
    Sorry, if I sounded argumentative. I just shared what somebody's perception could be when they read the title 'Workflow Engine'. It took me a while to understand that probably 'Rules Engine' would have been a better nomenclature.
Aaronontheweb 14y ago
Speaking as a dev who uses Python on Windows primarily, I like this a lot. Much more lightweight than Windows Workflow Foundation and doesn't feel like sysadmin work compared to doing stuff with Powershell + Scheduled Services + other parts of Windows automation. In other words, this feels like "developer work" rather than "Windows IT Pro" work :p Kudos!
socratic 14y ago
Does Python have a BPEL/BPM-style workflow engine like Ruby's ruote? This doesn't seem to be it, but I would love to have a workflow engine which is designed for long running tasks, with periodic human interruption, conditional flows, and so on.
- HerrMonnezza 14y ago
  I'm not sure how they compare to Ruby's "ruote", but there are quite a few Python workflow managers and libraries out there. GC3Pie (http://gc3pie.googlecode.com/ http://gc3pie.googlecode.com/) is a Python library for running many-task workflows featuring interruptible execution, interfaces to SGE, PBS and LSF clusters, and composition operators to build dynamic task dependencies (so, not just DAGs). Disclaimer: I'm one of the developers. Weaver (http://bitbucket.org/pbui/weaver http://bitbucket.org/pbui/weaver) is a Python front-end for building workflows that can run on top of the Makeflow engine, supporting SGE, Condor and WorkQueue as execution back-ends. (See the comment by its author "pbui" on another HN thread: http://news.ycombinator.com/item?id=4047100 http://news.ycombinator.com/item?id=4047100) NiPyPe (http://nipy.sourceforge.net/nipype/ http://nipy.sourceforge.net/nipype/) is a Python workflow engine especially targeted at Neuro-Imaging processing (but the core framework is generic, as far as I understand). I'm pretty sure this list is not exhaustive: many people seem to be re-writing the same core functionality, coming from different fields and/or with slightly different requirements.
Morg 14y ago
Indeed the title made the content look shitty. Workflow engine means an engine to drive workflows - at least for me. This thing is meh, no reason to use it over existing C tools, I wonder why someone rererererereinvented the (not actually round) wheel.

70 points by dungwiz 14y ago

How is this a workflow engine rather than an interpretation of a *nix daemon? I think this is a good implementation of a Python program, but there's - from what I've personally seen - a tremendous opportunity for an affordable, flexible workflow engine (that's not based on Microsoft SharePoint).

natep 14y ago

Isn't this what inotify and related programs (incron, etc) are for? I expected it to at least be based on python-pyinotify or python-inotifyx.

mdipierro 14y ago
Eventually it should be rewritten to use pyinotify. For now it was good enough for my needs.

zobzu 14y ago

So someone remade cron+incrond, in python, using more memory, with a more complicated syntax. Okay. I'll pass.

jdbernard 14y ago
Yeah, but it seems like one of those tools you write because you want it to work your way, not necessarily because it doesn't already exist. I have a few of those: tools that already exist but I wrote my own for fun or because it fit my workflow better. Also, the Windows version of cron and the like is not as developer-friendly, IMHO. I do not see why this could not work well on both systems.

arocks 14y ago

This is not a workflow engine but rather a rules engine. A workflow engine is like a navigating a directed graph or a flowchart. Executing one node would take you to the same or different node based on some rules. So a rules engine is typically a part of a workflow engine or invoked by it.

mdipierro 14y ago
There are two types of workflow engines: sequential and finite state machines. The latter is more general and includes the former. In a finite state machine external event let you move through the nodes of a graph. The present node is the state. In workflow.py the creation/update of a file is the external event. The state is also characterized by the files present in the file system. The change of state is associated to the execution of a program. I agree it is a little different than what people are used to and I am not arguing semantics. It was useful to me. Hope it is useful to others. That is all. I will be happy to take patches to improve functionality.
- arocks 14y ago
  Sorry, if I sounded argumentative. I just shared what somebody's perception could be when they read the title 'Workflow Engine'. It took me a while to understand that probably 'Rules Engine' would have been a better nomenclature.

Aaronontheweb 14y ago

Speaking as a dev who uses Python on Windows primarily, I like this a lot. Much more lightweight than Windows Workflow Foundation and doesn't feel like sysadmin work compared to doing stuff with Powershell + Scheduled Services + other parts of Windows automation. In other words, this feels like "developer work" rather than "Windows IT Pro" work :p Kudos!

socratic 14y ago

Does Python have a BPEL/BPM-style workflow engine like Ruby's ruote? This doesn't seem to be it, but I would love to have a workflow engine which is designed for long running tasks, with periodic human interruption, conditional flows, and so on.

HerrMonnezza 14y ago
I'm not sure how they compare to Ruby's "ruote", but there are quite a few Python workflow managers and libraries out there. GC3Pie (http://gc3pie.googlecode.com/ http://gc3pie.googlecode.com/) is a Python library for running many-task workflows featuring interruptible execution, interfaces to SGE, PBS and LSF clusters, and composition operators to build dynamic task dependencies (so, not just DAGs). Disclaimer: I'm one of the developers. Weaver (http://bitbucket.org/pbui/weaver http://bitbucket.org/pbui/weaver) is a Python front-end for building workflows that can run on top of the Makeflow engine, supporting SGE, Condor and WorkQueue as execution back-ends. (See the comment by its author "pbui" on another HN thread: http://news.ycombinator.com/item?id=4047100 http://news.ycombinator.com/item?id=4047100) NiPyPe (http://nipy.sourceforge.net/nipype/ http://nipy.sourceforge.net/nipype/) is a Python workflow engine especially targeted at Neuro-Imaging processing (but the core framework is generic, as far as I understand). I'm pretty sure this list is not exhaustive: many people seem to be re-writing the same core functionality, coming from different fields and/or with slightly different requirements.

Morg 14y ago

Indeed the title made the content look shitty. Workflow engine means an engine to drive workflows - at least for me. This thing is meh, no reason to use it over existing C tools, I wonder why someone rererererereinvented the (not actually round) wheel.