3 ms·

If you do not have all your stuff in AWS, and are not sure if you want to pay high amounts for managed or "serverless" solutions, I am currently developing a st

by teraku 6y ago

If you do not have all your stuff in AWS, and are not sure if you want to pay high amounts for managed or "serverless" solutions, I am currently developing a standard[0] for workflows and a distributed scheduler which is compliant to this standard. It's still WIP, but I've used my fair share of managed solutions, "serverless", open-source (Airflow/Luigi) and BI engines to wonder, why there is no general standard just for the definition of the workflows. So that moving between systems is much less friction... [0] https://github.com/OpenWorkflow/OpenWorkflow https://github.com/OpenWorkflow/OpenWorkflow

brightball 6y ago
Isn’t that what BPMN is supposed to offer?
- deleted 6y ago
  [deleted]
- teraku 6y ago
  Yes and no. So it's the same in a sense that I define a format for these workflows. This means that the workflow authoring tool is completly independant of the scheduler/execution engine, as long as both are compliant to the standard. BPMN tries to do a lot more, and I think it's overkill for most use-cases. The major difference is OpenWorkflow defines tasks very loosely: you specify the environment you seek, and what to do on it. You can incorporate it into your current set of jobs without much friction. I don't think BPMN is a good comparison, as companies who seek this are mostly huge and have slowly changing processes. This is more for cases when the world around you might be everchanging, incorporating APIs, shuffling files around or moving data for analytics/model training.
prpl 6y ago
There’s already CWL, which isn’t that successful.
- teraku 6y ago
  Just looked at it, since I never heard of it before. It looks kind of like what I'm trying to achieve, but looking at the specification is very detailed again. It even comes with its own data model for input and output. I don't feel like this should be the focus nor mandatory. It's very Java/Python heavy, which, to be fair, are the two most common languages for these use-cases. But I believe you either have a very static, strong system, as-in shipping a whole BPMN model onto your company and implementing SAP, or you want something light-weight that's easy to use, like e.g. Airflow. I'm more on the light-weight front, but I think systems like Airflow have the problem of being too shut-in into one language. I want my workflows to be quickly defineable (e.g. via Python), but I don't want the Engine to actually run on Python... I mean, worst case I just gained a lot of experience once I'm done with implementing. I do see a lot of similarities between what's already there and what I'm doing, but also some differences. And it's fun :D
kfk 6y ago
Don’t you think the big challenge is creating and managing events? Once the graph is built you can throw it at something like dask with fire and forget. The challenge is WHEN to run.
mfateev 6y ago
I personally think that any such standard would be inferior to any general purpose programming language. Independently of my opinion on the standard I think the best implementation platform for it would be temporal.io.

by teraku 6y ago

brightball 6y ago

Isn’t that what BPMN is supposed to offer?

deleted 6y ago
[deleted]
teraku 6y ago
Yes and no. So it's the same in a sense that I define a format for these workflows. This means that the workflow authoring tool is completly independant of the scheduler/execution engine, as long as both are compliant to the standard. BPMN tries to do a lot more, and I think it's overkill for most use-cases. The major difference is OpenWorkflow defines tasks very loosely: you specify the environment you seek, and what to do on it. You can incorporate it into your current set of jobs without much friction. I don't think BPMN is a good comparison, as companies who seek this are mostly huge and have slowly changing processes. This is more for cases when the world around you might be everchanging, incorporating APIs, shuffling files around or moving data for analytics/model training.

prpl 6y ago

There’s already CWL, which isn’t that successful.

teraku 6y ago
Just looked at it, since I never heard of it before. It looks kind of like what I'm trying to achieve, but looking at the specification is very detailed again. It even comes with its own data model for input and output. I don't feel like this should be the focus nor mandatory. It's very Java/Python heavy, which, to be fair, are the two most common languages for these use-cases. But I believe you either have a very static, strong system, as-in shipping a whole BPMN model onto your company and implementing SAP, or you want something light-weight that's easy to use, like e.g. Airflow. I'm more on the light-weight front, but I think systems like Airflow have the problem of being too shut-in into one language. I want my workflows to be quickly defineable (e.g. via Python), but I don't want the Engine to actually run on Python... I mean, worst case I just gained a lot of experience once I'm done with implementing. I do see a lot of similarities between what's already there and what I'm doing, but also some differences. And it's fun :D

kfk 6y ago

Don’t you think the big challenge is creating and managing events? Once the graph is built you can throw it at something like dask with fire and forget. The challenge is WHEN to run.

mfateev 6y ago

I personally think that any such standard would be inferior to any general purpose programming language. Independently of my opinion on the standard I think the best implementation platform for it would be temporal.io.