3 ms·

Show HN: A distributed workflow engine written in Go

9 points by kindaworkz 3y ago

I'd like to share a project I've been working on for the past few months. It's a distributed workflow engine written entirely in Go. Some highlights: * Tasks are executed in a Docker container * Can run stand-alone or distributed * Highly extensible * Able to enforce limits (CPU/RAM) per task * Web UI Would love the get your feedback on it, and find out if this could be useful.

jekude 3y ago
Would I be able to use this to run untrusted code from my users? I have had an idea for a game I'd like to eventually build where users write code to control their character. If I send the code to my server running Tork, what are the security risks?
- kindaworkz 3y ago
  I actually wrote a whole post about that exact use case using Tork: https://dev.to/acoh3n/lets-build-a-code-execution-engine-4kgi https://dev.to/acoh3n/lets-build-a-code-execution-engine-4kg... Let me know if this helps.
importsaas 3y ago
Potential frenemy here :) First up, congratulations on the launch, and project. The timing and need for such a project is certainly apt. Sharing my feedback based on spending 30 minutes as per the guide. - a config with an example could be introduced earlier. Currently it is deeper in the docs. - docker compose example could be introduced earlier. Currently it's just in the repo. - would be great if the destination of the job could also be dynamic. Currently you have a nice yaml way to run SQL/go/bash. I think a job is truly potent when it can pull data from multiple places, but also push that to multiple places as well. - it's great that it is thinking distributed from the get go. - having a queue support was smart. having S3 has a source/destination would also be interesting - how would you do safely do multi-tenant? Eg a SaaS decides to use tork, having 10 customers. I want to run a similiar job for each, and in a way that data doesn't get mixed up. - hit some errors using an old docker engine. Something I haven't run into for many other projects that can run a docker compose version:3 ERR node ... failed health check error="Error response from daemon: client version 1.42 is too new. Maximum supported API version is 1.41" WRN error getting CPU usage error="not implemented yet" DBG received heartbeat cpu-percent=0 heartbeat-time=1696246437 hostname=temp.local node-id=... status=DOWN - Am wondering if this could someday be a good alternative to swarm/portainer to run long living containers. That with the workflow may help in devops - how do you schedule jobs. Wasn't clear after seeing examples in 30m - I notice your git repo deletes branches every few days. So early in a project maybe it can be avoided? - web ui is promising. Reminds me of inngest territory. Didn't have time to see what the other two tabs. Maybe add those screenshots. - it's not clear what you mean by extensible. Because of being able to run any image, bash? - docs could have a section for CRUD like GET requests to see running jobs. Filter by job status, filter by date. Godspeed!

9 points by kindaworkz 3y ago

jekude 3y ago

Would I be able to use this to run untrusted code from my users? I have had an idea for a game I'd like to eventually build where users write code to control their character. If I send the code to my server running Tork, what are the security risks?

kindaworkz 3y ago
I actually wrote a whole post about that exact use case using Tork: https://dev.to/acoh3n/lets-build-a-code-execution-engine-4kgi https://dev.to/acoh3n/lets-build-a-code-execution-engine-4kg... Let me know if this helps.

importsaas 3y ago

Potential frenemy here :) First up, congratulations on the launch, and project. The timing and need for such a project is certainly apt. Sharing my feedback based on spending 30 minutes as per the guide. - a config with an example could be introduced earlier. Currently it is deeper in the docs. - docker compose example could be introduced earlier. Currently it's just in the repo. - would be great if the destination of the job could also be dynamic. Currently you have a nice yaml way to run SQL/go/bash. I think a job is truly potent when it can pull data from multiple places, but also push that to multiple places as well. - it's great that it is thinking distributed from the get go. - having a queue support was smart. having S3 has a source/destination would also be interesting - how would you do safely do multi-tenant? Eg a SaaS decides to use tork, having 10 customers. I want to run a similiar job for each, and in a way that data doesn't get mixed up. - hit some errors using an old docker engine. Something I haven't run into for many other projects that can run a docker compose version:3 ERR node ... failed health check error="Error response from daemon: client version 1.42 is too new. Maximum supported API version is 1.41" WRN error getting CPU usage error="not implemented yet" DBG received heartbeat cpu-percent=0 heartbeat-time=1696246437 hostname=temp.local node-id=... status=DOWN - Am wondering if this could someday be a good alternative to swarm/portainer to run long living containers. That with the workflow may help in devops - how do you schedule jobs. Wasn't clear after seeing examples in 30m - I notice your git repo deletes branches every few days. So early in a project maybe it can be avoided? - web ui is promising. Reminds me of inngest territory. Didn't have time to see what the other two tabs. Maybe add those screenshots. - it's not clear what you mean by extensible. Because of being able to run any image, bash? - docs could have a section for CRUD like GET requests to see running jobs. Filter by job status, filter by date. Godspeed!