13 ms·

Fast self-hostable open-source workflow engine

378 points by rubenfiszel 3y ago

me_bx 3y ago
The self-hosted pricing depends on the number of vCPUs used. Does this mean that Windmill installation pings back home with info about clients?
- sisve 3y ago
  So that is the self hosted enterprise version. You can selfhost for free. It does not ping back home. But if you download all the resource types from the windmill hub you will send a http request to them. I do not believe the EE version pings back either as of now. But I think the plan is to report back with usage. Or i need to report it manually.
zegoverno 3y ago
great product!
slig 3y ago
I've been following Windmill since their HN Launch (my first script was edited 540 ago, just checked). Less than a year ago I started using it more heavily and it's been a breeze. Their Discord server is very active and Ruben usually replies and fixes bugs in minutes, even on weekends.
- thelastparadise 3y ago
  What do you use it for?
  - slig 3y ago
    Daily puzzle generation, daily PDF updates/creation (if the original content is changed, it triggers a new generation), triggering and editing of Remotion videos.
Kiro 3y ago
I don't think this should be a Show HN. > Off topic: blog posts https://news.ycombinator.com/showhn.html https://news.ycombinator.com/showhn.html
- dang 3y ago
  Thanks, I've taken Show HN out of the title now.
  - rubenfiszel 3y ago
    Curious, what's the difference between an article and a blog post ?
    - pvg 3y ago
      Articles generally can't be show hn's either so it doesn't matter much for the purposes of Show HN. The full thing is "Off topic: blog posts, sign-up pages, newsletters, lists, and other reading material. Those can't be tried out, so can't be Show HNs. Make a regular submission instead." https://news.ycombinator.com/showhn.html https://news.ycombinator.com/showhn.html
knowsuchagency 3y ago
Windmill is fantastic. It manages to cover so much ground i.e. airflow, prefect, retool, budibase but better
- nextworddev 3y ago
  It’s mainly for workflows but not for building rest endpoints?
  - sisve 3y ago
    I think the workflow engine is the _best_ part of it, but together with the script editor (online code editor with LSP support) and a drag-n-drop app builder, that is for developers, so you "drop down" to code much of the logic and connect it to script or workflow, it's a really nice combo. (you see a lot of other companies trying to span out to workflow + script + appbuilder). It's really made for developers or at least people that want code as a "first class citizen" of the platform. So i has a cli to sync to/from a VCS, VS code extention. But at the same time I have gotten a tech savvy person on the customer success team to create a workflow where they connect to our API and actually create meaningful services way faster then the enterprise developer team would have done. The script and flows get automatically endpoints to connect to, but all request is saved to db, and picked up by a worker that has to save the result to disk, before the server again pick up the result and send it to a client. So I would not use it to customer. I have used it for a lot of internal tools to employees. Meaning they can wait 1-2 sec, because you can teach them that it works and you provide a lot of value.
    - ryanjshaw 3y ago
      > It's really made for developers or at least people that want code as a "first class citizen" of the platform. Power Automate doesn't have this (unless you hack it into custom connectors) and it's a really puzzling decision. I guess they want you to use Azure Functions but those need a PhD to understand and the deal with the security surface.
    - nextworddev 3y ago
hintymad 3y ago
Does being fast beyond certain threshold really matter for a workflow engine, though, especially given that many workflows have long-running tasks? What I find matters is multi-tenancy. That is, can a workflow system support as many jobs as a user desires, yet each job gets scheduled and executed as if it is the only one on the workflow engine. For instance, Amazon EC2 and EBS probably have millions of jobs and job queues running at any moment, yet each job has pretty predictable and stable startup delay and execution time. In contrast, each of Nomad and k8s has a global queue that sometimes gets stuck because of bad actors or too many pending jobs (I understand that an orchestrator is far more than just a workflow engine. I'm just contrasting the user experience).
- rubenfiszel 3y ago
  So I agree that for most jobs taking already more than 300ms, a 50ms per job will not be perceivable anymore. What matters is the DX and even though I'm very proud of our performance, a lot of our work is focused on providing a consistently great developer experience in all aspects (vscode extension, instant deployment, etc, etc). However, there is one big benefit to being performant which is that you can use the same stack for performance sensitive jobs such as event-streaming use cases. That remove the duplication of infrastructure. The other benefit is that it improve the developer experience overall with fast previews.
- 3cats-in-a-coat 3y ago
  A workflow engine is indistinguishable from a regular interpreter or runtime, except for the fact it's suspendable (hence can run tasks long). Ideally a workflow engine being fast means it can be just the platform you write everything in. The artificial separation between "normal" code and "workflow engines" is completely unnecessary. Our platforms are extremely fragmented, pointlessly so. Remote vs local, statically vs dynamically typed, async vs sync, short running vs long running... If an engine is written without these arbitrary limitations, it can be more.
- analyte123 3y ago
  Yes. Just think of how annoying it is when you have to wait 5 seconds instead of 0.5s for a 2FA message, then multiply that by everything that you ever do with your workflow engine. That's not even to speak of cases where running the workflow (e.g. acquiring data) faster is a competitive advantage, although this thing is still probably too slow for truly HFT-level tasks.
0xbadcafebee 3y ago
> we wanted to prove it's also the best system to deploy at scale in production > The queue is implemented in Postgresql itself Tell me you don't understand system design without telling me you don't understand system design
- dang 3y ago
  "Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." If you know more than others, that's great, but in that case please share some of what you know—without putdowns or snark—so the rest of us can learn. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sort=byDate&type=comment&query=share%20know%20learn%20by:dang https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor... https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- ryanjshaw 3y ago
  I would argue that implementing your internal queues inside your DB is sound system design. Your backups and system restores are as simple as it gets. You should have a specific reason to do anything more complicated. And no, "deploy at scale" is puffery, not a good reason.
- igortg 3y ago
  Any known issue on implementing a queue with Postgres? Do you at least know it has a pretty good pub/sub mechanism?
  - mmcclure 3y ago
    Postgres is having a moment with excellent queueing libraries right now, not sure what GP is referring to. We use Oban[1] at Mux for some workloads and love it, and a former colleague just built something similar in Go that was on the front page yesterday (River)[2]. [1] https://github.com/sorentwo/oban https://github.com/sorentwo/oban [2] https://github.com/riverqueue/river https://github.com/riverqueue/river
  - 0xbadcafebee 3y ago
deleted 3y ago
[deleted]
boxed 3y ago
It always amazes me that people spend this much time and effort on writing articles like this and don't run their text through a spell checker even once. It also amazes me that people still use text editors that by default doesn't spellcheck. Anyone know what editors that could be? Seems pretty crazy in 2023!
- andrewstuart2 3y ago
  If this is an engineer, they probably use whatever tool they're most familiar with, and that tool is probably set up for mostly engineering (code) work, in which case in my experience spell checking is a major distraction with an outsized ratio of false positives, because my identifiers aren't chosen for their validity in a wordlist. Even in markdown, it's more annoying than helpful most of the time, so I tend to leave it as a manual step that I mostly ignore. People know what I mean 99.99% of the time. So, that'd be my guess. Any sort of editor that's been tuned to an engineer's preferences which includes only manual spell checking.
  - j_maffe 3y ago
    Yeah, but this sort of article is what you get at the end of the day. Plus there are several ways of getting rid of false positives and as GP said: at least do it before posting.
  - boxed 3y ago
    PyCharm has a spell checker. It's pretty great. Not using it seems madness.
    - senorrib 3y ago
      Spellchecker in Jetbrains IDEs is terrible. It throws a lot of false positives.
      - boxed 3y ago
        It's pretty good imo. It's super easy to add words too, easier in fact than the spell checker in iOS or macOS. The one in iOS is the worst imo, as it adds stuff automatically based on common mistakes you make because "machine learning" and you can't even see the list of the "learned" words to remove things it learned that aren't actually words!
victorbjorklund 3y ago
Looks very similar to the UI of Pipedream that I used before. Not a negative thing. Will try it.
- slig 3y ago
  I migrated from Pipedream, Windmill is much, much more developer friendly.
gigatexal 3y ago
Hi Windmill devs -- why build this on relational Postgres at all? I've been toying with the idea of forking airflow and trying to swap out the backend for something document based either on Mongo or something else. The reasons being db migrations are a pain in the relational strongly typed schema world vs the more forgiving paradigm of documents.
- cyanydeez 3y ago
  what'd be wrong just storing your documents in postgres JsoN Or XML fields
  - DonnyV 3y ago
    Because Postgres is a relational database.
    - cyanydeez 3y ago
      that wasn't the concern being presented. The concern is "I want to store mostly unknown structured data, so why dont I use a DB already stupd to do that"
  - gigatexal 3y ago
    That could work too I guess.
- rubenfiszel 3y ago
  because 95% of the tables you need for this are relational (queue, completed_job, users, etc, etc) and for the few things that are unstructured, you can represent them well in PG with JSONB (which is what we do for inputs and outputs).
code_biologist 3y ago
Windmill is great. Definitely don't stray from the self-hostable + DX mission! I haven't had need to use it professionally, but I use it on a home server to run several small web crawlers and yt-dlp jobs. Really fun piece of tooling.
- darkteflon 3y ago
  Could you clarify what you mean by “definitely don't stray from the self-hostable + DX mission”?
  - code_biologist 3y ago
    I hate cron for running small personal scripts and jobs. I hate ad-hoc handling of logging / observability / success monitoring. I want a web UI for all of it. I hate trying to figure out how to handle event based job triggers that are more fancy than the 10 line bash script they end up running. I hate trying to wrap up solutions for all those into infrastructure as code — I will end up running an unpatched Ubuntu 20 for the next decade if I can't just nuke the box and reload my config + scripts. I hate the idea of relying on the continued existence of a hosted service, as well as them being able to see that I'm fine tuning an LLM for erotic stories (or whatever). Windmill solved all that for me. I write a short little Python script, paste it into the web UI and boom it's running with good answers for all the above issues. That's great DX. If workflow engines try to solve every problem they can devolve into plugin and complexity hell. Jenkins issues have caused a lot of headache at my day job lately. I don't want Windmill to fall into the same trap.
    - darkteflon 3y ago
      Thanks, that’s really helpful - agreed on all accounts!
DonnyV 3y ago
This looks interesting. I wonder if they will be adding compiled languages like C#?
- CuriousSkeptic 3y ago
  Might want to checkout DurableTasks[1] for that [1] https://github.com/Azure/durabletask https://github.com/Azure/durabletask
- rubenfiszel 3y ago
  you can always run compiled programs from bash as binaries (but yeah it's not as great as an experience than being able to run interpreted languages directly).
darkteflon 3y ago
While appreciating that they’re not direct substitutes, does anyone have experience comparing Windmill with Dagster for managing dependency resolution in complex data pipelines?
- bafe 3y ago
  I briefly tried windmill for a project that involved creating custom workflows on-demand from a configuration file. I can't recommend windmill for data pipelines, it is meant to be more of a low-code internal app platform like retool or budibase. It is meant for a relatively static workflow that requires human intervention, like a simple business process involving some API calls and humans approvals in the loop. For complex (and potentially reconfigurable) data pipelines, dragster is a much better choice
  - rubenfiszel 3y ago
    When you tried, we didn't have s3 integrations nor restartable flows. We will present all of this in day 5 of our launch week and it might be a good time to revisit. I agree we were not a good fit prior. I think we would now compare favorably as we will offer excellent ergonomics for data processing, leveraging polars, duckdb, and other OLAP libraries to their full extent.
    - bafe 3y ago
      It was several months ago, so it is entirely possible. Back then I got the feeling that windmill tried to be more of a low code business/internal tool platform than a data/ETL workflow tool, I especially missed an expressive way to define workflows programmatically (i think you had a JSON schema but nothing as powerful as dragster where you can define a whole workflow in pure python)
      - rubenfiszel 3y ago
        We added a vscode extension to build everything from code, including manipulating the yaml or UI and have them instantly impact each other: https://www.windmill.dev/blog/launch-week-1/vscode-extension https://www.windmill.dev/blog/launch-week-1/vscode-extension yaml is not real code and so the part that I consider to be real code is that each step is its separate file in python, typescript, that you can edit in your code editor and have your plugins working, testing frameworks. It's normal functions that you can run locally. It would be possible for us to do like dagster/prefect/airflow which is to use a macro-processing step/decorators to build dynamically the graph which is what our yaml is in the end, a 1:1 encoding of our dag spec called openflow: https://docs.windmill.dev/docs/openflow/ https://docs.windmill.dev/docs/openflow/ We didn't do it yet because in most cases, the decorators are a lot like yaml, they are a very rigid way of declaring that some functions are nodes that you can chain in limited ways. On the other hand, not providing that mode allow us to put more efforts in the low-code graph builder for now. But, as someone that love compilers and built a few, I'm very eager for us to provide such a mode so it's probably a few months away :) We'd love to have your input on the DX for data/ETL once we present it Friday so feel free to join our discord or shoot me an email at ruben@windmill.dev
jgil 3y ago
What are the units for the x-axis? I'm assuming milliseconds from context, but not having the axis labeled obfuscates the interpretation.
- rubenfiszel 3y ago
  It's seconds. You can find all the detailed numbers there: https://www.windmill.dev/docs/misc/benchmarks/competitors https://www.windmill.dev/docs/misc/benchmarks/competitors
haroldp 3y ago
Is this what I need? We have ad hoc business processes like: 1. Sell service to client 2. Email client with scheduling info 3. Agree to scheduling date with client 4. Send client email with documentation and request their data 5. Remind client to send data 6. Alert admins data hasn't been received 7. Beat data out of client 8. Prepare data 9. QA and checkoff service 10. Email client to review and approve 11. Publish/run service 12. Send client stats/results And I want to move that from spreadsheets, personal emails and admins keeping in their head where everything is at, to web forms and uploads, automated emails and dashboards. I looked at airtable, smartsheet, budibase and many others but they seem to be very project-based, where you are inventing a unique workflow for each project and using it to keep managers on top of where their team is at with the various steps. Project Management vs process management. None of them seem to have decent calendar integration. Few of them seem to be great about emails or scheduled scripts. I have APIs for my data, or I can code them if needed. I'd prefer a low-code to a no-code approach, where managers have a spreadsheet view and can do some of of the UI work and programmers can make it do stuff and handle integrations.
- goodpoint 3y ago
  It is not.
- erichocean 3y ago
  I don't think that'll achieve your goals, it's far too clunky and inflexible for actually running a business end-to-end. (No knock on Windmill, they're all like this in my experience as the way they model the world is how a programmer thinks about things, not a business owner. BPMN is straight up cancer—avoid.) I would suggest looking at workflow systems within the VFX space, which they call "pipelines". They are human-driven and maintained, but support very high levels of process automation. VFX pipelines are written in Python, but the key thing is, they're very good at a few things simultaneously: (1) Highly irregular work… (2) That nevertheless requires lots of process, including of the ad hoc variety… (3) With high human touch (human in the loop), so calendars, task assignments, timelines, approvals, producer dashboards, etc. (4) That's repeatable on the next project/opportunity, with as much customization as needed. As a bonus, pipelines handle billions of dollars of work annually under high pressure, with tens of thousands of users, so there's a lot of hard won experience built into them. I personally have found that there are three levels of VFX pipeline tech that need to be combined and customized to do everything. This is the most recent stack I've set up: * Kitsu for the producer/manager level, human task assignments. Mainly for managers, has the calendar functionality. * Prism Pipeline for software integration (i.e. keeping the humans following the process). This is what people actually do. * Kabaret for compute work. People usually kick off these jobs, but they're handled by the farm. You'll be responsible for everything that happens here. [0] Don't try to get by with just one or two of the three, even though feature-wise it looks like you might be able to. Incorporating computer-driven workflows is tricky and requires purpose-built libraries in my experience. Just use the right library for the job, and accept that they all have somewhat overlapping functionality. For most non-VFX work, you'll need to script access to the browser, since that's where people do stuff today (i.e. with the Prism Pipeline library). Launch Chrome, Firefox, or Edge with a debug port and use Playwright as your "plugin" mechanism, with Prism/Python as the driver. You can make your pages do whatever you want and script literally any web-based process, filling in fields from your database as needed, adding data to the page, etc. For the last six years, I've run a business that does around $5M of annual revenue using this approach, with 4 FTE people (including me). We handle around 40 million compute tasks a day, but also have all of the "daily work" where humans are in the loop. We do all of the tasks of the type you've listed. HTH! [0] If you need fully automated task generation (so, no human in the loop), write "pump" scripts in plain Python that push jobs into Kitsu or Kabaret at a particular interval. I personally run them under a god monitor, and have them exit after, say, 15 minutes (or whatever interval makes sense) to get cron-like functionality. This is useful because you can pause execution of any pump process easily, change the interval, restart them, etc. plus you get easy access to the logs for debugging. The scripts themselves are usually less than 100 LOC. I have around 60 of these that run the automatic parts of our business. Whenever I need custom, temporary behavior in this part of the business, I copy the pump script, edit the original to avoid the special case, and have the new script only handle the special case (with whatever new behavior I need). Done, takes very little time, and is easy to run temporarily (e.g. for the next week, a common occurrence because some downstream vendor has caused a problem we need to work around). Another approach to trigger tasks/workflows is to have a server that gets pinged by 3rd parties (e.g. Webhooks) and does the same whenever the endpoint is invoked. We have that too, but it's literally just a few scripts. (Most workflow systems seem to believe this is damn near 100% of the stuff that needs to be "automated.")
yungraft 3y ago
[dead]
deleted 3y ago
[deleted]
j_maffe 3y ago
That graph is utterly useless. How hard is it to place units on a freaking graph?
- rubenfiszel 3y ago
  Apologies, the units are seconds, fixing the graph right now. You will find more ample details on the benchmarks there: https://www.windmill.dev/docs/misc/benchmarks/competitors https://www.windmill.dev/docs/misc/benchmarks/competitors We made them as easy as possible to be reproduced
deleted 3y ago
[deleted]
alexchamberlain 3y ago
Interesting that Windmill's backend seems to be implemented in Rust, but the docs talk about supporting jobs written in Python, TypeScript and Go. I understand there is generic Docker support, but does anyone know if you can run Rust jobs natively?
- sisve 3y ago
  It can not. (It does support go)
- rubenfiszel 3y ago
  We could add support for Rust, and it would work the same as go, do a built step prior, and then cache the binary. If we have enough demand for it, we will add it (we are a small team and can only prioritize what's popular unfortunately). On a personal note, I write a lot of rust (and love it) but using rust for writing non-backend type of stuff is not something I would do myself. I'm more of a believer of hybrid like polars that optimize the compute in Rust but provide a compatible sdk in python.
paxys 3y ago
The devs of Windmill seem to have taken the winning advice of "do one thing and do it well" and done the exact opposite. Going over windmill.dev I have no idea what specifically the software should be used for. Is it a competitor to Retool? Airflow? Temporal? Apparently there's a no-code workflow builder? A drag and drop UI builder? An online IDE? Dozens of integrations? What on earth is going on??
- deleted 3y ago
  [deleted]
- debarshri 3y ago
  Doing one thing well is not a winning advice. You can do multiple things, it really depends on the execution and other factors. They are doing one that is workflows and doing it well.
  - sisve 3y ago
    Also they are using postgres for everything. I think that is a good example of picking one thing that you know very well and using that to the limits. But i do agree that windmill is doing so many things that the elevator pitch need some work. But a lot of companies are trying to do the same. So i do not think people will think of workflow engine + online ide + app builder as a natural grouping in 5 years
  - MuffinFlavored 3y ago
    I don't really understand what a "workflow" is / what it means. Like GitHub Actions is a workflow, but it's a convoluted YAML wrapper around shell scripts. Is a workflow shell scripts? I'm going to guess no. I'm going to guess it's "code". So... why is a workflow code but code isn't a workflow? Windmill is a "workflow (code?)" orchestrator?
    - jagtstronaut 3y ago
      Yeah I assumed this was a camunda competitor (a bpmn workflow product), but it seems like it is not that.
punnerud 3y ago
The GitHub repo for the open source project gives a better intro/sell: https://github.com/windmill-labs/windmill https://github.com/windmill-labs/windmill (Especially the video)
xyoer 3y ago
[dead]
majkinetor 3y ago
I am using Rundeck now to schedule project jobs, get notified on errors, provide UI for PowerShells/bash scripts, and have decent project and user management. It uses Quartz and can handle hundreds of jobs per minute without a problem. It can connect to git repository to keep job definitions there. I also use local file system a lot (on Windows or Linux) as I don't like to edit my scripts in browser, given that they are mini programs with lots of separate includes. I would like to try windmill instead. Can it cover all those cases, particularly the part where scripts are on the file system? Can I use automatic GUI creation for PowerShell scripts of windmill in such case? Rundeck transmits environment variables with GUI values when starting script.
- rubenfiszel 3y ago
  Not only it can cover those cases, you will likely be pleased with the advantages of such system (the automatic GUI creation is just one of those). The binary itself doesn't sandbox your processes unless you're in the nsjail mode. Hence, if you either use the binary raw (without docker) or mount the filesystem to be available to your container, it can run anything that is available on the filesystem.
  - majkinetor 3y ago
    I played a bit now on cloud and was able to run things in powershell and create a custom GUI form, connect button to script and show results in the text area. I didn't succeed to return all output from the powershell, just last value. The only way I got it is by using control "job log". This one however has some custom header of windmill (job guid etc.), so is there any way to show a raw log ? I generally do not like no-code tools, but first impression of this mix of no-code and code looks intriguing and app feels very nice overall, with great UI and easy to understand features. Good job.
    - rubenfiszel 3y ago
      So bash and powershell are a bit special since they are the only one that are not implemented as using a main function and thus have no result to return. However, we had that feedback a few time so there is a trick. If you write your result in ./result.json, we will process it as the result of your powershell script.
atlas_hugged 3y ago
It says open source, then it says limit of 10 SSO users. IANAL so this confuses me. Can you explain what this means? Actually, that sounds rude, excuse the ‘tism, I’m sorry. I’m just not super familiar with the license stuff, I generally only touch MIT licensed code when it’s work related, but I thought open source generally allows modification to code. So how do you enforce a limit of 10? I just skimmed the code a bit because of this confusion and I see stuff about enforcing a license in there. So couldn’t anyone just remove that license check code if it’s open-source? If that can’t be modified, then it’s just “source available”, isn’t it? Which to me is also fine, I just feel like I’m being either misled, if you get what I’m saying? Basically I think this project is pretty cool, so I wanted to bring it up to my boss because I think we have a need for this, and I had previously brought up Airflow, but I don’t know how to explain this to her. Sorry for the stream of consciousness. Typing on mobile while answering Teams messages and tickets.
- rubenfiszel 3y ago
  Some people will argue we are open-core because almost everything is open-source (AGPL) but our sso and enterprise plugins are under a source-available proprietary license. The reason that we didn't split them out of the codebase is that it's harder to maintain and would require to load the plugins. We didn't want to waste time on that when we had so much to build.
  - atlas_hugged 3y ago
    oh ok I see. Just the SSO part is like that. That makes more sense. So I guess I can just self-host for our team of 7 without SSO and we can finally organize these messy cron jobs we’ve accumulated. We would not be reselling at all. Just organizing some annoying aspects of our day to day. Thanks for the clarification. Much appreciated. If we end up expanding our usage down the road, I’ll see if I can convince her to consider shelling out funds for the SSO stuff too. I’d love to be able to support this project. Seems pretty cool!
    - rubenfiszel 3y ago
vlugorilla 3y ago
Been using windmill for a few months, and it's very pleasing. One of the best self-hosted tools I have!
MuffinFlavored 3y ago
https://www.windmill.dev/docs/misc/benchmarks/aws_lambda https://www.windmill.dev/docs/misc/benchmarks/aws_lambda I did not realize at first glance you can write a script and then trigger it (with an API key, not sure if you can make a public one or if that's a security nightmare) through HTTP to basically have your own self-hosted AWS Lambda / the whole "serverless HTTP triggers/functions" craze
- rubenfiszel 3y ago
  Yes absolutely, we're even faster than lambda with dedicated workers since the design is simpler, while loop jobs from the queue while still keeping all the error handling, observability, and easy deployment.
  - MuffinFlavored 3y ago
    Is there a workaround I'm missing where the URLs are exposed without needing an API key? (Something I'm pretty sure Lambda allows if you configure it properly)
    - slig 3y ago
      Not natively, one workaround is using a simple CF worker that proxies the request.
    - rubenfiszel 3y ago
      The api keys can be generated to be webhook specific (that's what they are by default when you generate them in the script's UX) and hence can be made public so there is no risk to expose them publicly, they are similar to a uuid of a lambda (not a key but impossible to guess). We require api key because every request is permissioned and hence we need to know who the script is executed on behalf of. For instance, scripts can fetch secrets/variables but those calls will be permissioned with the permissions of the caller which in this case we get from the webhook specific token.
iFire 3y ago
LICENSE ??? https://github.com/windmill-labs/windmill/blob/main/LICENSE https://github.com/windmill-labs/windmill/blob/main/LICENSE I'm a bit confused.
- rubenfiszel 3y ago
  We use a mix of AGPL for 95% of the code, Apache 2.0 for specs definitions and apis, and source-available for our enterprise plugins.
- SpaghettiCthulu 3y ago
  What on Earth is confusing about that?
brunoqc 3y ago
It's open-core (if you care about that). "The SSO is free up to 10ppl." It's weird that "Content Search" is such a pro feature.
- rubenfiszel 3y ago
  Only if you deem SSO to be of an essential nature to such products. To me it's all semantics at this point. But it's a complex software that require software engineers with salaries and we do not want to rely on donations so we do have to give reasons for people to pay. SSO and audit logs are good ways to segment enterprises to support the project.
  - brunoqc 3y ago
    > Only if you deem SSO to be of an essential nature to such products. I thought that multiple features were enterprise-only (like multiplayer, the content search...). In our world where we don't take security seriously enough, I do think that SSO is essential.
haolez 3y ago
I could use this as a bridge between developers and operation, but in my case the lack of localization is annoying for the staff in the ops side. I don't have time to contribute something like this right now, but I will keep an eye on this.
namtab00 3y ago
I'm trying to understand in what way this differs from node-red.
whoiscroberts 3y ago
How does this compare to Prefect?
- rubenfiszel 3y ago
  it's faster, it supports more languages (python, but also typescript, bash, go, and raw queries to different databases) The rest will be a matter of taste
ketozhang 3y ago
Did you guys considered existing standards when you chose what to use for representing workflow definitions before choosing OpenFlow? For example, Common Workflow Language
- rubenfiszel 3y ago
  We did and started with an existing standard but quickly, trying to fit to the standard was more complex than rolling our own. Agreed we just created yet one more standard but the bit about the input transforms being full javascript expressions or the way we encoded suspend steps was impossible to retrofit.
RandomBK 3y ago
One balance I've yet to find is how to handle the split between code-in-database (workflows stored in a database and edited via web IDE) vs code-in-git (workflows checked into the repo and only changed via the normal development+peer-review process). It looks like Workflow is primarily the former (code-in-database), but provides an API to sync things from a git repo. Is there a mechanism to enforce the rule that certain scripts/functionality/secrets are locked behind workflows sourced from a provided repository?
- sisve 3y ago
  I think most of windmills enterprises customers do it the code to git way. I have chosen another part where i deploy and then it's automatically push to git. Buy windmill proves you with 3-4 really good ways of deploying from 1 cowboy just doing what he wants to the normal EE way of doing software meaning code-in-git and people only have read access to production and can not modify things. You can have separate instance running or just separate workspaces and promote.
RandomBK 3y ago
I would love to begin using this system, but the licensing is giving me serious pause. While most of the software is under the AGPLv3, the Commercial License section of the README [0] implies that the company takes on a fairly broad interpretation of the AGPL. In particular, the line "[...] to build a feature on top of Windmill, to comply with AGPLv3 your product must be AGPLv3 [...]" seems to imply the company aligns with the stance taken by Google and other companies: that even calling the application via API is enough to trigger copyleft [1]. This implies that if I were to build a sign-up form that triggers a Windmill workflow in the backend, my entire application would either need to be AGPLv3 or I would need a commercial license. That's perfectly reasonable, as it means any non-AGPL use will have to contribute back to Windmill via a commercial license. However, it does mean positioning this as an "Fully Open-source" alternative to Airflow is only technically correct. This is much closer in practice to "source available" than how most developers would think as "open source". If this isn't how Windmill wants their license interpreted, I highly encourage clarifying things. [0] https://github.com/windmill-labs/windmill#commercial-license https://github.com/windmill-labs/windmill#commercial-license [1] https://opensource.google/documentation/reference/using/agpl-policy https://opensource.google/documentation/reference/using/agpl...
- rubenfiszel 3y ago
  Sorry for the lack of clarity. By product we mean white-labelling windmill, not triggering a script or flow by api.
  - RandomBK 3y ago
    Got it! That's great to hear. It'd be great to explicitly clarify that in the README :) Out of curiosity: Have you considered alternative licenses like SSPL or the Elastic license, which make a clear(er) delineation between whitelabelling/hosting vs simply using the application? I'm sure I'm not the first person to have written off Windmill because of the AGPL.
    - rubenfiszel 3y ago
      Those licenses are not open-source and likely to be more restrictive than what we currently offer. It was important for me to be truly open-source. If you are using windmill as a whole, you're free to use it however you please with the AGPL but for customers with doubts, we do sell an enterprise/commercial license.
felipemesquita 3y ago
This looks great! Are there any plans to include Ruby as a supported scripting language along with Python, Typescript, Go, Bash, and Sql? I currently use my own crude workflow runner for processes that are mostly sql, but with some ruby code. I’ve stuck with it because it works and we currently use Ruby for almost everything, so it’s not worth it to add another language to the mix so we could use a better tool.
- rubenfiszel 3y ago
  Ruby is next :)
  - felipemesquita 3y ago
    Looking forward to trying it as soon as possible!
hokkos 3y ago
A little late but after reading this thread I moved a airflow instance to windmill and I much much prefer windmin, airflow was clunky and heavy for my simple workflow that consist of running a few python script. Also is auto ui is great to play with the arguments in, the cron ui and next start date... Also is it written in rust so much more low usage than airflow that consistently cpu picked while doing nothing.

378 points by rubenfiszel 3y ago

The self-hosted pricing depends on the number of vCPUs used. Does this mean that Windmill installation pings back home with info about clients?

sisve 3y ago
So that is the self hosted enterprise version. You can selfhost for free. It does not ping back home. But if you download all the resource types from the windmill hub you will send a http request to them. I do not believe the EE version pings back either as of now. But I think the plan is to report back with usage. Or i need to report it manually.

zegoverno 3y ago

great product!

slig 3y ago

I've been following Windmill since their HN Launch (my first script was edited 540 ago, just checked). Less than a year ago I started using it more heavily and it's been a breeze. Their Discord server is very active and Ruben usually replies and fixes bugs in minutes, even on weekends.

thelastparadise 3y ago
What do you use it for?
- slig 3y ago
  Daily puzzle generation, daily PDF updates/creation (if the original content is changed, it triggers a new generation), triggering and editing of Remotion videos.

Kiro 3y ago

I don't think this should be a Show HN. > Off topic: blog posts https://news.ycombinator.com/showhn.html https://news.ycombinator.com/showhn.html

dang 3y ago
Thanks, I've taken Show HN out of the title now.
- rubenfiszel 3y ago
  Curious, what's the difference between an article and a blog post ?
  - pvg 3y ago
    Articles generally can't be show hn's either so it doesn't matter much for the purposes of Show HN. The full thing is "Off topic: blog posts, sign-up pages, newsletters, lists, and other reading material. Those can't be tried out, so can't be Show HNs. Make a regular submission instead." https://news.ycombinator.com/showhn.html https://news.ycombinator.com/showhn.html

knowsuchagency 3y ago

Windmill is fantastic. It manages to cover so much ground i.e. airflow, prefect, retool, budibase but better

nextworddev 3y ago
It’s mainly for workflows but not for building rest endpoints?
- sisve 3y ago
  I think the workflow engine is the _best_ part of it, but together with the script editor (online code editor with LSP support) and a drag-n-drop app builder, that is for developers, so you "drop down" to code much of the logic and connect it to script or workflow, it's a really nice combo. (you see a lot of other companies trying to span out to workflow + script + appbuilder). It's really made for developers or at least people that want code as a "first class citizen" of the platform. So i has a cli to sync to/from a VCS, VS code extention. But at the same time I have gotten a tech savvy person on the customer success team to create a workflow where they connect to our API and actually create meaningful services way faster then the enterprise developer team would have done. The script and flows get automatically endpoints to connect to, but all request is saved to db, and picked up by a worker that has to save the result to disk, before the server again pick up the result and send it to a client. So I would not use it to customer. I have used it for a lot of internal tools to employees. Meaning they can wait 1-2 sec, because you can teach them that it works and you provide a lot of value.
  - ryanjshaw 3y ago
    > It's really made for developers or at least people that want code as a "first class citizen" of the platform. Power Automate doesn't have this (unless you hack it into custom connectors) and it's a really puzzling decision. I guess they want you to use Azure Functions but those need a PhD to understand and the deal with the security surface.
  - nextworddev 3y ago

hintymad 3y ago

Does being fast beyond certain threshold really matter for a workflow engine, though, especially given that many workflows have long-running tasks? What I find matters is multi-tenancy. That is, can a workflow system support as many jobs as a user desires, yet each job gets scheduled and executed as if it is the only one on the workflow engine. For instance, Amazon EC2 and EBS probably have millions of jobs and job queues running at any moment, yet each job has pretty predictable and stable startup delay and execution time. In contrast, each of Nomad and k8s has a global queue that sometimes gets stuck because of bad actors or too many pending jobs (I understand that an orchestrator is far more than just a workflow engine. I'm just contrasting the user experience).

rubenfiszel 3y ago
So I agree that for most jobs taking already more than 300ms, a 50ms per job will not be perceivable anymore. What matters is the DX and even though I'm very proud of our performance, a lot of our work is focused on providing a consistently great developer experience in all aspects (vscode extension, instant deployment, etc, etc). However, there is one big benefit to being performant which is that you can use the same stack for performance sensitive jobs such as event-streaming use cases. That remove the duplication of infrastructure. The other benefit is that it improve the developer experience overall with fast previews.
3cats-in-a-coat 3y ago
A workflow engine is indistinguishable from a regular interpreter or runtime, except for the fact it's suspendable (hence can run tasks long). Ideally a workflow engine being fast means it can be just the platform you write everything in. The artificial separation between "normal" code and "workflow engines" is completely unnecessary. Our platforms are extremely fragmented, pointlessly so. Remote vs local, statically vs dynamically typed, async vs sync, short running vs long running... If an engine is written without these arbitrary limitations, it can be more.
analyte123 3y ago
Yes. Just think of how annoying it is when you have to wait 5 seconds instead of 0.5s for a 2FA message, then multiply that by everything that you ever do with your workflow engine. That's not even to speak of cases where running the workflow (e.g. acquiring data) faster is a competitive advantage, although this thing is still probably too slow for truly HFT-level tasks.

0xbadcafebee 3y ago

> we wanted to prove it's also the best system to deploy at scale in production > The queue is implemented in Postgresql itself Tell me you don't understand system design without telling me you don't understand system design

dang 3y ago
"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." If you know more than others, that's great, but in that case please share some of what you know—without putdowns or snark—so the rest of us can learn. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sort=byDate&type=comment&query=share%20know%20learn%20by:dang https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor... https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
ryanjshaw 3y ago
I would argue that implementing your internal queues inside your DB is sound system design. Your backups and system restores are as simple as it gets. You should have a specific reason to do anything more complicated. And no, "deploy at scale" is puffery, not a good reason.
igortg 3y ago
Any known issue on implementing a queue with Postgres? Do you at least know it has a pretty good pub/sub mechanism?
- mmcclure 3y ago
  Postgres is having a moment with excellent queueing libraries right now, not sure what GP is referring to. We use Oban[1] at Mux for some workloads and love it, and a former colleague just built something similar in Go that was on the front page yesterday (River)[2]. [1] https://github.com/sorentwo/oban https://github.com/sorentwo/oban [2] https://github.com/riverqueue/river https://github.com/riverqueue/river
- 0xbadcafebee 3y ago

deleted 3y ago

[deleted]

boxed 3y ago

It always amazes me that people spend this much time and effort on writing articles like this and don't run their text through a spell checker even once. It also amazes me that people still use text editors that by default doesn't spellcheck. Anyone know what editors that could be? Seems pretty crazy in 2023!

andrewstuart2 3y ago
If this is an engineer, they probably use whatever tool they're most familiar with, and that tool is probably set up for mostly engineering (code) work, in which case in my experience spell checking is a major distraction with an outsized ratio of false positives, because my identifiers aren't chosen for their validity in a wordlist. Even in markdown, it's more annoying than helpful most of the time, so I tend to leave it as a manual step that I mostly ignore. People know what I mean 99.99% of the time. So, that'd be my guess. Any sort of editor that's been tuned to an engineer's preferences which includes only manual spell checking.
- j_maffe 3y ago
  Yeah, but this sort of article is what you get at the end of the day. Plus there are several ways of getting rid of false positives and as GP said: at least do it before posting.
- boxed 3y ago
  PyCharm has a spell checker. It's pretty great. Not using it seems madness.
  - senorrib 3y ago
    Spellchecker in Jetbrains IDEs is terrible. It throws a lot of false positives.
    - boxed 3y ago
      It's pretty good imo. It's super easy to add words too, easier in fact than the spell checker in iOS or macOS. The one in iOS is the worst imo, as it adds stuff automatically based on common mistakes you make because "machine learning" and you can't even see the list of the "learned" words to remove things it learned that aren't actually words!

victorbjorklund 3y ago

Looks very similar to the UI of Pipedream that I used before. Not a negative thing. Will try it.

slig 3y ago
I migrated from Pipedream, Windmill is much, much more developer friendly.

gigatexal 3y ago

Hi Windmill devs -- why build this on relational Postgres at all? I've been toying with the idea of forking airflow and trying to swap out the backend for something document based either on Mongo or something else. The reasons being db migrations are a pain in the relational strongly typed schema world vs the more forgiving paradigm of documents.

cyanydeez 3y ago
what'd be wrong just storing your documents in postgres JsoN Or XML fields
- DonnyV 3y ago
  Because Postgres is a relational database.
  - cyanydeez 3y ago
    that wasn't the concern being presented. The concern is "I want to store mostly unknown structured data, so why dont I use a DB already stupd to do that"
- gigatexal 3y ago
  That could work too I guess.
rubenfiszel 3y ago
because 95% of the tables you need for this are relational (queue, completed_job, users, etc, etc) and for the few things that are unstructured, you can represent them well in PG with JSONB (which is what we do for inputs and outputs).

code_biologist 3y ago

Windmill is great. Definitely don't stray from the self-hostable + DX mission! I haven't had need to use it professionally, but I use it on a home server to run several small web crawlers and yt-dlp jobs. Really fun piece of tooling.

darkteflon 3y ago
Could you clarify what you mean by “definitely don't stray from the self-hostable + DX mission”?
- code_biologist 3y ago
  I hate cron for running small personal scripts and jobs. I hate ad-hoc handling of logging / observability / success monitoring. I want a web UI for all of it. I hate trying to figure out how to handle event based job triggers that are more fancy than the 10 line bash script they end up running. I hate trying to wrap up solutions for all those into infrastructure as code — I will end up running an unpatched Ubuntu 20 for the next decade if I can't just nuke the box and reload my config + scripts. I hate the idea of relying on the continued existence of a hosted service, as well as them being able to see that I'm fine tuning an LLM for erotic stories (or whatever). Windmill solved all that for me. I write a short little Python script, paste it into the web UI and boom it's running with good answers for all the above issues. That's great DX. If workflow engines try to solve every problem they can devolve into plugin and complexity hell. Jenkins issues have caused a lot of headache at my day job lately. I don't want Windmill to fall into the same trap.
  - darkteflon 3y ago
    Thanks, that’s really helpful - agreed on all accounts!

DonnyV 3y ago

This looks interesting. I wonder if they will be adding compiled languages like C#?

CuriousSkeptic 3y ago
Might want to checkout DurableTasks[1] for that [1] https://github.com/Azure/durabletask https://github.com/Azure/durabletask
rubenfiszel 3y ago
you can always run compiled programs from bash as binaries (but yeah it's not as great as an experience than being able to run interpreted languages directly).

darkteflon 3y ago

While appreciating that they’re not direct substitutes, does anyone have experience comparing Windmill with Dagster for managing dependency resolution in complex data pipelines?

bafe 3y ago
I briefly tried windmill for a project that involved creating custom workflows on-demand from a configuration file. I can't recommend windmill for data pipelines, it is meant to be more of a low-code internal app platform like retool or budibase. It is meant for a relatively static workflow that requires human intervention, like a simple business process involving some API calls and humans approvals in the loop. For complex (and potentially reconfigurable) data pipelines, dragster is a much better choice
- rubenfiszel 3y ago
  When you tried, we didn't have s3 integrations nor restartable flows. We will present all of this in day 5 of our launch week and it might be a good time to revisit. I agree we were not a good fit prior. I think we would now compare favorably as we will offer excellent ergonomics for data processing, leveraging polars, duckdb, and other OLAP libraries to their full extent.
  - bafe 3y ago
    It was several months ago, so it is entirely possible. Back then I got the feeling that windmill tried to be more of a low code business/internal tool platform than a data/ETL workflow tool, I especially missed an expressive way to define workflows programmatically (i think you had a JSON schema but nothing as powerful as dragster where you can define a whole workflow in pure python)
    - rubenfiszel 3y ago
      We added a vscode extension to build everything from code, including manipulating the yaml or UI and have them instantly impact each other: https://www.windmill.dev/blog/launch-week-1/vscode-extension https://www.windmill.dev/blog/launch-week-1/vscode-extension yaml is not real code and so the part that I consider to be real code is that each step is its separate file in python, typescript, that you can edit in your code editor and have your plugins working, testing frameworks. It's normal functions that you can run locally. It would be possible for us to do like dagster/prefect/airflow which is to use a macro-processing step/decorators to build dynamically the graph which is what our yaml is in the end, a 1:1 encoding of our dag spec called openflow: https://docs.windmill.dev/docs/openflow/ https://docs.windmill.dev/docs/openflow/ We didn't do it yet because in most cases, the decorators are a lot like yaml, they are a very rigid way of declaring that some functions are nodes that you can chain in limited ways. On the other hand, not providing that mode allow us to put more efforts in the low-code graph builder for now. But, as someone that love compilers and built a few, I'm very eager for us to provide such a mode so it's probably a few months away :) We'd love to have your input on the DX for data/ETL once we present it Friday so feel free to join our discord or shoot me an email at ruben@windmill.dev

jgil 3y ago

What are the units for the x-axis? I'm assuming milliseconds from context, but not having the axis labeled obfuscates the interpretation.

rubenfiszel 3y ago
It's seconds. You can find all the detailed numbers there: https://www.windmill.dev/docs/misc/benchmarks/competitors https://www.windmill.dev/docs/misc/benchmarks/competitors

haroldp 3y ago

Is this what I need? We have ad hoc business processes like: 1. Sell service to client 2. Email client with scheduling info 3. Agree to scheduling date with client 4. Send client email with documentation and request their data 5. Remind client to send data 6. Alert admins data hasn't been received 7. Beat data out of client 8. Prepare data 9. QA and checkoff service 10. Email client to review and approve 11. Publish/run service 12. Send client stats/results And I want to move that from spreadsheets, personal emails and admins keeping in their head where everything is at, to web forms and uploads, automated emails and dashboards. I looked at airtable, smartsheet, budibase and many others but they seem to be very project-based, where you are inventing a unique workflow for each project and using it to keep managers on top of where their team is at with the various steps. Project Management vs process management. None of them seem to have decent calendar integration. Few of them seem to be great about emails or scheduled scripts. I have APIs for my data, or I can code them if needed. I'd prefer a low-code to a no-code approach, where managers have a spreadsheet view and can do some of of the UI work and programmers can make it do stuff and handle integrations.

goodpoint 3y ago
It is not.
erichocean 3y ago
I don't think that'll achieve your goals, it's far too clunky and inflexible for actually running a business end-to-end. (No knock on Windmill, they're all like this in my experience as the way they model the world is how a programmer thinks about things, not a business owner. BPMN is straight up cancer—avoid.) I would suggest looking at workflow systems within the VFX space, which they call "pipelines". They are human-driven and maintained, but support very high levels of process automation. VFX pipelines are written in Python, but the key thing is, they're very good at a few things simultaneously: (1) Highly irregular work… (2) That nevertheless requires lots of process, including of the ad hoc variety… (3) With high human touch (human in the loop), so calendars, task assignments, timelines, approvals, producer dashboards, etc. (4) That's repeatable on the next project/opportunity, with as much customization as needed. As a bonus, pipelines handle billions of dollars of work annually under high pressure, with tens of thousands of users, so there's a lot of hard won experience built into them. I personally have found that there are three levels of VFX pipeline tech that need to be combined and customized to do everything. This is the most recent stack I've set up: * Kitsu for the producer/manager level, human task assignments. Mainly for managers, has the calendar functionality. * Prism Pipeline for software integration (i.e. keeping the humans following the process). This is what people actually do. * Kabaret for compute work. People usually kick off these jobs, but they're handled by the farm. You'll be responsible for everything that happens here. [0] Don't try to get by with just one or two of the three, even though feature-wise it looks like you might be able to. Incorporating computer-driven workflows is tricky and requires purpose-built libraries in my experience. Just use the right library for the job, and accept that they all have somewhat overlapping functionality. For most non-VFX work, you'll need to script access to the browser, since that's where people do stuff today (i.e. with the Prism Pipeline library). Launch Chrome, Firefox, or Edge with a debug port and use Playwright as your "plugin" mechanism, with Prism/Python as the driver. You can make your pages do whatever you want and script literally any web-based process, filling in fields from your database as needed, adding data to the page, etc. For the last six years, I've run a business that does around $5M of annual revenue using this approach, with 4 FTE people (including me). We handle around 40 million compute tasks a day, but also have all of the "daily work" where humans are in the loop. We do all of the tasks of the type you've listed. HTH! [0] If you need fully automated task generation (so, no human in the loop), write "pump" scripts in plain Python that push jobs into Kitsu or Kabaret at a particular interval. I personally run them under a god monitor, and have them exit after, say, 15 minutes (or whatever interval makes sense) to get cron-like functionality. This is useful because you can pause execution of any pump process easily, change the interval, restart them, etc. plus you get easy access to the logs for debugging. The scripts themselves are usually less than 100 LOC. I have around 60 of these that run the automatic parts of our business. Whenever I need custom, temporary behavior in this part of the business, I copy the pump script, edit the original to avoid the special case, and have the new script only handle the special case (with whatever new behavior I need). Done, takes very little time, and is easy to run temporarily (e.g. for the next week, a common occurrence because some downstream vendor has caused a problem we need to work around). Another approach to trigger tasks/workflows is to have a server that gets pinged by 3rd parties (e.g. Webhooks) and does the same whenever the endpoint is invoked. We have that too, but it's literally just a few scripts. (Most workflow systems seem to believe this is damn near 100% of the stuff that needs to be "automated.")

yungraft 3y ago

[dead]

deleted 3y ago

[deleted]

j_maffe 3y ago

That graph is utterly useless. How hard is it to place units on a freaking graph?

rubenfiszel 3y ago
Apologies, the units are seconds, fixing the graph right now. You will find more ample details on the benchmarks there: https://www.windmill.dev/docs/misc/benchmarks/competitors https://www.windmill.dev/docs/misc/benchmarks/competitors We made them as easy as possible to be reproduced

deleted 3y ago

[deleted]

alexchamberlain 3y ago

Interesting that Windmill's backend seems to be implemented in Rust, but the docs talk about supporting jobs written in Python, TypeScript and Go. I understand there is generic Docker support, but does anyone know if you can run Rust jobs natively?

sisve 3y ago
It can not. (It does support go)
rubenfiszel 3y ago
We could add support for Rust, and it would work the same as go, do a built step prior, and then cache the binary. If we have enough demand for it, we will add it (we are a small team and can only prioritize what's popular unfortunately). On a personal note, I write a lot of rust (and love it) but using rust for writing non-backend type of stuff is not something I would do myself. I'm more of a believer of hybrid like polars that optimize the compute in Rust but provide a compatible sdk in python.

paxys 3y ago

The devs of Windmill seem to have taken the winning advice of "do one thing and do it well" and done the exact opposite. Going over windmill.dev I have no idea what specifically the software should be used for. Is it a competitor to Retool? Airflow? Temporal? Apparently there's a no-code workflow builder? A drag and drop UI builder? An online IDE? Dozens of integrations? What on earth is going on??

deleted 3y ago
[deleted]
debarshri 3y ago
Doing one thing well is not a winning advice. You can do multiple things, it really depends on the execution and other factors. They are doing one that is workflows and doing it well.
- sisve 3y ago
  Also they are using postgres for everything. I think that is a good example of picking one thing that you know very well and using that to the limits. But i do agree that windmill is doing so many things that the elevator pitch need some work. But a lot of companies are trying to do the same. So i do not think people will think of workflow engine + online ide + app builder as a natural grouping in 5 years
- MuffinFlavored 3y ago
  I don't really understand what a "workflow" is / what it means. Like GitHub Actions is a workflow, but it's a convoluted YAML wrapper around shell scripts. Is a workflow shell scripts? I'm going to guess no. I'm going to guess it's "code". So... why is a workflow code but code isn't a workflow? Windmill is a "workflow (code?)" orchestrator?
  - jagtstronaut 3y ago
    Yeah I assumed this was a camunda competitor (a bpmn workflow product), but it seems like it is not that.

punnerud 3y ago

The GitHub repo for the open source project gives a better intro/sell: https://github.com/windmill-labs/windmill https://github.com/windmill-labs/windmill (Especially the video)

xyoer 3y ago

[dead]

majkinetor 3y ago

I am using Rundeck now to schedule project jobs, get notified on errors, provide UI for PowerShells/bash scripts, and have decent project and user management. It uses Quartz and can handle hundreds of jobs per minute without a problem. It can connect to git repository to keep job definitions there. I also use local file system a lot (on Windows or Linux) as I don't like to edit my scripts in browser, given that they are mini programs with lots of separate includes. I would like to try windmill instead. Can it cover all those cases, particularly the part where scripts are on the file system? Can I use automatic GUI creation for PowerShell scripts of windmill in such case? Rundeck transmits environment variables with GUI values when starting script.

rubenfiszel 3y ago
Not only it can cover those cases, you will likely be pleased with the advantages of such system (the automatic GUI creation is just one of those). The binary itself doesn't sandbox your processes unless you're in the nsjail mode. Hence, if you either use the binary raw (without docker) or mount the filesystem to be available to your container, it can run anything that is available on the filesystem.
- majkinetor 3y ago
  I played a bit now on cloud and was able to run things in powershell and create a custom GUI form, connect button to script and show results in the text area. I didn't succeed to return all output from the powershell, just last value. The only way I got it is by using control "job log". This one however has some custom header of windmill (job guid etc.), so is there any way to show a raw log ? I generally do not like no-code tools, but first impression of this mix of no-code and code looks intriguing and app feels very nice overall, with great UI and easy to understand features. Good job.
  - rubenfiszel 3y ago
    So bash and powershell are a bit special since they are the only one that are not implemented as using a main function and thus have no result to return. However, we had that feedback a few time so there is a trick. If you write your result in ./result.json, we will process it as the result of your powershell script.

atlas_hugged 3y ago

It says open source, then it says limit of 10 SSO users. IANAL so this confuses me. Can you explain what this means? Actually, that sounds rude, excuse the ‘tism, I’m sorry. I’m just not super familiar with the license stuff, I generally only touch MIT licensed code when it’s work related, but I thought open source generally allows modification to code. So how do you enforce a limit of 10? I just skimmed the code a bit because of this confusion and I see stuff about enforcing a license in there. So couldn’t anyone just remove that license check code if it’s open-source? If that can’t be modified, then it’s just “source available”, isn’t it? Which to me is also fine, I just feel like I’m being either misled, if you get what I’m saying? Basically I think this project is pretty cool, so I wanted to bring it up to my boss because I think we have a need for this, and I had previously brought up Airflow, but I don’t know how to explain this to her. Sorry for the stream of consciousness. Typing on mobile while answering Teams messages and tickets.

rubenfiszel 3y ago
Some people will argue we are open-core because almost everything is open-source (AGPL) but our sso and enterprise plugins are under a source-available proprietary license. The reason that we didn't split them out of the codebase is that it's harder to maintain and would require to load the plugins. We didn't want to waste time on that when we had so much to build.
- atlas_hugged 3y ago
  oh ok I see. Just the SSO part is like that. That makes more sense. So I guess I can just self-host for our team of 7 without SSO and we can finally organize these messy cron jobs we’ve accumulated. We would not be reselling at all. Just organizing some annoying aspects of our day to day. Thanks for the clarification. Much appreciated. If we end up expanding our usage down the road, I’ll see if I can convince her to consider shelling out funds for the SSO stuff too. I’d love to be able to support this project. Seems pretty cool!
  - rubenfiszel 3y ago

vlugorilla 3y ago

Been using windmill for a few months, and it's very pleasing. One of the best self-hosted tools I have!

MuffinFlavored 3y ago

https://www.windmill.dev/docs/misc/benchmarks/aws_lambda https://www.windmill.dev/docs/misc/benchmarks/aws_lambda I did not realize at first glance you can write a script and then trigger it (with an API key, not sure if you can make a public one or if that's a security nightmare) through HTTP to basically have your own self-hosted AWS Lambda / the whole "serverless HTTP triggers/functions" craze

rubenfiszel 3y ago
Yes absolutely, we're even faster than lambda with dedicated workers since the design is simpler, while loop jobs from the queue while still keeping all the error handling, observability, and easy deployment.
- MuffinFlavored 3y ago
  Is there a workaround I'm missing where the URLs are exposed without needing an API key? (Something I'm pretty sure Lambda allows if you configure it properly)
  - slig 3y ago
    Not natively, one workaround is using a simple CF worker that proxies the request.
  - rubenfiszel 3y ago
    The api keys can be generated to be webhook specific (that's what they are by default when you generate them in the script's UX) and hence can be made public so there is no risk to expose them publicly, they are similar to a uuid of a lambda (not a key but impossible to guess). We require api key because every request is permissioned and hence we need to know who the script is executed on behalf of. For instance, scripts can fetch secrets/variables but those calls will be permissioned with the permissions of the caller which in this case we get from the webhook specific token.

iFire 3y ago

LICENSE ??? https://github.com/windmill-labs/windmill/blob/main/LICENSE https://github.com/windmill-labs/windmill/blob/main/LICENSE I'm a bit confused.

rubenfiszel 3y ago
We use a mix of AGPL for 95% of the code, Apache 2.0 for specs definitions and apis, and source-available for our enterprise plugins.
SpaghettiCthulu 3y ago
What on Earth is confusing about that?

brunoqc 3y ago

It's open-core (if you care about that). "The SSO is free up to 10ppl." It's weird that "Content Search" is such a pro feature.

rubenfiszel 3y ago
Only if you deem SSO to be of an essential nature to such products. To me it's all semantics at this point. But it's a complex software that require software engineers with salaries and we do not want to rely on donations so we do have to give reasons for people to pay. SSO and audit logs are good ways to segment enterprises to support the project.
- brunoqc 3y ago
  > Only if you deem SSO to be of an essential nature to such products. I thought that multiple features were enterprise-only (like multiplayer, the content search...). In our world where we don't take security seriously enough, I do think that SSO is essential.

haolez 3y ago

I could use this as a bridge between developers and operation, but in my case the lack of localization is annoying for the staff in the ops side. I don't have time to contribute something like this right now, but I will keep an eye on this.

namtab00 3y ago

I'm trying to understand in what way this differs from node-red.

whoiscroberts 3y ago

How does this compare to Prefect?

rubenfiszel 3y ago
it's faster, it supports more languages (python, but also typescript, bash, go, and raw queries to different databases) The rest will be a matter of taste

ketozhang 3y ago

Did you guys considered existing standards when you chose what to use for representing workflow definitions before choosing OpenFlow? For example, Common Workflow Language

rubenfiszel 3y ago
We did and started with an existing standard but quickly, trying to fit to the standard was more complex than rolling our own. Agreed we just created yet one more standard but the bit about the input transforms being full javascript expressions or the way we encoded suspend steps was impossible to retrofit.

RandomBK 3y ago

One balance I've yet to find is how to handle the split between code-in-database (workflows stored in a database and edited via web IDE) vs code-in-git (workflows checked into the repo and only changed via the normal development+peer-review process). It looks like Workflow is primarily the former (code-in-database), but provides an API to sync things from a git repo. Is there a mechanism to enforce the rule that certain scripts/functionality/secrets are locked behind workflows sourced from a provided repository?

sisve 3y ago
I think most of windmills enterprises customers do it the code to git way. I have chosen another part where i deploy and then it's automatically push to git. Buy windmill proves you with 3-4 really good ways of deploying from 1 cowboy just doing what he wants to the normal EE way of doing software meaning code-in-git and people only have read access to production and can not modify things. You can have separate instance running or just separate workspaces and promote.

RandomBK 3y ago

I would love to begin using this system, but the licensing is giving me serious pause. While most of the software is under the AGPLv3, the Commercial License section of the README [0] implies that the company takes on a fairly broad interpretation of the AGPL. In particular, the line "[...] to build a feature on top of Windmill, to comply with AGPLv3 your product must be AGPLv3 [...]" seems to imply the company aligns with the stance taken by Google and other companies: that even calling the application via API is enough to trigger copyleft [1]. This implies that if I were to build a sign-up form that triggers a Windmill workflow in the backend, my entire application would either need to be AGPLv3 or I would need a commercial license. That's perfectly reasonable, as it means any non-AGPL use will have to contribute back to Windmill via a commercial license. However, it does mean positioning this as an "Fully Open-source" alternative to Airflow is only technically correct. This is much closer in practice to "source available" than how most developers would think as "open source". If this isn't how Windmill wants their license interpreted, I highly encourage clarifying things. [0] https://github.com/windmill-labs/windmill#commercial-license https://github.com/windmill-labs/windmill#commercial-license [1] https://opensource.google/documentation/reference/using/agpl-policy https://opensource.google/documentation/reference/using/agpl...

rubenfiszel 3y ago
Sorry for the lack of clarity. By product we mean white-labelling windmill, not triggering a script or flow by api.
- RandomBK 3y ago
  Got it! That's great to hear. It'd be great to explicitly clarify that in the README :) Out of curiosity: Have you considered alternative licenses like SSPL or the Elastic license, which make a clear(er) delineation between whitelabelling/hosting vs simply using the application? I'm sure I'm not the first person to have written off Windmill because of the AGPL.
  - rubenfiszel 3y ago
    Those licenses are not open-source and likely to be more restrictive than what we currently offer. It was important for me to be truly open-source. If you are using windmill as a whole, you're free to use it however you please with the AGPL but for customers with doubts, we do sell an enterprise/commercial license.

felipemesquita 3y ago

This looks great! Are there any plans to include Ruby as a supported scripting language along with Python, Typescript, Go, Bash, and Sql? I currently use my own crude workflow runner for processes that are mostly sql, but with some ruby code. I’ve stuck with it because it works and we currently use Ruby for almost everything, so it’s not worth it to add another language to the mix so we could use a better tool.

rubenfiszel 3y ago
Ruby is next :)
- felipemesquita 3y ago
  Looking forward to trying it as soon as possible!

hokkos 3y ago

A little late but after reading this thread I moved a airflow instance to windmill and I much much prefer windmin, airflow was clunky and heavy for my simple workflow that consist of running a few python script. Also is auto ui is great to play with the arguments in, the cron ui and next start date... Also is it written in rust so much more low usage than airflow that consistently cpu picked while doing nothing.