4 ms·

That's great. I agree with the whole thesis. We took an alternative approach with https://www.windmill.dev https://www.windmill.dev which is to consider the un

That's great. I agree with the whole thesis. We took an alternative approach with https://www.windmill.dev https://www.windmill.dev which is to consider the unit of abstraction to be at the source code level rather than the container level. We then parse the main function, and imports to extract the args and dependencies, and then run the code as is in the desired runtime (typescript, python, go, bash). Then all the secret sauce is to manage the cache efficiently so that the workers are always hot regardless of your imports It's not as integrated in the codebase as this, but the audience is different, our users build complex workflows from scratch, cron jobs, or just one-off scripts with the auto-generated UI. Indeed the whole context in FLAME seems to be snapshotted and then rehydrated on the target VM. Another approach would be to introduce syntax to specify what is required context from what is not and only loading the minimally required. That's what we are currently exploring for integrating better Windmill with existing codebase instead of having to rely on http calls.

bo0tzz 3y ago
> Indeed the whole context in FLAME seems to be snapshotted and then rehydrated on the target VM. Another approach would be to introduce syntax to specify what is required context from what is not and only loading the minimally required. This isn't strictly what is happening. FLAME just uses the BEAM's built in clustering features to call a function on a remote node. That implicitly handles transferring only the context that is necessary. From the article: > FLAME.call accepts the name of a runner pool, and a function. It then finds or boots a new copy of our entire application and runs the function there. Any variables the function closes over (like our %Video{} struct and interval) are passed along automatically.
- rubenfiszel 3y ago
  Fair point, TIL about another incredible capability of the BEAM. As long as you're willing to write Elixir, this is clearly a superior scheme for deferred tasks/background jobs. One issue I see with this scheme still is that you have to be careful of what you do at initialization of the app since now all your background jobs are gonna run that. For instance, maybe your task doesn't need to be connected to the db and as per the article it will if your app does. They mention having hot-modules, but what if you want to run 1M of those jobs on 100 workers, you now have a 100 unnecessary apps. It's probably a non-issue, the number of things done at initialization could be kept minimal, and FLAME could just have some checks to skip initialization code when in a flame context.
  - chrismccord 3y ago
    This is actually a feature. If you watch the screencast, I talk about Elixir supervision trees and how all Elixir programs carefully specify the order their services stop and stop in. So if your flame functions need DB access, you start your Ecto.Repo with a small or single DB connection pool. If not, you flip it off. > It's probably a non-issue, the number of things done at initialization could be kept minimal, and FLAME could just have some checks to skip initialization code when in a flame context. Exactly :)
    - jrmiii 3y ago
      So, Chris, how do you envision the FLAME child understanding what OTP children it needs to start on boot, because this could be FLAME.call dependent if you have multiple types of calls as described above. Is there a way to pass along that data or for it to be pulled from the parent? Acknowledging this is brand new; just curious what your thinking is. EDIT: Would it go in the pool config, and a runner as a member of the pool has access to that?
      - chrismccord 3y ago
        Good question. The pools themselves in your app will be per usecase, and you can reference the named pool you are a part of inside the runner, ie by looking in system env passed as pool options. That said, we should probably just encode the pool name along with the other parent info in the `%FLAME.Parent{}` for easier lookup
        jrmiii 3y ago
        Ah, that makes a lot of sense - I think the FLAME.Parent{} approach may enable backends that wouldn't be possible otherwise. For example, if I used the heroku api to do the equivalent of ps:scale to boot up more nodes - those new nodes (dynos in heroku parlance) could see what kind of pool members they are. I don't think there is a way to do dyno specific env vars - they apply at the app level. If anyone tries to do a Heroku backend before I do, an alternative might be to use distinct process types in the Procfile for each named pool and ps:scale those to 0 or more. Also, might need something like Supabase's libcluster_postgres[1] to fully pull it off. EDIT2: So the heroku backend would be a challenge. You'd maybe have to use something like the formation api[2] to spawn the pool, but even then you can't idle them down because Heroku will try to start them back. I.e. there's no `restart: false` from what I can tell from the docs or you could use the dyno api[3] with a timeout set up front (no idle awareness) [1] https://github.com/supabase/libcluster_postgres https://github.com/supabase/libcluster_postgres [2] https://devcenter.heroku.com/articles/platform-api-reference#formation https://devcenter.heroku.com/articles/platform-api-reference... [3] https://devcenter.heroku.com/articles/platform-api-reference#dyno https://devcenter.heroku.com/articles/platform-api-reference...
Nezteb 3y ago
Oops you've got an extra w, here is the URL for anyone looking: https://www.windmill.dev/ https://www.windmill.dev/ I love the project's goals; I'm really hoping Windmill becomes a superior open-source Retool/Airtable alternative!
- rubenfiszel 3y ago
  Thanks, fixed! (and thanks)

bo0tzz 3y ago

> Indeed the whole context in FLAME seems to be snapshotted and then rehydrated on the target VM. Another approach would be to introduce syntax to specify what is required context from what is not and only loading the minimally required. This isn't strictly what is happening. FLAME just uses the BEAM's built in clustering features to call a function on a remote node. That implicitly handles transferring only the context that is necessary. From the article: > FLAME.call accepts the name of a runner pool, and a function. It then finds or boots a new copy of our entire application and runs the function there. Any variables the function closes over (like our %Video{} struct and interval) are passed along automatically.

rubenfiszel 3y ago
Fair point, TIL about another incredible capability of the BEAM. As long as you're willing to write Elixir, this is clearly a superior scheme for deferred tasks/background jobs. One issue I see with this scheme still is that you have to be careful of what you do at initialization of the app since now all your background jobs are gonna run that. For instance, maybe your task doesn't need to be connected to the db and as per the article it will if your app does. They mention having hot-modules, but what if you want to run 1M of those jobs on 100 workers, you now have a 100 unnecessary apps. It's probably a non-issue, the number of things done at initialization could be kept minimal, and FLAME could just have some checks to skip initialization code when in a flame context.
- chrismccord 3y ago
  This is actually a feature. If you watch the screencast, I talk about Elixir supervision trees and how all Elixir programs carefully specify the order their services stop and stop in. So if your flame functions need DB access, you start your Ecto.Repo with a small or single DB connection pool. If not, you flip it off. > It's probably a non-issue, the number of things done at initialization could be kept minimal, and FLAME could just have some checks to skip initialization code when in a flame context. Exactly :)
  - jrmiii 3y ago
    So, Chris, how do you envision the FLAME child understanding what OTP children it needs to start on boot, because this could be FLAME.call dependent if you have multiple types of calls as described above. Is there a way to pass along that data or for it to be pulled from the parent? Acknowledging this is brand new; just curious what your thinking is. EDIT: Would it go in the pool config, and a runner as a member of the pool has access to that?
    - chrismccord 3y ago
      Good question. The pools themselves in your app will be per usecase, and you can reference the named pool you are a part of inside the runner, ie by looking in system env passed as pool options. That said, we should probably just encode the pool name along with the other parent info in the `%FLAME.Parent{}` for easier lookup
      - jrmiii 3y ago
        Ah, that makes a lot of sense - I think the FLAME.Parent{} approach may enable backends that wouldn't be possible otherwise. For example, if I used the heroku api to do the equivalent of ps:scale to boot up more nodes - those new nodes (dynos in heroku parlance) could see what kind of pool members they are. I don't think there is a way to do dyno specific env vars - they apply at the app level. If anyone tries to do a Heroku backend before I do, an alternative might be to use distinct process types in the Procfile for each named pool and ps:scale those to 0 or more. Also, might need something like Supabase's libcluster_postgres[1] to fully pull it off. EDIT2: So the heroku backend would be a challenge. You'd maybe have to use something like the formation api[2] to spawn the pool, but even then you can't idle them down because Heroku will try to start them back. I.e. there's no `restart: false` from what I can tell from the docs or you could use the dyno api[3] with a timeout set up front (no idle awareness) [1] https://github.com/supabase/libcluster_postgres https://github.com/supabase/libcluster_postgres [2] https://devcenter.heroku.com/articles/platform-api-reference#formation https://devcenter.heroku.com/articles/platform-api-reference... [3] https://devcenter.heroku.com/articles/platform-api-reference#dyno https://devcenter.heroku.com/articles/platform-api-reference...

Nezteb 3y ago

Oops you've got an extra w, here is the URL for anyone looking: https://www.windmill.dev/ https://www.windmill.dev/ I love the project's goals; I'm really hoping Windmill becomes a superior open-source Retool/Airtable alternative!

rubenfiszel 3y ago
Thanks, fixed! (and thanks)