4 ms·

Fair point, TIL about another incredible capability of the BEAM. As long as you're willing to write Elixir, this is clearly a superior scheme for deferred tasks

Fair point, TIL about another incredible capability of the BEAM. As long as you're willing to write Elixir, this is clearly a superior scheme for deferred tasks/background jobs. One issue I see with this scheme still is that you have to be careful of what you do at initialization of the app since now all your background jobs are gonna run that. For instance, maybe your task doesn't need to be connected to the db and as per the article it will if your app does. They mention having hot-modules, but what if you want to run 1M of those jobs on 100 workers, you now have a 100 unnecessary apps. It's probably a non-issue, the number of things done at initialization could be kept minimal, and FLAME could just have some checks to skip initialization code when in a flame context.

chrismccord 3y ago
This is actually a feature. If you watch the screencast, I talk about Elixir supervision trees and how all Elixir programs carefully specify the order their services stop and stop in. So if your flame functions need DB access, you start your Ecto.Repo with a small or single DB connection pool. If not, you flip it off. > It's probably a non-issue, the number of things done at initialization could be kept minimal, and FLAME could just have some checks to skip initialization code when in a flame context. Exactly :)
- jrmiii 3y ago
  So, Chris, how do you envision the FLAME child understanding what OTP children it needs to start on boot, because this could be FLAME.call dependent if you have multiple types of calls as described above. Is there a way to pass along that data or for it to be pulled from the parent? Acknowledging this is brand new; just curious what your thinking is. EDIT: Would it go in the pool config, and a runner as a member of the pool has access to that?
  - chrismccord 3y ago
    Good question. The pools themselves in your app will be per usecase, and you can reference the named pool you are a part of inside the runner, ie by looking in system env passed as pool options. That said, we should probably just encode the pool name along with the other parent info in the `%FLAME.Parent{}` for easier lookup
    - jrmiii 3y ago
      Ah, that makes a lot of sense - I think the FLAME.Parent{} approach may enable backends that wouldn't be possible otherwise. For example, if I used the heroku api to do the equivalent of ps:scale to boot up more nodes - those new nodes (dynos in heroku parlance) could see what kind of pool members they are. I don't think there is a way to do dyno specific env vars - they apply at the app level. If anyone tries to do a Heroku backend before I do, an alternative might be to use distinct process types in the Procfile for each named pool and ps:scale those to 0 or more. Also, might need something like Supabase's libcluster_postgres[1] to fully pull it off. EDIT2: So the heroku backend would be a challenge. You'd maybe have to use something like the formation api[2] to spawn the pool, but even then you can't idle them down because Heroku will try to start them back. I.e. there's no `restart: false` from what I can tell from the docs or you could use the dyno api[3] with a timeout set up front (no idle awareness) [1] https://github.com/supabase/libcluster_postgres https://github.com/supabase/libcluster_postgres [2] https://devcenter.heroku.com/articles/platform-api-reference#formation https://devcenter.heroku.com/articles/platform-api-reference... [3] https://devcenter.heroku.com/articles/platform-api-reference#dyno https://devcenter.heroku.com/articles/platform-api-reference...

chrismccord 3y ago

This is actually a feature. If you watch the screencast, I talk about Elixir supervision trees and how all Elixir programs carefully specify the order their services stop and stop in. So if your flame functions need DB access, you start your Ecto.Repo with a small or single DB connection pool. If not, you flip it off. > It's probably a non-issue, the number of things done at initialization could be kept minimal, and FLAME could just have some checks to skip initialization code when in a flame context. Exactly :)

jrmiii 3y ago
So, Chris, how do you envision the FLAME child understanding what OTP children it needs to start on boot, because this could be FLAME.call dependent if you have multiple types of calls as described above. Is there a way to pass along that data or for it to be pulled from the parent? Acknowledging this is brand new; just curious what your thinking is. EDIT: Would it go in the pool config, and a runner as a member of the pool has access to that?
- chrismccord 3y ago
  Good question. The pools themselves in your app will be per usecase, and you can reference the named pool you are a part of inside the runner, ie by looking in system env passed as pool options. That said, we should probably just encode the pool name along with the other parent info in the `%FLAME.Parent{}` for easier lookup
  - jrmiii 3y ago
    Ah, that makes a lot of sense - I think the FLAME.Parent{} approach may enable backends that wouldn't be possible otherwise. For example, if I used the heroku api to do the equivalent of ps:scale to boot up more nodes - those new nodes (dynos in heroku parlance) could see what kind of pool members they are. I don't think there is a way to do dyno specific env vars - they apply at the app level. If anyone tries to do a Heroku backend before I do, an alternative might be to use distinct process types in the Procfile for each named pool and ps:scale those to 0 or more. Also, might need something like Supabase's libcluster_postgres[1] to fully pull it off. EDIT2: So the heroku backend would be a challenge. You'd maybe have to use something like the formation api[2] to spawn the pool, but even then you can't idle them down because Heroku will try to start them back. I.e. there's no `restart: false` from what I can tell from the docs or you could use the dyno api[3] with a timeout set up front (no idle awareness) [1] https://github.com/supabase/libcluster_postgres https://github.com/supabase/libcluster_postgres [2] https://devcenter.heroku.com/articles/platform-api-reference#formation https://devcenter.heroku.com/articles/platform-api-reference... [3] https://devcenter.heroku.com/articles/platform-api-reference#dyno https://devcenter.heroku.com/articles/platform-api-reference...