4 ms·

We’re experiencing this at my place of work. Our backend stack is Python, and coming from an Elixir/Phoenix background, adding a durable queue to our infrastruc

by shijie 3y ago

We’re experiencing this at my place of work. Our backend stack is Python, and coming from an Elixir/Phoenix background, adding a durable queue to our infrastructure should be trivial but because of the nature of Python, it is all but impossible to run the queue in the same codebase and application server. Elixir and the BEAM make it so easy and pleasant to run extremely complicated infrastructure in the same codebase and in the same application context. Hard problems made easy. I miss it.

lionkor 3y ago
FYI "all but impossible" usually means the opposite of what you think it means - "all but" meaning "everything except". Seen this usage a lot and it bothers me, because both usages are common and theyre the exact opposites. If I didn't have context clues, it'd be difficult to know which one you mean. Edit: not native but I've always understood it as the first def here https://idioms.thefreedictionary.com/all+but https://idioms.thefreedictionary.com/all+but
- pakitan 3y ago
  It's an idiom and means "nearly impossible". There is no other usage.
- shijie 3y ago
  In this context I meant “all but” == “everything except” i.e it is almost impossible. Are you saying it means the opposite of how I used it? I’m not certain what you mean.
- TJSomething 3y ago
  I'm not sure how it makes sense to claim that an action is "everything except impossible." This is more in the classic sense of "the battle was all but lost" for a battle was practically but not actually lost. You could implement a queue in a Python server, but it would be technically awful to do.
- yodon 3y ago
  100% of the usage I've encountered as a native speaker is "all but impossible" = "almost impossible". This stems from "all but" = "almost"[0] [0]https://www.merriam-webster.com/dictionary/all%20but https://www.merriam-webster.com/dictionary/all%20but
  - Terr_ 3y ago
    Agreed, to me the meaning unpacks like: "it meets or exceeds every category of difficulty that is less severe than Impossible." Compare to a disaster described as "the destruction is all-but-total". That phrase does not mean "you could describe the destruction by any label except total." For example, whatever just happened it definitely not "tiny" or "moderate".
- andyferris 3y ago
  “All but” = “almost” (which presumably historically was “all most”). Like “all but dead”. Everything except dead, as in as close to death as possible without actually being dead (and certainly not the opposite of dead, as in perfect health).
catlover76 3y ago
> it is all but impossible to run the queue in the same codebase and application server. I am surprised, both in terms of the difficulty you faced and that you would want to; I have beef with the fact that using Celery with Django runs "worker" process on the application server by default.
- andy_ppp 3y ago
  It sounds mad but because of the process isolation within the language you can trust running services within your application and message passing can be local or to a different machine in the BEAM cluster. It opens up a whole new way to think about a runtime once everything is an actor model process with a mailbox you can send messages to. It's much better than having to worry about extra infrastructure, just deploy your app and because it's code you control you can customise what the queue is going in really bespoke and application specific ways.
  - catlover76 3y ago
    > to a different machine in the BEAM cluster. Wow, that's pretty cool
    - andy_ppp 3y ago
      Wait until you find out you can connect an interactive shell to production and run functions in your application to debug things live…
Yoric 3y ago
Every time I see microservices, or Kafka, or Redis, ... I can't help being disappointed because we seem to be stuck reinventing BEAM/Erlang (and Elixir) 30 years after the fact, except with technologies that are slower, more fragile, more complicated to use and harder to deploy. :sigh:
- rdtsc 3y ago
  Someone in the Erlang community named our processes “nanoservices” as a joke. I thought that was pretty apt though, a gen_server is pretty close. And you can have 3M of those on one node!
  - sodapopcan 3y ago
    I think it's more like Erlang processes are microservices and microservices as we know them today are usually "regularly-sized services" :D
thibaut_barrere 3y ago
Something I often "draw" to other developers when showing Elixir is the difference in terms of "boundaries". It is common to have: frontend <-JSON-> backend <-db/queue-> background jobs Each boundary introduces complexity (marshalling / impedance mismatch / but also different HR/recruiting needs). In Elixir you can more or less do: front/back/background with no boundaries or reduced boundaries (e.g. using LiveView & OBAN in the same process, even if going through Postgres as a queue). Someone coined the term "deepstack engineer" as well recently I think, which I could draw as: frontend <-> backend <-> background <-> machine learning Now with Elixir you can do everything in the same process (but with isolation as required): front/back/background/ml And this is something I find really interesting about this stack.
- itronitron 3y ago
  Interesting, most of my work has been 'across the stack' but for desktop applications. This looks like a better match for me than most other approaches to web applications.
  - thibaut_barrere 3y ago
    It is a very good point: I did a lot of desktop apps at one point (the largest being a mixture of C# and legacy C++ wrappers), and the first time I tried LiveView it somehow reminded me of WinForms! So I can relate :-)
- Yoric 3y ago
  I'm curious, how do you run ML in the same process?
  - thibaut_barrere 3y ago
    It is just the way it is with Elixir - everything can run in the same process (or in different connected nodes): See: - https://www.youtube.com/watch?v=g3oyh3g1AtQ https://www.youtube.com/watch?v=g3oyh3g1AtQ - https://www.youtube.com/watch?v=HK38-HIK6NA https://www.youtube.com/watch?v=HK38-HIK6NA
Spivak 3y ago
But why? I have one project that is a single Python codebase that embeds the whole http server, task scheduler, a durable queue implementation, and job processing implementation using those queues all in a single process. I'm sure you're right that the nature of Elixir makes wrangling all the different services easier but is there something special it does to make embedding easier as well?
- keep_reading 3y ago
  Your performance is going to be comparatively terrible because of the way those tasks are scheduled in Python
- Yoric 3y ago
  How do you dispatch to multiple cores in a single process in Python?
- OhHiMarkos 3y ago
  Care to share the code of this project? I am looking to build something similar.
indigo0086 3y ago
Are there any examples online of doing these things with simple (or just not as complex) elixir code compared to using dedicated libraries in other languages

by shijie 3y ago

lionkor 3y ago

FYI "all but impossible" usually means the opposite of what you think it means - "all but" meaning "everything except". Seen this usage a lot and it bothers me, because both usages are common and theyre the exact opposites. If I didn't have context clues, it'd be difficult to know which one you mean. Edit: not native but I've always understood it as the first def here https://idioms.thefreedictionary.com/all+but https://idioms.thefreedictionary.com/all+but

pakitan 3y ago
It's an idiom and means "nearly impossible". There is no other usage.
shijie 3y ago
In this context I meant “all but” == “everything except” i.e it is almost impossible. Are you saying it means the opposite of how I used it? I’m not certain what you mean.
TJSomething 3y ago
I'm not sure how it makes sense to claim that an action is "everything except impossible." This is more in the classic sense of "the battle was all but lost" for a battle was practically but not actually lost. You could implement a queue in a Python server, but it would be technically awful to do.
yodon 3y ago
100% of the usage I've encountered as a native speaker is "all but impossible" = "almost impossible". This stems from "all but" = "almost"[0] [0]https://www.merriam-webster.com/dictionary/all%20but https://www.merriam-webster.com/dictionary/all%20but
- Terr_ 3y ago
  Agreed, to me the meaning unpacks like: "it meets or exceeds every category of difficulty that is less severe than Impossible." Compare to a disaster described as "the destruction is all-but-total". That phrase does not mean "you could describe the destruction by any label except total." For example, whatever just happened it definitely not "tiny" or "moderate".
andyferris 3y ago
“All but” = “almost” (which presumably historically was “all most”). Like “all but dead”. Everything except dead, as in as close to death as possible without actually being dead (and certainly not the opposite of dead, as in perfect health).

catlover76 3y ago

> it is all but impossible to run the queue in the same codebase and application server. I am surprised, both in terms of the difficulty you faced and that you would want to; I have beef with the fact that using Celery with Django runs "worker" process on the application server by default.

andy_ppp 3y ago
It sounds mad but because of the process isolation within the language you can trust running services within your application and message passing can be local or to a different machine in the BEAM cluster. It opens up a whole new way to think about a runtime once everything is an actor model process with a mailbox you can send messages to. It's much better than having to worry about extra infrastructure, just deploy your app and because it's code you control you can customise what the queue is going in really bespoke and application specific ways.
- catlover76 3y ago
  > to a different machine in the BEAM cluster. Wow, that's pretty cool
  - andy_ppp 3y ago
    Wait until you find out you can connect an interactive shell to production and run functions in your application to debug things live…

Yoric 3y ago

Every time I see microservices, or Kafka, or Redis, ... I can't help being disappointed because we seem to be stuck reinventing BEAM/Erlang (and Elixir) 30 years after the fact, except with technologies that are slower, more fragile, more complicated to use and harder to deploy. :sigh:

rdtsc 3y ago
Someone in the Erlang community named our processes “nanoservices” as a joke. I thought that was pretty apt though, a gen_server is pretty close. And you can have 3M of those on one node!
- sodapopcan 3y ago
  I think it's more like Erlang processes are microservices and microservices as we know them today are usually "regularly-sized services" :D

thibaut_barrere 3y ago

Something I often "draw" to other developers when showing Elixir is the difference in terms of "boundaries". It is common to have: frontend <-JSON-> backend <-db/queue-> background jobs Each boundary introduces complexity (marshalling / impedance mismatch / but also different HR/recruiting needs). In Elixir you can more or less do: front/back/background with no boundaries or reduced boundaries (e.g. using LiveView & OBAN in the same process, even if going through Postgres as a queue). Someone coined the term "deepstack engineer" as well recently I think, which I could draw as: frontend <-> backend <-> background <-> machine learning Now with Elixir you can do everything in the same process (but with isolation as required): front/back/background/ml And this is something I find really interesting about this stack.

itronitron 3y ago
Interesting, most of my work has been 'across the stack' but for desktop applications. This looks like a better match for me than most other approaches to web applications.
- thibaut_barrere 3y ago
  It is a very good point: I did a lot of desktop apps at one point (the largest being a mixture of C# and legacy C++ wrappers), and the first time I tried LiveView it somehow reminded me of WinForms! So I can relate :-)
Yoric 3y ago
I'm curious, how do you run ML in the same process?
- thibaut_barrere 3y ago
  It is just the way it is with Elixir - everything can run in the same process (or in different connected nodes): See: - https://www.youtube.com/watch?v=g3oyh3g1AtQ https://www.youtube.com/watch?v=g3oyh3g1AtQ - https://www.youtube.com/watch?v=HK38-HIK6NA https://www.youtube.com/watch?v=HK38-HIK6NA

Spivak 3y ago

But why? I have one project that is a single Python codebase that embeds the whole http server, task scheduler, a durable queue implementation, and job processing implementation using those queues all in a single process. I'm sure you're right that the nature of Elixir makes wrangling all the different services easier but is there something special it does to make embedding easier as well?

keep_reading 3y ago
Your performance is going to be comparatively terrible because of the way those tasks are scheduled in Python
Yoric 3y ago
How do you dispatch to multiple cores in a single process in Python?
OhHiMarkos 3y ago
Care to share the code of this project? I am looking to build something similar.

indigo0086 3y ago

Are there any examples online of doing these things with simple (or just not as complex) elixir code compared to using dedicated libraries in other languages