3 ms·

Can you give an example of what you consider a good solution in a different language? That will help us see exactly what you're looking for and how/whether it m

Can you give an example of what you consider a good solution in a different language? That will help us see exactly what you're looking for and how/whether it might be achieved in Python.

bmitc 3y ago
Erlang and Elixir. :)
- staticautomatic 3y ago
  Well RabbitMQ is built in Erlang so there you go :)
  - bmitc 3y ago
    But RabbitMQ is just a messaging bus, isn't it? That doesn't handle the processes (operating system processes or virtual processes) actually processing the messages and those processes handling fault-tolerance for the things they're connected to.
    - staticautomatic 3y ago
      That is correct. Honestly I’m not sure this is a good use case for Python but it’s definitely possible. I’ve used the RabbitMQ + gevent + multiprocessing pattern in the past and it works but I find the code extremely hard to reason about. If I were doing it again from scratch I’d probably choose another language with better concurrency primitives.
      - bmitc 3y ago
        > Honestly I’m not sure this is a good use case for Python but it’s definitely possible It definitely isn't, but this is a new role with a rushed timeline in a place dominated by Python. > and it works but I find the code extremely hard to reason about This is also a major concern of mine.
        staticautomatic 3y ago
        Rather than handling the multiprocessing and message passing yourself it would be much easier to use celery + gevent and let celery do the work of spinning up processes for executing the tasks. It may feel limiting but my advice would be to keep all the celery job queue stuff isolated from your server, especially if you are using an async web framework. Have your web server just put all the jobs in the celery queue and let it handle executing them, regardless of whether they’re cpu or io-bound. If you try to optimize too much by doing something like leaning on celery for cpu-bound tasks but letting your web server handle the io-bound ones you’re going to be in for a world of hurt when it comes to both debugging and enforcing the order of execution. Celery has its warts but you’ll at least know where in the system your problem is and have reasonably good control over the pipeline.
        bmitc 3y ago
        Can Celery help handle state internal to the workers? What about using Pkykka alongside Pyro such that each Pyro remote object is actually a Pykka actor? Such that Pyro allows splitting workers across separate Python OS processes and the "messaging" while Pykka handles the internal state.
        staticautomatic 3y ago
        Could you give an example of state internal to the worker?
        bmitc 3y ago
        Similar to how one can pass around state inside an OTP GenServer in Erlang. In Python, it could be a class managing a session type of object, like a TCP socket connection, or managing some piece of hardware that is doing something independently of the other parts of the system, or a database writer, etc.
        staticautomatic 3y ago
- da39a3ee 3y ago
  That was what I was worried you'd say. So it doesn't seem like your question is about Python at all, and all your criticism of Python seems entirely unjustified. What you're actually saying is "I have to work in a language other than Erlang/Elixir and I don't like it" But Erlang/Elixir are defined by the sort of concurrency model that you say you want, so as far as I can see your question would be just the same if you were asking about Java or Go. If you have a question like "I can do this in Java or Go. How do I do it in Python?" Then people might be able to help. But please stop with the criticism of Python for not being Erlang/Elixir, since Java and Go also are not that.
  - bmitc 3y ago
    Do you have any particularly helpful advice? This is honestly pretty frustrating trying to build a system like this in Python. If you think asyncio, coroutines, tasks, multiprocessing, threads, subprocesses, etc. are simple and understandable in Python, then I would love for someone to explain them to me in a way that is manageable. And no, the question would not be the same. Java and Go have proper concurrency substrates. Python does not.

Can you give an example of what you consider a good solution in a different language? That will help us see exactly what you're looking for and how/whether it might be achieved in Python.

bmitc 3y ago

Erlang and Elixir. :)

staticautomatic 3y ago
Well RabbitMQ is built in Erlang so there you go :)
- bmitc 3y ago
  But RabbitMQ is just a messaging bus, isn't it? That doesn't handle the processes (operating system processes or virtual processes) actually processing the messages and those processes handling fault-tolerance for the things they're connected to.
  - staticautomatic 3y ago
    That is correct. Honestly I’m not sure this is a good use case for Python but it’s definitely possible. I’ve used the RabbitMQ + gevent + multiprocessing pattern in the past and it works but I find the code extremely hard to reason about. If I were doing it again from scratch I’d probably choose another language with better concurrency primitives.
    - bmitc 3y ago
      > Honestly I’m not sure this is a good use case for Python but it’s definitely possible It definitely isn't, but this is a new role with a rushed timeline in a place dominated by Python. > and it works but I find the code extremely hard to reason about This is also a major concern of mine.
      - staticautomatic 3y ago
        Rather than handling the multiprocessing and message passing yourself it would be much easier to use celery + gevent and let celery do the work of spinning up processes for executing the tasks. It may feel limiting but my advice would be to keep all the celery job queue stuff isolated from your server, especially if you are using an async web framework. Have your web server just put all the jobs in the celery queue and let it handle executing them, regardless of whether they’re cpu or io-bound. If you try to optimize too much by doing something like leaning on celery for cpu-bound tasks but letting your web server handle the io-bound ones you’re going to be in for a world of hurt when it comes to both debugging and enforcing the order of execution. Celery has its warts but you’ll at least know where in the system your problem is and have reasonably good control over the pipeline.
        bmitc 3y ago
        Can Celery help handle state internal to the workers? What about using Pkykka alongside Pyro such that each Pyro remote object is actually a Pykka actor? Such that Pyro allows splitting workers across separate Python OS processes and the "messaging" while Pykka handles the internal state.
        staticautomatic 3y ago
        Could you give an example of state internal to the worker?
        bmitc 3y ago
        Similar to how one can pass around state inside an OTP GenServer in Erlang. In Python, it could be a class managing a session type of object, like a TCP socket connection, or managing some piece of hardware that is doing something independently of the other parts of the system, or a database writer, etc.
        staticautomatic 3y ago
da39a3ee 3y ago
That was what I was worried you'd say. So it doesn't seem like your question is about Python at all, and all your criticism of Python seems entirely unjustified. What you're actually saying is "I have to work in a language other than Erlang/Elixir and I don't like it" But Erlang/Elixir are defined by the sort of concurrency model that you say you want, so as far as I can see your question would be just the same if you were asking about Java or Go. If you have a question like "I can do this in Java or Go. How do I do it in Python?" Then people might be able to help. But please stop with the criticism of Python for not being Erlang/Elixir, since Java and Go also are not that.
- bmitc 3y ago
  Do you have any particularly helpful advice? This is honestly pretty frustrating trying to build a system like this in Python. If you think asyncio, coroutines, tasks, multiprocessing, threads, subprocesses, etc. are simple and understandable in Python, then I would love for someone to explain them to me in a way that is manageable. And no, the question would not be the same. Java and Go have proper concurrency substrates. Python does not.