3 ms·

Distributed software, which accesses and depends on remote resources, is harder to run than local-only software. I'm working on making it easier to run distribu

by catern 5y ago

Distributed software, which accesses and depends on remote resources, is harder to run than local-only software. I'm working on making it easier to run distributed software for both programmers and end-users. See the main article describing my current vision: http://catern.com/integration.html http://catern.com/integration.html My most significant project in this area: https://github.com/catern/rsyscall https://github.com/catern/rsyscall My website in general: http://catern.com/ http://catern.com/ (happy to collaborate on anything on there) And my current plans: http://catern.com/plan.html http://catern.com/plan.html If you're interested, feel free to contact me using the information on my website.

remram 5y ago
I read this comment 3 times and skimmed both links but I still have no concrete idea what you're building.
- catern 5y ago
  Can you say what left you confused? I'm surprised, I thought https://github.com/catern/rsyscall https://github.com/catern/rsyscall is pretty self-contained and self-explanatory (if unclear on what exactly rsyscall is useful for), and I thought http://catern.com/integration.html http://catern.com/integration.html is quite detailed (perhaps excessively so).
  - remram 5y ago
    I understand the problem and the use-case, but what does your library provide? Running processes remotely? Supervising remote processes? Transparently running Python functions in remote processes?
    - catern 5y ago
      >Running processes remotely? Supervising remote processes? These things. This is mentioned in http://catern.com/integration.html#thread http://catern.com/integration.html#thread
      - remram 5y ago
        So this offers capabilities like `multiprocessing` but possibly across the network, e.g. queues and shared memory? Can it spawn the remote processes too? With SSH, or its own system? You keep pointing me to that document that you feel is compelling, but all it does is explicitly run a Python function in a local thread... I know it's meant as a shortcut in the example but it's hard to infer what the real capabilities are, when what you're doing is not much more than what `trio` or stdlib can do.
        catern 5y ago
        >Can it spawn the remote processes too? With SSH, or its own system? Yes, with rsyscall, is that not clear? It's pretty explicit in http://catern.com/integration.html#thread http://catern.com/integration.html#thread >but all it does is explicitly run a Python function in a local thread... That's not all that's happening. I think you might be pattern-matching what's happening in the article to something you've already seen, when in fact it's something novel.
        remram 5y ago
        It's clear that it "may operate on a local or remote host" and in this case you run a local thread. Just how much is actually implemented is not clear at all. A lot of frameworks are written to be "extensible" but the extensions are left as an exercise to the reader... It is nice to have a self-contained example that you can run on your machine, but I think an example that actually shows the capabilities of the framework will serve you better. E.g. show an example of serving network requests by running services on multiple nodes, rather than a toy function with a toy database. Showing that it's as easy as `run_in_executor()` is nice, but not if it does the exact same thing as `run_in_executor()`.
        catern 5y ago
        >Showing that it's as easy as `run_in_executor()` is nice, but not if it does the exact same thing as `run_in_executor()`. That's fair and a good point! I suppose reading this, you have no evidence that anything but local_thread exists. I'll change that, somehow, to show that indeed there exist values other than local_thread, and that passing those will run things on remote hosts. This article originally grew out of a tutorial on writing tests using this style, for which running over multiple hosts is not necessary and maybe undesirable... but that's not good for what the article is today.

by catern 5y ago

remram 5y ago

I read this comment 3 times and skimmed both links but I still have no concrete idea what you're building.

catern 5y ago
Can you say what left you confused? I'm surprised, I thought https://github.com/catern/rsyscall https://github.com/catern/rsyscall is pretty self-contained and self-explanatory (if unclear on what exactly rsyscall is useful for), and I thought http://catern.com/integration.html http://catern.com/integration.html is quite detailed (perhaps excessively so).
- remram 5y ago
  I understand the problem and the use-case, but what does your library provide? Running processes remotely? Supervising remote processes? Transparently running Python functions in remote processes?
  - catern 5y ago
    >Running processes remotely? Supervising remote processes? These things. This is mentioned in http://catern.com/integration.html#thread http://catern.com/integration.html#thread
    - remram 5y ago
      So this offers capabilities like `multiprocessing` but possibly across the network, e.g. queues and shared memory? Can it spawn the remote processes too? With SSH, or its own system? You keep pointing me to that document that you feel is compelling, but all it does is explicitly run a Python function in a local thread... I know it's meant as a shortcut in the example but it's hard to infer what the real capabilities are, when what you're doing is not much more than what `trio` or stdlib can do.
      - catern 5y ago
        >Can it spawn the remote processes too? With SSH, or its own system? Yes, with rsyscall, is that not clear? It's pretty explicit in http://catern.com/integration.html#thread http://catern.com/integration.html#thread >but all it does is explicitly run a Python function in a local thread... That's not all that's happening. I think you might be pattern-matching what's happening in the article to something you've already seen, when in fact it's something novel.
        remram 5y ago
        It's clear that it "may operate on a local or remote host" and in this case you run a local thread. Just how much is actually implemented is not clear at all. A lot of frameworks are written to be "extensible" but the extensions are left as an exercise to the reader... It is nice to have a self-contained example that you can run on your machine, but I think an example that actually shows the capabilities of the framework will serve you better. E.g. show an example of serving network requests by running services on multiple nodes, rather than a toy function with a toy database. Showing that it's as easy as `run_in_executor()` is nice, but not if it does the exact same thing as `run_in_executor()`.
        catern 5y ago
        >Showing that it's as easy as `run_in_executor()` is nice, but not if it does the exact same thing as `run_in_executor()`. That's fair and a good point! I suppose reading this, you have no evidence that anything but local_thread exists. I'll change that, somehow, to show that indeed there exist values other than local_thread, and that passing those will run things on remote hosts. This article originally grew out of a tutorial on writing tests using this style, for which running over multiple hosts is not necessary and maybe undesirable... but that's not good for what the article is today.