8 ms·

This is a clever use of fuse/sshfs and chroots to allow you to run programs on remote computers. However, for many workloads, isn't the network I/O going to out

This is a clever use of fuse/sshfs and chroots to allow you to run programs on remote computers. However, for many workloads, isn't the network I/O going to outweigh the benefits of remote CPU cycles? I really think this is a clever hack, but I'm not sure how useful it is. Is it really good for the transcoding example? I could see that being good if the remote machine has a better GPU, but I'm struggling to see how this could work for other use-cases. Maybe compiling (which was one of the main use-cases behind Sun's SGE)? I do most of my work on HPC clusters, so moving compute out to remote nodes is something I'm very interested in... and I'd love to learn more about what kinds of workflows the authors had in mind.

fuzzer37 6y ago
I can totally imagine wanting to work on a thin client on your home network over a 1gbps network connection. Hell, if you optimized specifically for this and ran a 10gbps line right from your thin client to the server, it would be blazing fast. Another application I could see is putting a loud server into a closet somewhere and just running network cables to it. Make a 4 GPU render node, then just send jobs to it over your thin MacBook.
- stefan_ 6y ago
  A 4k 60hz 30bpc display requires 14Gbps to feed. The thin client thing is such a meme, all the computing requirements continuously scale - there is nothing "seamless" about them.
  - deleted 6y ago
    [deleted]
  - junipertea 6y ago
    I agree that computing requirements scale as we go, but surely you don't need 4k at 60Hz to browse the web now? I regularly play games using steam link now and while it's not the full quality and framerate, it's just convenient (and cheaper than having a full desktop pc in every room). Thin computing seems much more seamless and workable now than it did in my college years with Sun thin clients.
  - alpaca128 6y ago
    A ton of tasks don't require any graphical data streaming, in fact aside from gaming most resource intensive applications can be run in the terminal and thus easily offloaded to another machine. Text editors are already getting distributed code checkers via LSP, it wouldn't surprise me if in a couple years we could run the whole backend of IDEs on another computer and the local part can be more lightweight than most websites. Then you only need to drive a 4k screen for displaying a desktop and maybe video playback, which can already be achieved with a passively cooled Raspberry Pi.
    - herbstein 6y ago
      > it wouldn't surprise me if in a couple years we could run the whole backend of IDEs on another computer and the local part can be more lightweight than most websites. Checkout the official SSH/Docker/WSL Remote Development extensions to VSCode
      - taeric 6y ago
        Not surprisingly, emacs has been able to transparently do this for a long time, now. Granted, this all falls flat when your network connection is garbage. :(
        TeMPOraL 6y ago
        Also could use some more polish around handling SSH connections; I often manage to lock up TRAMP when I step away from the computer for 5 minutes. It's something that can probably be fixed with the right ssh_config/sshd_config, but I can't for the life of me figure out the right incantations. For now, I just habitually do M-x tramp-cleanup-all-connections whenever I stop actively working with remote files, and let Emacs reconnect automatically when I save a remote file or refresh a remote folder listing. (Incidentally, half of my use of TRAMP went away once I managed to configure a 256-color terminal on my sidearm; with that, Emacs in terminal mode looks almost indistinguishable from a GUI session.)
        taeric 6y ago
  - mrmuagi 6y ago
    I think if you were to send the raw frames sure, but wouldn't they just send the delta of the frames (only areas which changed) -- with some sort of compression algorithm and parameters like VNC does?
  - rwmj 6y ago
    SPICE solved this (years ago in fact) by doing various sorts of compression, including detection of video, and streaming that over the network. (https://www.spice-space.org/features.html https://www.spice-space.org/features.html)
- jayd16 6y ago
  The thing is you need a client fat enough for normal use but too thin to run the command in question. For the workflow you describe, why not just ssh or RDP into the beefy machine from the thin client? Wouldn't that make more sense?
  - zozbot234 6y ago
    ssh and RDP doesn't easily extend to live workload migration or other forms of location transparency.
  - fuzzer37 6y ago
    Well for starters, no scp command at the beginning and end of each job. I'm sure you could work out an alternative solution, this is just one such solution.
    - eightails 6y ago
      Yeah, this feels like a good solution for when you only occasionally need to get the remote machine to do the work, e.g. the macbook example for simple dev work. In that case it probably is more convenient. If you were relying heavily on the remote machine, maybe using a pinebook pro or something, ssh or rdp might make more sense. It's cool to have more options though, I have considered workflows like this before especially in the context of replacing my laptop with something like a pinebook.
      - fuzxi 6y ago
        Per the readme, architectures aren't cross-compatible, so you can't run commands from an x86 box on an ARM box (and presumably vice-versa). That would rule out the pinebook and most SBCs as potential use-cases.
  - GekkePrutser 6y ago
    The thin machine could be your laptop where you'd occasionally want to do a heavy operation on a workstation or server on the data you happen to work on on your laptop. Sounds like a decent usecase. Of course running it there will work too but will need more setup. So the occasional thing is what really makes this worthwhile. It's a bit of a niche but I see the benefit.
- liuliu 6y ago
  The said method doesn't support cross-platform executions (at least not without some qemu hackery). I can totally see how I would use this to transcode videos from my NAS using my main workstation CPU since both are connected through a 10gbs switch though (need to figure out how to integrate this with Plex transcoding flow).
  - eightails 6y ago
    I believe you could get a similar result using Jellyfin and rffmpeg [0]. From a quick glance it seems like it should work from ARM->x86 https://github.com/joshuaboniface/rffmpeg https://github.com/joshuaboniface/rffmpeg
npsimons 6y ago
It reminds me of GNU Parallel[0], so I have to imagine it fits for some use cases. I stumbled upon Parallel when I was trying to do user-level "distributed" computing at the day job for running hundreds of simulations in a shorter amount of time. Never really did get it working (I had a hard enough time just ginning up a Python program to orchestrate test cases across local cores; still took 3.5 hours for a full run of tests), and they're currently trying to get a SLURM cluster setup while I'm debugging NVidia binary drivers and overloading our Gitlab instance. [0] - https://www.gnu.org/software/parallel/ https://www.gnu.org/software/parallel/
baybal2 6y ago
> Maybe compiling (which was one of the main use-cases behind Sun's SGE)? The problem is that your task will still be limited by power of a single machine. You need means to run a task across more than one server, and preferably great many. It's possible, but completely impractical to run a shared memory system across network. If somebody figures out how to do that usable in practice, and without complete rewrite of the program that will be really impressive, and an ACM medal material.
- stonogo 6y ago
  ScaleMP, SGI UV, and HP Superdome all disagree with you. Arguably, plain old RDMA qualifies under this phrasing.
  - ericbarrett 6y ago
    I took the parent comment as meaning on a regular Ethernet+IP network. SGI UV and HP Superdome are semi-exotic hardware architectures. ScaleMP requires Infiniband, which is at least 10 times, if not 100, the price of Ethernet; it also requires fiber optic cable for runs > 5 meters.
    - nonameiguess 6y ago
      I should add to my above comment about the IC doing this kind of thing in Kubernetes on EC2 instances that they utilize the Amazon Elastic Fabric Adapter, which bypasses the ethernet NIC to avoid all of the queueing delay you can't tolerate in applications that need to be basically unaware that some threads are local and some are remote. And obviously they make sure your servers are in the same or physically nearby racks.
    - zozbot234 6y ago
      I thought modern variations of Ethernet (Gbps+ range) were competing with traditional uses of Infiniband nowadays.
  - Jouvence 6y ago
    UV and Superdome are custom hardware for huge NUMA boxes,so not a great comparison. ScaleMP is definitely valid though - a real shame it is stuck behind a license fee, would be interesting to experiment with.
jvanderbot 6y ago
Sometimes I like to transcode videos I recorded on my windows machine during a gaming session to post to our group chat. If I have to do those steps locally, I can't be gaming. That's not cool. Still, kinda makes more sense to just fuse/smb the videos and have the remote processing node pull, processes, push.
chris_wot 6y ago
I regularly build LibreOffice. It would be great to use a faster computer. Edit: yeah, I posted too fast. I am not their use-case.
AdamJacobMuller 6y ago
> for many workloads, isn't the network I/O going to outweigh the benefits of remote CPU cycles? Definitely, but, there are also many workloads (and situations) where it doesn't and this works great.
- ddtaylor 6y ago
  One that comes to mind is intense build jobs.
jxy 6y ago
For a typical HPC cluster, there is a shared filesystem with exactly the same file structures across the cluster over a high bandwidth network, which basically replaces "a clever use of fuse/sshfs and chroots" with something far better. OP's project is clearly useless in HPC setting. On the other hand, OP's project is exactly an linux/posix version of the same as what Plan 9 was designed to do and has been doing.
- mbreese 6y ago
  > project is clearly useless in HPC setting Which is why I'm curious about the applications they had in mind. I know that my workflow isn't typical for most people, so hopefully others have better uses in mind. For me, I just submit adhoc jobs to SLURM and call it a day. Not everyone has access to an HPC cluster or is comfortable setting such a system up for a small, home cluster. Anything (like this project) that makes HPC-like processing more accessible, I'm interested in.
  - Overv 6y ago
    When I developed this I was very much thinking of using this to run one-off compute heavy tasks, e.g. compressing a video, running scientific computations or rendering something in Blender. You would just rent a VM for a few hours with the hardware you need, set it up in a few minutes, run your job there and delete it again.
  - zekrioca 6y ago
    OP’s project is actually a good idea at development stage, where data is not the main thing yet..
- e12e 6y ago
  I miss openMosix[1]... Its been so long, but I remember running kernel compiles on two workstations a long time ago... Double the core count! It actually gave a significant speedup on 100mbps switched lan (I think it was just a crossed cable, no actual switch). I couldn't remember setting up NFS or anything like that - apparently openMosix did some tricks too: http://www.firewall.cx/linux-knowledgebase-tutorials/openmosix-linux-supercomputer/307-linux-openmosix-file-system.html http://www.firewall.cx/linux-knowledgebase-tutorials/openmos... [1] https://en.m.wikipedia.org/wiki/OpenMosix https://en.m.wikipedia.org/wiki/OpenMosix
ipsum2 6y ago
Compilation is a good example of where using remote machines can help substantially: https://docs.bazel.build/versions/master/remote-execution.html https://docs.bazel.build/versions/master/remote-execution.ht...
- account42 6y ago
  See also: https://github.com/distcc/distcc https://github.com/distcc/distcc
kelnos 6y ago
> for many workloads, isn't the network I/O going to outweigh the benefits of remote CPU cycles? Yes, the README addresses this: > File system performance remains a bottleneck, so the most suitable workloads are computationally constrained tasks like ray tracing and video encoding. Using outrun for something like git status works, but is not recommended.
xtracto 6y ago
Somehow I think this may make it possible to "horizontally Scale" the desktop experience. What if I have my main computer/cpu at home and have additional headless computers configured and I set my Linux system so that everything besides the Kernel/boot stuff is spread through the other computers?
ur-whale 6y ago
> isn't the network I/O going to outweigh the benefits of remote CPU cycles? Not if: a) your network B/W is higher than your local disks (not as unusual as you might think, spinning medium is awful slow) b) the app you're using chunks up data in a reasonably efficient manner so as to not get bitten by latency issues (ffmpeg encoding likely meets that bar).

fuzzer37 6y ago

I can totally imagine wanting to work on a thin client on your home network over a 1gbps network connection. Hell, if you optimized specifically for this and ran a 10gbps line right from your thin client to the server, it would be blazing fast. Another application I could see is putting a loud server into a closet somewhere and just running network cables to it. Make a 4 GPU render node, then just send jobs to it over your thin MacBook.

stefan_ 6y ago
A 4k 60hz 30bpc display requires 14Gbps to feed. The thin client thing is such a meme, all the computing requirements continuously scale - there is nothing "seamless" about them.
- deleted 6y ago
  [deleted]
- junipertea 6y ago
  I agree that computing requirements scale as we go, but surely you don't need 4k at 60Hz to browse the web now? I regularly play games using steam link now and while it's not the full quality and framerate, it's just convenient (and cheaper than having a full desktop pc in every room). Thin computing seems much more seamless and workable now than it did in my college years with Sun thin clients.
- alpaca128 6y ago
  A ton of tasks don't require any graphical data streaming, in fact aside from gaming most resource intensive applications can be run in the terminal and thus easily offloaded to another machine. Text editors are already getting distributed code checkers via LSP, it wouldn't surprise me if in a couple years we could run the whole backend of IDEs on another computer and the local part can be more lightweight than most websites. Then you only need to drive a 4k screen for displaying a desktop and maybe video playback, which can already be achieved with a passively cooled Raspberry Pi.
  - herbstein 6y ago
    > it wouldn't surprise me if in a couple years we could run the whole backend of IDEs on another computer and the local part can be more lightweight than most websites. Checkout the official SSH/Docker/WSL Remote Development extensions to VSCode
    - taeric 6y ago
      Not surprisingly, emacs has been able to transparently do this for a long time, now. Granted, this all falls flat when your network connection is garbage. :(
      - TeMPOraL 6y ago
        Also could use some more polish around handling SSH connections; I often manage to lock up TRAMP when I step away from the computer for 5 minutes. It's something that can probably be fixed with the right ssh_config/sshd_config, but I can't for the life of me figure out the right incantations. For now, I just habitually do M-x tramp-cleanup-all-connections whenever I stop actively working with remote files, and let Emacs reconnect automatically when I save a remote file or refresh a remote folder listing. (Incidentally, half of my use of TRAMP went away once I managed to configure a 256-color terminal on my sidearm; with that, Emacs in terminal mode looks almost indistinguishable from a GUI session.)
        taeric 6y ago
- mrmuagi 6y ago
  I think if you were to send the raw frames sure, but wouldn't they just send the delta of the frames (only areas which changed) -- with some sort of compression algorithm and parameters like VNC does?
- rwmj 6y ago
  SPICE solved this (years ago in fact) by doing various sorts of compression, including detection of video, and streaming that over the network. (https://www.spice-space.org/features.html https://www.spice-space.org/features.html)
jayd16 6y ago
The thing is you need a client fat enough for normal use but too thin to run the command in question. For the workflow you describe, why not just ssh or RDP into the beefy machine from the thin client? Wouldn't that make more sense?
- zozbot234 6y ago
  ssh and RDP doesn't easily extend to live workload migration or other forms of location transparency.
- fuzzer37 6y ago
  Well for starters, no scp command at the beginning and end of each job. I'm sure you could work out an alternative solution, this is just one such solution.
  - eightails 6y ago
    Yeah, this feels like a good solution for when you only occasionally need to get the remote machine to do the work, e.g. the macbook example for simple dev work. In that case it probably is more convenient. If you were relying heavily on the remote machine, maybe using a pinebook pro or something, ssh or rdp might make more sense. It's cool to have more options though, I have considered workflows like this before especially in the context of replacing my laptop with something like a pinebook.
    - fuzxi 6y ago
      Per the readme, architectures aren't cross-compatible, so you can't run commands from an x86 box on an ARM box (and presumably vice-versa). That would rule out the pinebook and most SBCs as potential use-cases.
- GekkePrutser 6y ago
  The thin machine could be your laptop where you'd occasionally want to do a heavy operation on a workstation or server on the data you happen to work on on your laptop. Sounds like a decent usecase. Of course running it there will work too but will need more setup. So the occasional thing is what really makes this worthwhile. It's a bit of a niche but I see the benefit.
liuliu 6y ago
The said method doesn't support cross-platform executions (at least not without some qemu hackery). I can totally see how I would use this to transcode videos from my NAS using my main workstation CPU since both are connected through a 10gbs switch though (need to figure out how to integrate this with Plex transcoding flow).
- eightails 6y ago
  I believe you could get a similar result using Jellyfin and rffmpeg [0]. From a quick glance it seems like it should work from ARM->x86 https://github.com/joshuaboniface/rffmpeg https://github.com/joshuaboniface/rffmpeg

npsimons 6y ago

It reminds me of GNU Parallel[0], so I have to imagine it fits for some use cases. I stumbled upon Parallel when I was trying to do user-level "distributed" computing at the day job for running hundreds of simulations in a shorter amount of time. Never really did get it working (I had a hard enough time just ginning up a Python program to orchestrate test cases across local cores; still took 3.5 hours for a full run of tests), and they're currently trying to get a SLURM cluster setup while I'm debugging NVidia binary drivers and overloading our Gitlab instance. [0] - https://www.gnu.org/software/parallel/ https://www.gnu.org/software/parallel/

baybal2 6y ago

> Maybe compiling (which was one of the main use-cases behind Sun's SGE)? The problem is that your task will still be limited by power of a single machine. You need means to run a task across more than one server, and preferably great many. It's possible, but completely impractical to run a shared memory system across network. If somebody figures out how to do that usable in practice, and without complete rewrite of the program that will be really impressive, and an ACM medal material.

stonogo 6y ago
ScaleMP, SGI UV, and HP Superdome all disagree with you. Arguably, plain old RDMA qualifies under this phrasing.
- ericbarrett 6y ago
  I took the parent comment as meaning on a regular Ethernet+IP network. SGI UV and HP Superdome are semi-exotic hardware architectures. ScaleMP requires Infiniband, which is at least 10 times, if not 100, the price of Ethernet; it also requires fiber optic cable for runs > 5 meters.
  - nonameiguess 6y ago
    I should add to my above comment about the IC doing this kind of thing in Kubernetes on EC2 instances that they utilize the Amazon Elastic Fabric Adapter, which bypasses the ethernet NIC to avoid all of the queueing delay you can't tolerate in applications that need to be basically unaware that some threads are local and some are remote. And obviously they make sure your servers are in the same or physically nearby racks.
  - zozbot234 6y ago
    I thought modern variations of Ethernet (Gbps+ range) were competing with traditional uses of Infiniband nowadays.
- Jouvence 6y ago
  UV and Superdome are custom hardware for huge NUMA boxes,so not a great comparison. ScaleMP is definitely valid though - a real shame it is stuck behind a license fee, would be interesting to experiment with.

jvanderbot 6y ago

Sometimes I like to transcode videos I recorded on my windows machine during a gaming session to post to our group chat. If I have to do those steps locally, I can't be gaming. That's not cool. Still, kinda makes more sense to just fuse/smb the videos and have the remote processing node pull, processes, push.

chris_wot 6y ago

I regularly build LibreOffice. It would be great to use a faster computer. Edit: yeah, I posted too fast. I am not their use-case.

AdamJacobMuller 6y ago

> for many workloads, isn't the network I/O going to outweigh the benefits of remote CPU cycles? Definitely, but, there are also many workloads (and situations) where it doesn't and this works great.

ddtaylor 6y ago
One that comes to mind is intense build jobs.

jxy 6y ago

For a typical HPC cluster, there is a shared filesystem with exactly the same file structures across the cluster over a high bandwidth network, which basically replaces "a clever use of fuse/sshfs and chroots" with something far better. OP's project is clearly useless in HPC setting. On the other hand, OP's project is exactly an linux/posix version of the same as what Plan 9 was designed to do and has been doing.

mbreese 6y ago
> project is clearly useless in HPC setting Which is why I'm curious about the applications they had in mind. I know that my workflow isn't typical for most people, so hopefully others have better uses in mind. For me, I just submit adhoc jobs to SLURM and call it a day. Not everyone has access to an HPC cluster or is comfortable setting such a system up for a small, home cluster. Anything (like this project) that makes HPC-like processing more accessible, I'm interested in.
- Overv 6y ago
  When I developed this I was very much thinking of using this to run one-off compute heavy tasks, e.g. compressing a video, running scientific computations or rendering something in Blender. You would just rent a VM for a few hours with the hardware you need, set it up in a few minutes, run your job there and delete it again.
- zekrioca 6y ago
  OP’s project is actually a good idea at development stage, where data is not the main thing yet..
e12e 6y ago
I miss openMosix[1]... Its been so long, but I remember running kernel compiles on two workstations a long time ago... Double the core count! It actually gave a significant speedup on 100mbps switched lan (I think it was just a crossed cable, no actual switch). I couldn't remember setting up NFS or anything like that - apparently openMosix did some tricks too: http://www.firewall.cx/linux-knowledgebase-tutorials/openmosix-linux-supercomputer/307-linux-openmosix-file-system.html http://www.firewall.cx/linux-knowledgebase-tutorials/openmos... [1] https://en.m.wikipedia.org/wiki/OpenMosix https://en.m.wikipedia.org/wiki/OpenMosix

ipsum2 6y ago

Compilation is a good example of where using remote machines can help substantially: https://docs.bazel.build/versions/master/remote-execution.html https://docs.bazel.build/versions/master/remote-execution.ht...

account42 6y ago
See also: https://github.com/distcc/distcc https://github.com/distcc/distcc

kelnos 6y ago

> for many workloads, isn't the network I/O going to outweigh the benefits of remote CPU cycles? Yes, the README addresses this: > File system performance remains a bottleneck, so the most suitable workloads are computationally constrained tasks like ray tracing and video encoding. Using outrun for something like git status works, but is not recommended.

xtracto 6y ago

Somehow I think this may make it possible to "horizontally Scale" the desktop experience. What if I have my main computer/cpu at home and have additional headless computers configured and I set my Linux system so that everything besides the Kernel/boot stuff is spread through the other computers?

ur-whale 6y ago

> isn't the network I/O going to outweigh the benefits of remote CPU cycles? Not if: a) your network B/W is higher than your local disks (not as unusual as you might think, spinning medium is awful slow) b) the app you're using chunks up data in a reasonably efficient manner so as to not get bitten by latency issues (ffmpeg encoding likely meets that bar).