3 ms·

For a typical HPC cluster, there is a shared filesystem with exactly the same file structures across the cluster over a high bandwidth network, which basically

by jxy 6y ago

For a typical HPC cluster, there is a shared filesystem with exactly the same file structures across the cluster over a high bandwidth network, which basically replaces "a clever use of fuse/sshfs and chroots" with something far better. OP's project is clearly useless in HPC setting. On the other hand, OP's project is exactly an linux/posix version of the same as what Plan 9 was designed to do and has been doing.

mbreese 6y ago
> project is clearly useless in HPC setting Which is why I'm curious about the applications they had in mind. I know that my workflow isn't typical for most people, so hopefully others have better uses in mind. For me, I just submit adhoc jobs to SLURM and call it a day. Not everyone has access to an HPC cluster or is comfortable setting such a system up for a small, home cluster. Anything (like this project) that makes HPC-like processing more accessible, I'm interested in.
- Overv 6y ago
  When I developed this I was very much thinking of using this to run one-off compute heavy tasks, e.g. compressing a video, running scientific computations or rendering something in Blender. You would just rent a VM for a few hours with the hardware you need, set it up in a few minutes, run your job there and delete it again.
- zekrioca 6y ago
  OP’s project is actually a good idea at development stage, where data is not the main thing yet..
e12e 6y ago
I miss openMosix[1]... Its been so long, but I remember running kernel compiles on two workstations a long time ago... Double the core count! It actually gave a significant speedup on 100mbps switched lan (I think it was just a crossed cable, no actual switch). I couldn't remember setting up NFS or anything like that - apparently openMosix did some tricks too: http://www.firewall.cx/linux-knowledgebase-tutorials/openmosix-linux-supercomputer/307-linux-openmosix-file-system.html http://www.firewall.cx/linux-knowledgebase-tutorials/openmos... [1] https://en.m.wikipedia.org/wiki/OpenMosix https://en.m.wikipedia.org/wiki/OpenMosix

by jxy 6y ago

mbreese 6y ago

> project is clearly useless in HPC setting Which is why I'm curious about the applications they had in mind. I know that my workflow isn't typical for most people, so hopefully others have better uses in mind. For me, I just submit adhoc jobs to SLURM and call it a day. Not everyone has access to an HPC cluster or is comfortable setting such a system up for a small, home cluster. Anything (like this project) that makes HPC-like processing more accessible, I'm interested in.

Overv 6y ago
When I developed this I was very much thinking of using this to run one-off compute heavy tasks, e.g. compressing a video, running scientific computations or rendering something in Blender. You would just rent a VM for a few hours with the hardware you need, set it up in a few minutes, run your job there and delete it again.
zekrioca 6y ago
OP’s project is actually a good idea at development stage, where data is not the main thing yet..

e12e 6y ago

I miss openMosix[1]... Its been so long, but I remember running kernel compiles on two workstations a long time ago... Double the core count! It actually gave a significant speedup on 100mbps switched lan (I think it was just a crossed cable, no actual switch). I couldn't remember setting up NFS or anything like that - apparently openMosix did some tricks too: http://www.firewall.cx/linux-knowledgebase-tutorials/openmosix-linux-supercomputer/307-linux-openmosix-file-system.html http://www.firewall.cx/linux-knowledgebase-tutorials/openmos... [1] https://en.m.wikipedia.org/wiki/OpenMosix https://en.m.wikipedia.org/wiki/OpenMosix