4 ms·

Does anyone have a link for building a Raspberry Pi cluster over gigabit ethernet? A quick search turned up some stuff: https://medium.com/@glmdev/building-a-r

Does anyone have a link for building a Raspberry Pi cluster over gigabit ethernet? A quick search turned up some stuff: https://medium.com/@glmdev/building-a-raspberry-pi-cluster-784f0df9afbd https://medium.com/@glmdev/building-a-raspberry-pi-cluster-7... https://epcced.github.io/wee_archlet/ https://epcced.github.io/wee_archlet/ https://www.raspberrypi.org/magpi/cluster-computer-raspberry-pi-3/ https://www.raspberrypi.org/magpi/cluster-computer-raspberry... Specifically, I'd like to find software that makes the cluster appear as a single memory address space and N identical CPUs/cores. When I'm in that OS/VM/kernel, whatever you want to call it, I want to be able to experiment with running stuff like Elixir or Go and have the runtime handle the virtual memory and cache coherency stuff between the nodes. I don't want to deal with any manual memory management at all. I just want rules of thumb regarding rough latency between N nodes and M routers for whatever network topology it uses. Icing on the cake would be if I could run something like IPFS (or another hash tree) and have data distribution handled under the hood and appear as a single directory structure. The goal being to play around with stuff like neural nets and ray tracing without having to use any proprietary frameworks. It should just appear as say a 256 core computer with however many GB or ram and however many TB of hard drive space I give it.

zackmorris 7y ago
Just searched a bit more for MPI integration with various languages. I find things like this kind of heartbreaking: https://godoc.org/github.com/cpmech/gosl/mpi https://godoc.org/github.com/cpmech/gosl/mpi It provides MPI bindings in Go, but they really should have provided an MPI layer internally so that the Go metaphors of things like channels and goroutines "just work" with no special syntax. I think this is where I'm getting stuck. So much cluster software provides interfaces to send/receive data and get the current thread's CPU id and total number of CPUs. But I'm not finding much info on doing this directly in the kernel or language runtime so that the client can be written in a topology-agnostic fashion. Edit: there doesn't seem to be a Go-native MPI runtime (yet) https://groups.google.com/forum/#!topic/golang-nuts/t7Vjpfu0sjQ https://groups.google.com/forum/#!topic/golang-nuts/t7Vjpfu0...
- AceJohnny2 7y ago
  I believe Erlang's messaging/multithreaded primitives work over the network?
cesarb 7y ago
> Specifically, I'd like to find software that makes the cluster appear as a single memory address space and N identical CPUs/cores. This is called "single system image". A couple of coworkers played with openMosix on our desktops several years ago, it was fun seeing bash processes moving on their own from one desktop to the other. But I haven't heard anything about that in a while, it seems single system image clusters have fallen out of fashion.

zackmorris 7y ago

Just searched a bit more for MPI integration with various languages. I find things like this kind of heartbreaking: https://godoc.org/github.com/cpmech/gosl/mpi https://godoc.org/github.com/cpmech/gosl/mpi It provides MPI bindings in Go, but they really should have provided an MPI layer internally so that the Go metaphors of things like channels and goroutines "just work" with no special syntax. I think this is where I'm getting stuck. So much cluster software provides interfaces to send/receive data and get the current thread's CPU id and total number of CPUs. But I'm not finding much info on doing this directly in the kernel or language runtime so that the client can be written in a topology-agnostic fashion. Edit: there doesn't seem to be a Go-native MPI runtime (yet) https://groups.google.com/forum/#!topic/golang-nuts/t7Vjpfu0sjQ https://groups.google.com/forum/#!topic/golang-nuts/t7Vjpfu0...

AceJohnny2 7y ago
I believe Erlang's messaging/multithreaded primitives work over the network?

cesarb 7y ago

> Specifically, I'd like to find software that makes the cluster appear as a single memory address space and N identical CPUs/cores. This is called "single system image". A couple of coworkers played with openMosix on our desktops several years ago, it was fun seeing bash processes moving on their own from one desktop to the other. But I haven't heard anything about that in a while, it seems single system image clusters have fallen out of fashion.