4 ms·

> emulate a "real" super-computer and really use MPI and suc Wouldn’t containers be a easier way to do that?

dragontamer 8y ago
No. Containers on a single node are too fast. Super-computers have high-latency communications through thick pipes. True, Super Computers have 40Gbit or 100Gbit connections between nodes, but it can take multiple microseconds to send messages around. A bunch of containers all sitting on the same box would be able to handle communications within dozens of nanoseconds. So its a bad "emulator" for SuperComputers. Coordinating all your nodes to compute a problem, while achieving high utilization, is tricky. Its not like programming a normal computer where threads share RAM and can communicate in nanoseconds.
- fluxty 8y ago
  Had to read your answer twice... "High latency???" But I see you consider 1-2us high latency. :P It is an interesting question what people who don't have access to a supercomputer, but would like to learn and optimize for HPC-style distributed memory programming should use. I've found AWS to be pretty nice, except there are no RDMA drivers for the elastic NIC and the BW is a bit low. (25Gbit vs. 100Gbit). For MPI bulk synchronous programs, it's probably a pretty close model, though.
  - laumars 8y ago
    > Had to read your answer twice... "High latency???" But I see you consider 1-2us high latency. :P Relative to accessing the hardware resources on a host, it very much is. Just as accessing the RAM is slow relative to accessing the L2 cache on a CPU
- skissane 8y ago
  > No. Containers on a single node are too fast. You can add artificial delay to your local container network to better simulate a production environment. For example using https://github.com/alexei-led/pumba https://github.com/alexei-led/pumba and "pumba netem delay" you can add networking delay between Docker containers, and "pumba netem rate" can limit the bandwidth between them as well. ("pumba" is just using the underlying Linux networking traffic control technologies, such as the "tc" command from "iproute2", so you don't have to use "pumba", you can set this up manually, but a tool like "pumba" makes it a lot easier.)
  - geezerjay 8y ago
    Thank you for mentioning pumba. Sounds very interesting.
  - dragontamer 8y ago
    I've used netem to emulate millisecond delays before. But I'm not sure if it has the granularity to emulate micro-second level delays. Basically, netem is designed to provide milliseconds of delay, emulating a worldwide network. Supercomputers are thousands of times faster than that. I'd have to play with netem before I was certain that it could handle a sub ~10uS delay that supercomputers have node-to-node. Considering that Linux task-switching is on the order of ~10mS or so, I have severe doubts that uS level delays will work with netem. The NanoPi-Fire3 uses normal Gigabit Ethernet, which probably has latencies in the ~50uS region. Which is slower than a real supercomputer, but "proportionally" should be representative of supercomputers (since the tiny embedded ARM chips are around 50x slower than a real supercomputer node anyway). A bunch of Rasp Pis on Gigabit Ethernet seems like a better overall "cheap supercomputer" architecture, for students of supercomputing. Better than containers or software-emulated network delays
- marmaduke 8y ago
  > handle communications within dozens of nanoseconds. So its a bad "emulator" for SuperComputers But ping on local host reports .3 ms latency..? In any case it’s still an easy way to get started, and arguably when exact latency starts counting for something, you have to be tuning your code on the system it’s going to run on. An RPi cluster could skew that in all sorts of ways, eg the TCP stack being disproportionately slower etc

No. Containers on a single node are too fast. Super-computers have high-latency communications through thick pipes. True, Super Computers have 40Gbit or 100Gbit connections between nodes, but it can take multiple microseconds to send messages around. A bunch of containers all sitting on the same box would be able to handle communications within dozens of nanoseconds. So its a bad "emulator" for SuperComputers. Coordinating all your nodes to compute a problem, while achieving high utilization, is tricky. Its not like programming a normal computer where threads share RAM and can communicate in nanoseconds.

fluxty 8y ago
Had to read your answer twice... "High latency???" But I see you consider 1-2us high latency. :P It is an interesting question what people who don't have access to a supercomputer, but would like to learn and optimize for HPC-style distributed memory programming should use. I've found AWS to be pretty nice, except there are no RDMA drivers for the elastic NIC and the BW is a bit low. (25Gbit vs. 100Gbit). For MPI bulk synchronous programs, it's probably a pretty close model, though.
- laumars 8y ago
  > Had to read your answer twice... "High latency???" But I see you consider 1-2us high latency. :P Relative to accessing the hardware resources on a host, it very much is. Just as accessing the RAM is slow relative to accessing the L2 cache on a CPU
skissane 8y ago
> No. Containers on a single node are too fast. You can add artificial delay to your local container network to better simulate a production environment. For example using https://github.com/alexei-led/pumba https://github.com/alexei-led/pumba and "pumba netem delay" you can add networking delay between Docker containers, and "pumba netem rate" can limit the bandwidth between them as well. ("pumba" is just using the underlying Linux networking traffic control technologies, such as the "tc" command from "iproute2", so you don't have to use "pumba", you can set this up manually, but a tool like "pumba" makes it a lot easier.)
- geezerjay 8y ago
  Thank you for mentioning pumba. Sounds very interesting.
- dragontamer 8y ago
  I've used netem to emulate millisecond delays before. But I'm not sure if it has the granularity to emulate micro-second level delays. Basically, netem is designed to provide milliseconds of delay, emulating a worldwide network. Supercomputers are thousands of times faster than that. I'd have to play with netem before I was certain that it could handle a sub ~10uS delay that supercomputers have node-to-node. Considering that Linux task-switching is on the order of ~10mS or so, I have severe doubts that uS level delays will work with netem. The NanoPi-Fire3 uses normal Gigabit Ethernet, which probably has latencies in the ~50uS region. Which is slower than a real supercomputer, but "proportionally" should be representative of supercomputers (since the tiny embedded ARM chips are around 50x slower than a real supercomputer node anyway). A bunch of Rasp Pis on Gigabit Ethernet seems like a better overall "cheap supercomputer" architecture, for students of supercomputing. Better than containers or software-emulated network delays
marmaduke 8y ago
> handle communications within dozens of nanoseconds. So its a bad "emulator" for SuperComputers But ping on local host reports .3 ms latency..? In any case it’s still an easy way to get started, and arguably when exact latency starts counting for something, you have to be tuning your code on the system it’s going to run on. An RPi cluster could skew that in all sorts of ways, eg the TCP stack being disproportionately slower etc