5 ms·
Specifically distributed would be helped (multiple nodes). Right now the typical paradigm is this really clunky lock-step across nodes waiting for each other (
by brrrrrm 4y ago
Specifically distributed would be helped (multiple nodes).
Right now the typical paradigm is this really clunky lock-step across nodes waiting for each other (everything is blocking). If your hardware is non-homogenous (both interconnects and accelerators) or your program needs to be run in a pipelined way, you're fighting an extremely uphill battle.
Go, JavaScript etc are the usual languages of choice for this type of stuff because it's easy to express non-blocking code.
- nine_k 4y agoI suspect multiprocessing + shared memory could help this. The stdlib has provisions for both, but a coordination layer is needed. That would be closest to true multithreading.
- brrrrrm 4y agoit's not a matter of performance but expressibility. const y = await remote_model_a(x) // different machine const z = await remote_model_b(y) // different machine await z.backward() is trivially pipelined when run in parallel. With multithreading suddenly the backing C++ library has to be aware of this and figure things out for you