4 ms·
All of distributed machine learning would benefit from async, but it's unfortunately a Python-only ecosystem for the time being. Probably gonna be a couple yea
by brrrrrm 4y ago
All of distributed machine learning would benefit from async, but it's unfortunately a Python-only ecosystem for the time being. Probably gonna be a couple years before that has a chance of changing
- nine_k 4y agoHow would ML benefit from async? I thought everything interesting in things like TF or PyTorch is done by native code which is free to use multithreading and anything at all. (I know little about ML.)
- brrrrrm 4y agoSpecifically distributed would be helped (multiple nodes). Right now the typical paradigm is this really clunky lock-step across nodes waiting for each other (everything is blocking). If your hardware is non-homogenous (both interconnects and accelerators) or your program needs to be run in a pipelined way, you're fighting an extremely uphill battle. Go, JavaScript etc are the usual languages of choice for this type of stuff because it's easy to express non-blocking code.
- nine_k 4y agoI suspect multiprocessing + shared memory could help this. The stdlib has provisions for both, but a coordination layer is needed. That would be closest to true multithreading.
- brrrrrm 4y agoit's not a matter of performance but expressibility. const y = await remote_model_a(x) // different machine const z = await remote_model_b(y) // different machine await z.backward() is trivially pipelined when run in parallel. With multithreading suddenly the backing C++ library has to be aware of this and figure things out for you