4 ms·
Those are real issues though.
by substation13 3y ago
Those are real issues though.
- antupis 3y agoyes but generally they are not that kind make or break type issues like eg Julia correctness problems.
- lopatin 3y agoIs concurrency useful for ML?
- not-my-account 3y agoYou end up having to do a lot of things in a ML training run, some of which you can do in parallel because it’s not important now (eg saving metadata) or because you’d otherwise be resource limited (eg loading data and formatting batches for training)
- davidktr 3y agoAnd for this you cannot use Python's multiprocessing because ... ? Sure, moving data between processes is slow because of pickling [0]. However, I'm using parallel processing for the things you suggested, and for these it works great. If I really had the use case and needed threads, I'd much rather use C++ bindings in a Python package than rebuilding the whole thing. Guess it depends on the scale we are talking about. [0] https://pythonspeed.com/articles/faster-multiprocessing-pickle/ https://pythonspeed.com/articles/faster-multiprocessing-pick...
- hosh 3y agoIt’s handling all the things that can go wrong when communicating and coordinating across processes, across machines, or troubleshooting bottlenecks on running systems that Elixir (and Erlang) excels.
- formulathree 3y agoNo, parallelism is useful, concurrency without parallelism is not useful. Go and elixir provide some parallelism but the primary focus for both languages is concurrency.
- NaiveBayesian 3y agoIf your data loading pipeline grows even slightly complex, then yes, you absolutely need concurrency in order to deliver your samples to the GPU fast enough. The current workarounds to make this happen in python are quite ugly imho, e.g. Pytorch spawns multiple python processes and then pushes data between the processes through shared memory, which incurs quite some overhead. Tensorflow on the other hand requires you to stick to their Tensor-dsl so that it can run within their graph engine. If native concurrency were a thing, data loading would be much more straightforward to implement without such hacks.
- substation13 3y agoYes, it can be. 1. Loading data 2. Running algorithms that benefit from shared memory 3. Serving the model (if it's not being output to some portable format) There are also general benefits of using one language across a project. Because Python is weak on these things, we end up using multiple languages.
- throwawaymaths 3y agoIt's not. Until you need to deploy it.
- itronitron 3y agoConcurrency generally makes things run faster. If you test your ML methods your tests will complete faster if the ML methods are able to use and take advantage of concurrency. Some people consider that useful.