6 ms·
I'm trying to understand, what kind of scripting requires first-class concurrency that isn't fulfilled by say Python?
by weff_ 7y ago
I'm trying to understand, what kind of scripting requires first-class concurrency that isn't fulfilled by say Python?
- Grinnz 7y agoI am not that familiar with Python, but the GIL has long prevented any real in-process concurrency. Perl has concurrency but it's complex, heavy, and poorly supported. Raku's approach to this is built to avoid all these problems (like Elixir).
- weff_ 7y agoI agree the GIL is a problem but it's only an issue for CPU-bound problems. Is there really an important amount of CPU-bound work that is written in a scripting language? If it's CPU-bound, wouldn't you want to use something lower level?
- karmakaze 7y agoIt's not just CPU bound problems, handling multiple overlapping i/o operations is more trouble than it ought to be.
- weff_ 7y agoCan you expand a bit on that? I'm not familiar with the issue you're describing.
- Grinnz 7y agoIndeed and as a Perl developer I make use of XS/external libraries, cooperative multitasking (event loops/promises), and forking to cover these use cases. It doesn't preclude wanting the additional option to take advantage of threads in a useful way, since they do exist.
- nuclear_eclipse 7y agoIf it's entirely CPU bound, you can use multiprocessing to negate most of the GIL issues, and transparently send inputs/outputs between the parent and child processes. If it's I/O bound, then AsyncIO is a great way to express asynchronous workflows. If it's a combination of both I/O and CPU bound workloads, there are ways to mix multiprocessing and AsyncIO to better saturate your cores without losing the simplicity or readability of Python: https://github.com/jreese/aiomultiprocess https://github.com/jreese/aiomultiprocess
- weff_ 7y agoVery true, I had not even thought about the multiprocessing package; it's sometimes not as convenient as multithreading but it'll get those other cores working.
- deleted 7y ago[deleted]
- ajoseps 7y agoI'm guessing because Python's concurrency relies on the Global Interpreter Lock. Although I think concurrent.futures might address that. Haven't worked with python concurrency libraries in a bit.
- weff_ 7y agoas I posted above: I agree the GIL is a problem but it's only an issue for CPU-bound problems. Is there really an important amount of CPU-bound work that is written in a scripting language? If it's CPU-bound, wouldn't you want to use something lower level?
- zaphirplane 7y agoThere is machine learning, which usually calls into numpy or other c extensions With a lot of the data preparation done in python
- rubyn00bie 7y agotldr; Using a scripting language that allows for native threads or has a strong concurrency model builtin to the core would be beneficial for any CPU bound scripting task... ----- Python's concurrency model is good for waiting on network or disk I/O because of its GIL (Global Interpreter Lock): https://realpython.com/python-gil/#the-impact-on-multi-threaded-python-programs https://realpython.com/python-gil/#the-impact-on-multi-threa... If your program is CPU bound the GIL will slow you down. I'm sure since the python ecosystem is quite vast there are ways around the GIL... but then you have to worry about every library you use not supporting "real" multi-threading, or (b)locking for no reason and shitting all over your efforts.
- bjornjaja 7y agos/tldr;/summary:/g
- weff_ 7y agoAs I've posted above, I'm a bit confused by CPU-bound work being processed in a scripting language. If you're planning on doing intense CPU-bound work, maybe use a lower-level language? I'm not saying abandon Python: you can extend Python with C or just use IPC to transfer data between a Python front-end and a computation back-end.
- rubyn00bie 7y agoI totally you feel you, I guess I thought your question was substantially more surface level than it was. My apologies. I'm personally with you. I also don't tend to think object boxing is really the performance bottleneck for most applications, and if/when it is, likely the other requirements should've already ruled out using one (a scripting language). It's like writing Nifs for Elixir, yeah sure you _can_, they have their purpose, but you could also just write another application to do that one thing and like you said, use IPC. So in summary, we agree with each other, here's to: the right tool for the job!
- weff_ 7y ago> the right tool for the job! Hear, hear!
- vaer-k 7y agoWhy should first-class concurrency needs be required to script in Elixir? This question seems to imply that Python is somehow a default language and special requirements must be needed to justify writing in something else. Elixir is general-use and pleasant to write scripts in so seems reasonable to me for someone to do so if that's their thing.
- Grinnz 7y agoWhich is a similar response to the comments asking "why Perl over Python?" I ask, why not (both)?
- b2gills 7y agoEspecially since you can use the Inline::Python module.
- weff_ 7y agoOvid2 said Perl6 has a good concurrency model lliamander replied that Elixir does too and it's worth a look To that, 7thaccount replied that Perl6 and Elixir fill different niches. So far, it seems Perl6 fills a niche that requires scripting and first-class concurrency. My question then is: what is this niche that requires very solid concurrency but also scripting. In other words, what does Perl6 have in terms of concurrency that Python does not (given they are both scripting languages)?
- dnautics 7y agoWell just from experience untangling async calls in python is a nightmare and sometimes hard to reason about. The red/blue function problem is real. Meanwhile dispatching concurrent long-running scripting tasks is basically trivial in elixir (Enum.map over Task.async, then Enum.map over Task.await)
- dragonwriter 7y agoFor concurrency I think Raku has slightly more developed async support than Python, but the bigger advantage, I think, is in parallelism where, aside from the CPython GIL limiting practical parallelism in the main implementation (which is a big deal), Python as a language lacks the parallel iterables produced by the hyper (parallel but constrained to return in order) and race (parallel and not constrained in order) methods on the Iterable role in Raku, or anything like Raku’s hyperoperators that are semantically concurrent and parallelizable at the discretion of the optimizer. (Come to think of it, while also parallel, all those are also high-level concurrency constructs that Python lacks equivalents to.) Python as a language can support parallelism via threads, and CPython as an implementation can via multiprocessing, but those are both very low level abstractions; Raku has higher level abstractions which allow much more succinct expression of parallel operations.
- Too 7y agoSounds like multiprocessing.Pool.imap and imap_unordered? When dealing with io you also have async/await equivalent iterators like asyncio.as_completed().
- b2gills 7y agoRaku has `await` but it doesn't need `async`. sub foo ( Int \n where 1..100 ) { # ___ start is a prefix operator that creates a Promise # V start { sleep n.rand; (1..n).pick } } my $result = foo(10); say $result.status; # Planned say await $result; # 6 (after a delay) # this is nearly identical to foo above sub bar ( Int \n where 1..100 ) { Promise.in( n.rand ).then({ (1..n).pick }) } (Note that `start` does not need a block unless you are grouping more than one statement as I have above.) There are combinators like `Promise.allof` and `Promise.anyof`. You usually don't need to use `Promise.allof`, because you can use `await` on a list. my @promises = Promise.in(1.rand) xx 10; my @results = await @promises; (Note that the left side of `xx` is _thunky_ so there are ten different Promises with ten different random wait times.) --- You should see the `react`/`supply` and `whenever` feature.