3 ms·
I understand you are writing a server. To be fair, not all compute loads are embarrassingly parallel. For those tasks single thread performance - especially fro
by fsloth 11y ago
I understand you are writing a server. To be fair, not all compute loads are embarrassingly parallel. For those tasks single thread performance - especially from the point of view of memory bandwidth - is critical.
Single core is not 'past'. 1. CPU:s are not getting that much faster every generation (one could argue that this means parallellization is critical, but...) 2. The number of cores on the average desktop is not that much. One of the reasons I think is that in many cases the individual tasks that must be performed sequentially in a typical desktop program are so tiny or hard to split that that the platform overhead and complexity of multiple threads cause the benefits from actually utilizing multiple threads to be a bit harder to reach than just using a language with a nice parallellization story. By 'benefits' I mean making the program actually faster for the user.
Of course, I'm talking about Amdahl's law https://en.m.wikipedia.org/wiki/Amdahl%27s_law https://en.m.wikipedia.org/wiki/Amdahl%27s_law
- vegabook 11y agoYep - my use case analyses the streaming incoming data using compute-intensive algorithms. I need very high performance single core, for which Erlang is unsuitable. Therefore I'm using calls to numpy. But then I need to distribute data widely in heterogenous, changeable ways, and for that, Erlang wins hands down.
- twic 11y agoAre you calling from Erlang to Numpy? If so, wow, and how is that working out?
- vegabook 11y agoNothing fancy - I message pass via zeromq to a cluster running python/numpy instances. It's pretty coarse granularity - the (BFGS curve fitting) optimizations I need from numpy will take between 0.5 and 3 seconds usually so I'm not suffering too much on serialization time in comparison. The optimizations involved do not need to be run in real time against the very latest streaming data, they just need to be as recent as possible, which makes this solution quite workable.
- tadlan 11y agoHave you tried dask?