3 ms·
Hi thanks for raising this. I will add more details around benchmarking and the points you have raised to the documentation. Some answers for now: - One may use
by jtatton 6y ago
Hi thanks for raising this. I will add more details around benchmarking and the points you have raised to the documentation. Some answers for now:
- One may use the `shared` keyword in order to override the default copying semantic of Concurnas for mutable data. In this way one may spawn 10 isolates operating on an array without copying it 10 times. Of course this may introduce non deterministic behavior if they overlap in terms of how data is proceed in that array, so care must be applied. Also, a class may be marked as being shared, again suppressing copying, this is handy if if one is importing already thread safe Java code.
- In terms of off heap support all classes are augmented with runtime generated serializers and deserializers (https://concurnas.com/docs/offHeap.html#serialization-of-objects https://concurnas.com/docs/offHeap.html#serialization-of-obj...). Off heap memory support is an area where I'd like to improve Concurnas some more. I think that some form of value type support would be beneficial here. The goal being for it to be possible for one to implement an efficient RDBMS or NoSQL system in Concurnas itself.
- For remote computing, yes there is caching of code to avoid retransmission (the documentation could be better regarding this, I will update the following in due course: https://concurnas.com/docs/distComp.html#request-dependencies https://concurnas.com/docs/distComp.html#request-dependencie...)
- The point about work unit size for remote computing is interesting, I wonder if there is a framework/general method for solving this sort of problem (feels like a solved problem).
- viluon 6y agoThe solution to avoiding unnecessary copying via `shared` sounds naive to me. I don't want the language's safety guarantees to disappear once I try to add such simple optimisations, especially when writing high-performance concurrent code.
- krapht 6y agoThis isn't really easy to solve. I'm only (a little) familiar with Chapel which I think is the most widely used high level HPC language; how data is distributed needs a lot of thought and is specified by the programmer according to language constructs of "locale" and "distribution". The other hard problem an HPC language needs to handle is error handling. What happens when a processor goes down, data isn't received in a timely fashion, etc. How a data processing job might get rerouted to a different server and how intermediate data calculations might get checkpointed so the loss of one result doesn't mean redoing the whole computation.