3 ms·

Well, it sounds like a good start - at least for the cache example - would be a shared memory hash table. Building one in pure python with the existing multipro

Well, it sounds like a good start - at least for the cache example - would be a shared memory hash table. Building one in pure python with the existing multiprocessing primitives is doable, though obviously not trivial. Building one in C is also possible, though obviously there's the development time / performance tradeoff there. A shared memory hash table actually seems like a really useful thing to eventually make its way into the multiprocessing module. For the event example, parsing the data structures into shared memory and then accessing from there would remove the parsing and memory overhead he complains about. I don't really want to dig in to the standards he cites and see how difficult that solution would be, but again, I can't see a reason why it isn't possible. At any rate, in both cases, the solution is definitely doable; there's nothing inherent in python's concurrency model that "prevents you from properly making use of modern multicore hardware" Side note: shared memory access in modern multicore hardware may not get you the performance gains you would initially think. Especially for write-heavy workloads, cache invalidation is a huge problem.

yummyfajitas 13y ago
For the event example, garbage collection is the tricky bit, not sharing. Sharing could be handled merely by writing C structs linearly to an array - it's cleaning them up when the message is done being processed that's the problem. A shared memory hash table is a) tricky to do and b) would require repeated polling of the hash table by client processes. Or perhaps you'd need to write to the hash table all the clients who are listening to that future.
- AnIrishDuck 13y ago
  > For the event example, garbage collection is the tricky bit, not sharing. Sharing could be handled merely by writing C structs linearly to an array - it's cleaning them up when the message is done being processed that's the problem. I agree; however, this problem could still be bootstrapped to each individual process's GC. The parser sends the shared memory id to all worker processes, and the shared memory representation contains a reference counter. Each object decrements the reference counter in __del__; when the reference counter reaches zero, the memory is unmapped and reclaimed by the kernel. This would probably cause major fragmentation issues with small objects, so pooling might be necessary. This sounds like a job that could be abstracted by a library. > A shared memory hash table is a) tricky to do and b) would require repeated polling of the hash table by client processes. Or perhaps you'd need to write to the hash table all the clients who are listening to that future. I won't argue that shared memory hash tables (and concurrent data structures in general) are easy to implement. But that leads back to my original point; that kind of code should be in a library. Admittedly, Python today lacks libraries providing concurrent data structures. And it lacks modules that handle concurrent memory management. That doesn't mean there's something inherent in its design that prevents such libraries from existing.

yummyfajitas 13y ago

For the event example, garbage collection is the tricky bit, not sharing. Sharing could be handled merely by writing C structs linearly to an array - it's cleaning them up when the message is done being processed that's the problem. A shared memory hash table is a) tricky to do and b) would require repeated polling of the hash table by client processes. Or perhaps you'd need to write to the hash table all the clients who are listening to that future.

AnIrishDuck 13y ago
> For the event example, garbage collection is the tricky bit, not sharing. Sharing could be handled merely by writing C structs linearly to an array - it's cleaning them up when the message is done being processed that's the problem. I agree; however, this problem could still be bootstrapped to each individual process's GC. The parser sends the shared memory id to all worker processes, and the shared memory representation contains a reference counter. Each object decrements the reference counter in __del__; when the reference counter reaches zero, the memory is unmapped and reclaimed by the kernel. This would probably cause major fragmentation issues with small objects, so pooling might be necessary. This sounds like a job that could be abstracted by a library. > A shared memory hash table is a) tricky to do and b) would require repeated polling of the hash table by client processes. Or perhaps you'd need to write to the hash table all the clients who are listening to that future. I won't argue that shared memory hash tables (and concurrent data structures in general) are easy to implement. But that leads back to my original point; that kind of code should be in a library. Admittedly, Python today lacks libraries providing concurrent data structures. And it lacks modules that handle concurrent memory management. That doesn't mean there's something inherent in its design that prevents such libraries from existing.