4 ms·

Except when your use case requires a massive shared data cache that needs to be atomically updated.

zepolen 9y ago
Of which there are plenty well defined ones to do the job already, and as a plus they can communicate with any language not just Python.
omarforgotpwd 9y ago
Redis could help. Obviously not perfect for every use case but covers many of them.
- optimusclimb 9y ago
  It doesn't if you need to manage atomic data across the processes, as there's no way to lock and block the other cache consumers (think the data you need to handle cache evictions, etc.) Also, you're describing multiple python processes + an extra server (redis) process - as a "simpler" solution for the limitation that Python doesn't do multi-threads well. Of course there are a ton of use cases out there where you can scale in other ways, but threads and shared memory exist for a reason - there's no reason not to call a spade a spade and say the GIL is still a limitation.
  - tinix 9y ago
    Blocking workers in a Redis queue is not hard... You can simply put them all on a pubsub control channel and then orchestrate them that way you need to do shit. Or literally just take down the processes, or the network, so they disconnect and stop BLPOPing the queue. Cache evictions can be handled by Redis natively with TTL. For retries and failure mitigation, you can still lean on Redis via BRPOPLPUSH/RPOPLPUSH. If you want to scale beyond one machine, you can't rely on threading to help you. So why not just do it right to begin with, and use a parallel worker queue? It's not a matter of the GIL being a limitation, a single machine is a limitation too. Don't blame your tools because you're misusing them. As for threading in Python... on a single machine, for one reason or another... I would still rather use multiple processes, or at the very least, would just simply use eventlet and greenthreads. Not saying it covers all use cases, it's not a silver bullet, and it doesn't replace threading natively, but damnit, it scales better, and it's the right way to do the task at hand.
    - optimusclimb 9y ago
      Wasn't reading data from or using a redis queue, this wasn't a blocking queue issue. Second, my use case wasn't a simple cache, I was omitting details. So redis having a TTL eviction policy for the values it stores is a moot point. The resources I was dealing with ranged from around 0.5GB to several gigabytes. That was the important working data - but whether or not these objects were available was what had to be coordinated (as well as some other bookkeeping data.) Also, in this case - of course scaling beyond one machine was important. We were. The issue is that for each machine you allocate, you want to maximize usage of its resources. So each machine gets its own data cache, but nonetheless, we still wanted to max CPU usage per machine. So again, it's multi-process, vs. multi-thread, and in this case - multi-threaded with shared memory was a much easier paradigm than handling co-ordination among separate python processes. I was just giving an example of reasons one would want true multi-threading in python. I wasn't trying to go into explicit details of an entire project. Please consider this when you reply to people and tell them they're "misusing their tools." Good day stranger.
abecedarius 9y ago
I think it's ok to not write everything in Python, and this is a long way from the top of my problems with it.
- optimusclimb 9y ago
  Of course it is - and that's what people do. The reason for the parent article is that there ARE people that would like to continue to use Python the language, and their existing source code/libraries, but would like not to deal with the GIL. Just because it is not a priority for you doesn't mean it isn't for others.
- jhayward 9y ago
  This is self-fulfilling. As long as Python is useless for a set of tasks that are intrinsic and important to some domains, they won't use it.
jhayward 9y ago
> Except when your use case requires a massive shared data cache that needs to be atomically updated You can delete the last 6 words. Anything where multiple processes would have to read in/acquire a massive dataset to do some independent work qualifies. For instance, running some number (e.g., hundreds to hundreds of thousands) of analytical or statistical tests over a set to pick parameters, etc.
- abecedarius 9y ago
  Read-only shared memory can cover that. Python's ref counting does make it a nuisance: you can't share it as a Python object graph.

Of which there are plenty well defined ones to do the job already, and as a plus they can communicate with any language not just Python.

omarforgotpwd 9y ago

Redis could help. Obviously not perfect for every use case but covers many of them.

optimusclimb 9y ago
It doesn't if you need to manage atomic data across the processes, as there's no way to lock and block the other cache consumers (think the data you need to handle cache evictions, etc.) Also, you're describing multiple python processes + an extra server (redis) process - as a "simpler" solution for the limitation that Python doesn't do multi-threads well. Of course there are a ton of use cases out there where you can scale in other ways, but threads and shared memory exist for a reason - there's no reason not to call a spade a spade and say the GIL is still a limitation.
- tinix 9y ago
  Blocking workers in a Redis queue is not hard... You can simply put them all on a pubsub control channel and then orchestrate them that way you need to do shit. Or literally just take down the processes, or the network, so they disconnect and stop BLPOPing the queue. Cache evictions can be handled by Redis natively with TTL. For retries and failure mitigation, you can still lean on Redis via BRPOPLPUSH/RPOPLPUSH. If you want to scale beyond one machine, you can't rely on threading to help you. So why not just do it right to begin with, and use a parallel worker queue? It's not a matter of the GIL being a limitation, a single machine is a limitation too. Don't blame your tools because you're misusing them. As for threading in Python... on a single machine, for one reason or another... I would still rather use multiple processes, or at the very least, would just simply use eventlet and greenthreads. Not saying it covers all use cases, it's not a silver bullet, and it doesn't replace threading natively, but damnit, it scales better, and it's the right way to do the task at hand.
  - optimusclimb 9y ago
    Wasn't reading data from or using a redis queue, this wasn't a blocking queue issue. Second, my use case wasn't a simple cache, I was omitting details. So redis having a TTL eviction policy for the values it stores is a moot point. The resources I was dealing with ranged from around 0.5GB to several gigabytes. That was the important working data - but whether or not these objects were available was what had to be coordinated (as well as some other bookkeeping data.) Also, in this case - of course scaling beyond one machine was important. We were. The issue is that for each machine you allocate, you want to maximize usage of its resources. So each machine gets its own data cache, but nonetheless, we still wanted to max CPU usage per machine. So again, it's multi-process, vs. multi-thread, and in this case - multi-threaded with shared memory was a much easier paradigm than handling co-ordination among separate python processes. I was just giving an example of reasons one would want true multi-threading in python. I wasn't trying to go into explicit details of an entire project. Please consider this when you reply to people and tell them they're "misusing their tools." Good day stranger.

abecedarius 9y ago

I think it's ok to not write everything in Python, and this is a long way from the top of my problems with it.

optimusclimb 9y ago
Of course it is - and that's what people do. The reason for the parent article is that there ARE people that would like to continue to use Python the language, and their existing source code/libraries, but would like not to deal with the GIL. Just because it is not a priority for you doesn't mean it isn't for others.
jhayward 9y ago
This is self-fulfilling. As long as Python is useless for a set of tasks that are intrinsic and important to some domains, they won't use it.

jhayward 9y ago

> Except when your use case requires a massive shared data cache that needs to be atomically updated You can delete the last 6 words. Anything where multiple processes would have to read in/acquire a massive dataset to do some independent work qualifies. For instance, running some number (e.g., hundreds to hundreds of thousands) of analytical or statistical tests over a set to pick parameters, etc.

abecedarius 9y ago
Read-only shared memory can cover that. Python's ref counting does make it a nuisance: you can't share it as a Python object graph.