4 ms·

I personally quite dislike Redis, which is highly unconcurrent. And don't get me started on Node.js. Of course people do use coroutines - I simple don't think

by mdomans 10y ago

I personally quite dislike Redis, which is highly unconcurrent. And don't get me started on Node.js. Of course people do use coroutines - I simple don't think they're a good idea. I think they're a bad idea. They're neither readable nor introduce better performance. And yes, I really don't think people think they need concurrency - they just want this_A and this_B to happen ASAP and with no blocking. On your explanation of coroutines and preemptive - I agree, though I consider both equally bad. The point I was making that Celery runs lockless - and that'd be an interesting idea to back port to Python. And I know queues are old idea. What I like about them is the ability to get rid of "locks everywhere". An example of what I consider good concurrency model is Grand Central Dispatch with atomic ops running via queues.

btilly 10y ago
This is your response to having some of your glaring mistakes pointed out? You ignore having been proven wrong, then proceed to state your personal opinions as if people should care? I have news for you. Redis and Node.js are widely used because they are able to solve real problems that real people have. They work in practice despite your personal opinions to the contrary. Moving on, you're dead wrong about concurrency. How many programs are running at once on your computer? While you are loading a web page, do you want your browser to continue to be able to respond to you? Those are two simple cases where people want concurrency. Those are also two cases that Celery doesn't handle. At one point they would have been written with cooperative multi-tasking. Today these are generally written with preemptive instead. (Though there are some exceptions. PuTTY is a fun one. That implemented coroutines in C with an impressive preprocessor hack. See http://www.chiark.greenend.org.uk/~sgtatham/coroutines.html http://www.chiark.greenend.org.uk/~sgtatham/coroutines.html for details.) And back to Celery, things that don't require locking, can and should get by without locking. This is a useful observation, but hardly a universally applicable one. And waving the magic Celery want doesn't change that. If I have 500 files to process, and that data needs to wind up nicely summarized in a database, Celery will only let me get rid of locking in my code because I rely on locking being implemented in the database. If you care about data and race conditions, at some point you have to solve hard problems. And you'll need to lock. But only do it when you need to.
- mdomans 10y ago
  I disagree with you and here's why; Opinions of mine were made after trying both Redis and Node.js and finding them in poor state. You can read about Redis's problems here: http://redis.io/topics/latency http://redis.io/topics/latency and here http://redis.io/topics/transactions http://redis.io/topics/transactions Node.js is better in regard to handling high load. And the concept of having the heavy lifting happening mostly away from programmer is very sensible. I only find the implementation lacking and there seem to be a fair amount of people now migrating away from Node.js In regard to your point about two programs running "concurrently" - that's actually multitasking, a feature of OS, not the programming language. And if your OS doesn't support that, you can't write a web browser that will make the email client work concurrently. Concurrency only means that two different things seem to be happening at once. Cooperative multitasking and cooperative concurrency are two different things. Cooperative multitasking means that if your browser is poorly written and blocks - your whole machine blocks. On the other hand cooperative concurrency is a paradigm under which if your coroutine blocks, your program blocks. The argument between whether to choose threads or coroutines is an argument whether it is better to have blocking or race conditions. So, having traversed "why I'm wrong", let's consider if there are better models. Arguably the best language in the class of highly concurrent ones is Erlang. Erlang for years dominated the space of stable concurrency. And how it works? Message passing. You can still deadlock Erlang program - it's just much harder. Another good example I like is Grand Central Dispatch and it's OS version libdispatch. The whole point there is to have blocks or operations that, and this is guaranteed by the programmer, mutate the state of memory in an atomic way. Both of those examples are really about having a queue of changes and applying them. And you could, quite sensibly, argue that a mutex is nothing more internally than an ordered queue, so why bother. Well, my problem is really with the API language gives to the programmer. The point I'm trying to make is that concerning the programmer with how his work is executed is wrong. If we can avoid locks inside Python interpreter, and we can, by having a different syntax to describe what needs to happen, I'd consider it a much better solution. It's much easier to say: "drive me to the Station" than to drive there.
  - btilly 10y ago
    I am not sure what your trying involved. However I have found use cases that both Redis and Node.js were great for. And I'm well aware of how to make both suck. Every programming tool has limitations. Use them within the right domains, and they are likely to work well. Use them for something they are poorly suited to, and your experience is likely to be horrible. That is why Redis talks honestly about what the potential problems and strengths are. Those aren't problems in the right use case, but they are things you need to know to help figure out if you have the right use case. Moving on, cooperative multitasking is a form of cooperative concurrency. It has all of the same strengths and weaknesses as cooperative concurrency in any other setting. Hence a single badly coded network call could freeze up Mac OS 9, or Win 3.1. That said if every application is coded correctly, then you can do a lot of things concurrently. I remember using email, Usenet and a browser at the same time on Mac OS 9, and it worked fine..most of the time. Once the system gets complex enough, the fact that any mistake can lock everything becomes unacceptable. This is why operating systems do not generally use cooperative multitasking. An individual application may reasonably choose to go either way. But as they get more complex, there is pressure to go preemptive. Moving on, you only have half the story for Erlang. Erlang uses message passing AND immutable memory. The fact that you can't modify memory in place is a huge limitation on the programmer. However it also eliminates large classes of race conditions. If Python made all data immutable, it could also get rid of most of the uses of the GIL. For a less extreme approach, look at Go. In Go data is mutable. But by convention, you pass objects around using channels, and only one goroutine owns an object at a time. Since only the owner accesses it, that eliminates most race conditions. However if you violate the convention, a Go program dump can dump core very quickly!

by mdomans 10y ago

btilly 10y ago

This is your response to having some of your glaring mistakes pointed out? You ignore having been proven wrong, then proceed to state your personal opinions as if people should care? I have news for you. Redis and Node.js are widely used because they are able to solve real problems that real people have. They work in practice despite your personal opinions to the contrary. Moving on, you're dead wrong about concurrency. How many programs are running at once on your computer? While you are loading a web page, do you want your browser to continue to be able to respond to you? Those are two simple cases where people want concurrency. Those are also two cases that Celery doesn't handle. At one point they would have been written with cooperative multi-tasking. Today these are generally written with preemptive instead. (Though there are some exceptions. PuTTY is a fun one. That implemented coroutines in C with an impressive preprocessor hack. See http://www.chiark.greenend.org.uk/~sgtatham/coroutines.html http://www.chiark.greenend.org.uk/~sgtatham/coroutines.html for details.) And back to Celery, things that don't require locking, can and should get by without locking. This is a useful observation, but hardly a universally applicable one. And waving the magic Celery want doesn't change that. If I have 500 files to process, and that data needs to wind up nicely summarized in a database, Celery will only let me get rid of locking in my code because I rely on locking being implemented in the database. If you care about data and race conditions, at some point you have to solve hard problems. And you'll need to lock. But only do it when you need to.

mdomans 10y ago
I disagree with you and here's why; Opinions of mine were made after trying both Redis and Node.js and finding them in poor state. You can read about Redis's problems here: http://redis.io/topics/latency http://redis.io/topics/latency and here http://redis.io/topics/transactions http://redis.io/topics/transactions Node.js is better in regard to handling high load. And the concept of having the heavy lifting happening mostly away from programmer is very sensible. I only find the implementation lacking and there seem to be a fair amount of people now migrating away from Node.js In regard to your point about two programs running "concurrently" - that's actually multitasking, a feature of OS, not the programming language. And if your OS doesn't support that, you can't write a web browser that will make the email client work concurrently. Concurrency only means that two different things seem to be happening at once. Cooperative multitasking and cooperative concurrency are two different things. Cooperative multitasking means that if your browser is poorly written and blocks - your whole machine blocks. On the other hand cooperative concurrency is a paradigm under which if your coroutine blocks, your program blocks. The argument between whether to choose threads or coroutines is an argument whether it is better to have blocking or race conditions. So, having traversed "why I'm wrong", let's consider if there are better models. Arguably the best language in the class of highly concurrent ones is Erlang. Erlang for years dominated the space of stable concurrency. And how it works? Message passing. You can still deadlock Erlang program - it's just much harder. Another good example I like is Grand Central Dispatch and it's OS version libdispatch. The whole point there is to have blocks or operations that, and this is guaranteed by the programmer, mutate the state of memory in an atomic way. Both of those examples are really about having a queue of changes and applying them. And you could, quite sensibly, argue that a mutex is nothing more internally than an ordered queue, so why bother. Well, my problem is really with the API language gives to the programmer. The point I'm trying to make is that concerning the programmer with how his work is executed is wrong. If we can avoid locks inside Python interpreter, and we can, by having a different syntax to describe what needs to happen, I'd consider it a much better solution. It's much easier to say: "drive me to the Station" than to drive there.
- btilly 10y ago
  I am not sure what your trying involved. However I have found use cases that both Redis and Node.js were great for. And I'm well aware of how to make both suck. Every programming tool has limitations. Use them within the right domains, and they are likely to work well. Use them for something they are poorly suited to, and your experience is likely to be horrible. That is why Redis talks honestly about what the potential problems and strengths are. Those aren't problems in the right use case, but they are things you need to know to help figure out if you have the right use case. Moving on, cooperative multitasking is a form of cooperative concurrency. It has all of the same strengths and weaknesses as cooperative concurrency in any other setting. Hence a single badly coded network call could freeze up Mac OS 9, or Win 3.1. That said if every application is coded correctly, then you can do a lot of things concurrently. I remember using email, Usenet and a browser at the same time on Mac OS 9, and it worked fine..most of the time. Once the system gets complex enough, the fact that any mistake can lock everything becomes unacceptable. This is why operating systems do not generally use cooperative multitasking. An individual application may reasonably choose to go either way. But as they get more complex, there is pressure to go preemptive. Moving on, you only have half the story for Erlang. Erlang uses message passing AND immutable memory. The fact that you can't modify memory in place is a huge limitation on the programmer. However it also eliminates large classes of race conditions. If Python made all data immutable, it could also get rid of most of the uses of the GIL. For a less extreme approach, look at Go. In Go data is mutable. But by convention, you pass objects around using channels, and only one goroutine owns an object at a time. Since only the owner accesses it, that eliminates most race conditions. However if you violate the convention, a Go program dump can dump core very quickly!