4 ms·

Are you sure you couldn't write it?

by jnoller 16y ago

fauigerzigerk 16y ago
I could write it, but it would either be dog slow or take 10 times as much time to write. The only way of using multiple cores would be to implement everything on top of shared memory. That implies using no pointers in any of the data structures and implementing a custom garbage collector.
- jnoller 16y ago
  Alright, so it's not the optimal language for something with a large amount of mutable shared state being hacked at by threads. No one ever tried to bill it as the "one true language" - maybe it's better for glue in this case, or APIs or clients, etc. Maybe it needs some jython, etc. This is why multiple languages (a diverse toolbox) and multiple interpreters each with their pros and cons is also a good thing. Each one will have its shortcomings, and it's almost impossible to achieve all of the goals of all of the modern programming languages in a single language. So, sure - I'd love to see python used for this problem - and all problems but thats me being irrational, that will never be true. What is true is that eventually the GIL will go away, or a non cPython interpreter will rise above cPython as the defacto standard.
  - fauigerzigerk 16y ago
    I don't see why it is irrational to wish that a general purpose programming language be able to run data intensive, parallel server code. It's not exactly asking for HTML to be modified so you can write signal processing code in it. The sad thing is that a Python + C combination would actually be _more_ suitable for the task than any of the JVM based language implementations, because the JVM has the unfortunate feature of eating memory like crazy due to the lack of structured value types. This type of application is all about conserving memory. The core data structures must be extremely tight. But they make up only like 5% of the code. So Python + C would be way superior to Java for something like Cassandra if it wasn't for the GIL. I guess I have to practice meditation for a few years so I can one day be as relaxed about these things as you are ;-)
    - jnoller 16y ago
      Well, if it was I/O bound - the GIL shouldn't bother it too much ;) Don't mistake me for calm though, it drives me nuts :)
    - rbanffy 16y ago
      It's not irrational. But, since you are willing to rely on C in addition to Python, why not code the parallelizable parts in C and leave the rest in Python. And you can always write a proof-of-concept in Python, prove the algorithm is right and rewrite the performance-sensitive parts in C. If your tests are thorough, all is good.
      - fauigerzigerk 16y ago
        That's unfortunately not an option because all requests would have to go through that single threaded bottleneck to get to the parallelized C code, which is only acceptable for long running operations. For a request/response type of scenario like Cassandra, Lucene or the kind of analytics systems I'm working on it would make the whole parallelization pointless. The realistic option is to use a separate C based process behind the usual multi-process mod_python setup. That works but is more complicated and error prone than an in-process solution. So I think that's why so many new projects that have these kinds of scenarios (NoSQL for instance) go with Java in spite of its flaws.

Are you sure you couldn't write it?

by jnoller 16y ago

Are you sure you couldn't write it?

I could write it, but it would either be dog slow or take 10 times as much time to write. The only way of using multiple cores would be to implement everything on top of shared memory. That implies using no pointers in any of the data structures and implementing a custom garbage collector.

jnoller 16y ago
Alright, so it's not the optimal language for something with a large amount of mutable shared state being hacked at by threads. No one ever tried to bill it as the "one true language" - maybe it's better for glue in this case, or APIs or clients, etc. Maybe it needs some jython, etc. This is why multiple languages (a diverse toolbox) and multiple interpreters each with their pros and cons is also a good thing. Each one will have its shortcomings, and it's almost impossible to achieve all of the goals of all of the modern programming languages in a single language. So, sure - I'd love to see python used for this problem - and all problems but thats me being irrational, that will never be true. What is true is that eventually the GIL will go away, or a non cPython interpreter will rise above cPython as the defacto standard.
- fauigerzigerk 16y ago
  I don't see why it is irrational to wish that a general purpose programming language be able to run data intensive, parallel server code. It's not exactly asking for HTML to be modified so you can write signal processing code in it. The sad thing is that a Python + C combination would actually be _more_ suitable for the task than any of the JVM based language implementations, because the JVM has the unfortunate feature of eating memory like crazy due to the lack of structured value types. This type of application is all about conserving memory. The core data structures must be extremely tight. But they make up only like 5% of the code. So Python + C would be way superior to Java for something like Cassandra if it wasn't for the GIL. I guess I have to practice meditation for a few years so I can one day be as relaxed about these things as you are ;-)
  - jnoller 16y ago
    Well, if it was I/O bound - the GIL shouldn't bother it too much ;) Don't mistake me for calm though, it drives me nuts :)
  - rbanffy 16y ago
    It's not irrational. But, since you are willing to rely on C in addition to Python, why not code the parallelizable parts in C and leave the rest in Python. And you can always write a proof-of-concept in Python, prove the algorithm is right and rewrite the performance-sensitive parts in C. If your tests are thorough, all is good.
    - fauigerzigerk 16y ago
      That's unfortunately not an option because all requests would have to go through that single threaded bottleneck to get to the parallelized C code, which is only acceptable for long running operations. For a request/response type of scenario like Cassandra, Lucene or the kind of analytics systems I'm working on it would make the whole parallelization pointless. The realistic option is to use a separate C based process behind the usual multi-process mod_python setup. That works but is more complicated and error prone than an in-process solution. So I think that's why so many new projects that have these kinds of scenarios (NoSQL for instance) go with Java in spite of its flaws.