5 ms·

> Time of "Getting that shit done" and cost of long-term maintenance are two of the main reasons why people use Ruby/Python, not for the raw performance. Plent

> Time of "Getting that shit done" and cost of long-term maintenance are two of the main reasons why people use Ruby/Python, not for the raw performance. Plenty of other languages "get shit done". > And anyway, from my experience, Python has never been the cause of the slowness. Then you're not actually performing performance analysis. Python is S-L-O-W. I've written large systems in Python. I've profiled them to find the hotspots. I've micro-benchmarked them against implementations in other languages (C, Java). Python lost. Always. By a lot. So then I had to rewrite critical sections of the Python in C (or accept that we'd need 32 servers instead of 2). Eventually I stopped bothering, and used a different language, because the aggregate cost of a thousand small performance issues (and a few big ones) is actually quite high.

scott_w 14y ago
I think you missed the point. There are plenty of use cases where Python isn't the cause of slowness. It doesn't matter if C is 100x faster than Python if your app is running 49/50 seconds running SQL queries.
- coldtea 14y ago
  No. But it DOES matter to me than Python restricts you to using it for those cases, whereas I would want (and could technically be able to) use it for far more stuff.
  - pekk 14y ago
    So you prematurely optimize Python out of everything you write, because it might not be suitable for... some use case yet to be determined?
    - coldtea 14y ago
      No, I don't "prematurely optimize Python out of everything I write". The exact opposite, I would like to be able to use Python for more stuff but I can't because it's slow. Nothing about doing it "prematurely" here. I have tried it for specific problems and it was slow. If anything, having built in in Python first is the reverse of "prematurely". And I have, as many programmers have, a very good general idea of where it's slow and what it can be used adequately for without resorting to C. And what I argued is that people constrain themselves unconsciously, based on that very knowledge, to the kind of projects they do with pure Python. Nobody writes a AAA game in Python. Nobody tries to build a browser in Python. Nobody writes a performant Apache/Nginx class web-server in Python. Nobody writes a database engine in Python. Etc etc. (Note for the "Vulcan"-style literal readers: by "nobody" here I mean "not many" / "not a sane person". I'm sure you can find some experimental, bug ridden, half-developed example of all of those done in Python, with maybe 100 users at most. Oh, and I specifically wrote: "pure python").
      - deleted 14y ago
        [deleted]
      - voltagex_ 14y ago
        >Nobody writes a AAA game in Python. No, but the scripting/UI/mod toolkit for the game might be in Python >Nobody writes a performant Apache/Nginx class web-server in Python No, but Python can do quite well serving WSGI to Nginx >Nobody writes a database engine in Python No, but Python ORMs work pretty well. The point I'm trying to make is that I don't think that it's a bad thing that Python sometimes gets relegated to a "support" language. I think the time of the single language programmer is over. If you need something faster than Python, use something else.
        coldtea 14y ago
        >The point I'm trying to make is that I don't think that it's a bad thing that Python sometimes gets relegated to a "support" language. Of course it is bad. It would only be good if the inverse (being able to write the whole thing in Python and get the same speed) would be worse. But it clearly is not -- hence, it's bad. It might be pragmatic and necessary compromise at the moment, but that doesn't make it "not bad". >I think the time of the single language programmer is over. If you need something faster than Python, use something else. On the contrary, multi language projects are a kludge born out of (pragmatic) necessity. Nothing inherently inevitable (or good) about them.
        pwang 14y ago
        > On the contrary, multi language projects are a kludge born out of (pragmatic) necessity. > Nothing inherently inevitable (or good) about them. Um, except that coherent planes/tiers of abstraction is the foundation of robust software engineering. You write languages to a spec; compiler authors emit bytecodes for another spec; hardware engineers optimize transistors and lay traces on silicon to meet yet another spec. Etc. Multi-language is the only thing that makes the world work today. The closest single-language runtime that exists is FORTH and things like that. Every other "modern" language where you get to pretend your process has a contiguous address space, or that there even exists a thing such as "process", is s pragmatic compromise between compiler and runtime technology.
nostrademons 14y ago
"So then I had to rewrite critical sections of the Python in C" I think that's kinda the point; rewriting critical sections in C is pretty much Working As Intended. That's why it's a scripting language. The point is that you didn't have to write the whole thing in C.
- fauigerzigerk 14y ago
  Do you think that most Python developers can write safe C code? I doubt it but I have no data on it either.
  - nilliams 14y ago
    A fairly moot point. Do you think most C developers can write safe C code? The 2 I've encountered in my short career basically have no idea what they're doing (which I suspect is partly why they stuck with C instead of exploring alternative languages).
    - fauigerzigerk 14y ago
      If that were the case it would make the situation even worse. It means the whole "write performance critical parts in C" argument is bogus.
      - nostrademons 14y ago
        To answer your original question - no, but there's less C code than the equivalent C-only program, so you're still better off. Or if your argument is that C is a security nightmare (which you won't find much argument from me on), replace C/CPython with Java/Jython.
      - pcwalton 14y ago
        You could write performance critical parts in a safer language, though, such as Rust (disclaimer: I work on Rust). C isn't the only option here.
- 14y ago
vidarh 14y ago
> Then you're not actually performing performance analysis. Python is S-L-O-W. You totally miss the point. There are large classes of systems where CPU time in your app is not a significant factor. E.g. my go-to example of this is the first Ruby project I deployed in a production environment: A queueing middleware server. If we served the queues entirely out of memory we could serve a few million messages a day out of 10% of a single 2005-era Xeon core (and while Ruby doesn't scale across cores, this system could easily spread load by simply running multiple queue processes).But of of that 10%, 90% was spent in kernel space. So even if switching to C (or whatever) were to speed up the user-space part of the code 90%, it would only cut CPU time spent from 10% to 9.1%. This is fairly typical. If your code is CPU-bound, and it's not CPU bound because of stupid algorithm choices, then things can look very different. The first thing to do is find out if you're CPU or IO bound.
- pifflesnort 14y ago
  > You totally miss the point. There are large classes of systems where CPU time in your app is not a significant factor. E.g. my go-to example of this is the first Ruby project I deployed in a production environment: This is only true if you ignore response latency as a comparative metric entirely. > But of of that 10%, 90% was spent in kernel space. So even if switching to C (or whatever) were to speed up the user-space part of the code 90%, it would only cut CPU time spent from 10% to 9.1%. I'm really curious what you were doing in Ruby that spent 90% of your runtime in the kernel. Having implemented similar middleware, I found that nearly all my runtime was spent just performing basic decoding/encoding of framed packet data, and as a kernel developer, I'm having a hard time imagining what an in memory queue broker could have been doing to incur that overhead
  - vidarh 14y ago
    > This is only true if you ignore response latency as a comparative metric entirely. Response latency makes scripting language unacceptable for some types of problems. But in my experience very few latency problems I've come across are down to language choice vs. badly written database queries, lack of caching etc., that'd still be unacceptable regardless of language. Basically, the moment a database is involved, that's where you're most likely to see your low latency going out the window. Of course there are situations where language choice definitively matters in terms of latency. Despite using mostly Ruby now, there are certainly places where I'd use C. E.g. I'd never try to replace Sphinx search with pure Ruby code, for example. > I'm really curious what you were doing in Ruby that spent 90% of your runtime in the kernel. Having implemented similar middleware, I found that nearly all my runtime was spent just performing basic decoding/encoding of framed packet data, and as a kernel developer, I'm having a hard time imagining what an in memory queue broker could have been doing to incur that overhead The moment you find yourself "decoding/encoding framed packet data", you have lost if your goal is a really efficient message queue if said decoding/encoding consists of more than check a length field and possibly verify a magic boundary value. If the userspace part of your queueing system does more than wait on sockets to be available and read/write large-ish blocks, you'll be slow. Of course there are plenty of situations where you don't get to choose the protocol, and you're stuck doing things the slow way, and sometimes that may make a language like Ruby impractical. But there are also plenty of situations where in-memory queues are also not acceptable, and the moment you hit disk, the entire performance profile shifts so dramatically that you suddenly often have much more flexibility.

scott_w 14y ago

I think you missed the point. There are plenty of use cases where Python isn't the cause of slowness. It doesn't matter if C is 100x faster than Python if your app is running 49/50 seconds running SQL queries.

coldtea 14y ago
No. But it DOES matter to me than Python restricts you to using it for those cases, whereas I would want (and could technically be able to) use it for far more stuff.
- pekk 14y ago
  So you prematurely optimize Python out of everything you write, because it might not be suitable for... some use case yet to be determined?
  - coldtea 14y ago
    No, I don't "prematurely optimize Python out of everything I write". The exact opposite, I would like to be able to use Python for more stuff but I can't because it's slow. Nothing about doing it "prematurely" here. I have tried it for specific problems and it was slow. If anything, having built in in Python first is the reverse of "prematurely". And I have, as many programmers have, a very good general idea of where it's slow and what it can be used adequately for without resorting to C. And what I argued is that people constrain themselves unconsciously, based on that very knowledge, to the kind of projects they do with pure Python. Nobody writes a AAA game in Python. Nobody tries to build a browser in Python. Nobody writes a performant Apache/Nginx class web-server in Python. Nobody writes a database engine in Python. Etc etc. (Note for the "Vulcan"-style literal readers: by "nobody" here I mean "not many" / "not a sane person". I'm sure you can find some experimental, bug ridden, half-developed example of all of those done in Python, with maybe 100 users at most. Oh, and I specifically wrote: "pure python").
    - deleted 14y ago
      [deleted]
    - voltagex_ 14y ago
      >Nobody writes a AAA game in Python. No, but the scripting/UI/mod toolkit for the game might be in Python >Nobody writes a performant Apache/Nginx class web-server in Python No, but Python can do quite well serving WSGI to Nginx >Nobody writes a database engine in Python No, but Python ORMs work pretty well. The point I'm trying to make is that I don't think that it's a bad thing that Python sometimes gets relegated to a "support" language. I think the time of the single language programmer is over. If you need something faster than Python, use something else.
      - coldtea 14y ago
        >The point I'm trying to make is that I don't think that it's a bad thing that Python sometimes gets relegated to a "support" language. Of course it is bad. It would only be good if the inverse (being able to write the whole thing in Python and get the same speed) would be worse. But it clearly is not -- hence, it's bad. It might be pragmatic and necessary compromise at the moment, but that doesn't make it "not bad". >I think the time of the single language programmer is over. If you need something faster than Python, use something else. On the contrary, multi language projects are a kludge born out of (pragmatic) necessity. Nothing inherently inevitable (or good) about them.
        pwang 14y ago
        > On the contrary, multi language projects are a kludge born out of (pragmatic) necessity. > Nothing inherently inevitable (or good) about them. Um, except that coherent planes/tiers of abstraction is the foundation of robust software engineering. You write languages to a spec; compiler authors emit bytecodes for another spec; hardware engineers optimize transistors and lay traces on silicon to meet yet another spec. Etc. Multi-language is the only thing that makes the world work today. The closest single-language runtime that exists is FORTH and things like that. Every other "modern" language where you get to pretend your process has a contiguous address space, or that there even exists a thing such as "process", is s pragmatic compromise between compiler and runtime technology.

nostrademons 14y ago

"So then I had to rewrite critical sections of the Python in C" I think that's kinda the point; rewriting critical sections in C is pretty much Working As Intended. That's why it's a scripting language. The point is that you didn't have to write the whole thing in C.

fauigerzigerk 14y ago
Do you think that most Python developers can write safe C code? I doubt it but I have no data on it either.
- nilliams 14y ago
  A fairly moot point. Do you think most C developers can write safe C code? The 2 I've encountered in my short career basically have no idea what they're doing (which I suspect is partly why they stuck with C instead of exploring alternative languages).
  - fauigerzigerk 14y ago
    If that were the case it would make the situation even worse. It means the whole "write performance critical parts in C" argument is bogus.
    - nostrademons 14y ago
      To answer your original question - no, but there's less C code than the equivalent C-only program, so you're still better off. Or if your argument is that C is a security nightmare (which you won't find much argument from me on), replace C/CPython with Java/Jython.
    - pcwalton 14y ago
      You could write performance critical parts in a safer language, though, such as Rust (disclaimer: I work on Rust). C isn't the only option here.
14y ago

vidarh 14y ago

> Then you're not actually performing performance analysis. Python is S-L-O-W. You totally miss the point. There are large classes of systems where CPU time in your app is not a significant factor. E.g. my go-to example of this is the first Ruby project I deployed in a production environment: A queueing middleware server. If we served the queues entirely out of memory we could serve a few million messages a day out of 10% of a single 2005-era Xeon core (and while Ruby doesn't scale across cores, this system could easily spread load by simply running multiple queue processes).But of of that 10%, 90% was spent in kernel space. So even if switching to C (or whatever) were to speed up the user-space part of the code 90%, it would only cut CPU time spent from 10% to 9.1%. This is fairly typical. If your code is CPU-bound, and it's not CPU bound because of stupid algorithm choices, then things can look very different. The first thing to do is find out if you're CPU or IO bound.

pifflesnort 14y ago
> You totally miss the point. There are large classes of systems where CPU time in your app is not a significant factor. E.g. my go-to example of this is the first Ruby project I deployed in a production environment: This is only true if you ignore response latency as a comparative metric entirely. > But of of that 10%, 90% was spent in kernel space. So even if switching to C (or whatever) were to speed up the user-space part of the code 90%, it would only cut CPU time spent from 10% to 9.1%. I'm really curious what you were doing in Ruby that spent 90% of your runtime in the kernel. Having implemented similar middleware, I found that nearly all my runtime was spent just performing basic decoding/encoding of framed packet data, and as a kernel developer, I'm having a hard time imagining what an in memory queue broker could have been doing to incur that overhead
- vidarh 14y ago
  > This is only true if you ignore response latency as a comparative metric entirely. Response latency makes scripting language unacceptable for some types of problems. But in my experience very few latency problems I've come across are down to language choice vs. badly written database queries, lack of caching etc., that'd still be unacceptable regardless of language. Basically, the moment a database is involved, that's where you're most likely to see your low latency going out the window. Of course there are situations where language choice definitively matters in terms of latency. Despite using mostly Ruby now, there are certainly places where I'd use C. E.g. I'd never try to replace Sphinx search with pure Ruby code, for example. > I'm really curious what you were doing in Ruby that spent 90% of your runtime in the kernel. Having implemented similar middleware, I found that nearly all my runtime was spent just performing basic decoding/encoding of framed packet data, and as a kernel developer, I'm having a hard time imagining what an in memory queue broker could have been doing to incur that overhead The moment you find yourself "decoding/encoding framed packet data", you have lost if your goal is a really efficient message queue if said decoding/encoding consists of more than check a length field and possibly verify a magic boundary value. If the userspace part of your queueing system does more than wait on sockets to be available and read/write large-ish blocks, you'll be slow. Of course there are plenty of situations where you don't get to choose the protocol, and you're stuck doing things the slow way, and sometimes that may make a language like Ruby impractical. But there are also plenty of situations where in-memory queues are also not acceptable, and the moment you hit disk, the entire performance profile shifts so dramatically that you suddenly often have much more flexibility.