8 ms·

Python is optimized for small programs being easy and quick (and pleasant) to write. Every other use case it fails in some way. For being slow. For lacking sta

by hasenj 9y ago

Python is optimized for small programs being easy and quick (and pleasant) to write. Every other use case it fails in some way. For being slow. For lacking static typing. For lacking compilation. For having a GIL. etc.

anewhnaccount2 9y ago
This is too categorical. And also it fails to define when small stops being small (and so is easy to defend by moving the goalposts). Yes, these are real problems, but some of them have (partial) solutions. You can use one of the many solutions to write a C, C++, cython and so on extension. You can have type checking with mypy. You can produce executable with cx_freeze (not compilation, but compilation is more of a solution than a problem). You can create parallel programs using multiprocessing. (etc.)
- weberc2 9y ago
  Writing extensions in c/c++ largely negates all stated benefits of using Python in the first place, and I'm proficient in all three languages. Further, mypy is hopeful, but it's not quite there yet (lots of holes and bugs--notably no support for recursive types). Lastly multiprocessing for parallelism is a poor man's solution, and far from what I've become accustomed to in Go, Rust, etc. Python is a nice language and it pays my bills, but Go beats it at many of its own purported strengths, in my opinion.
- hasenj 9y ago
  Using these solution makes the development process much more troublesome than it needs to be, thus defeating the whole point of using python.
ptero 9y ago
However, as posters in other thread said, the number and availability of libraries makes functionality of those small programs exceed that of much larger ones written in C or a similar language. If you have to write a large project from scratch in Python it will be slow, but Python has gone pretty far in the direction where for many large tasks 90% of the functionality will be obtainable through standard libraries. Doing the remaining 10% in basic Python often keeps it competitive at runtime with compiled languages but produce much simpler, shorter and easier to maintain code.
- weberc2 9y ago
  Python has great libraries, but the claim that you can use them and have code that competes with compiled languages is flat out wrong. CPython is at least an order of magnitude slower than Java/Go/Haskell/etc and two orders slower than C/C++/Rust/etc. PyPy closes the gap a bit, but it's still noticeably slower than the aforementioned compiled languages. Note to responders--HN won't let me respond to you, because "I'm posting too fast".
  - acdha 9y ago
    Those statements are too broad to be anything other than flamebait. Which specific code did you benchmark? That’s not to say that CPython isn’t slower than a compiled language but in common usage the gaps usually end up being much smaller because most of us aren’t paid to run fib() and things like I/O dominate or code ends up being in the same C libraries (libjpeg, OpenSSL, etc.). I’ve seen multiple cases where Python beat C/Go because the app used something like dictionaries which was heavily optimized C code in the standard library. Again, compiled languages should usually win but the details matter, especially since most people don’t do the exact same work you do. Simply tossing around big numbers with no support contributes noise rather than insight to the conversation.
    - hasenj 9y ago
      > Which specific code did you benchmark? I had a csv file with ~16k entries. Each entry has a lat, lon, name and id. The task: given a (lat, lon) pair, list all entries within 5km. The simplest straight forward solution is to iterate the list, measure the distance of each item to the given point, and add it to the return list if the distance is within the desired range. No crazy optimization or anything. The results were something roughly like this: The D version runs in 2~4ms when compiled with optimizations turned on to the max, and about 6~8ms without optimizations (just default compiler settings). The Go version runs in about 6~8ms. (The Go compiler has no option to produce optimized code). The python version runs in about 50ms. That's 10x slower than Go and 20x slower than D. This is python 3. Using pypy shows absolutely no difference in performance. Obviously the time to load the data from the csv file into a list is not counted in the above results. > I’ve seen multiple cases where Python beat C/Go because the app used something like dictionaries which was heavily optimized C code in the standard library. That's exactly backwards. The Go and D versions use structs which are compact data structures that pack the data together. So the entire array fits in one contiguous block of memory. Python has no concept of structs, so you have to deal with dictionaries. In general, writing optimized code is not a good idea. You should almost always write the most plain and straight forward code. Because fast languages are fast by default, there's no optimization required. The straight forward solution is fast enough. If you are writing Python, your code will be slow by default. You will need to think about ways to optimize it if you want better performance.
dr_zoidberg 9y ago
Working in digital forensics, I read your comment and immediately thought of Volatility and Rekall, the most important memory forensics frameworks. Both monsters that can analize memory dumps from very different operating systems and microprocessor architectures, supporting a wealth of plugins/extensions to analyze every bit of information you can think of. Both written in Python. Please, tell me how they are failing.
- datawarrior 9y ago
  That's largely because Rekall is a fork of Volatility, not because Python was better suited to the task for both projects. Python is used heavily in security/forensics work so it was just a natural choice for the developers. I'm not sure what your argument is otherwise. If there was a tool written in C or Rust to benchmark against then we could better argue which is better for the task. I don't think anyone said Python was "failing."
  - dr_zoidberg 9y ago
    My point is that both of them are big, succesful projects, written in Python, in contrast to hasenjs comment that anything that isn't "a small program being easy and quick (and pleasant) to write... fails in some way.".
bluedino 9y ago
Is it really slow? I ported a bunch of code from PHP to Python, nothing incredibly complicated, but it was easy to switch and the Python code all ended up running about 25-40% faster. I'm sure it would have been even faster to use Java or something like that but it would have taken 5 times as long, and I would have needed to know a lot more Java than Python.
- weberc2 9y ago
  Yeah, slow is relative. PHP is no one's standard for "fast". I program in Python by day, and I find Go much easier to work with, and it's roughly 10 times faster. There are other performant options besides Java these days.
  - zerkten 9y ago
    What are you actually programming? You mention some languages, but you don't actually give any context.
    - deleted 9y ago
      [deleted]
  - jack9 9y ago
    > PHP is no one's standard for "fast". PHP is my standard for "fast" in the realm of scripting. Just behind Perl. Python is slower in every prototype benchmark we have ever tried. It's a practical problem that hand waving cannot remedy.
    - hasenj 9y ago
      Fast would be something like Go or D or any of the other compiled languages.
      - jack9 9y ago
        I'm not sure how that relates. None of those are scripting languages, although when you start talking about "compiled" it gets a little messy with definitions. Either way, it's not what I'm talking about.
baldfat 9y ago
DEPLOYMENT My ONE HUGE Gripe with Python. This and that Pandas (I understand why BUT it drove me away) Zero based for statistical work. Your math is 1 based and the language is 0 based.
- peatmoss 9y ago
  It’s funny because deployment is one of the reasons I hear Python data people trash on R. In general, I’d agree with them that it’s slightly harder for systems people to grok R, but on the other hand I don’t exactly see Python as a shining beacon.
  - baldfat 9y ago
    R as a Domain Specific Language has great output for Reports, Word Documents or Power Point slides and a million of reports. There is also the shinny package and shinny server for web publishing. This is why I went to R from Pandas. My company was impressd with all the reports but asked why they couldn't edit the themes. Then I did it in R in 2 days and haven't looked back. I love Python but I have been using it less and less and more Racket.
    - peatmoss 9y ago
      Haha, yes, I keep encountering your comments about R and Racket and have to do a double take to make sure you’re not me. Would love to hear more about any data Racketeering you’re doing. I’ve been thinking of documenting up my wishlist for quanty things in Racket and maybe try implementing a data dsl in Racket.
      - baldfat 9y ago
        I am thinking of doing a munging and data cleaning dsl and using OpenRefine as a way to implement the code. I really think that would be super handy for doing data cleaning. A few people have done some R calls within Racket that is intresting.

by hasenj 9y ago

anewhnaccount2 9y ago

This is too categorical. And also it fails to define when small stops being small (and so is easy to defend by moving the goalposts). Yes, these are real problems, but some of them have (partial) solutions. You can use one of the many solutions to write a C, C++, cython and so on extension. You can have type checking with mypy. You can produce executable with cx_freeze (not compilation, but compilation is more of a solution than a problem). You can create parallel programs using multiprocessing. (etc.)

weberc2 9y ago
Writing extensions in c/c++ largely negates all stated benefits of using Python in the first place, and I'm proficient in all three languages. Further, mypy is hopeful, but it's not quite there yet (lots of holes and bugs--notably no support for recursive types). Lastly multiprocessing for parallelism is a poor man's solution, and far from what I've become accustomed to in Go, Rust, etc. Python is a nice language and it pays my bills, but Go beats it at many of its own purported strengths, in my opinion.
hasenj 9y ago
Using these solution makes the development process much more troublesome than it needs to be, thus defeating the whole point of using python.

ptero 9y ago

However, as posters in other thread said, the number and availability of libraries makes functionality of those small programs exceed that of much larger ones written in C or a similar language. If you have to write a large project from scratch in Python it will be slow, but Python has gone pretty far in the direction where for many large tasks 90% of the functionality will be obtainable through standard libraries. Doing the remaining 10% in basic Python often keeps it competitive at runtime with compiled languages but produce much simpler, shorter and easier to maintain code.

weberc2 9y ago
Python has great libraries, but the claim that you can use them and have code that competes with compiled languages is flat out wrong. CPython is at least an order of magnitude slower than Java/Go/Haskell/etc and two orders slower than C/C++/Rust/etc. PyPy closes the gap a bit, but it's still noticeably slower than the aforementioned compiled languages. Note to responders--HN won't let me respond to you, because "I'm posting too fast".
- acdha 9y ago
  Those statements are too broad to be anything other than flamebait. Which specific code did you benchmark? That’s not to say that CPython isn’t slower than a compiled language but in common usage the gaps usually end up being much smaller because most of us aren’t paid to run fib() and things like I/O dominate or code ends up being in the same C libraries (libjpeg, OpenSSL, etc.). I’ve seen multiple cases where Python beat C/Go because the app used something like dictionaries which was heavily optimized C code in the standard library. Again, compiled languages should usually win but the details matter, especially since most people don’t do the exact same work you do. Simply tossing around big numbers with no support contributes noise rather than insight to the conversation.
  - hasenj 9y ago
    > Which specific code did you benchmark? I had a csv file with ~16k entries. Each entry has a lat, lon, name and id. The task: given a (lat, lon) pair, list all entries within 5km. The simplest straight forward solution is to iterate the list, measure the distance of each item to the given point, and add it to the return list if the distance is within the desired range. No crazy optimization or anything. The results were something roughly like this: The D version runs in 2~4ms when compiled with optimizations turned on to the max, and about 6~8ms without optimizations (just default compiler settings). The Go version runs in about 6~8ms. (The Go compiler has no option to produce optimized code). The python version runs in about 50ms. That's 10x slower than Go and 20x slower than D. This is python 3. Using pypy shows absolutely no difference in performance. Obviously the time to load the data from the csv file into a list is not counted in the above results. > I’ve seen multiple cases where Python beat C/Go because the app used something like dictionaries which was heavily optimized C code in the standard library. That's exactly backwards. The Go and D versions use structs which are compact data structures that pack the data together. So the entire array fits in one contiguous block of memory. Python has no concept of structs, so you have to deal with dictionaries. In general, writing optimized code is not a good idea. You should almost always write the most plain and straight forward code. Because fast languages are fast by default, there's no optimization required. The straight forward solution is fast enough. If you are writing Python, your code will be slow by default. You will need to think about ways to optimize it if you want better performance.

dr_zoidberg 9y ago

Working in digital forensics, I read your comment and immediately thought of Volatility and Rekall, the most important memory forensics frameworks. Both monsters that can analize memory dumps from very different operating systems and microprocessor architectures, supporting a wealth of plugins/extensions to analyze every bit of information you can think of. Both written in Python. Please, tell me how they are failing.

datawarrior 9y ago
That's largely because Rekall is a fork of Volatility, not because Python was better suited to the task for both projects. Python is used heavily in security/forensics work so it was just a natural choice for the developers. I'm not sure what your argument is otherwise. If there was a tool written in C or Rust to benchmark against then we could better argue which is better for the task. I don't think anyone said Python was "failing."
- dr_zoidberg 9y ago
  My point is that both of them are big, succesful projects, written in Python, in contrast to hasenjs comment that anything that isn't "a small program being easy and quick (and pleasant) to write... fails in some way.".

bluedino 9y ago

Is it really slow? I ported a bunch of code from PHP to Python, nothing incredibly complicated, but it was easy to switch and the Python code all ended up running about 25-40% faster. I'm sure it would have been even faster to use Java or something like that but it would have taken 5 times as long, and I would have needed to know a lot more Java than Python.

weberc2 9y ago
Yeah, slow is relative. PHP is no one's standard for "fast". I program in Python by day, and I find Go much easier to work with, and it's roughly 10 times faster. There are other performant options besides Java these days.
- zerkten 9y ago
  What are you actually programming? You mention some languages, but you don't actually give any context.
  - deleted 9y ago
    [deleted]
- jack9 9y ago
  > PHP is no one's standard for "fast". PHP is my standard for "fast" in the realm of scripting. Just behind Perl. Python is slower in every prototype benchmark we have ever tried. It's a practical problem that hand waving cannot remedy.
  - hasenj 9y ago
    Fast would be something like Go or D or any of the other compiled languages.
    - jack9 9y ago
      I'm not sure how that relates. None of those are scripting languages, although when you start talking about "compiled" it gets a little messy with definitions. Either way, it's not what I'm talking about.

baldfat 9y ago

DEPLOYMENT My ONE HUGE Gripe with Python. This and that Pandas (I understand why BUT it drove me away) Zero based for statistical work. Your math is 1 based and the language is 0 based.

peatmoss 9y ago
It’s funny because deployment is one of the reasons I hear Python data people trash on R. In general, I’d agree with them that it’s slightly harder for systems people to grok R, but on the other hand I don’t exactly see Python as a shining beacon.
- baldfat 9y ago
  R as a Domain Specific Language has great output for Reports, Word Documents or Power Point slides and a million of reports. There is also the shinny package and shinny server for web publishing. This is why I went to R from Pandas. My company was impressd with all the reports but asked why they couldn't edit the themes. Then I did it in R in 2 days and haven't looked back. I love Python but I have been using it less and less and more Racket.
  - peatmoss 9y ago
    Haha, yes, I keep encountering your comments about R and Racket and have to do a double take to make sure you’re not me. Would love to hear more about any data Racketeering you’re doing. I’ve been thinking of documenting up my wishlist for quanty things in Racket and maybe try implementing a data dsl in Racket.
    - baldfat 9y ago
      I am thinking of doing a munging and data cleaning dsl and using OpenRefine as a way to implement the code. I really think that would be super handy for doing data cleaning. A few people have done some R calls within Racket that is intresting.