8 ms·

I really really don't been to pile on Python... but every time I've had to interact with it I've been shocked at how slow it is compared with C or C++. I tend t

by new299 9y ago

I really really don't been to pile on Python... but every time I've had to interact with it I've been shocked at how slow it is compared with C or C++. I tend to write scientific code to process datasets in the range on 10Gb, for simple operations Python code can take hours as opposed to just taking minutes or seconds in C. I'm sure it's possible to write more highly optimized code in Python, but it never seems to be the case with the code bases I've worked with. With my own code (a few years ago now), I spent significant time optimizing and the final result was something that was still significantly slower than C/C++. Overall, for my work, I couldn't see an advantage. Am I just doing it wrong? Currently Go seems far more appealing to me and I've enjoyed using it (maybe I should also try Rust).

fspeech 9y ago
Have you tried Cython? That seems like a much smaller leap than a new language.
saghm 9y ago
Python is a high-level interpreted language, and C/C++ are low-level (even compared to other) compiled languages. While you might be able to optimize your Python code to run faster than it does now, it's never going to match the performance of C/C++, nor is it intended to. Go will be a significant speedup over Python, but likely won't quite match the speed of C/C++ for most tasks. Then again, the ease of development in Go will likely be noticeably better than in C or C++. Rust could potentially match the speed of C++, but it's a much more complex language than Go, so it will take some time to master. (Personally, having spent a large amount of time programming in Rust, I find myself more productive than in Go due to the powerful abstractions present in the former and lacking in the latter, but conventional wisdom is that this won't be the case for most people, and it certainly won't be when you're first learning Rust). EDIT: Anecdotally I've heard that D is a nice language, but I have no experience with it, so I can't comment on how productive it is or how fast D code will run.
- pjmlp 9y ago
  You can easily get best of both worlds with Lisp derived languages.
  - sythe2o0 9y ago
    It's been my understanding that Lisp and its relatives aren't designed to compete against C for speed either. Is this wrong?
    - pjmlp 9y ago
      C was designed on a PDP-11, a computer much more powerful than the mainframes where Lisp was already running since 10 years, so Lisp implementations were already quite good for most tasks by then. While C designers were trying to create UNIX at AT&T, Xerox PARC was busy creating Smalltalk, Interlisp-D and Mesa/Cedar. Other companies were creating Genera and the Connection Machine, In the end Worse is Better won, because the machines were too expensive for most pockets, normal developers could not grasp Lisp, mismanagement from those companies and UNIX was kind of cheap when comparing prices, with source code available almost for free (AT&T was prevented from selling it early on).
    - steinuil 9y ago
      Common Lisp has some very fast implementations (I think SBCL is the fastest) and it was designed to be a systems language. It can be optimized to be as fast as C.
      - rifung 9y ago
        > It can be optimized to be as fast as C Sorry if dumb question but how is this possible when it's garbage collected? Now I really feel like I need to learn a Lisp though.. Is there a certain Lisp you would recommend that's practical enough I'd use it for projects? Racket?
        pjmlp 9y ago
        You can start by reading about Genera and Interlisp-D https://en.wikipedia.org/wiki/Genera_(operating_system) https://en.wikipedia.org/wiki/Genera_(operating_system) https://www.ics.uci.edu/~andre/ics228s2006/teitelmanmasinter.pdf https://www.ics.uci.edu/~andre/ics228s2006/teitelmanmasinter... http://www.softwarepreservation.org/projects/LISP http://www.softwarepreservation.org/projects/LISP Just check the hardware capabilities of systems having Lisp 1.5 support, several years before C was created. Lisp is not only manipulating lists, it grew up to natively support all relevant data structures, including primitives to handle systems programming tasks. Regarding GC and systems programming, there are plenty of GC enabled systems programming languages, the main point is that they offer features to control the GC and also do stack and GC free allocation if required. Some examples would be Mesa/Cedar, Modula-2+, Modula-3, Oberon, Oberon-2, Oberon-07, Active Oberon, Component Pascal, Sing#, System C#, D
- 0xdeadbeefbabe 9y ago
  > Go will be a significant speedup over Python, but likely won't quite match the speed of C/C++ for most tasks. Then again, the ease of development in Go will likely be noticeably better than in C or C++. In my experience, you don't mean: learn to make C and C++ development just as easy as Go development and enjoy the best of both worlds. I agree with you, but it seems illogical on the surface.
- deleted 9y ago
  [deleted]
- adrianN 9y ago
  Javascript is also high-level interpreted language, but it's much faster than Python. Python is pretty hard to optimize in a JIT (and it gets much less attention than Javascript).
  - saghm 9y ago
    True, but it's also miles behind C/C++. While Python is arguably the slowest of the mainstream interpreted language, they're all orders of magnitude slower than C.
    - new299 9y ago
      Hmmm. I kind of expect them to be 1 or 2x slower. Maybe even a single order of magnitude... But Python seems often to be 100 or 1000x slower. For my applications this is kind of unfortunate.
- nindalf 9y ago
  > Go will be a significant speedup over Python, but likely won't quite match the speed of C/C++ for most tasks. Its true, Go won't match the speed of C, but it comes pretty damn close. Say you have 4 cores, 1 would be dedicated to GC and the other 3 would be executing your program. That sounds sub-optimal, meaning that a Go program could only be 75% as fast as an equivalent C program. But you have to wonder how many developers are capable of writing multi-threaded, correct C code. Go is simple to learn, lightning quick to compile and runs reasonably fast. By no means is it perfect, and it is not the tool I'd use for every task in front of me, but it is good enough for most tasks. I agree that Rust is the future in the low level space.
  - vardump 9y ago
    > Say you have 4 cores, 1 would be dedicated to GC In my opinion, idiomatic golang is not that GC heavy. A slice (a bit like ArrayList) of structs can be a contiguous chunk of memory as opposed to array of object references, which requires allocating each object separately. There are an order of magnitude less allocated objects than in Java for example. (Perhaps the situation will change once Java gains value types in maybe Java 9).
- vardump 9y ago
  > Go...but likely won't quite match the speed of C/C++ for most tasks. Last I checked golang generated code 3 years ago, it was full of redundant instructions, most notably redundant range checks. Seemed to be about half as fast as equivalent C/C++ code. I believe golang can approach something like 80-95% of C/C++ performance once codegen is good.
messe 9y ago
Python is interpreted, C and C++ are compiled. They target different niches. Python is more focused on ease of use than performance. Usually when writing scientific code in Python you're going to want to at the very least use numpy. It wraps various C functions for many time consuming tasks. To get the most out of numpy you're going to have to vectorize you're code, as python's looping constructs are notoriously slow. Instead of writing something like for i in range(0, len(v)): v[i] = w[i]**2 + w[i] You'd write something more along the lines of v[:] = w**2 + w
- pmoriarty 9y ago
  It is up to the implementation of a language whether that language is compiled or interpreted. For instance, there are C interpreters and Python compilers. Neither language is limited to being one or the other. It's just that usually people use C compilers and Python interpreters, but there's nothing inherent in those languages that limits them to those. It's all up to the implementation.
  - kibwen 9y ago
    This is true, but not always in a useful sense. There exist language features whose semantics are expressly dynamic, and if your language includes those features it will frustrate static compilation. For example, if your language includes `eval` (as Python does), then any precompiled binary would also need to include a complete Python interpreter. This is why efforts to compile Python (and other languages that are traditionally interpreted) often omit such features, and why languages that intend to be compiled tend to forgo them in the first place.
    - pjmlp 9y ago
      Which is why for me the best way is to be able to choose between JIT/AOT deployments, instead of a plain interpreter. Maybe one day PyPy will finally become the default option.
  - messe 9y ago
    > It is up to the implementation of a language whether that language is compiled or interpreted. While that's perfectly true, it didn't seem necessary to go into it.
jononor 9y ago
The trick to high-performance scientific calculations in Python is to use libraries like NumPy (possibly via Pandas), and their large set of vectorized operations. Then majority of the number crunching happens in optimized C/C++, with Python primarily 'orchestrating' these. For the cases where the data-manipulation functionality desired is missing and pure Python is problematic performance wise, one can use Cython to bridge the gap. Or if that is not enough, write those functions in C against the NumPy C API, and expose that as Python APIs.
ziotom78 9y ago
I am in a similar situation, as I develop a lot of scientific code. I've tested several solutions, and so far the most viable for me (though far from perfect) is Python+NumPy+Fortran, the latter exposed to Python using f2py (much easier than binding C/C++ to Python). Not sure if this might be a good solution for you, it depends whether the bottleneck in your code is in I/O or in raw calculations (in the former case you're not going to get any significant advantage from Fortran). I've tested several other languages that might be good for my purposes. From what I have seen, Rust is not ideal because of the acknowledged poor support for multi dimensional arrays [1] and the strange semantics for floats (due to the need to be «correct» in presence of NaNs). Two options that might become interesting once they gain a stable status are Julia [2] and Chapel [3]. The former might be exactly what you are you looking for; the latter is ok if you usually run your codes on HPC platforms. But be prepared to face some friction when sharing your work with colleagues or publishing papers: the little penetration of these two languages in the scientific world implies that people will face difficulties in using/understanding how your code works. [1] Look for «array» in the page https://users.rust-lang.org/t/on-rust-goals-in-2018-and-beyond/12336 https://users.rust-lang.org/t/on-rust-goals-in-2018-and-beyo... [2] https://julialang.org/ https://julialang.org/ [3] http://chapel.cray.com/ http://chapel.cray.com/
- zombinedev 9y ago
  D has quite good multi-dimensional array support, though probably not as flexible as Python. One of the best libraries in this area is ndslice (part of mir-algorithm) - http://docs.algorithm.dlang.io/latest/index.html http://docs.algorithm.dlang.io/latest/index.html. See the "Image Processing" example to get a sense of API - http://docs.algorithm.dlang.io/latest/mir_ndslice.html http://docs.algorithm.dlang.io/latest/mir_ndslice.html. mir-algorithm is the base for the high-performance glas implementation written in pure D that outperforms libraries with hand-written assembly like OpenBLAS in single-threaded mode (multi-threaded is not yet ready). See http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/glas-gemm-benchmark.html http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/.... (Note that this blog post is bit outdated - since then the author rewrote large parts of the library, IIRC.)
- nnq 9y ago
zombinedev 9y ago
If you're interested in high-performance scientific computing, then I guess you'll pleased to know that many use D for this domain, specifically for its high-performance and also high-level features that make prototyping as easy (if not easier) as in Python. In addition to good C/C++ interop, there are also libraries for interfacing with Python and R. Be sure to check http://blog.mir.dlang.io/ http://blog.mir.dlang.io/, https://github.com/kaleidicassociates/lubeck https://github.com/kaleidicassociates/lubeck, https://github.com/libmir https://github.com/libmir, https://github.com/DlangScience https://github.com/DlangScience and https://www.youtube.com/watch?v=edjrSDjkfko https://www.youtube.com/watch?v=edjrSDjkfko
vram22 9y ago
Have you tried out any of the options for speeding up Python apps, like Cython, writing parts that are in the perf bottleneck in C (if that is applicable), using ctypes / cffi to call C libraries for some tasks, etc.? You can use ctypes or cffi to call either existing libraries or ones you write.
deleted 9y ago
[deleted]
Recurecur 9y ago
I suggest taking a long, hard look at Julia. It's squarely aimed at high performance numerics and scientific programming, but is in fact a general purpose language with great potential. It's already very, very efficient (written using LLVM), yet has a lot of the clean elegance of Python.
lwitt51 9y ago
Try out the Numba package: https://numba.pydata.org/ https://numba.pydata.org/
Houshalter 9y ago
What everyone in this subthread is looking for is LuaJIT. It's the fastest JITed language. Comparable to C for many tasks. It has a very tiny memory footprint and integrates into C and C++ well. And there's torch if you need to do fast matrix math or machine learning.
- rurban 9y ago
  That's up to interpretation. Eg. Pony is faster and safer than LuaJIT or C for example.

by new299 9y ago

fspeech 9y ago

Have you tried Cython? That seems like a much smaller leap than a new language.

saghm 9y ago

Python is a high-level interpreted language, and C/C++ are low-level (even compared to other) compiled languages. While you might be able to optimize your Python code to run faster than it does now, it's never going to match the performance of C/C++, nor is it intended to. Go will be a significant speedup over Python, but likely won't quite match the speed of C/C++ for most tasks. Then again, the ease of development in Go will likely be noticeably better than in C or C++. Rust could potentially match the speed of C++, but it's a much more complex language than Go, so it will take some time to master. (Personally, having spent a large amount of time programming in Rust, I find myself more productive than in Go due to the powerful abstractions present in the former and lacking in the latter, but conventional wisdom is that this won't be the case for most people, and it certainly won't be when you're first learning Rust). EDIT: Anecdotally I've heard that D is a nice language, but I have no experience with it, so I can't comment on how productive it is or how fast D code will run.

pjmlp 9y ago
You can easily get best of both worlds with Lisp derived languages.
- sythe2o0 9y ago
  It's been my understanding that Lisp and its relatives aren't designed to compete against C for speed either. Is this wrong?
  - pjmlp 9y ago
    C was designed on a PDP-11, a computer much more powerful than the mainframes where Lisp was already running since 10 years, so Lisp implementations were already quite good for most tasks by then. While C designers were trying to create UNIX at AT&T, Xerox PARC was busy creating Smalltalk, Interlisp-D and Mesa/Cedar. Other companies were creating Genera and the Connection Machine, In the end Worse is Better won, because the machines were too expensive for most pockets, normal developers could not grasp Lisp, mismanagement from those companies and UNIX was kind of cheap when comparing prices, with source code available almost for free (AT&T was prevented from selling it early on).
  - steinuil 9y ago
    Common Lisp has some very fast implementations (I think SBCL is the fastest) and it was designed to be a systems language. It can be optimized to be as fast as C.
    - rifung 9y ago
      > It can be optimized to be as fast as C Sorry if dumb question but how is this possible when it's garbage collected? Now I really feel like I need to learn a Lisp though.. Is there a certain Lisp you would recommend that's practical enough I'd use it for projects? Racket?
      - pjmlp 9y ago
        You can start by reading about Genera and Interlisp-D https://en.wikipedia.org/wiki/Genera_(operating_system) https://en.wikipedia.org/wiki/Genera_(operating_system) https://www.ics.uci.edu/~andre/ics228s2006/teitelmanmasinter.pdf https://www.ics.uci.edu/~andre/ics228s2006/teitelmanmasinter... http://www.softwarepreservation.org/projects/LISP http://www.softwarepreservation.org/projects/LISP Just check the hardware capabilities of systems having Lisp 1.5 support, several years before C was created. Lisp is not only manipulating lists, it grew up to natively support all relevant data structures, including primitives to handle systems programming tasks. Regarding GC and systems programming, there are plenty of GC enabled systems programming languages, the main point is that they offer features to control the GC and also do stack and GC free allocation if required. Some examples would be Mesa/Cedar, Modula-2+, Modula-3, Oberon, Oberon-2, Oberon-07, Active Oberon, Component Pascal, Sing#, System C#, D
0xdeadbeefbabe 9y ago
> Go will be a significant speedup over Python, but likely won't quite match the speed of C/C++ for most tasks. Then again, the ease of development in Go will likely be noticeably better than in C or C++. In my experience, you don't mean: learn to make C and C++ development just as easy as Go development and enjoy the best of both worlds. I agree with you, but it seems illogical on the surface.
deleted 9y ago
[deleted]
adrianN 9y ago
Javascript is also high-level interpreted language, but it's much faster than Python. Python is pretty hard to optimize in a JIT (and it gets much less attention than Javascript).
- saghm 9y ago
  True, but it's also miles behind C/C++. While Python is arguably the slowest of the mainstream interpreted language, they're all orders of magnitude slower than C.
  - new299 9y ago
    Hmmm. I kind of expect them to be 1 or 2x slower. Maybe even a single order of magnitude... But Python seems often to be 100 or 1000x slower. For my applications this is kind of unfortunate.
nindalf 9y ago
> Go will be a significant speedup over Python, but likely won't quite match the speed of C/C++ for most tasks. Its true, Go won't match the speed of C, but it comes pretty damn close. Say you have 4 cores, 1 would be dedicated to GC and the other 3 would be executing your program. That sounds sub-optimal, meaning that a Go program could only be 75% as fast as an equivalent C program. But you have to wonder how many developers are capable of writing multi-threaded, correct C code. Go is simple to learn, lightning quick to compile and runs reasonably fast. By no means is it perfect, and it is not the tool I'd use for every task in front of me, but it is good enough for most tasks. I agree that Rust is the future in the low level space.
- vardump 9y ago
  > Say you have 4 cores, 1 would be dedicated to GC In my opinion, idiomatic golang is not that GC heavy. A slice (a bit like ArrayList) of structs can be a contiguous chunk of memory as opposed to array of object references, which requires allocating each object separately. There are an order of magnitude less allocated objects than in Java for example. (Perhaps the situation will change once Java gains value types in maybe Java 9).
vardump 9y ago
> Go...but likely won't quite match the speed of C/C++ for most tasks. Last I checked golang generated code 3 years ago, it was full of redundant instructions, most notably redundant range checks. Seemed to be about half as fast as equivalent C/C++ code. I believe golang can approach something like 80-95% of C/C++ performance once codegen is good.

messe 9y ago

Python is interpreted, C and C++ are compiled. They target different niches. Python is more focused on ease of use than performance. Usually when writing scientific code in Python you're going to want to at the very least use numpy. It wraps various C functions for many time consuming tasks. To get the most out of numpy you're going to have to vectorize you're code, as python's looping constructs are notoriously slow. Instead of writing something like for i in range(0, len(v)): v[i] = w[i]**2 + w[i] You'd write something more along the lines of v[:] = w**2 + w

pmoriarty 9y ago
It is up to the implementation of a language whether that language is compiled or interpreted. For instance, there are C interpreters and Python compilers. Neither language is limited to being one or the other. It's just that usually people use C compilers and Python interpreters, but there's nothing inherent in those languages that limits them to those. It's all up to the implementation.
- kibwen 9y ago
  This is true, but not always in a useful sense. There exist language features whose semantics are expressly dynamic, and if your language includes those features it will frustrate static compilation. For example, if your language includes `eval` (as Python does), then any precompiled binary would also need to include a complete Python interpreter. This is why efforts to compile Python (and other languages that are traditionally interpreted) often omit such features, and why languages that intend to be compiled tend to forgo them in the first place.
  - pjmlp 9y ago
    Which is why for me the best way is to be able to choose between JIT/AOT deployments, instead of a plain interpreter. Maybe one day PyPy will finally become the default option.
- messe 9y ago
  > It is up to the implementation of a language whether that language is compiled or interpreted. While that's perfectly true, it didn't seem necessary to go into it.

jononor 9y ago

The trick to high-performance scientific calculations in Python is to use libraries like NumPy (possibly via Pandas), and their large set of vectorized operations. Then majority of the number crunching happens in optimized C/C++, with Python primarily 'orchestrating' these. For the cases where the data-manipulation functionality desired is missing and pure Python is problematic performance wise, one can use Cython to bridge the gap. Or if that is not enough, write those functions in C against the NumPy C API, and expose that as Python APIs.

ziotom78 9y ago

I am in a similar situation, as I develop a lot of scientific code. I've tested several solutions, and so far the most viable for me (though far from perfect) is Python+NumPy+Fortran, the latter exposed to Python using f2py (much easier than binding C/C++ to Python). Not sure if this might be a good solution for you, it depends whether the bottleneck in your code is in I/O or in raw calculations (in the former case you're not going to get any significant advantage from Fortran). I've tested several other languages that might be good for my purposes. From what I have seen, Rust is not ideal because of the acknowledged poor support for multi dimensional arrays [1] and the strange semantics for floats (due to the need to be «correct» in presence of NaNs). Two options that might become interesting once they gain a stable status are Julia [2] and Chapel [3]. The former might be exactly what you are you looking for; the latter is ok if you usually run your codes on HPC platforms. But be prepared to face some friction when sharing your work with colleagues or publishing papers: the little penetration of these two languages in the scientific world implies that people will face difficulties in using/understanding how your code works. [1] Look for «array» in the page https://users.rust-lang.org/t/on-rust-goals-in-2018-and-beyond/12336 https://users.rust-lang.org/t/on-rust-goals-in-2018-and-beyo... [2] https://julialang.org/ https://julialang.org/ [3] http://chapel.cray.com/ http://chapel.cray.com/

zombinedev 9y ago
D has quite good multi-dimensional array support, though probably not as flexible as Python. One of the best libraries in this area is ndslice (part of mir-algorithm) - http://docs.algorithm.dlang.io/latest/index.html http://docs.algorithm.dlang.io/latest/index.html. See the "Image Processing" example to get a sense of API - http://docs.algorithm.dlang.io/latest/mir_ndslice.html http://docs.algorithm.dlang.io/latest/mir_ndslice.html. mir-algorithm is the base for the high-performance glas implementation written in pure D that outperforms libraries with hand-written assembly like OpenBLAS in single-threaded mode (multi-threaded is not yet ready). See http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/glas-gemm-benchmark.html http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/.... (Note that this blog post is bit outdated - since then the author rewrote large parts of the library, IIRC.)
nnq 9y ago

zombinedev 9y ago

If you're interested in high-performance scientific computing, then I guess you'll pleased to know that many use D for this domain, specifically for its high-performance and also high-level features that make prototyping as easy (if not easier) as in Python. In addition to good C/C++ interop, there are also libraries for interfacing with Python and R. Be sure to check http://blog.mir.dlang.io/ http://blog.mir.dlang.io/, https://github.com/kaleidicassociates/lubeck https://github.com/kaleidicassociates/lubeck, https://github.com/libmir https://github.com/libmir, https://github.com/DlangScience https://github.com/DlangScience and https://www.youtube.com/watch?v=edjrSDjkfko https://www.youtube.com/watch?v=edjrSDjkfko

vram22 9y ago

Have you tried out any of the options for speeding up Python apps, like Cython, writing parts that are in the perf bottleneck in C (if that is applicable), using ctypes / cffi to call C libraries for some tasks, etc.? You can use ctypes or cffi to call either existing libraries or ones you write.

deleted 9y ago

[deleted]

Recurecur 9y ago

I suggest taking a long, hard look at Julia. It's squarely aimed at high performance numerics and scientific programming, but is in fact a general purpose language with great potential. It's already very, very efficient (written using LLVM), yet has a lot of the clean elegance of Python.

lwitt51 9y ago

Try out the Numba package: https://numba.pydata.org/ https://numba.pydata.org/

Houshalter 9y ago

What everyone in this subthread is looking for is LuaJIT. It's the fastest JITed language. Comparable to C for many tasks. It has a very tiny memory footprint and integrates into C and C++ well. And there's torch if you need to do fast matrix math or machine learning.

rurban 9y ago
That's up to interpretation. Eg. Pony is faster and safer than LuaJIT or C for example.