9 ms·
The better Python market isn't an easy one to crack because its a bit crowded. Go (despite its perceived and real faults) has succeeded in this space by deliver
by nindalf 9y ago
The better Python market isn't an easy one to crack because its a bit crowded. Go (despite its perceived and real faults) has succeeded in this space by delivering better GC, good libraries, static typing and faster programs. Python itself is improving rapidly, for example with the addition of Type Hints. Its pretty difficult to be a better Python in 2017.
The better C market, on the other hand, hasn't seen any real contender other than C++ gain traction for decades. Rust is trying now and it could get there, but its still a massive opportunity.
- pjmlp 9y agoAnd on embedded even C++ hardly managed to overtake C. https://youtu.be/D7Sd8A6_fYU https://youtu.be/D7Sd8A6_fYU
- deleted 9y ago[deleted]
- new299 9y agoI really really don't been to pile on Python... but every time I've had to interact with it I've been shocked at how slow it is compared with C or C++. I tend to write scientific code to process datasets in the range on 10Gb, for simple operations Python code can take hours as opposed to just taking minutes or seconds in C. I'm sure it's possible to write more highly optimized code in Python, but it never seems to be the case with the code bases I've worked with. With my own code (a few years ago now), I spent significant time optimizing and the final result was something that was still significantly slower than C/C++. Overall, for my work, I couldn't see an advantage. Am I just doing it wrong? Currently Go seems far more appealing to me and I've enjoyed using it (maybe I should also try Rust).
- fspeech 9y agoHave you tried Cython? That seems like a much smaller leap than a new language.
- saghm 9y agoPython is a high-level interpreted language, and C/C++ are low-level (even compared to other) compiled languages. While you might be able to optimize your Python code to run faster than it does now, it's never going to match the performance of C/C++, nor is it intended to. Go will be a significant speedup over Python, but likely won't quite match the speed of C/C++ for most tasks. Then again, the ease of development in Go will likely be noticeably better than in C or C++. Rust could potentially match the speed of C++, but it's a much more complex language than Go, so it will take some time to master. (Personally, having spent a large amount of time programming in Rust, I find myself more productive than in Go due to the powerful abstractions present in the former and lacking in the latter, but conventional wisdom is that this won't be the case for most people, and it certainly won't be when you're first learning Rust). EDIT: Anecdotally I've heard that D is a nice language, but I have no experience with it, so I can't comment on how productive it is or how fast D code will run.
- pjmlp 9y agoYou can easily get best of both worlds with Lisp derived languages.
- sythe2o0 9y agoIt's been my understanding that Lisp and its relatives aren't designed to compete against C for speed either. Is this wrong?
- pjmlp 9y agoC was designed on a PDP-11, a computer much more powerful than the mainframes where Lisp was already running since 10 years, so Lisp implementations were already quite good for most tasks by then. While C designers were trying to create UNIX at AT&T, Xerox PARC was busy creating Smalltalk, Interlisp-D and Mesa/Cedar. Other companies were creating Genera and the Connection Machine, In the end Worse is Better won, because the machines were too expensive for most pockets, normal developers could not grasp Lisp, mismanagement from those companies and UNIX was kind of cheap when comparing prices, with source code available almost for free (AT&T was prevented from selling it early on).
- messe 9y agoPython is interpreted, C and C++ are compiled. They target different niches. Python is more focused on ease of use than performance. Usually when writing scientific code in Python you're going to want to at the very least use numpy. It wraps various C functions for many time consuming tasks. To get the most out of numpy you're going to have to vectorize you're code, as python's looping constructs are notoriously slow. Instead of writing something like for i in range(0, len(v)): v[i] = w[i]**2 + w[i] You'd write something more along the lines of v[:] = w**2 + w
- pmoriarty 9y agoIt is up to the implementation of a language whether that language is compiled or interpreted. For instance, there are C interpreters and Python compilers. Neither language is limited to being one or the other. It's just that usually people use C compilers and Python interpreters, but there's nothing inherent in those languages that limits them to those. It's all up to the implementation.
- kibwen 9y agoThis is true, but not always in a useful sense. There exist language features whose semantics are expressly dynamic, and if your language includes those features it will frustrate static compilation. For example, if your language includes `eval` (as Python does), then any precompiled binary would also need to include a complete Python interpreter. This is why efforts to compile Python (and other languages that are traditionally interpreted) often omit such features, and why languages that intend to be compiled tend to forgo them in the first place.
- pjmlp 9y agoWhich is why for me the best way is to be able to choose between JIT/AOT deployments, instead of a plain interpreter. Maybe one day PyPy will finally become the default option.
- messe 9y ago> It is up to the implementation of a language whether that language is compiled or interpreted. While that's perfectly true, it didn't seem necessary to go into it.
- jononor 9y agoThe trick to high-performance scientific calculations in Python is to use libraries like NumPy (possibly via Pandas), and their large set of vectorized operations. Then majority of the number crunching happens in optimized C/C++, with Python primarily 'orchestrating' these. For the cases where the data-manipulation functionality desired is missing and pure Python is problematic performance wise, one can use Cython to bridge the gap. Or if that is not enough, write those functions in C against the NumPy C API, and expose that as Python APIs.
- ziotom78 9y agoI am in a similar situation, as I develop a lot of scientific code. I've tested several solutions, and so far the most viable for me (though far from perfect) is Python+NumPy+Fortran, the latter exposed to Python using f2py (much easier than binding C/C++ to Python). Not sure if this might be a good solution for you, it depends whether the bottleneck in your code is in I/O or in raw calculations (in the former case you're not going to get any significant advantage from Fortran). I've tested several other languages that might be good for my purposes. From what I have seen, Rust is not ideal because of the acknowledged poor support for multi dimensional arrays [1] and the strange semantics for floats (due to the need to be «correct» in presence of NaNs). Two options that might become interesting once they gain a stable status are Julia [2] and Chapel [3]. The former might be exactly what you are you looking for; the latter is ok if you usually run your codes on HPC platforms. But be prepared to face some friction when sharing your work with colleagues or publishing papers: the little penetration of these two languages in the scientific world implies that people will face difficulties in using/understanding how your code works. [1] Look for «array» in the page https://users.rust-lang.org/t/on-rust-goals-in-2018-and-beyond/12336 https://users.rust-lang.org/t/on-rust-goals-in-2018-and-beyo... [2] https://julialang.org/ https://julialang.org/ [3] http://chapel.cray.com/ http://chapel.cray.com/
- zombinedev 9y agoD has quite good multi-dimensional array support, though probably not as flexible as Python. One of the best libraries in this area is ndslice (part of mir-algorithm) - http://docs.algorithm.dlang.io/latest/index.html http://docs.algorithm.dlang.io/latest/index.html. See the "Image Processing" example to get a sense of API - http://docs.algorithm.dlang.io/latest/mir_ndslice.html http://docs.algorithm.dlang.io/latest/mir_ndslice.html. mir-algorithm is the base for the high-performance glas implementation written in pure D that outperforms libraries with hand-written assembly like OpenBLAS in single-threaded mode (multi-threaded is not yet ready). See http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/glas-gemm-benchmark.html http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/.... (Note that this blog post is bit outdated - since then the author rewrote large parts of the library, IIRC.)
- nnq 9y ago
- zombinedev 9y agoIf you're interested in high-performance scientific computing, then I guess you'll pleased to know that many use D for this domain, specifically for its high-performance and also high-level features that make prototyping as easy (if not easier) as in Python. In addition to good C/C++ interop, there are also libraries for interfacing with Python and R. Be sure to check http://blog.mir.dlang.io/ http://blog.mir.dlang.io/, https://github.com/kaleidicassociates/lubeck https://github.com/kaleidicassociates/lubeck, https://github.com/libmir https://github.com/libmir, https://github.com/DlangScience https://github.com/DlangScience and https://www.youtube.com/watch?v=edjrSDjkfko https://www.youtube.com/watch?v=edjrSDjkfko
- vram22 9y agoHave you tried out any of the options for speeding up Python apps, like Cython, writing parts that are in the perf bottleneck in C (if that is applicable), using ctypes / cffi to call C libraries for some tasks, etc.? You can use ctypes or cffi to call either existing libraries or ones you write.
- deleted 9y ago[deleted]
- Recurecur 9y agoI suggest taking a long, hard look at Julia. It's squarely aimed at high performance numerics and scientific programming, but is in fact a general purpose language with great potential. It's already very, very efficient (written using LLVM), yet has a lot of the clean elegance of Python.
- lwitt51 9y agoTry out the Numba package: https://numba.pydata.org/ https://numba.pydata.org/
- Houshalter 9y agoWhat everyone in this subthread is looking for is LuaJIT. It's the fastest JITed language. Comparable to C for many tasks. It has a very tiny memory footprint and integrates into C and C++ well. And there's torch if you need to do fast matrix math or machine learning.
- rurban 9y agoThat's up to interpretation. Eg. Pony is faster and safer than LuaJIT or C for example.
- saghm 9y agoAccording to their respective Wikipedia articles, D is 8 years older than Go. Given that you're saying that Go had a lot of success breaking into the Python market, why couldn't D have instead if they had put their efforts there from the beginning?
- mattnewton 9y agoI suspect a large part of go's initial success are the names attached to it, both Google and the team have great reputations. Go's continued success is a credit to the language, but maybe D has had some discoverability issues?
- vram22 9y ago>According to their respective Wikipedia articles, D is 8 years older than Go. And Go itself is some years old now (created in either 2007 or 2009 - according to its Wikipedia article - maybe they mean initial creation and first release for public use, by those two year values). So Go is about either 8 or 10 years old. And that makes D either 16 or 18 years old. >Go had a lot of success breaking into the Python market, why couldn't D have instead if they had put their efforts there from the beginning? So when D was new or just a few years old (13 to 16 years ago, say), the Python market was a lot smaller that it is today or has been for the last few years. It might not have even been a target for them, for that or any other reason - another reason could be that Walter is from a systems and compiler background, so might not have been too interested in the domain of interpreted languages (just guessing here, maybe he will comment on this). But BTW, D can be used almost like a scripting language due to its speed of compilation. See rdmd command and the "Why D?" and "D as a scripting language" articles on the Infognition blog (a google away).
- WalterBright 9y agoI've always been attracted to systems programming, and it's what I know best. Hence D is angled that way.
- vram22 9y ago
- vram22 9y ago>Go (despite its perceived and real faults) has succeeded in this space by delivering better GC, good libraries, static typing and faster programs. Also fast compilation, which D also has.