4 ms·
I feel for you. Perhaps some of this may help: It sounds like you're mostly having trouble with OpenCV, which has a horrid API, especially the python version a
by makmanalp 10y ago
I feel for you. Perhaps some of this may help:
It sounds like you're mostly having trouble with OpenCV, which has a horrid API, especially the python version adds another layer of confusion on top of it, and dumping things back and forth from opencv Mat objects to numpy arrays sucks and wastes time. The documentation also is fairly bad. The error messages, like you say, kinda suck. So I can't help you much with that but know that the opencv python bindings fall WAY below the standard of what's considered good. Stuff like simplecv is a bit better but doesn't cover nearly as much as OpenCV does.
Numpy itself is great though, for how little code you often need for a complex, vectorized matrix operation. I've used stuff like Eigen (or even boost matrices or raw BLAS stuff) which I think is way more painful to deal with. For the annoying cruft that any of those bring, the minor speed boost over numpy isn't worth it for me most of the time.
The other thing is, when folks complain about the slowness of python when doing computationally heavy stuff like this, it's often the situation that they have a bunch of vectorized operations, and in the middle somewhere, they stick in something like a regular python for loop, which is orders of magnitude slower. The trick, especially for hot loops, is to just avoid using anything but vectorized operations. In something like C++ of course you're less likely to notice this happening because there is much less overhead, but there'd still be an impact of doing stuff the non-vectorized way anyway.
For profiling, take a look at this: https://www.huyng.com/posts/python-performance-analysis https://www.huyng.com/posts/python-performance-analysis plus the internal cprofile module that does call-graph profiling. It's nothing amazing like vtune or whatever, but it gets the job done.
As far as architecture goes - the relative lack of typing structure in python definitely makes life easier in many ways and cuts down on boilerplate, but it also makes it much easier to shoot yourself in the foot like you say. One thing to keep in mind is that when writing data heavy code there is an additional layer of concerns in play. Stuff like the "shape" of your data. It sounds like if you're running into this, what you want is to make a big diagram / graph of how data is flowing through your code, each stage of computation, and what the data looks like before and after. When you understand this well, it's much easier to reason about the code.
One thing that helps to know is that the style of programming in the python world is a bit different. In C++ land, the instinct is to start with building architecture, and when you inevitably realize that you've misunderstood the problem domain, you do a massive refactor. This usually constitutes large changes in how your code is structured but C++ gives you tools, like strong types, to do this rather easily. With python, unless you're setting of on a massive project, you start with minimal architecture. Write code that does the thing you need. Then, as you find yourself repeating yourself or doing unnecessary things, or dealing with complexity you don't need to know, only then do you start giving structure to your code bit by bit, as you need it. It's a different philosophy that's the result of a different set of functionalities.
In the larger context, I mostly have the opposite experience as you - I can pull out C++ whenever I need it, but I almost never do. The set of tools I have for computation an analysis in python are so much richer. For example, imagine c++ code for reading an excel sheet, and scraping some data off the web, cleaning both, joining them together, running a complex mathematical calculation, and saving a few plots. And I can do this from scratch in half an hour, around 100 lines of code. I also have an interactive shell that I can explore the data and the libraries I have, and I'm not stuck in the code-compile-code-compile loop. The productivity boost is amazing. In contrast, I don't even want to imagine what it would be like to attempt to do this in C++. Python is great glue. Admittedly, you're pushing at the edges of what python is useful for. It's definitely possible (I've written computer vision stuff with python before, and it worked OK), but if you're not making use of any of these benefits of using python, you might as well use C++ and enjoy the speed and strong typing.
Hope this helps your python experience be a bit smoother!