5 ms·
Profiling Python with cProfile
- RogerL 13y agoYou'll perhaps take this as snark, but I am not going to read light gray text on a white background with a poorly rendered font (windows 7, firefox 27, 47yo eyes).
- nwenzel 13y agoFor less snark and more helpfulness, consider removing "I am not going to read". I think you'll be happier and less likely to be perceived as snarky.
- ymichael 13y agoAhh sorry about that. I'm like the most terrible UI person. ever... thanks for the feedback. made the text darker.
- deleted 13y ago[deleted]
- mdaniel 13y agoFWIW, I've had good luck with using the "select all" keyboard shortcut in those circumstances. I believe that you have an additional advantage in that Firefox has an option(1) to deactivate their stylesheets (one at a time, IIRC) which can also get you back into normal styling without a great deal of effort. 1= or it used to but without it in front of me I can't promise it still does
- nwenzel 13y agoVery helpful to have a UI for digging through profiler results. I use the django debug toolbar but I think your profiler will be incredibly helpful for finding bottlenecks. Thanks for posting, thanks for sharing.
- jbaiter 13y agoThen you might really enjoy runsnakerun: http://www.vrplumber.com/programming/runsnakerun/ http://www.vrplumber.com/programming/runsnakerun/
- tgb 13y agoKernprof and lineprof are also helpful.
- pekk 13y agoI'm glad that there wasn't a "complain about the GIL" or "write everything in Go because Python is too slow" step in this optimization workflow
- habitue 13y agoI'd say you had it right the first time with the list comprehension. List comprehensions in python are way faster than while/append loops because they're implemented in C. The issue is that you had the "not in" which is O(n) over a list (making the comprehension O(n^2) ). If the documents are hashable, I'd suggest making results a set. "not in" is constant time on a set. If a document is something not hashable, you can make results a dictionary and get the same constant time access. In python function/method calls are expensive, so avoid them at all costs inside tight loops.
- m_ke 13y agoThumbs up to this guy. Function calls in tight loops are a killer. Property and global lookups are also "much" less efficient than stuff in the local scope.
- fijal 13y agoer. "as long as you use CPython", yes. If you use PyPy (which you really should for such an example), there are no such restrictions.
- js2 13y agoIf a document is something not hashable, you can make results a dictionary Huh? Dictionary keys have to be hashable. Oh, maybe you mean coming up with a hashable key for each result, then just taking the values at the end. Something like: # A set of all the documents all_docs = dict(doc.key, doc for doc in dictionary.all_docs()) # A set of the documents matching `operand` results = set(doc.key for doc in get_results(operand, dictionary, pfile, force_list=True)) return [all_docs[key] for key in all_docs if key not in results] (This loses the "sorted" property the author has in the original comments. If that's important, just make all_docs an OrderedDict.)
- habitue 13y agoYeah, that was what I was talking about. Without typing up an example its cumbersome to explain. But the idea is use a constant time access data structure. The easiest would be a set, but if (as I suspect) the documents are dicts, you'd need to lookup by a hashable key. Thanks for giving the example