4 ms·

A Brief Intro to Profiling in Python

81 points by craigkerstiens 14y ago

CognitiveLens 14y ago
looks like the deck is down? Any status update?
- sn6uv 14y ago
  It's up again now
deleted 14y ago
[deleted]
Kaedon 14y ago
Definitely useful, I'll revisit the slides later when I'm looking at optimizing. I'm working through some of CPU overhead issues with Django now and wishing occasionally that I had chosen to use Sqlalchemy as my primary database tool. Django seemed like the easier solution at the time because of the steep learning curve of Sqlalchemy. I think it's worth learning, especially now that I've hit some edges in Django's ORM. I suppose it's good to know I can always use a crazy hybrid of the two libraries if needed.
sherjilozair 14y ago
No intro to profiling in Python is complete without any mention of line_profiler (http://pythonhosted.org/line_profiler/ http://pythonhosted.org/line_profiler/). As the name suggests, it gives you line-by-line profiling information. In a scripting language such as Python, I find this much more useful.
- pizza 14y ago
  Unfortunately, line_profiler doesn't have Python 3 support.
- bsimpson 14y ago
  I discovered line_profiler last week when I was trying to learn how to profile in Python. You show it your code, and it shows you what's slow - this is exactly what a profiler should be. I really want to start a campaign to get something similar built for JavaScript. JavaScript has the most diverse user base (from people who barely know how to use their computers all the way up to Googlers), yet our profiling tools can be intimidating/hard-to-decipher even for people who are quite comfortable with the rest of the language. My mental model for my code is my code. Showing me a data-grid with a tree of function calls (many of which may be in external libraries I don't control) isn't nearly as helpful as showing me, line-by-line, how fast my code runs. (This is especially true since not every line that takes time to run makes a function call.)
alexpopescu 14y ago
I don't think that using a client side generated id is actually a real step in optimizing. Except the case you block all writes to the db.
- wiredfool 14y ago
  What I wound up doing in a case like this was dump (using copy) into a temp table, then run a sql function on the data. That way, you get it done in as few round trips as possible, essentially 3 statements (create temp table, copy, select dostuff()). The database is real fast about doing what it needs to do when you give it one big block of stuff to work with. And, if you've got a serial column, you're going go have a primary key/uniqueness constraint on it. Retrying that when it fails at interesting times is going to cost far more time and brain cells than letting the db do it.
- mrgoldenbrown 14y ago
  I think it is interesting to have it in there for academic purposes, to see how fast you could go sticking with the high level ORM API. Once he drops down into the expression language, he goes back to generating the ids server side.
gshanmuganandh 14y ago
I found Yappi(http://code.google.com/p/yappi/ http://code.google.com/p/yappi/) helpful when i had to profile a multi-threaded application.
zanny 14y ago
I got real sad that a presentation in 2013 is being written against Python2 syntax still. I do admit I still have to go out of my way sometimes to find a complete p3 stack, but I've never failed to find the tools I needed in versions supporting 3 somewhere.
- Nursie 14y ago
  Unfortunately there are still a lot of bindings and third party libraries that need python2. Recently I've been playing with python-alsaaudio and python-cwiid on linux, and they both seem to be require python 2. It seems like it's not going to go away any time soon. I'd be surprised if most new work was even being done in python 3, even though it's been out for almost 5 years now.
  - andy_boot 14y ago
    I'm hoping that now that we have the major framework - Django supporting python 3 in Alpha (now) and fully (in v1.6) we'll finally see a slow move to python 3.
- lucian1900 14y ago
  It depends greatly on what one is doing. Py3 needlessly restricts what you can use, a particularly compelling thing being PyPy.

CognitiveLens 14y ago

looks like the deck is down? Any status update?

sn6uv 14y ago
It's up again now

deleted 14y ago

[deleted]

Kaedon 14y ago

Definitely useful, I'll revisit the slides later when I'm looking at optimizing. I'm working through some of CPU overhead issues with Django now and wishing occasionally that I had chosen to use Sqlalchemy as my primary database tool. Django seemed like the easier solution at the time because of the steep learning curve of Sqlalchemy. I think it's worth learning, especially now that I've hit some edges in Django's ORM. I suppose it's good to know I can always use a crazy hybrid of the two libraries if needed.

sherjilozair 14y ago

No intro to profiling in Python is complete without any mention of line_profiler (http://pythonhosted.org/line_profiler/ http://pythonhosted.org/line_profiler/). As the name suggests, it gives you line-by-line profiling information. In a scripting language such as Python, I find this much more useful.

pizza 14y ago
Unfortunately, line_profiler doesn't have Python 3 support.
bsimpson 14y ago
I discovered line_profiler last week when I was trying to learn how to profile in Python. You show it your code, and it shows you what's slow - this is exactly what a profiler should be. I really want to start a campaign to get something similar built for JavaScript. JavaScript has the most diverse user base (from people who barely know how to use their computers all the way up to Googlers), yet our profiling tools can be intimidating/hard-to-decipher even for people who are quite comfortable with the rest of the language. My mental model for my code is my code. Showing me a data-grid with a tree of function calls (many of which may be in external libraries I don't control) isn't nearly as helpful as showing me, line-by-line, how fast my code runs. (This is especially true since not every line that takes time to run makes a function call.)

alexpopescu 14y ago

I don't think that using a client side generated id is actually a real step in optimizing. Except the case you block all writes to the db.

wiredfool 14y ago
What I wound up doing in a case like this was dump (using copy) into a temp table, then run a sql function on the data. That way, you get it done in as few round trips as possible, essentially 3 statements (create temp table, copy, select dostuff()). The database is real fast about doing what it needs to do when you give it one big block of stuff to work with. And, if you've got a serial column, you're going go have a primary key/uniqueness constraint on it. Retrying that when it fails at interesting times is going to cost far more time and brain cells than letting the db do it.
mrgoldenbrown 14y ago
I think it is interesting to have it in there for academic purposes, to see how fast you could go sticking with the high level ORM API. Once he drops down into the expression language, he goes back to generating the ids server side.

gshanmuganandh 14y ago

I found Yappi(http://code.google.com/p/yappi/ http://code.google.com/p/yappi/) helpful when i had to profile a multi-threaded application.

zanny 14y ago

I got real sad that a presentation in 2013 is being written against Python2 syntax still. I do admit I still have to go out of my way sometimes to find a complete p3 stack, but I've never failed to find the tools I needed in versions supporting 3 somewhere.

Nursie 14y ago
Unfortunately there are still a lot of bindings and third party libraries that need python2. Recently I've been playing with python-alsaaudio and python-cwiid on linux, and they both seem to be require python 2. It seems like it's not going to go away any time soon. I'd be surprised if most new work was even being done in python 3, even though it's been out for almost 5 years now.
- andy_boot 14y ago
  I'm hoping that now that we have the major framework - Django supporting python 3 in Alpha (now) and fully (in v1.6) we'll finally see a slow move to python 3.
lucian1900 14y ago
It depends greatly on what one is doing. Py3 needlessly restricts what you can use, a particularly compelling thing being PyPy.