How to Profile a Python Script

How do I profile a Python script?

Python includes a profiler called cProfile. It not only gives the total running time, but also times each function separately, and tells you how many times each function was called, making it easy to determine where you should make optimizations.

You can call it from within your code, or from the interpreter, like this:

import cProfile
cProfile.run('foo()')

Even more usefully, you can invoke the cProfile when running a script:

python -m cProfile myscript.py

To make it even easier, I made a little batch file called 'profile.bat':

python -m cProfile %1

So all I have to do is run:

profile euler048.py

And I get this:

1007 function calls in 0.061 CPU seconds

Ordered by: standard name
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    1    0.000    0.000    0.061    0.061 <string>:1(<module>)
 1000    0.051    0.000    0.051    0.000 euler048.py:2(<lambda>)
    1    0.005    0.005    0.061    0.061 euler048.py:2(<module>)
    1    0.000    0.000    0.061    0.061 {execfile}
    1    0.002    0.002    0.053    0.053 {map}
    1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler objects}
    1    0.000    0.000    0.000    0.000 {range}
    1    0.003    0.003    0.003    0.003 {sum}

EDIT: Updated link to a good video resource from PyCon 2013 titled
Python Profiling

Also via YouTube.

How can I profile Python code line-by-line?

I believe that's what Robert Kern's line_profiler is intended for. From the link:

File: pystone.py
Function: Proc2 at line 149
Total time: 0.606656 s

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   149                                           @profile
   150                                           def Proc2(IntParIO):
   151     50000        82003      1.6     13.5      IntLoc = IntParIO + 10
   152     50000        63162      1.3     10.4      while 1:
   153     50000        69065      1.4     11.4          if Char1Glob == 'A':
   154     50000        66354      1.3     10.9              IntLoc = IntLoc - 1
   155     50000        67263      1.3     11.1              IntParIO = IntLoc - IntGlob
   156     50000        65494      1.3     10.8              EnumLoc = Ident1
   157     50000        68001      1.4     11.2          if EnumLoc == Ident1:
   158     50000        63739      1.3     10.5              break
   159     50000        61575      1.2     10.1      return IntParIO

profiling simple python script

You are doing 1000x more work in foo than in bar.

Assuming they had the same speed, 0.136 / 1000 = 0.000136 which is too small of a number for this display, and bar()'s time just rounds to 0.00.

Profile a python script using cProfile into an external file

The output file generated by the cProfile -o module isn't plaintext; it's a serialized pstats.Stats object. Rather than using the -o option, I generally just redirect stdout into the file I want to create.

python -m cProfile -s time my_script.py > profile.text 2>&1

Otherwise, you just need to use the pstats module to read the file and inspect its contents (see the documentation linked above).

How to profile my code?

The standard answer to this question is to use cProfile.

You'll find though that without having your code separated out into methods that cProfile won't give you particularly rich information.

Instead, you might like to try what another poster here calls Monte Carlo Profiling. To quote from another answer:

If you're in a hurry and you can
manually interrupt your program under
the debugger while it's being
subjectively slow, there's a simple
way to find performance problems.

Just halt it several times, and each
time look at the call stack. If there
is some code that is wasting some
percentage of the time, 20% or 50% or
whatever, that is the probability that
you will catch it in the act on each
sample. So that is roughly the
percentage of samples on which you
will see it. There is no educated
guesswork required. If you do have a
guess as to what the problem is, this
will prove or disprove it.

You may have multiple performance
problems of different sizes. If you
clean out any one of them, the
remaining ones will take a larger
percentage, and be easier to spot, on
subsequent passes.

Caveat: programmers tend to be
skeptical of this technique unless
they've used it themselves. They will
say that profilers give you this
information, but that is only true if
they sample the entire call stack.
Call graphs don't give you the same
information, because 1) they don't
summarize at the instruction level,
and 2) they give confusing summaries
in the presence of recursion. They
will also say it only works on toy
programs, when actually it works on
any program, and it seems to work
better on bigger programs, because
they tend to have more problems to
find [emphasis added].

It's not orthodox, but I've used it very successfully in a project where profiling using cProfile was not giving me useful output.

The best thing about it is that this is dead easy to do in Python. Simply run your Python script in the interpreter, press [Control-C], note the traceback and repeat a number of times.

How i can profile a python script consisting of multiple modules and classes?

pycallgraph is a beautiful tool for profiling python code http://pycallgraph.slowchop.com/en/master/