Is there any simple way to benchmark Python script?
Have a look at timeit, the python profiler and pycallgraph. Also make sure to have a look at the comment below by nikicc
mentioning "SnakeViz". It gives you yet another visualisation of profiling data which can be helpful.
timeit
def test():
"""Stupid test function"""
lst = []
for i in range(100):
lst.append(i)
if __name__ == '__main__':
import timeit
print(timeit.timeit("test()", setup="from __main__ import test"))
# For Python>=3.5 one can also write:
print(timeit.timeit("test()", globals=locals()))
Essentially, you can pass it python code as a string parameter, and it will run in the specified amount of times and prints the execution time. The important bits from the docs:
timeit.timeit(stmt='pass', setup='pass', timer=<default timer>, number=1000000, globals=None)
Create aTimer
instance with the given statement, setup
code and timer function and run itstimeit
method with
number executions. The optional globals argument specifies a namespace in which to execute the code.
... and:
Timer.timeit(number=1000000)
Time number executions of the main statement. This executes the setup
statement once, and then returns the time it takes to execute the main
statement a number of times, measured in seconds as a float.
The argument is the number of times through the loop, defaulting to one
million. The main statement, the setup statement and the timer function
to be used are passed to the constructor.Note:
By default,timeit
temporarily turns offgarbage collection
during the timing. The advantage of this approach is that
it makes independent timings more comparable. This disadvantage is
that GC may be an important component of the performance of the
function being measured. If so, GC can be re-enabled as the first
statement in the setup string. For example:
timeit.Timer('for i in xrange(10): oct(i)', 'gc.enable()').timeit()
Profiling
Profiling will give you a much more detailed idea about what's going on. Here's the "instant example" from the official docs:
import cProfile
import re
cProfile.run('re.compile("foo|bar")')
Which will give you:
197 function calls (192 primitive calls) in 0.002 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.001 0.001 <string>:1(<module>)
1 0.000 0.000 0.001 0.001 re.py:212(compile)
1 0.000 0.000 0.001 0.001 re.py:268(_compile)
1 0.000 0.000 0.000 0.000 sre_compile.py:172(_compile_charset)
1 0.000 0.000 0.000 0.000 sre_compile.py:201(_optimize_charset)
4 0.000 0.000 0.000 0.000 sre_compile.py:25(_identityfunction)
3/1 0.000 0.000 0.000 0.000 sre_compile.py:33(_compile)
Both of these modules should give you an idea about where to look for bottlenecks.
Also, to get to grips with the output of profile
, have a look at this post
pycallgraph
NOTE pycallgraph has been officially abandoned since Feb. 2018. As of Dec. 2020 it was still working on Python 3.6 though. As long as there are no core changes in how python exposes the profiling API it should remain a helpful tool though.
This module uses graphviz to create callgraphs like the following:
You can easily see which paths used up the most time by colour. You can either create them using the pycallgraph API, or using a packaged script:
pycallgraph graphviz -- ./mypythonscript.py
The overhead is quite considerable though. So for already long-running processes, creating the graph can take some time.
Benchmarking run times in Python
From the profile
docs that you linked to:
Note The profiler modules are designed to provide an execution profile for a given program, not for benchmarking purposes (for that, there is
timeit
for reasonably accurate results). This particularly applies to benchmarking Python code against C code: the profilers introduce overhead for Python code, but not for C-level functions, and so the C code would seem faster than any Python one.
So, no, you do not want to use profile
to benchmark your code. What you want to use profile
for is to figure out why your code is too slow, after you already know that it is.
And you do not want to output a timestamp before and after the function call, either. There are just way too many things you can get wrong that way if you're not careful (using the wrong timestamp function, letting the GC run a cycle collection in the middle of your test run, including test overhead in the loop timing, etc.), and timeit
takes care of all of that for you.
Something like this is a common way to benchmark things:
for impl in 'mycode', 'googlecode', 'thriftcode':
t = timeit.timeit('serialize(data)',
setup='''from {} import serialize;
with open('data.txt') as f: data=f.read()
'''.format(impl),
number=10000)
print('{}: {}'.format(impl, t)
(I'm assuming here that you can write three modules that wrap the three different serialization tools in the same API, a single serialize
function that takes a string and does something or other with it. Obviously there are different ways to organize things.)
How to benchmark a C program from a python script?
timeit
will measure the CPU time used by the Python process in which it runs. Execution time of external processes will not be "credited" to those times.
Test computer processing speed with a simple Python script
You can do any complex calculation problem in a loop:
- Calculate factorial for some big number (easy to implement)
- Calculate chain SHA1 hash 100 000 times (very easy to implement)
- Invert big matrix (no so easy to implement)
- ...
- etc.
Some of those problems use CPU (factorial, SHA1), some others - CPU and memory (matrix invert). So first you need to decide, which part of computer you want to benchmark.
Use Python Popen to benchmark a script?
I'd also just use the time
utility:
time python foo.py
You can then make a bash script to run this multiple times, recording the time taken for each run. Then, just average them with a shell utility or a Python script.
How do I profile a Python script?
Python includes a profiler called cProfile. It not only gives the total running time, but also times each function separately, and tells you how many times each function was called, making it easy to determine where you should make optimizations.
You can call it from within your code, or from the interpreter, like this:
import cProfile
cProfile.run('foo()')
Even more usefully, you can invoke the cProfile when running a script:
python -m cProfile myscript.py
To make it even easier, I made a little batch file called 'profile.bat':
python -m cProfile %1
So all I have to do is run:
profile euler048.py
And I get this:
1007 function calls in 0.061 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.061 0.061 <string>:1(<module>)
1000 0.051 0.000 0.051 0.000 euler048.py:2(<lambda>)
1 0.005 0.005 0.061 0.061 euler048.py:2(<module>)
1 0.000 0.000 0.061 0.061 {execfile}
1 0.002 0.002 0.053 0.053 {map}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler objects}
1 0.000 0.000 0.000 0.000 {range}
1 0.003 0.003 0.003 0.003 {sum}
EDIT: Updated link to a good video resource from PyCon 2013 titled
Python Profiling
Also via YouTube.
How can I time a code segment for testing performance with Pythons timeit?
You can use time.time()
or time.clock()
before and after the block you want to time.
import time
t0 = time.time()
code_block
t1 = time.time()
total = t1-t0
This method is not as exact as timeit
(it does not average several runs) but it is straightforward.
time.time()
(in Windows and Linux) and time.clock()
(in Linux) are not precise enough for fast functions (you get total = 0). In this case or if you want to average the time elapsed by several runs, you have to manually call the function multiple times (As I think you already do in you example code and timeit does automatically when you set its number argument)
import time
def myfast():
code
n = 10000
t0 = time.time()
for i in range(n): myfast()
t1 = time.time()
total_n = t1-t0
In Windows, as Corey stated in the comment, time.clock()
has much higher precision (microsecond instead of second) and is preferred over time.time()
.
Related Topics
Ssl Insecureplatform Error When Using Requests Package
Too Many Different Python Versions on My System and Causing Problems
Pipe Subprocess Standard Output to a Variable
Numpy: Find First Index of Value Fast
Configuring So That Pip Install Can Work from Github
How to Manage Third-Party Python Libraries with Google App Engine? (Virtualenv? Pip)
How to Parse a Website Using Selenium and Beautifulsoup in Python
How to Update the Image of a Tkinter Label Widget
Where to Put Freeze_Support() in a Python Script
Force Numpy Ndarray to Take Ownership of Its Memory in Cython
Why Not Generate the Secret Key Every Time Flask Starts
Is There Any Simple Way to Benchmark Python Script
How to Filter a Date of a Datetimefield in Django