Which Python Memory Profiler Is Recommended

Which Python memory profiler is recommended?

guppy3 is quite simple to use. At some point in your code, you have to write the following:

from guppy import hpy
h = hpy()
print(h.heap())

This gives you some output like this:

Partition of a set of 132527 objects. Total size = 8301532 bytes.
Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
0  35144  27  2140412  26   2140412  26 str
1  38397  29  1309020  16   3449432  42 tuple
2    530   0   739856   9   4189288  50 dict (no owner)

You can also find out from where objects are referenced and get statistics about that, but somehow the docs on that are a bit sparse.

There is a graphical browser as well, written in Tk.

For Python 2.x, use Heapy.

Which PyPy memory profiler is recommended?

I only know about one profiler, which is provided by python community itself, memory profiler.

You can download it using either pip or easy_install,

$ easy_install -U memory_profiler # pip install -U memory_profiler

psutil is recommended module to have with this profiler.

If you want to install it from source,

$ python setup.py install

The latest sources can be found on https://github.com/fabianp/memory_profiler.

Documentation, samples, API help can be found on https://pypi.python.org/pypi/memory_profiler.

How do I profile memory usage in Python?

This one has been answered already here: Python memory profiler

Basically you do something like that (cited from Guppy-PE):

>>> from guppy import hpy; h=hpy()
>>> h.heap()
Partition of a set of 48477 objects. Total size = 3265516 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0  25773  53  1612820  49   1612820  49 str
     1  11699  24   483960  15   2096780  64 tuple
     2    174   0   241584   7   2338364  72 dict of module
     3   3478   7   222592   7   2560956  78 types.CodeType
     4   3296   7   184576   6   2745532  84 function
     5    401   1   175112   5   2920644  89 dict of class
     6    108   0    81888   3   3002532  92 dict (no owner)
     7    114   0    79632   2   3082164  94 dict of type
     8    117   0    51336   2   3133500  96 type
     9    667   1    24012   1   3157512  97 __builtin__.wrapper_descriptor
<76 more rows. Type e.g. '_.more' to view.>
>>> h.iso(1,[],{})
Partition of a set of 3 objects. Total size = 176 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0      1  33      136  77       136  77 dict (no owner)
     1      1  33       28  16       164  93 list
     2      1  33       12   7       176 100 int
>>> x=[]
>>> h.iso(x).sp
 0: h.Root.i0_modules['__main__'].__dict__['x']
>>> 

Trying to understand python memory profiler

Peak memory refers to the peak memory usage of your system (including memory usage of other processes) during the program runtime.

Increment is the increment in memory usage relative to the memory usage just before the program is run (i.e. increment = peak memory - starting memory).

So you'd report increment. Peak memory just helps you figure how close you are to using all your RAM during a program run.

If you refer to the line-by-line example in the usage section of the readme:

Line #    Mem usage  Increment   Line Contents
==============================================
     3                           @profile
     4      5.97 MB    0.00 MB   def my_func():
     5     13.61 MB    7.64 MB       a = [1] * (10 ** 6)
     6    166.20 MB  152.59 MB       b = [2] * (2 * 10 ** 7)
     7     13.61 MB -152.59 MB       del b
     8     13.61 MB    0.00 MB       return a

%memit is essentially giving you the memory usage from line 6, but reports the increment relative to line 4 (actually line 1, but presumably, there's no computation in lines 1-3).

python: will memory_profiler affect runtime?

It depends how you are using memory_profiler. This can be used in two different ways:

• To get memory usage line-by-line (run with python -m memory_profiler my_script.py). This needs to get memory information (from the OS) for every line executed within the profiled function. How this affects run-time depends on the amount of lines in the function: if it has a lot of lines with fast execution times, it might suppose a significant overhead. On the other hand, if the function to profile has few lines, and each lines has a significant computing time, then the overhead will be negligible.

• To get memory as a function of time (run with mprof run my_script.py and plot with mprof plot). In this case the function that collects the memory usage is in a different process as the one that runs your script, hence the overhead is minimal (unless you are using all CPUs).

Python memory_profiler: @profile not working on multithreading

The docs for memory_profiler : https://pypi.org/project/memory-profiler/ say the following if you use the decorator (@profile):

In this case the script can be run without specifying -m
memory_profiler in the command line.

So I think you just need to run python MyScript.py

---

Related Topics