How to Use Valgrind with Python

How to use valgrind with python?

I found the answer here.

Python also needs to be compiled in debug mode, i.e.

./configure --prefix=/home/dejan/workspace/python --without-pymalloc --with-pydebug --with-valgrind

In addition, numpy has a suppresion file that gets rid of the extra valgrind warnings.

Python memory leaks?

There's a whole README.valgrind in the Python sources that explains the various caveats trying to use Valgrind with Python:

http://svn.python.org/projects/python/trunk/Misc/README.valgrind

Python uses its own small-object allocation scheme on top of malloc,
called PyMalloc.

Valgrind may show some unexpected results when PyMalloc is used.
Starting with Python 2.3, PyMalloc is used by default. You can disable
PyMalloc when configuring python by adding the --without-pymalloc option.
If you disable PyMalloc, most of the information in this document and
the supplied suppressions file will not be useful. As discussed above,
disabling PyMalloc can catch more problems.

If you use valgrind on a default build of Python, you will see
many errors like:

==6399== Use of uninitialised value of size 4
==6399== at 0x4A9BDE7E: PyObject_Free (obmalloc.c:711)
==6399== by 0x4A9B8198: dictresize (dictobject.c:477)

These are expected and not a problem.

Is it normal that running python under valgrind shows many errors with memory?

You could try using the suppression file that comes with the python source

Reading the Python Valgrind README is a good idea too!

Valgrind error and memory leaks with Python/C API

I initially suggested this as a duplicate. I don't think that's the case any more, but it still gives useful advice on removing false positives from the Valgrind output for Python programs.

Beyond that there are a number of specific issues with your program:

  • No error checking - the Python C API typically uses a NULL return value to indicate an error. There are clearly various ways of writing error checking code, but I'd be tempted to go for something like

    IA::IA() :
    pModule(NULL), pDict(NULL), pFunc(NULL), pName(NULL) // initially null initialize everything
    {
    setenv("PYTHONPATH",".",1);
    Py_Initialize();

    // I don't think you actually use this line, so maybe remove it
    if (PyRun_SimpleString("import sys") == -1) goto error;

    pName = PyBytes_FromString((char*)"Test");
    if (!pName) goto error;
    pModule = PyImport_Import(pName);
    if (!pModule) goto error;
    pDict = PyModule_GetDict(pModule);
    if (!pDict) goto error;
    pFunc = PyDict_GetItemString(pDict, "push_f");
    if (!pFunc) goto error;

    return; // completed OK

    error:
    Py_XDECREF(pName); // XDECREF is OK with NULL...
    Py_XDECREF(pModule);
    Py_XDECREF(pDict);
    Py_XDECREF(pFunc);
    PyErr_Print();
    throw std::runtime_error(""); // ??? - possibly get and use the error string
    // see https://stackoverflow.com/a/1418703/4657412
    }

    I'm aware people are suspicious of goto but in this case it's a reasonably clean way of jumping to an error handling block. You can structure it differently if you like.

  • The destructor doesn't decref pFunc, which looks like a memory leak.

  • IA::LaunchIA similarly is lacking working error checking.

  • IA::LaunchIA never decrefs toSend, pResult, TlistMob, TlistPlayer, pDPosIA. Some of these are related to the incompleteness of the code you show, but if they aren't decrefed then you're leaking memory.

  • You call PyErr_Print() without checking for an error. The documentation says:

    Call this function only when the error indicator is set. (Otherwise it will cause a fatal error!)

    I suspect that this might be your biggest problem.


Those are all the issues I can see. Without a minimal complete example it's impossible to actually check what you're seeing. Given you're using C++ you might be well advised to use/make a decent, object oriented wrapper for PyObject* to avoid having to worry about reference counting yourself - Boost Python has one.

Python: Ctypes how to check memory management

If you want to use Valgrind, then this readme might be helpful. Probably, this could be another good resource to make Valgrind friendly python and use it in your program.

But if you consider something else like tracemalloc, then you can easily get some example usage of it here. The examples are pretty easy to interpret. For example according to their doc,

  import tracemalloc
tracemalloc.start()

# ... run your application ...
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')
print("[ Top 10 ]")
for stat in top_stats[:10]:
print(stat)

This will output something like.

 <frozen importlib._bootstrap>:716: size=4855 KiB, count=39328, average=126 B
<frozen importlib._bootstrap>:284: size=521 KiB, count=3199, average=167 >

You can either parse this to plot memory usage for your investigation or you may
use the reference doc to get a more concrete idea.

In this case your program could be something like the following:

 from tkinter import *
import tracemalloc
root = Tk() # New GUI
# some code here

def destructorMethods:
tracemalloc.start()
myFunctions.destructorLinkedList() # Destructor method of my allocated memory in my C file
# Here is where I would want to run a Valgrind/Memory management check before closing
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')
print("[ Top 10 ]")
for stat in top_stats[:10]:
print(stat)

root.destroy() # close the program

root.protocol("WM_DELETE_WINDOW", destructorMethods)

Another option is, you can use a memory profiler to see memory usage at a variable time. The package is available here. After the installation of this package, you can probably use the following command in your script to get the memory usage over time in a png file.

 mprof run --include-children python your_filename.py
mprof plot --output timelyplot.png

or you may use different functions available on memory_profiler package according to your need. Maybe this tutorial can be an interesting one for you.



Related Topics



Leave a reply



Submit