How to use valgrind with python?
I found the answer here.
Python also needs to be compiled in debug mode, i.e.
./configure --prefix=/home/dejan/workspace/python --without-pymalloc --with-pydebug --with-valgrind
In addition, numpy has a suppresion file that gets rid of the extra valgrind warnings.
Python memory leaks?
There's a whole README.valgrind in the Python sources that explains the various caveats trying to use Valgrind with Python:
http://svn.python.org/projects/python/trunk/Misc/README.valgrind
Python uses its own small-object allocation scheme on top of malloc,
called PyMalloc.
Valgrind may show some unexpected results when PyMalloc is used.
Starting with Python 2.3, PyMalloc is used by default. You can disable
PyMalloc when configuring python by adding the --without-pymalloc option.
If you disable PyMalloc, most of the information in this document and
the supplied suppressions file will not be useful. As discussed above,
disabling PyMalloc can catch more problems.
If you use valgrind on a default build of Python, you will see
many errors like:
==6399== Use of uninitialised value of size 4
==6399== at 0x4A9BDE7E: PyObject_Free (obmalloc.c:711)
==6399== by 0x4A9B8198: dictresize (dictobject.c:477)
These are expected and not a problem.
Is it normal that running python under valgrind shows many errors with memory?
You could try using the suppression file that comes with the python source
Reading the Python Valgrind README is a good idea too!
Valgrind error and memory leaks with Python/C API
I initially suggested this as a duplicate. I don't think that's the case any more, but it still gives useful advice on removing false positives from the Valgrind output for Python programs.
Beyond that there are a number of specific issues with your program:
No error checking - the Python C API typically uses a NULL return value to indicate an error. There are clearly various ways of writing error checking code, but I'd be tempted to go for something like
IA::IA() :
pModule(NULL), pDict(NULL), pFunc(NULL), pName(NULL) // initially null initialize everything
{
setenv("PYTHONPATH",".",1);
Py_Initialize();
// I don't think you actually use this line, so maybe remove it
if (PyRun_SimpleString("import sys") == -1) goto error;
pName = PyBytes_FromString((char*)"Test");
if (!pName) goto error;
pModule = PyImport_Import(pName);
if (!pModule) goto error;
pDict = PyModule_GetDict(pModule);
if (!pDict) goto error;
pFunc = PyDict_GetItemString(pDict, "push_f");
if (!pFunc) goto error;
return; // completed OK
error:
Py_XDECREF(pName); // XDECREF is OK with NULL...
Py_XDECREF(pModule);
Py_XDECREF(pDict);
Py_XDECREF(pFunc);
PyErr_Print();
throw std::runtime_error(""); // ??? - possibly get and use the error string
// see https://stackoverflow.com/a/1418703/4657412
}I'm aware people are suspicious of
goto
but in this case it's a reasonably clean way of jumping to an error handling block. You can structure it differently if you like.The destructor doesn't decref
pFunc
, which looks like a memory leak.IA::LaunchIA
similarly is lacking working error checking.IA::LaunchIA
never decrefstoSend
,pResult
,TlistMob
,TlistPlayer
,pDPosIA
. Some of these are related to the incompleteness of the code you show, but if they aren't decrefed then you're leaking memory.You call
PyErr_Print()
without checking for an error. The documentation says:Call this function only when the error indicator is set. (Otherwise it will cause a fatal error!)
I suspect that this might be your biggest problem.
Those are all the issues I can see. Without a minimal complete example it's impossible to actually check what you're seeing. Given you're using C++ you might be well advised to use/make a decent, object oriented wrapper for PyObject*
to avoid having to worry about reference counting yourself - Boost Python has one.
Python: Ctypes how to check memory management
If you want to use Valgrind
, then this readme might be helpful. Probably, this could be another good resource to make Valgrind
friendly python and use it in your program.
But if you consider something else like tracemalloc
, then you can easily get some example usage of it here. The examples are pretty easy to interpret. For example according to their doc,
import tracemalloc
tracemalloc.start()
# ... run your application ...
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')
print("[ Top 10 ]")
for stat in top_stats[:10]:
print(stat)
This will output something like.
<frozen importlib._bootstrap>:716: size=4855 KiB, count=39328, average=126 B
<frozen importlib._bootstrap>:284: size=521 KiB, count=3199, average=167 >
You can either parse this to plot memory usage for your investigation or you may
use the reference doc to get a more concrete idea.
In this case your program could be something like the following:
from tkinter import *
import tracemalloc
root = Tk() # New GUI
# some code here
def destructorMethods:
tracemalloc.start()
myFunctions.destructorLinkedList() # Destructor method of my allocated memory in my C file
# Here is where I would want to run a Valgrind/Memory management check before closing
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')
print("[ Top 10 ]")
for stat in top_stats[:10]:
print(stat)
root.destroy() # close the program
root.protocol("WM_DELETE_WINDOW", destructorMethods)
Another option is, you can use a memory profiler to see memory usage at a variable time. The package is available here. After the installation of this package, you can probably use the following command in your script to get the memory usage over time in a png file.
mprof run --include-children python your_filename.py
mprof plot --output timelyplot.png
or you may use different functions available on memory_profiler
package according to your need. Maybe this tutorial can be an interesting one for you.
Related Topics
Take Multiple Lists into Dataframe
How to Set Headers Using Python's Urllib
Attributeerror: Can't Set Attribute When Connecting to SQLite Database with Flask-Sqlalchemy
Numpy.Where() Detailed, Step-By-Step Explanation/Examples
Best Way to Determine If a Sequence Is in Another Sequence
How to Ssh Connect Through Python Paramiko with Ppk Public Key
Python Equivalent to 'Hold On' in Matlab
Distributing My Python Scripts as Jar Files with Jython
MySQL "Incorrect String Value" Error When Save Unicode String in Django
What's a Good Equivalent to Subprocess.Check_Call That Returns the Contents of Stdout
Python - Initializing Multiple Lists/Line
Python:When Is a Variable Passed by Reference and When by Value
Possibilities for Python Classes Organized Across Files
Why Doesn't Django's Model.Save() Call Full_Clean()
Any Reason Not to Use '+' to Concatenate Two Strings
Keep Persistent Variables in Memory Between Runs of Python Script