Python Ctypes Cdll.Loadlibrary, Instantiate an Object, Execute Its Method, Private Variable Address Truncated

Python ctypes cdll.LoadLibrary, instantiate an object, execute its method, private variable address truncated

Always (CORRECTLY) specify argtypes and restype for functions defined in C, otherwise (C89 style) they will default to int (generally 32bit), generating !!! Undefined Behavior !!!. On 64bit, addresses (larger than 2 GiB) will be truncated (which is exactly what you're experiencing). Check [SO]: C function called from Python via ctypes returns incorrect value (@CristiFati's answer) for more details.

Also, when running into issues, don't forget about [Python.Docs]: ctypes - A foreign function library for Python.

Below it's an adapted version of your code.

detector.cpp:

#include <stdio.h>
#include <memory.h>

#include <fstream>

#define SIM_EXPORT __declspec(dllexport)

#define C_TAG "From C"
#define PRINT_MSG_3SPI(ARG0, ARG1, ARG2) printf("%s - [%s] (%d) - [%s]:  %s: 0x%0p(%d)\n", C_TAG, __FILE__, __LINE__, __FUNCTION__, ARG0, ARG1, ARG2)

using std::endl;

std::ofstream outFile;

class Detector {
    public:
        Detector();
        void process(int *pIn, int *pOut, int n);

    private:
        int m_var;
};

Detector::Detector()
: m_var(25) {
    outFile.open("addr_debug.txt");
    outFile << "m_var init address: " << &m_var << endl;
    PRINT_MSG_3SPI("&m_var(m_var)", &m_var, m_var);
}

void Detector::process(int *pIn, int *pOut, int n)
{
    outFile << "m_var process address: " << &m_var << endl;
    outFile.close();
    PRINT_MSG_3SPI("&m_var(m_var)", &m_var, m_var);
}

#if defined(__cplusplus)
extern "C" {
#endif

SIM_EXPORT Detector* DetectorNew() { return new Detector(); }

SIM_EXPORT void DetectorProcess(Detector *pDet, int *pIn, int *pOut, int n)
{
    pDet->process(pIn, pOut, n);
}

SIM_EXPORT void DetectorDelete(Detector *pDet) { delete pDet; }

#if defined(__cplusplus)
}
#endif

code00.py:

#!/usr/bin/env python

import ctypes as ct
import sys

import numpy as np

IntPtr = ct.POINTER(ct.c_int)

sim_dll = ct.CDLL("./sim.dll")

detector_new_func = sim_dll.DetectorNew
detector_new_func.argtypes = ()
detector_new_func.restype = ct.c_void_p

detector_process_func = sim_dll.DetectorProcess
detector_process_func.argtypes = (ct.c_void_p, IntPtr, IntPtr, ct.c_int)
detector_process_func.restype = None

detector_delete_func = sim_dll.DetectorDelete
detector_delete_func.argtypes = (ct.c_void_p,)
detector_delete_func.restype = None

class Detector():
    def __init__(self):
        self.obj = detector_new_func()

    def process(self, pin, pout, n):
        detector_process_func(self.obj, pin, pout, n)

    def __del__(self):
        detector_delete_func(self.obj)

def main(*argv):
    detector = Detector()

    n = 1024
    a = np.arange(n, dtype=np.uint32)
    b = np.zeros(n, dtype=np.int32)

    aptr = a.ctypes.data_as(IntPtr)
    bptr = b.ctypes.data_as(IntPtr)

    detector.process(aptr, bptr, n)

if __name__ == "__main__":
    print("Python {:s} {:03d}bit on {:s}\n".format(" ".join(elem.strip() for elem in sys.version.split("\n")),
                                                   64 if sys.maxsize > 0x100000000 else 32, sys.platform))
    rc = main(*sys.argv[1:])
    print("\nDone.")
    sys.exit(rc)

Notes:

• As I stated at the beginning, the problem was argtypes and restype not being specified (e.g. for DetectorNew: comment detector_new_func.restype = ct.c_void_p, and you'll run into the problem again)

• Code in the question is missing parts (#includes, imports, ...), also there are some syntax errors, so it doesn't compile, and therefore doesn't follow [SO]: How to create a Minimal, Complete, and Verifiable example (mcve) guidelines. Please when make sure to have MCVE when asking

• The object that you allocate (new Detector()), must also be deallocated (otherwise, it will generate a memory leak), so I added a function (DetectorDelete - to do that), which is called from (Python) Detector's destructor

• Other (non critical) changes (identifiers renaming, a bit of refactoring, printing to stdout, ...)

Output:

(py35x64_tes1) e:\Work\Dev\StackOverflow\q052268294>"c:\Install\x86\Microsoft\Visual Studio Community\2015\vc\vcvarsall.bat" x64

(py35x64_test) e:\Work\Dev\StackOverflow\q052268294>dir /b
code00.py
detector.cpp

(py35x64_test) e:\Work\Dev\StackOverflow\q052268294>cl /nologo /DDLL /EHsc detector.cpp  /link /DLL /OUT:sim.dll
detector.cpp
   Creating library sim.lib and object sim.exp

(py35x64_test) e:\Work\Dev\StackOverflow\q052268294>dir /b
code00.py
detector.cpp
detector.obj
sim.dll
sim.exp
sim.lib

(py35x64_test) e:\Work\Dev\StackOverflow\q052268294>"e:\Work\Dev\VEnvs\py35x64_test\Scripts\python.exe" ./code.py
Python 3.5.4 (v3.5.4:3f56838, Aug  8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] 064bit on win32

From C - [detector.cpp] (28) - [Detector::Detector]:  &m_var: 0x0000020CE366E270
From C - [detector.cpp] (34) - [Detector::process]:  &m_var: 0x0000020CE366E270

Done.

Resultpointer in function call

Check [Python 3.Docs]: ctypes - A foreign function library for Python. It contains (almost) every piece of info that you need.

There are a number of problems:

1. ctypes doesn't support pascal calling convention, only cdecl and stdcall (applies to 32bit only). That means (after reading the manual) that you shouldn't use the p* functions, but the c* (or s*)

2. You didn't specify argtypes (and restype) for your function. This results in UB. Some effects of this:

   • [SO]: Python ctypes cdll.LoadLibrary, instantiate an object, execute its method, private variable address truncated (@CristiFati's answer)
   • [SO]: python ctypes issue on different OSes (@CristiFati's answer)

3. It is a procedure (a function that returns void). Anyway this is a minor one

Here's some sample code (of course it's blind, as I didn't test it):

#!/usr/bin/env python3

import sys
import ctypes

dll = ctypes.CDLL("raccd32a.dll")

cGetCallInfo = dll.cGetCallInfo
cGetCallInfo.argtypes = [ctypes.c_char_p, ctypes.c_char_p, ctypes.c_char_p, ctypes.POINTER(ctypes.c_short)]
cGetCallInfo.restype = None

ADriveInfo = self.txt_CallBookPath.text().encode()
#ADriveInfo = b"C:\\callbook2019\\"  # Notice the double bkslashes
ACall = b"DG1ATN"
AInfo = ctypes.create_string_buffer(400)
result = ctypes.c_short(0)
cGetCallInfo(ADriveInfo, ACall, AInfo, ctypes.byref(result))

@EDIT0:

From the beginning, I wanted yo say that the 1^st argument passed to the function doesn't make much sense. Then, there are problems regarding the 2^nd one as well. According to the manual ([AMT-I]: TECHNICAL INFORMATION about RACCD32a.DLL (emphasis is mine)):

ADriveInfo, ACall and AInfo are pointers to zero-terminated strings. These
strings has to exist at the moment of calling xGetCallInfo. The calling
program is responsible for creating them. AInfo must be long enough to
comfort xGetCallInfo (at least 400 characters).
Note: "Length of AInfo" refers to the length of the string AInfo points at.
ADriveInfo and ACall are treated in the same manner for short.

In ADriveInfo the procedure expects the path to the CD ROM drive. Use
"G:\"
if "G:" designates the CD ROM drive with the callbook CD ROM.
Keep in mind that this information is a *must* and the calling program
has to know it.
Note: If the active directory on drive G: is not the root, ADriveInfo = "G:"
will lead to an error 3. So always use "G:\".
The calling program has to ensure that the length of ADriveInfo does not
exceed 80 characters.

ACall contains the call you are looking for, all letters in lower case,
no additional spaces etc. The calling program has to ensure that ACall is
not longer than 15 characters. However, there is no call longer than 6
characters in the database.

python ctypes issue on different OSes

In 99% of the cases, inconsistencies between arguments (and / or return) type inconsistencies are the cause (check [SO]: C function called from Python via ctypes returns incorrect value (@CristiFati's answer) for more details).

Always have [Python.Docs]: ctypes - A foreign function library for Python open when working with CTypes.

I found [GitHub]: erikssm/futronics-fingerprint-reader - (master) futronics-fingerprint-reader/ftrScanAPI.h (I don't know how different it's from what you currently have, but things you posted so far seem to match), and I did some changes to your code:

• Define argtypes and restype for functions

• Define missing types (for clarity only)

• Some other insignificant changes (renames)

• One other thing that I noticed in the above file, is a #pragma pack(push, 1) macro (check [MS.Docs]: pack for more details). For this structure it makes no difference (thanks @AnttiHaapala for the hint), as the 3 int (4 byte) members alignment doesn't change, but for other structures (with "smaller" member types (e.g. char, short)) you might want to add: _pack_ = 1

Your modified code (needless to say, I didn't run it as I don't have the .dll):

#!/usr/bin/env python

import ctypes as ct
from ctypes import wintypes as wt

# ...

class FTRSCAN_IMAGE_SIZE(ct.Structure):
    # _pack_ = 1
    _fields_ = (
        ("nWidth", ct.c_int),
        ("nHeight", ct.c_int),
        ("nImageSize", ct.c_int),
    )

PFTRSCAN_IMAGE_SIZE = ct.POINTER(FTRSCAN_IMAGE_SIZE)
FTRHANDLE = ct.c_void_p

lib = ct.WinDLL("ftrScanAPI.dll")  # provided by fingerprint scanner

ftrScanOpenDevice = lib.ftrScanOpenDevice
ftrScanOpenDevice.argtypes = ()
ftrScanOpenDevice.restype = FTRHANDLE

ftrScanGetImageSize = lib.ftrScanGetImageSize
ftrScanGetImageSize.argtypes = (FTRHANDLE, PFTRSCAN_IMAGE_SIZE)
ftrScanGetImageSize.restype = wt.BOOL

print("Open device and get device handle...")
h_device = ftrScanOpenDevice()
print("Handle is", h_device)
print("Get image size...")
image_size = FTRSCAN_IMAGE_SIZE(0, 0, 0)

if ftrScanGetImageSize(h_device, ct.byref(image_size)):
    print("Get image size succeed...")
    print("  W", image_size.nWidth)
    print("  H", image_size.nHeight)
    print("  Size", image_size.nImageSize)
else:
    print("Get image size failed...")

Access Violation when using Ctypes to Interface with Fortran DLL

I see 2 problems with the code (and a potential 3^rd one):

1. argtypes (and restype) not being specified, as described in [Python 3]: Specifying the required argument types (function prototypes). There are many examples of what happens in this case, here are 2 of them:

   • [SO]: python ctypes issue on different OSes (@CristiFati's answer)
   • [SO]: Python ctypes cdll.LoadLibrary, instantiate an object, execute its method, private variable address truncated (@CristiFati's answer)

2. This may be a consequence (or at least it's closely related to) the previous. I can only guess without the Fortran (or better: C) function prototype, but anyway there is certainly something wrong. I assume that for the input data things should be same as for the output data, so the function would take 2 arrays (same size), and the input one's elements would be void *s (since their type is not consistent). Then, you'd need something like (although I can't imagine how would Fortran know which element contains an int and which a double):

   in_data (ctypes.c_void_p * len(self.data))()
   for idx, item in enumerate(self.data):
       item.convert_to_ctypes()
       in_data[index] = ctypes.addressof(item.c_val)
   

3. Since you're on 32 bit, you should also take calling convention into account (ctypes.CDLL vs ctypes.WinDLL)

But again, without the function prototype, everything is just a speculation.

Also, why "MyFunction_".lower() instead of "myfunction_"?

C function called from Python via ctypes returns incorrect value

Listing [Python.Docs]: ctypes - A foreign function library for Python.

In order for everything to be properly converted (Python <=> C) when calling the function (residing in a .dll (.so)), 2 things need to be specified (leaving x86 (pc032) calling convention (Win) aside):

1. Argument types

2. Return type

In CTypes, this is achieved by specifying:

1. argtypes - a sequence (tuple, list) containing each function's argument (CTypes) type, in the order they appear in the function declaration (if the function only has one argument, there should be one element sequence)

2. restype - a single CTypes type (function's return type)

Side note: an alternative to the above, is prototyping foreign functions (CFUNCTYPE, WINFUNCTYPE, PYFUNCTYPE - check the Function prototypes section (in the URL at the beginning)).

Anyway:

• Failing to specify

• Misspelling (which basically is same thing as previous bullet)

any of them (where required ⁽¹⁾), will result in defaults being applied: all are treated (C89 style) as ints, which (on most systems) are 32 bits long.

This generates Undefined Behavior ⁽²⁾ (also applies when incorrectly specifying them), especially on 064bit CPU / OS (and Python process), where values of larger types (e.g. pointers) could be truncated (check [SO]: Maximum and minimum value of C types integers from Python (@CristiFati's answer) for details (limits -> size) regarding C integral types).

The displayed errors can be numerous and sometimes misleading.

You misspelled argtype (lacking an s at the end).

Correct that, and you should be fine

Example:

For a function func exported by libdll.dll (libdll.so) with the following header:

double func(uint32_t ui32,
            float f,
            long long vll[8],
            void *pv,
            char *pc,
            uint8_t *pui8);

The Python equivalent would be:

import ctypes as cts

func = libdll.func
func.argtypes = (cts.c_uint32,
                 cts.c_float,
                 cts.c_longlong * 8,
                 cts.c_void_p,
                 cts.c_char_p,
                 cts.POINTER(cts.c_ubyte))
'''
Last 2 argument types are almost the same (1st is signed while 2nd is unsigned).
  - 1st is used for NUL terminated strings (that work with strcpy, strlen, ...).
    Python automatically converts them to bytes

  - 2nd is used for buffers in general (which may contain NUL (0x00) characters)
'''

func.restype = cts.c_double

Some (more destructive) outcomes of the same (or very similar) scenario (there are many others):

• [SO]: Python ctypes cdll.LoadLibrary, instantiate an object, execute its method, private variable address truncated (@CristiFati's answer)

• [SO]: python ctypes issue on different OSes (@CristiFati's answer)

---

Footnotes

• #1: Technically, there are situations where it's not required to specify them. But even then, it's best to have them specified in order to eliminate any possible confusion:

  • Function with no arguments:

    function_from_dll.argtypes = ()
    

  • Function returning void:

    function_from_dll.restype = None
    

• #2: Undefined Behavior ([Wikipedia]: Undefined behavior) as the name suggests, is a situation when the outcome of a piece of code can't be "predicted" (or guaranteed by the C (C++) standard). The main cases:

  • Works as expected

  • Doesn't work as expected

    • Has some funny outputs / side effects

    • Crashes

  
  The "beauty" of it is that sometimes it seems totally random, sometimes it "only reproduces" under some specific circumstances (different machines, different OSes, different environments, ...). Bottom line is that all of them are purely coincidental! The problem lies in the code (can be the current code (highest chances) or other code it uses (libraries, compilers)).

Passing audio data from Python to C with ctypes

This should work for you. Use array for a writable buffer and create a ctypes array that references the buffer.

data = array.array('h',wav_bytes)
addr,size = data.buffer_info()
arr = (c_short * size).from_address(addr)
_native_function(arr,size)

Alternatively, to skip the copy of wav_bytes into data array, you could lie about the pointer type in argtypes. ctypes knows how convert a byte string to a c_char_p. A pointer is just an address, so the _native_function will receive the address but use it as an int* internally:

_native_function.argtypes = c_char_p,c_size_t
_native_function(wav_bytes,len(wav_bytes) // 2)

Another way to work around the "underlying buffer is not writable" error is to leverage c_char_p, which allows an immutable byte string to used, and then explicitly cast it to the pointer type you want:

_native_function.argtypes = POINTER(c_short),c_size_t
p = cast(c_char_p(wav_bytes),POINTER(c_short))
_native_function(p,len(wav_bytes) // 2)

In these latter cases you must ensure you don't actually write to the buffer as it will corrupt the immutable Python object holding the data.

Wrong ctypes assignation

[Python 3.Docs]: ctypes - A foreign function library for Python.

You misspelled restypes (it should be restype). By doing so, restype is not initialized, and defaults to int (this wouldn't be a problem on 32bit), and you ran into:

• [SO]: Python ctypes cdll.LoadLibrary, instantiate an object, execute its method, private variable address truncated (@CristiFati's answer)
• [SO]: python ctypes issue on different OSes (@CristiFati's answer)

Besides that, there are several problems in the code:

• If the C function specifies a pointer (double* in this case), don't use ctypes.c_void_p (in argtypes or restype) to map it, as it might be too wide, use (for this case) ctypes.POINTER(ctypes.c_double) instead

• For me this doesn't even compile (I wonder how were you able to run that code). I'm going to exemplify on XTrain only, but applies to YTrain and Xpredict as well. ctypes doesn't know to convert a Python list to a ctypes.POINTER(ctypes.c_double) (or ctypes.c_void_p), and the conversion must be made manually (to a ctypes.c_double array):

  XTrain = [1.0, 1.0, 1.0, 3.0, 3.0, 3.0]
  xtrain_ctypes = (ctypes.c_double * len(XTrain))(*XTrain)
  

  and pass xtrain_ctypes to the functions.

---

Related Topics