Ctypes Loading a C Shared Library That Has Dependencies

importing dll with dependencies using ctypes

I ended up solving this on my own so here is what I found out:

libb.so is a C compiled library, so when libd.so was compiled as C++ it tried to expose one of the functions it used from libb.so however libb.so was not compiled with extern "C" (I didn't think to do this since I was compiling it as C) so C++ tried exposing the name-mangled symbol for the C function and when CDLL loaded libd.so it couldn't find the symbol since it didn't exist as a c++ function.

The solution was to use this in the header files of my c code

#ifdef __cplusplus
extern "C" {
#endif
...my c code definitions...
#ifdef __cplusplus
}
#endif

python ctypes link multiple shared library with example gsl gslcblas

This old answer tells to apply mode=ctypes.RTLD_GLOBAL, i.e. in this case

import ctypes

dll1 = ctypes.CDLL('libgslcblas.so', mode=ctypes.RTLD_GLOBAL)
dll2 = ctypes.CDLL('libgsl.so')

Unload shared library inside ctypes loaded shared library

The rule Clean up after yourself always applies (although modern technologies take care of the cleaning aspect for you).

[Python 3.5]: ctypes - A foreign function library for Python contains lots of useful info, and should be your friend.

ctypes uses dlopen whel loading a .dll. As I noticed, it doesn't call the corresponding dlclose meaning that the .dll (and all of its dependents that were loaded when loading it) will remain in memory until the process terminates (or until explicitly unloaded).

From [man7]: DLOPEN(3):

If the object specified by filename has dependencies on other shared objects, then these are also automatically loaded by the dynamic linker using the same rules. (This process may occur recursively, if those objects in turn have dependencies, and so on.)

...

If the same shared object is loaded again with dlopen(), the same object handle is returned. The dynamic linker maintains reference counts for object handles, so a dynamically loaded shared object is not deallocated until dlclose() has been called on it as many times as dlopen() has succeeded on it. Any initialization returns (see below) are called just once.

So, I don't think you'd have a problem (of course, everything depends on the context). As you noticed, loading a library multiple times doesn't actually load it every time, so the chance to run out of memory is pretty small (well unless you are loading a huge number of different .dlls, each with lots of different dependencies).

One case that I can think of is loading a .dll that uses a symbol from another .dll. If that symbol is also defined in another (3rd) .dll, which was loaded before, then the code would behave differently than expected.

Anyway, you can manually unload (or better: decrease its refcount) a .dll (I'm not sure how this fits into the recommended ways or best practices ), like shown in the example below.

dll.c:

#include <stdio.h>

int test() {
    printf("[%s] (%d) - [%s]\n", __FILE__, __LINE__, __FUNCTION__);
    return 0;
}

code.py:

import sys
from ctypes import CDLL, \
    c_int, c_void_p

DLL = "./dll.so"

dlclose_func = CDLL(None).dlclose  # This WON'T work on Win
dlclose_func.argtypes = [c_void_p]

def _load_dll(dll_name):
    dll_dll = CDLL(dll_name)
    print("{:}".format(dll_dll))
    return dll_dll

def _load_test_func(dll):
    test_func = dll.test
    test_func.restype = c_int
    return test_func

def main():
    print("Loading a dll via `ctypes`, then delete the object. The dll is not unloaded. Call `dlclose` to unload. A 2nd call will fail.")
    dll_dll = _load_dll(DLL)
    dll_handle = dll_dll._handle
    del dll_dll
    print("{:} returned {:d}".format(dlclose_func.__name__, dlclose_func(dll_handle)))  # Even if the ctypes dll object was destroyed, the dll wasn't unloaded
    print("{:} returned {:d}".format(dlclose_func.__name__, dlclose_func(dll_handle)))  # A new dlclose call will fail

    print("\nUse `ctypes` to load the dll twice. The dll is not actually loaded only the 1st time (both have the same handle), but its ref count is increased. `dlclose` must be also called twice.")
    dll0_dll = _load_dll(DLL)
    dll1_dll = _load_dll(DLL)
    print("{:} returned {:d}".format(dlclose_func.__name__, dlclose_func(dll0_dll._handle)))
    print("{:} returned {:d}".format(dlclose_func.__name__, dlclose_func(dll1_dll._handle)))
    print("{:} returned {:d}".format(dlclose_func.__name__, dlclose_func(dll1_dll._handle)))

    print("\nLoad a dll via `ctypes`, and load one of its funcs. Try calling it before and after unloading the dll.")
    dll_dll = _load_dll(DLL)
    test_func = _load_test_func(dll_dll)
    print("{:} returned {:d}".format(test_func.__name__, test_func()))
    print("{:} returned {:d}".format(dlclose_func.__name__, dlclose_func(dll_dll._handle)))
    print("{:} returned {:d}".format(test_func.__name__, test_func()))  # Comment this line as it would segfault !!!

if __name__ == "__main__":
    print("Python {:s} on {:s}\n".format(sys.version, sys.platform))
    main()

Output:

[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q052179325]> ls
code.py  dll.c
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q052179325]> gcc -fPIC -shared -o dll.so dll.c
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q052179325]> python3 ./code.py
Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux

Loading a dll via `ctypes`, then delete the object. The dll is not unloaded. Call `dlclose` to unload. A 2nd call will fail.
<CDLL './dll.so', handle 1d7aa20 at 0x7fa38715f240>
dlclose returned 0
dlclose returned -1

Use `ctypes` to load the dll twice. The dll is not actually loaded only the 1st time (both have the same handle), but its ref count is increased. `dlclose` must be also called twice.
<CDLL './dll.so', handle 1de2c80 at 0x7fa38715f240>
<CDLL './dll.so', handle 1de2c80 at 0x7fa38715f278>
dlclose returned 0
dlclose returned 0
dlclose returned -1

Load a dll via `ctypes`, and load one of its funcs. Try calling it before and after unloading the dll.
<CDLL './dll.so', handle 1de39c0 at 0x7fa38715f8d0>
[dll.c] (5) - [test]
test returned 0
dlclose returned 0
Segmentation fault (core dumped)

Resolving circular shared-object dependencies with ctypes/cffi

This (if I understood the problem correctly,) is a perfectly normal usecase on Nix, and should run without problems.

When dealing with problems related to ctypes ([Python 3]: ctypes - A foreign function library for Python), the best (generic) way to tackle them is:

  • Write a (small) C application that does the required job (and of course, works)
  • Only then move to ctypes (basically this is translating the above application)

I prepared a small (and dummy) example:

  • defines.h:

    #pragma once

    #include <stdio.h>

    #define PRINT_MSG_0() printf("From C: [%s] (%d) - [%s]\n", __FILE__, __LINE__, __FUNCTION__)

  • libC:

    • libC.h:

      #pragma once

      size_t funcC();

    • libC.c:

      #include "defines.h"
      #include "libC.h"
      #include "libA.h"

      size_t funcC() {
          PRINT_MSG_0();
          for (size_t i = 0; i < ARRAY_DIM; i++)
          {
              printf("%zu - %c\n", i, charArray[i]);
          }
          printf("\n");
          return ARRAY_DIM;
      }

  • libB:

    • libB.h:

      #pragma once

      size_t funcB();

    • libB.c:

      #include "defines.h"
      #include "libB.h"
      #include "libC.h"

      size_t funcB() {
          PRINT_MSG_0();
          return funcC();
      }

  • libA:

    • libA.h:

      #pragma once

      #define ARRAY_DIM 3

      extern char charArray[ARRAY_DIM];

      size_t funcA();

    • libA.c:

      #include "defines.h"
      #include "libA.h"
      #include "libB.h"

      char charArray[ARRAY_DIM] = {'A', 'B', 'C'};

      size_t funcA() {
          PRINT_MSG_0();
          return funcB();
      }

  • code.py:

    #!/usr/bin/env python3

    import sys
    from ctypes import CDLL, \
        c_size_t

    DLL = "./libA.so"

    def main():
        lib_a = CDLL(DLL)
        func_a = lib_a.funcA
        func_a.restype = c_size_t

        ret = func_a()
        print("{:s} returned {:d}".format(func_a.__name__, ret))

    if __name__ == "__main__":
        print("Python {:s} on {:s}\n".format(sys.version, sys.platform))
        main()

Output:

[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q053327620]> ls
code.py  defines.h  libA.c  libA.h  libB.c  libB.h  libC.c  libC.h
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q053327620]> gcc -fPIC -shared -o libC.so libC.c
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q053327620]> gcc -fPIC -shared -o libB.so libB.c -L. -lC
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q053327620]> gcc -fPIC -shared -o libA.so libA.c -L. -lB
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q053327620]> ls
code.py  defines.h  libA.c  libA.h  libA.so  libB.c  libB.h  libB.so  libC.c  libC.h  libC.so
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q053327620]> LD_LIBRARY_PATH=. ldd libC.so
        linux-vdso.so.1 =>  (0x00007ffdfb1f4000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f56dcf23000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f56dd4ef000)
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q053327620]> LD_LIBRARY_PATH=. ldd libB.so
        linux-vdso.so.1 =>  (0x00007ffc2e7fd000)
        libC.so => ./libC.so (0x00007fdc90a9a000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fdc906d0000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fdc90e9e000)
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q053327620]> LD_LIBRARY_PATH=. ldd libA.so
        linux-vdso.so.1 =>  (0x00007ffd20d53000)
        libB.so => ./libB.so (0x00007fdbee95a000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fdbee590000)
        libC.so => ./libC.so (0x00007fdbee38e000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fdbeed5e000)
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q053327620]> nm -S libC.so | grep charArray
                 U charArray
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q053327620]> nm -S libA.so | grep charArray
0000000000201030 0000000000000003 D charArray
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q053327620]> LD_LIBRARY_PATH=. python3 code.py
Python 3.5.2 (default, Nov 12 2018, 13:43:14)
[GCC 5.4.0 20160609] on linux

From C: [libA.c] (9) - [funcA]
From C: [libB.c] (7) - [funcB]
From C: [libC.c] (7) - [funcC]
0 - A
1 - B
2 - C

funcA returned 3

But if your array is declared as static ([CPPReference]: C keywords: static) (and thus, as a consequence it can't be extern as in the example), then you're kind of toasted.

@EDIT0: Extending the example so that it better fits the description.

Since ldd doesn't show dependencies between the .sos, I'm going to assume that each is loaded dynamically.

  • utils.h:

    #pragma once

    #include <dlfcn.h>

    void *loadLib(char id);

  • utils.c:

    #include "defines.h"
    #include "utils.h"

    void *loadLib(char id) {
        PRINT_MSG_0();
        char libNameFormat[] = "lib%c.so";
        char libName[8];
        sprintf(libName, libNameFormat, id);
        int load_flags = RTLD_LAZY | RTLD_GLOBAL;  // !!! @TODO - @CristiFati: Note RTLD_LAZY: if RTLD_NOW would be here instead, there would be nothing left to do. Same thing if RTLD_GLOBAL wouldn't be specified. !!!
        void *ret = dlopen(libName, load_flags);
        if (ret == NULL) {
            char *err = dlerror();
            printf("Error loading lib (%s): %s\n", libName, (err != NULL) ? err : "(null)");
        }
        return ret;
    }

Below is a modified version of libB.c. Note that the same pattern should also be applied to the original libA.c.

  • libB.c:

    #include "defines.h"
    #include "libB.h"
    #include "libC.h"
    #include "utils.h"

    size_t funcB() {
        PRINT_MSG_0();
        void *mod = loadLib('C');
        size_t ret = funcC();
        dlclose(mod);
        return ret;
    }

Output:

[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q053327620]> ls
code.py  defines.h  libA.c  libA.h  libB.c  libB.h  libC.c  libC.h  utils.c  utils.h
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q053327620]> gcc -fPIC -shared -o libC.so libC.c utils.c
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q053327620]> gcc -fPIC -shared -o libB.so libB.c utils.c
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q053327620]> gcc -fPIC -shared -o libA.so libA.c utils.c
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q053327620]> ls
code.py  defines.h  libA.c  libA.h  libA.so  libB.c  libB.h  libB.so  libC.c  libC.h  libC.so  utils.c  utils.h
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q053327620]> ldd libA.so
        linux-vdso.so.1 =>  (0x00007ffe5748c000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f4d9e3f6000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f4d9e9c2000)
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q053327620]> ldd libB.so
        linux-vdso.so.1 =>  (0x00007ffe22fe3000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe93ce8a000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fe93d456000)
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q053327620]> ldd libC.so
        linux-vdso.so.1 =>  (0x00007fffe85c3000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2d47453000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f2d47a1f000)
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q053327620]> nm -S libC.so | grep charArray
                 U charArray
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q053327620]> nm -S libA.so | grep charArray
0000000000201060 0000000000000003 D charArray
[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q053327620]> LD_LIBRARY_PATH=. python3 code.py
Python 3.5.2 (default, Nov 12 2018, 13:43:14)
[GCC 5.4.0 20160609] on linux

Traceback (most recent call last):
  File "code.py", line 22, in <module>
    main()
  File "code.py", line 12, in main
    lib_a = CDLL(DLL)
  File "/usr/lib/python3.5/ctypes/__init__.py", line 347, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: ./libA.so: undefined symbol: funcB

I believe that this reproduces the problem. Now, if you modify (the 1st part of) code.py to:

#!/usr/bin/env python3

import sys
from ctypes import CDLL, \
    RTLD_GLOBAL, \
    c_size_t

RTLD_LAZY = 0x0001

DLL = "./libA.so"

def main():
    lib_a = CDLL(DLL, RTLD_LAZY | RTLD_GLOBAL)
    func_a = lib_a.funcA
    func_a.restype = c_size_t

    ret = func_a()
    print("{:s} returned {:d}".format(func_a.__name__, ret))

if __name__ == "__main__":
    print("Python {:s} on {:s}\n".format(sys.version, sys.platform))
    main()

you'd get the following output:

[cfati@cfati-ubtu16x64-0:~/Work/Dev/StackOverflow/q053327620]> LD_LIBRARY_PATH=. python3 code.py
Python 3.5.2 (default, Nov 12 2018, 13:43:14)
[GCC 5.4.0 20160609] on linux

From C: [libA.c] (11) - [funcA]
From C: [utils.c] (6) - [loadLib]
From C: [libB.c] (8) - [funcB]
From C: [utils.c] (6) - [loadLib]
From C: [libC.c] (7) - [funcC]
0 - A
1 - B
2 - C

funcA returned 3

Notes:

  • It's very important that in C RTLD_LAZY | RTLD_GLOBAL are there. if RTLD_LAZY is replaced by RTLD_NOW, it won't work
    • Also, if RTLD_GLOBAL isn't specified, it won't work either. I didn't check whether there are other RTLD_ flags that could be specified instead of RTLD_GLOBAL for things to still work
  • Creating that wrapper library that deals with all libraries loading and initialization, would be a good thing (workaround), especially if you plan to use them from multiple places (that way, the whole process would happen in one place only). But, previous bullet would still apply
  • For some reason, ctypes doesn't expose RTLD_LAZY (and many other related flags, as a matter of fact). Defining it in the code.py, is kind of a workaround, and on different (Nix) platforms (flavors), its value might differ

Using ctypes to wrap compiled library with dependancies

Try loading the required library beforehand:

ctypes.CDLL('libraw1394.so.X.Y', mode=ctypes.RTLD_GLOBAL)
ctypes.CDLL('libpvcam.so.2.7.4.2', mode=ctypes.RTLD_GLOBAL)

Related Topics

How to Recover the Return Value of a Function Passed to Multiprocessing.Process

Datastax Python Cassandra Driver Build Fails on Ubuntu

Loading Animation in Python

Opencv Error on Python

Where Does Python Tempfile Writes Its Files

How to Benchmark Part of Tensorflow Graph

Trying to Import Pypyodbc Module Gives Error 'Odbc Library Is Not Found. Is Ld_Library_Path Set'

How to Speed-Up Python Io

Automate Firefox with Python

Python Memory Debugging with Gdb

How to Handle Os.System Sigkill Signal Inside Python

Turn Off Buffering

Gae " No Attribute 'Httpshandler' " Dev_Appserver.Py

Python Script Prints Output of Os.System Before Print

Subprocess.Popen(): Oserror: [Errno 8] Exec Format Error in Python

Why Does Simple Echo in Subprocess Not Working

Find the Similarity Metric Between Two Strings

Python Gdal 2.1 Installation on Ubuntu 16.04


Leave a reply



Submit