Does Python Have a Bitfield Type

Does Python have a bitfield type?

Bitarray was the best answer I found, when I recently had a similar need. It's a C extension (so much faster than BitVector, which is pure python) and stores its data in an actual bitfield (so it's eight times more memory efficient than a numpy boolean array, which appears to use a byte per element.)

Struct w/ bit field: CPython and C padding differs

The problem is that implementation of bit fields is explicitely implementation dependant. C standard (ref n1256 draft for C11) says in 6.7.2.1 Structure and union specifiers:

...
§4 A bit-field shall have a type that is a qualified or unqualified version of _Bool, signed
int, unsigned int, or some other implementation-defined type.

...
9 A bit-field is interpreted as a signed or unsigned integer type consisting of the specified
number of bits

ok, you want an unsigned type using one single bit, but next paragraph says (emphasize mine):

10 An implementation may allocate any addressable storage unit large enough to hold a bitfield.

Here a byte is enough, so gcc will use one single byte. But MSVC in 32 bit mode will use 4 bytes and the output of the C program would be:

  4 str offset
 12 total
  4 str_pack offset
 12 total

exactly what the CPython implentations outputs

That means that ctypes implementers choosed to follow MSVC here. Two sentences in the modules documentation give a hint on that, in addition to the fact that most examples use Windows:

• a warning in 16.16.1.10. Structures and unions¶

Warning: ctypes does not support passing unions or structures with bit-fields to functions by value. While this may work on 32-bit x86, it’s not guaranteed by the library to work in the general case.

and next paragraph states in 16.16.1.11. Structure/union alignment and byte order

By default, Structure and Union fields are aligned in the same way the C compiler does it... This is what #pragma pack(n) also does in MSVC

The problem is that the ctypes module implementers choosed a convention - they cannot directly rely on the compiler here because the size of a bit field has to be a constant - independantly of the platform. You may think that is is a problem for non MSVC implementation and fill a bug report but I also think that the ctypes module is mainly used in Windows.

So the only way to be able to process your C structure here is to explicitely force the underlying types for the bit field and declare the first field to be an unsigned byte, and eventually specifies an explicit padding:

class Test(Structure):
    _fields_ = [
        ('bit', c_byte),
        ('str', c_ubyte * 8),
        ('', c_ubyte * 3),      # explicit padding
    ]

class TestPacked(Structure): 
    _fields_ = [
        ('bit_p', c_ubyte), 
        ('str_p', c_ubyte * 8),
    ]

But that is only a workaround because here bit and bit_p are plain bytes while the C code requires that only 1 bit should be used, the 7 other bits being padding bits.

Accessing bitfields while reading/writing binary data structures

Using bitstring (which you mention you're looking at) it should be easy enough to implement. First to create some data to decode:

>>> myheader = "3, 2, 3, 14, 4"
>>> a = bitstring.pack(myheader, 1, 0, 5, 1000, 2)
>>> a.bin
'00100101000011111010000010'
>>> a.tobytes()
'%\x0f\xa0\x80'

And then decoding it again is just

>>> a.readlist(myheader)
[1, 0, 5, 1000, 2]

Your main concern might well be the speed. The library is well optimised Python, but that's not nearly as fast as a C library would be.

Bit field specialization in python

In your code s is a class and the class x member actually represents the field type, so assigning s.x = 29 essentially destroys that object and assigns a normal Python int to it. Example:

>>> from ctypes import *
>>> class S(Structure):
...     _fields_ = [('x',c_int,5)]
... 
>>> S.x
<Field type=c_long, ofs=0:0, bits=5>
>>> S.x = 29
>>> S.x
29

Also, even if you create an instance first, r = s.x = 29 does not do s.x = 29 then r = s.x as in C/C++ but essentially r=29 and s.x=29. Example:

>>> from ctypes import *
>>> class S(Structure):
...     _fields_ = [('x',c_int,5)]
...     
>>> s=S()
>>> r=s.x=29
>>> s.x
-3
>>> r
29

So to fix, instantiate the class, assign s.x = 29 and return it:

from ctypes import *
def sign_extending(x, b):
    class S(Structure):
        _fields_ = [("x", c_int, b)]
  s=S()
  s.x = x
  return s.x

x = 29; #this 29 is -3 ( 11101 ) in 5 bits. 
r = sign_extending(x, 5) #Convert this from using 5 bits to a full int
print r

Output:

-3

---

Related Topics