Linux: Update Embedded Resource from Executable

Linux: update embedded resource from executable

Nothing special and nothing hard at all. I'll give you correct sequence below, but first let me to correct slightly your embedding method. Lets not use objcopy explicitly, lets use GNU LD instead to got correct entry inside ELF file.

Lets begin. This is test-emb.c file:

#include <stdio.h>

extern unsigned char data[] asm("_binary_data_txt_start");

int
main (void)
{
  fprintf(stderr, "%u, %u, %u\n", data[0] - '0', data[1] - '0', data[2] - '0');
  return 0;
}

This is resource called data.txt

12345678

This is another resource called newdata.txt

98765432

Now compile and link:

$ gcc test-emb.c -c -m32
$ gcc -o test-emb test-emb.o -Wl,--format=binary -Wl,data.txt -Wl,--format=default -m32

Try:

$ ./test-emb 
1, 2, 3

Now start dancing. Step one: determine logical and physical address of data section:

$ readelf -S test-emb | grep "\.data" | awk '{print $4}'
080496b8

$ readelf -S test-emb | grep "\.data" | awk '{print $5}'
0006b8

Step two: start and size fo binary data:

$ readelf -s test-emb | grep _binary_data_txt_start | awk '{print $2}'
080496c0

$readelf -s test-emb | grep _binary_data_txt_size | awk '{print $2}'
00000009

Step three: doing math. We do need: find offset of binary data in data, and convert it to physical starting point:

$ echo $((0x080496c0 - 0x080496b8))
8
echo $((0x0006b8 + 8))
1728

Step four: actual replacement (count value is binary data size, taht is 9):

cat newdata.txt | dd of=test-emb bs=1 seek=1728 count=9 conv=notrunc

Now check again:

$ ./test-emb 
9, 8, 7

Everything works. You may easily fold this method into script, not harder in use, that UpdateResource under Windows, but I want to give you understanding of how things are going on.

Is there any standard way of embedding resources into Linux executable image?

Make yourself an assembler file, blob.S:

    .global blob
    .global blob_size
    .section .rodata
blob:
    .incbin "blob.bin"
1:
blob_size:
    .int 1b - blob

Compile with gcc -c blob.S -o blob.o
The blob can now be accessed from within your C program with:

extern uint8_t blob[];
extern int blob_size;

Using a bin2c converter usually works fine, but if the blob is large, the incbin solution is much faster, and uses much less memory (compile time)

Performance of embedding resource files into executable

The disadvantage to this resource allocation method is that all resources will be present in virtual memory for the entire duration of your application.

For example, if you have 10 levels in your game, using a more common file-based storage solution you would only load the data for whichever level you are going to play. By loading all the game data at runtime you are allocating more RAM than necessary. A solution which uses an optimally minimal amount of RAM would only load the resources it needs for the duration it needs them.

However, if you were to use a file-based storage solution for your resources and load them all at the start of execution your RAM usage would be the same as the statically allocated resource method.

Also, if your target machine is a simple game console that does not do multitasking then you are usually free to use the available resources in whatever means suits you the best. There is no requirement to "play nice" as there's typically only one game running at a time.

If you're dealing with a few megabytes on a modern system the penalty is negligible, but the performance penalty when you get into multiple gigabytes of data is that you will be creating an unnecessary strain on the available memory resources of the system.

edit: be sure to see points raised in the comments

Embedding resources in executable using GCC

There are a couple possibilities:

• use ld's capability to turn any file into an object (Embedding binary blobs using gcc mingw):

  ld -r -b binary -o binary.o foo.bar  # then link in binary.o
  

• use a bin2c/bin2h utility to turn any file into an array of bytes (Embed image in code, without using resource section or external images)

---

Update: Here's a more complete example of how to use data bound into the executable using ld -r -b binary:

#include <stdio.h>

// a file named foo.bar with some example text is 'imported' into 
// an object file using the following command:
//
//      ld -r -b binary -o foo.bar.o foo.bar
//
// That creates an bject file named "foo.bar.o" with the following 
// symbols:
//
//      _binary_foo_bar_start
//      _binary_foo_bar_end
//      _binary_foo_bar_size
//
// Note that the symbols are addresses (so for example, to get the 
// size value, you have to get the address of the _binary_foo_bar_size
// symbol).
//
// In my example, foo.bar is a simple text file, and this program will
// dump the contents of that file which has been linked in by specifying
// foo.bar.o as an object file input to the linker when the progrma is built

extern char _binary_foo_bar_start[];
extern char _binary_foo_bar_end[];

int main(void)
{
    printf( "address of start: %p\n", &_binary_foo_bar_start);
    printf( "address of end: %p\n", &_binary_foo_bar_end);

    for (char* p = _binary_foo_bar_start; p != _binary_foo_bar_end; ++p) {
        putchar( *p);
    }

    return 0;
}

---

Update 2 - Getting the resource size: I could not read the _binary_foo_bar_size correctly. At runtime, gdb shows me the right size of the text resource by using display (unsigned int)&_binary_foo_bar_size. But assigning this to a variable gave always a wrong value. I could solve this issue the following way:

unsigned int iSize =  (unsigned int)(&_binary_foo_bar_end - &_binary_foo_bar_start)

It is a workaround, but it works good and is not too ugly.

QT Applications - Replacing embedded resources

This answer is specific for Qt's resource system (.qrc, rcc).

From the docs:

Currently, Qt always stores the data directly in the executable, even on Windows, macOS, and iOS, where the operating system provides native support for resources. This might change in a future Qt release.

So yes, the Qt resources are contained in the binary.

rcc'ing a .qrc file yields a .cpp file containing (mainly) simple char arrays which represent resource data, the resource names and some other metadata.

Compiling such a .cpp file creates byte fields in the binary.

You can alter such resources within a binary, but only in very limited ways.

• For starters, if the binary contains any kind of self-check (like hashing the data section and comparing it to some pre-calculated hash), you will not be able to change the data in a reasonable way.

• If your data doesn't have the same byte length as the original data, you can't simply replace it because it would alter the internal layout of the binary and invalidate relative addresses.

• In case of replacing with shorter strings you might get away with zero-padding at the end.

Resources are compressed by default (in the ZIP format). It is possible to turn off compression.

• If compression was turned on during compilation (which you don't control, as it seems), you'd need to create new data which compresses to the same length as the original.

Embedding a filesystem in an executable?

There exist so-called virtual file system implementations, which were designed with custom storage in mind. To name a few: our Solid File System, CodeBase File System, Single File System by AidAim.

Most of them use files as the container, some (eg. our SolFS) let you have container in custom places and access it via callbacks.

Depending on practical requirements you can use ZIP or even TAR format as well.

If you want to expose the filesystem contained in your executable to the OS so that other applications could read it, then one of our virtual storage solutions (CallbackDisk or Callback File System) will do the job for you on Windows. On Linux and MacOS X there exist other ways to do this, including FUSE and MacFUSE libraries (analogs to our CBFS).

put together all assemblies and use it as embedded resource in exe

I solved this issue. When the host app runs and doesn't find required assembly, it calls AssemblyResolve event. So, I have to use event handler called MainResolver to load MyAssembly.dll. Then before using methods of MyAssembly.dll, it's necessary to remove it from AssemblyResolve, because the app tries to resolve dependencies using ResolveEventHandler in order which they were added (It calls MainResolver and then Resolver). As a result the host app fails, because it can't find required assemblies in MainResolver. The solution is to reorder ResolveEventHandler or remove MainResolver from AssemblyResolve after it was called. I think that exclude useless handler is easier.

So, I don't need to change anything in MyClass. Everything I need is to add following code in host app before I create instance of MyClass.

static MyApp(){
  AppDomain.CurrentDomain.AssemblyResolve += new ResolveEventHandler(MainResolver);
}

static void main(string[] args){
  //remove MainResolver
  AppDomain.CurrentDomain.AssemblyResolve -= MainResolver;

  MyClass my = new MyClass();
  my.DoWork();
}

Now it works like a charm!

---

Related Topics