How to Invoke a System Call Via Syscall or Sysenter in Inline Assembly

How to invoke a system call via syscall or sysenter in inline assembly?

First of all, you can't safely use GNU C Basic asm(""); syntax for this (without input/output/clobber constraints). You need Extended asm to tell the compiler about registers you modify. See the inline asm in the GNU C manual and the inline-assembly tag wiki for links to other guides for details on what things like "D"(1) means as part of an asm() statement.

You also need asm volatile because that's not implicit for Extended asm statements with 1 or more output operands.


I'm going to show you how to execute system calls by writing a program that writes Hello World! to standard output by using the write() system call. Here's the source of the program without an implementation of the actual system call :

#include <sys/types.h>

ssize_t my_write(int fd, const void *buf, size_t size);

int main(void)
{
const char hello[] = "Hello world!\n";
my_write(1, hello, sizeof(hello));
return 0;
}

You can see that I named my custom system call function as my_write in order to avoid name clashes with the "normal" write, provided by libc. The rest of this answer contains the source of my_write for i386 and amd64.

i386

System calls in i386 Linux are implemented using the 128th interrupt vector, e.g. by calling int 0x80 in your assembly code, having set the parameters accordingly beforehand, of course. It is possible to do the same via SYSENTER, but actually executing this instruction is achieved by the VDSO virtually mapped to each running process. Since SYSENTER was never meant as a direct replacement of the int 0x80 API, it's never directly executed by userland applications - instead, when an application needs to access some kernel code, it calls the virtually mapped routine in the VDSO (that's what the call *%gs:0x10 in your code is for), which contains all the code supporting the SYSENTER instruction. There's quite a lot of it because of how the instruction actually works.

If you want to read more about this, have a look at this link. It contains a fairly brief overview of the techniques applied in the kernel and the VDSO. See also The Definitive Guide to (x86) Linux System Calls - some system calls like getpid and clock_gettime are so simple the kernel can export code + data that runs in user-space so the VDSO never needs to enter the kernel, making it much faster even than sysenter could be.


It's much easier to use the slower int $0x80 to invoke the 32-bit ABI.

// i386 Linux
#include <asm/unistd.h> // compile with -m32 for 32 bit call numbers
//#define __NR_write 4
ssize_t my_write(int fd, const void *buf, size_t size)
{
ssize_t ret;
asm volatile
(
"int $0x80"
: "=a" (ret)
: "0"(__NR_write), "b"(fd), "c"(buf), "d"(size)
: "memory" // the kernel dereferences pointer args
);
return ret;
}

As you can see, using the int 0x80 API is relatively simple. The number of the syscall goes to the eax register, while all the parameters needed for the syscall go into respectively ebx, ecx, edx, esi, edi, and ebp. System call numbers can be obtained by reading the file /usr/include/asm/unistd_32.h.

Prototypes and descriptions of the functions are available in the 2nd section of the manual, so in this case write(2).

The kernel saves/restores all the registers (except EAX) so we can use them as input-only operands to the inline asm. See What are the calling conventions for UNIX & Linux system calls (and user-space functions) on i386 and x86-64

Keep in mind that the clobber list also contains the memory parameter, which means that the instruction listed in the instruction list references memory (via the buf parameter). (A pointer input to inline asm does not imply that the pointed-to memory is also an input. See How can I indicate that the memory *pointed* to by an inline ASM argument may be used?)

amd64

Things look different on the AMD64 architecture which sports a new instruction called SYSCALL. It is very different from the original SYSENTER instruction, and definitely much easier to use from userland applications - it really resembles a normal CALL, actually, and adapting the old int 0x80 to the new SYSCALL is pretty much trivial. (Except it uses RCX and R11 instead of the kernel stack to save the user-space RIP and RFLAGS so the kernel knows where to return).

In this case, the number of the system call is still passed in the register rax, but the registers used to hold the arguments now nearly match the function calling convention: rdi, rsi, rdx, r10, r8 and r9 in that order. (syscall itself destroys rcx so r10 is used instead of rcx, letting libc wrapper functions just use mov r10, rcx / syscall.)

// x86-64 Linux
#include <asm/unistd.h> // compile without -m32 for 64 bit call numbers
// #define __NR_write 1
ssize_t my_write(int fd, const void *buf, size_t size)
{
ssize_t ret;
asm volatile
(
"syscall"
: "=a" (ret)
// EDI RSI RDX
: "0"(__NR_write), "D"(fd), "S"(buf), "d"(size)
: "rcx", "r11", "memory"
);
return ret;
}

(See it compile on Godbolt)

Do notice how practically the only thing that needed changing were the register names, and the actual instruction used for making the call. This is mostly thanks to the input/output lists provided by gcc's extended inline assembly syntax, which automagically provides appropriate move instructions needed for executing the instruction list.

The "0"(callnum) matching constraint could be written as "a" because operand 0 (the "=a"(ret) output) only has one register to pick from; we know it will pick EAX. Use whichever you find more clear.


Note that non-Linux OSes, like MacOS, use different call numbers. And even different arg-passing conventions for 32-bit.

syscall from within GCC inline assembly

Something like


char p = 'P';

int main()
{
__asm__ __volatile__
(
" movl $1, %%edx \n\t"
" leal p , %%ecx \n\t"
" movl $0, %%ebx \n\t"
" movl $4, %%eax \n\t"
" int $0x80 \n\t"
::: "%eax", "%ebx", "%ecx", "%edx"
);
}

Add: note that I've used lea to Load the Effective Address of the char into ecx register; for the value of ebx I tried $0 and $1 and it seems to work anyway ...

Avoid the use of external char

int main()
{
__asm__ __volatile__
(
" movl $1, %%edx \n\t"
" subl $4, %%esp \n\t"
" movl $80, (%%esp)\n\t"
" movl %%esp, %%ecx \n\t"
" movl $1, %%ebx \n\t"
" movl $4, %%eax \n\t"
" int $0x80 \n\t"
" addl $4, %%esp\n\t"
::: "%eax", "%ebx", "%ecx", "%edx"
);
}

N.B.: it works because of the endianness of intel processors! :D

C++ SYSENTER x86 calls in inline assembly

Making system calls in x86 Windows is different from x64. You need to specify the correct arguments length in ret otherwise you will get illegal instruction and/or runtime esp.

Furthermore I don't recommend you to use inline assembly, instead use it inside an .asm file or as shellcode.

To make a correct x86 system call on x86 Windows:

mov eax, SYSCALL_INDEX
call sysentry
ret ARGUMENTS_LENGTH_SIZE
mov edx,esp
sysenter
retn

To make a correct x64 system call on x64 Windows:

mov eax, SYSCALL_INDEX
mov r10,rcx
syscall
retn

The above will work 100% correctly on any x86 and x64 Windows (tested). Can't help you with inline assembly though, because I never used it that way.

Enjoy.

How do I call the write syscall using inline assembler in GCC under MacOS X?

A generic solution to this type of question: Write a short test program doing the write() call you are interested in, and then use gcc -S to produce assembly or use otool to disassemble the binary; find out how the write() call was assembled and transform that into the appropriate inline assembly.

(edit) To see the actual syscall, follow the library call in the assembly code. In the example of write(), following the indirections will lead you via libSystem.B.dylib to _write in libsystem_kernel.dylib, which you can disassemble using otool.

(edit2) Full example below, using this test program test.c:

#include <stdio.h>
int main(void)
{
char buf[] = "test\n";
ssize_t n;
n = write(2, buf, sizeof(buf));
return n;
}

Compile and test, using -O1 -fverbose-asm to get concise, readable machine code:

% gcc -O1 -fverbose-asm -o test test.c
% ./test
test

Use otool to disassemble main() in the test binary:

% otool -p _main -tvV test
test:
(__TEXT,__text) section
_main:
0000000100000ef0 pushq %rbp
0000000100000ef1 movq %rsp,%rbp
0000000100000ef4 subq $0x10,%rsp
0000000100000ef8 leaq 0xfa(%rbp),%rsi
0000000100000efc movb $0x74,0xfa(%rbp)
0000000100000f00 movb $0x65,0xfb(%rbp)
0000000100000f04 movb $0x73,0xfc(%rbp)
0000000100000f08 movb $0x74,0xfd(%rbp)
0000000100000f0c movb $0x0a,0xfe(%rbp)
0000000100000f10 movb $0x00,0xff(%rbp)
0000000100000f14 movl $0x00000002,%edi
0000000100000f19 movl $0x00000006,%edx
0000000100000f1e xorb %al,%al
0000000100000f20 callq 0x100000f32 ; symbol stub for: _write
0000000100000f25 addq $0x10,%rsp
0000000100000f29 popq %rbp
0000000100000f2a ret

List the libraries test is dynamically linked against to find _write:

% otool -L test
test:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0)

Try to locate _write in libSystem.B.dylib:

% otool -p _write -tvV /usr/lib/libSystem.B.dylib
/usr/lib/libSystem.B.dylib:
(__TEXT,__text) section
Can't find -p symbol: _write

Check the dependencies of libSystem.B.dylib:

% otool -L /usr/lib/libSystem.B.dylib
/usr/lib/libSystem.B.dylib:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0)
/usr/lib/system/libcache.dylib (compatibility version 1.0.0, current version 47.0.0)
/usr/lib/system/libcommonCrypto.dylib (compatibility version 1.0.0, current version 55010.0.0)
/usr/lib/system/libcompiler_rt.dylib (compatibility version 1.0.0, current version 6.0.0)
/usr/lib/system/libcopyfile.dylib (compatibility version 1.0.0, current version 85.1.0)
/usr/lib/system/libdispatch.dylib (compatibility version 1.0.0, current version 187.9.0)
/usr/lib/system/libdnsinfo.dylib (compatibility version 1.0.0, current version 395.11.0)
/usr/lib/system/libdyld.dylib (compatibility version 1.0.0, current version 195.6.0)
/usr/lib/system/libkeymgr.dylib (compatibility version 1.0.0, current version 23.0.0)
/usr/lib/system/liblaunch.dylib (compatibility version 1.0.0, current version 392.38.0)
/usr/lib/system/libmacho.dylib (compatibility version 1.0.0, current version 800.0.0)
/usr/lib/system/libmathCommon.A.dylib (compatibility version 1.0.0, current version 2026.0.0)
/usr/lib/system/libquarantine.dylib (compatibility version 1.0.0, current version 36.6.0)
/usr/lib/system/libremovefile.dylib (compatibility version 1.0.0, current version 21.1.0)
/usr/lib/system/libsystem_blocks.dylib (compatibility version 1.0.0, current version 53.0.0)
/usr/lib/system/libsystem_c.dylib (compatibility version 1.0.0, current version 763.13.0)
/usr/lib/system/libsystem_dnssd.dylib (compatibility version 1.0.0, current version 1.0.0)
/usr/lib/system/libsystem_info.dylib (compatibility version 1.0.0, current version 1.0.0)
/usr/lib/system/libsystem_kernel.dylib (compatibility version 1.0.0, current version 1699.26.8)
/usr/lib/system/libsystem_network.dylib (compatibility version 1.0.0, current version 1.0.0)
/usr/lib/system/libsystem_notify.dylib (compatibility version 1.0.0, current version 80.1.0)
/usr/lib/system/libsystem_sandbox.dylib (compatibility version 1.0.0, current version 1.0.0)
/usr/lib/system/libunc.dylib (compatibility version 1.0.0, current version 24.0.0)
/usr/lib/system/libunwind.dylib (compatibility version 1.0.0, current version 30.0.0)
/usr/lib/system/libxpc.dylib (compatibility version 1.0.0, current version 77.19.0)

Guess that _write is probably contained in libsystem_kernel.dylib (or alternatively try all of them):

% otool -p _write -tvV /usr/lib/system/libsystem_kernel.dylib | head -20
(__TEXT,__text) section
_write:
0000000000017fd4 movl $0x02000004,%eax
0000000000017fd9 movq %rcx,%r10
0000000000017fdc syscall
0000000000017fde jae 0x00017fe5
0000000000017fe0 jmp cerror
0000000000017fe5 ret
0000000000017fe6 nop
0000000000017fe7 nop
[...]

Now we have all the assembly we need to construct the inline assembly version of test:

#include <stdio.h>
int main(void)
{
char buf[] = "test\n";
ssize_t n;
asm volatile (
"movl $0x00000002, %%edi\n" /* first argument */
"movl $0x00000006, %%edx\n" /* third argument */
"movl $0x02000004, %%eax\n" /* syscall number */
"syscall\n"
: "=A"(n) /* %rax: return value */
: "S"(buf)); /* %rsi: second argument */
return n;
}

Compile and test:

% gcc -O1 -fverbose-asm -o test-asm test-asm.c
% ./test-asm
test

That seems to work. The inline assembly above is not very refined; for example, you could pass in the first and third arguments dynamically as well, instead of hardcoding them in the assembly code.

Callling the brk syscall using inline assembler

I'm not on linux, so I can't test this. But based on http://blog.rchapman.org/post/36801038863/linux-system-call-table-for-x86-64, I would expect it to be something like this:

__asm__ __volatile__("syscall"
: "=a" (ret)
: "0" (12), "D" (a1)
: "rcx", "r11", "cc");

You may also need the "memory" clobber.

To learn about constraints, check out the i386 section here: https://gcc.gnu.org/onlinedocs/gcc/Machine-Constraints.html



Related Topics



Leave a reply



Submit