X86_64 Assembly Linux System Call Confusion

x86_64 Assembly Linux System Call Confusion

You're running into one surprising difference between i386 and x86_64: they don't use the same system call mechanism. The correct code is:

movq $60, %rax
movq $2,  %rdi   ; not %rbx!
syscall

Interrupt 0x80 always invokes 32-bit system calls. It's used to allow 32-bit applications to run on 64-bit systems.

For the purposes of learning, you should probably try to follow the tutorial exactly, rather than translating on the fly to 64-bit -- there are a few other significant behavioral differences that you're likely to run into. Once you're familiar with i386, then you can pick up x86_64 separately.

Error handling for system calls in x86 assembly, under Linux

The legitimate return values from system calls are always either positive (signed) integers or addresses. When they are positive integers, the negative values can be used as error codes, so any negative value is an error.

So the only tricky case is when the return value is an address. It turns out that the addresses corresponding to integers in the range -4096..-1 are all in a kernel reserved page that will never be returned by the kernel -- so any bit pattern in that range will only ever be returned as an error code, and not as a valid address.

In addition, ALL addresses that correspond to negative integers in x86_64 are reserved for the kernel or invalid -- user addresses will always be in the range 0..2⁴⁷-1. So for x86_64 you need only check the sign bit (top bit) of %rax -- if it is set, there was an error.

test %rax, %rax
js   error

Fo 32-bit x86 code, this is not the case -- some valid addresses are negative numbbers. So in that case, you need to explicitly check for the error range, which is actually easiest to do with an unsigned comparison

cmpl  %eax, 0xfffff000   # unsigned 2^32 - 4096, aka signed -4096
ja    error              # -4095 .. -1 is an error, anything else is non-error

How to properly pass buffer pointers to Linux system calls in x86_64 assembly?

The actual problem is not with the buffer but with its length. Notice in the prototype you have socklen_t *addrlen so that should be a pointer. The value 15 that you pass is not a pointer hence the -EFAULT.

You should change the .length: equ $-ip_buff to ip_length: dd $-ip_buff and then use syscall getpeername,r12,ip_buff,ip_length

Assembly , syscall not work as expected. Ubuntu Linux x86_64 , using AT&T syntax

mov     $output,%rsi     # address of string to output moved to rsi
                                      ^^^^^^

Address of string. The value $12 is not the character sequence "12". If you wanted to print the string 12, you would need to load 0x31 and 0x32 ('1' and '2') into the memory area (making it big enough) the use 2 as the length.

For example, movw $0x3231, output or better movw $0x3231, output(%rip) to use RIP-relative addressing for static data, like normal for x86-64. (Unlike NASM, GAS syntax doesn't $'12' as a way to write the same integer constant.)

If you want to print an integer as a string, you'll probably want to manipulate it mathematically so you can do it one digit at a time. (Printing an integer as a string with AT&T syntax, with Linux system calls instead of printf)

Confused about 64-bit registers - ASM

Run strace ./my_program - you make a bogus stat system call, then write which succeeds, then fall off the end and segfault.

$ strace ./foo 
execve("./foo", ["./foo"], 0x7ffe6b91aa00 /* 51 vars */) = 0
stat(0x1, 0x401000)                     = -1 EFAULT (Bad address)
write(0, "Hello, World\n", 13Hello, World
)          = 13
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0xd} ---
+++ killed by SIGSEGV (core dumped) +++
Segmentation fault (core dumped)

It's not register names that are your problem, it's call numbers. You're using 32-bit call numbers but calling the 64-bit syscall ABI.

Call numbers and calling convention both differ.

int 0x80 system calls only ever look at the low 32 bits of registers which is why you shouldn't use them in 64-bit code.

The code you posted in a comment with mov rcx, message would work fine with mov ecx, message and so on, if it works with mov rcx, message. See What happens if you use the 32-bit int 0x80 Linux ABI in 64-bit code?.

Note that writing a 32-bit register zero-extends into the full 64-bit register so you should always use mov edi, 1 instead of mov rdi, 1. (Although NASM will do this optimization for you to save code-size; they're so equivalent that some assemblers will silently do it for you.)

Difference in ABI between x86_64 Linux functions and syscalls

The syscall instruction is intended to provide a quicker method of entering Ring-0 in order to carry out a system call. This is meant to be an improvement over the old method, which was to raise a software interrupt (int 0x80 on Linux).

Part of the reason the instruction is faster is because it does not change memory, or even change rsp to point at a kernel stack. Unlike a software interrupt, where the CPU is forced to allow the OS to resume operation without clobbering anything, for this command the CPU is allowed to assume the software is aware that something is happening here.

In particular, syscall stores two parts of the user-space state in registers. The RIP to return to after the call is stored in rcx, and the flags are stored in R11 (because RFLAGS is masked with a kernel-supplied value before entry to the kernel). This means that both those registers are clobbered by the instruction.

Since they are clobbered, the syscall ABI uses another register instead of rcx, hence the use of r10 for the 4th argument.

r10 is a natural choice, since in the x86-64 SystemV ABI it's not used for passing function args, and functions don't need to preserve their caller's value of r10. So a syscall wrapper function can mov %rcx, %r10 without any save/restore. This wouldn't be possible with any other register, for 6-arg syscalls and the SysV ABI's function calling convention.

---

BTW, the 32-bit system call ABI is also accessible with sysenter, which requires cooperation between user-space and kernel-space to allow returning to user-space after a sysenter. (i.e. storing some state in user-space before running sysenter). This is higher performance than int 0x80, but awkward. Still, glibc uses it (by jumping to user-space code in the vdso pages that the kernel maps into the address space of every process).

AMD's syscall is another approach to the same idea as Intel's sysenter: to make entry/exit from the kernel less expensive by not preserving absolutely everything.

How are system calls interpreted in x86 assembly linux

Simply the value (number) 4 is loaded into eax, no magic there. The operating system will look at the value in eax to figure out what function you want. System call number is a code that identifies the various available kernel functions you can use.

close() x86_64 system call weird return value

As expected, a rollback to a previous kernel version solved the problem. I'm not really a kernel specialist but as far as I understand, the answer given by @R.. makes sense:

This is a 64-bit machine, so 1<<32-9 should never appear. The problem is that the kernel is internally using unsigned instead of int for the return value of some of these functions, then returning -EBADF which gets reduced modulo 2^32 rather than modulo 2^64

The problem is that the generic code in the libc syscall wrappers that handles syscall error returns has to treat the return value as a long (since it could be a pointer or long for some syscalls) when making the comparison to see if it's a small negative value that would indicate an error. But the kernel returned (long)(unsigned)-9 which is very different from (long)-9. or (unsigned long)-9 (either of which would have worked).

---

Related Topics