How to Allocate a New Tls Area with Clone System Call

How can a caller properly use the clone() system call by specifying multiple arguments?

The additional arguments are optional arguments used to specify additional operations of the clone operation. They're only used if particular flags are set in the flags argument. If you don't set any of those flags, you don't need to supply the additional arguments.

If you set the CLONE_PARENT_SETTID flag, the child's thread ID will be stored in the location that parent_tid points to in the parent process.

If you set the CLONE_SETTLS flag, the tls argument will be used as the address of the thread-local storage descriptor.

If you set the CLONE_CHILD_SETTID flag, the child's thread ID will be stored in the location that child_tid points to in the child process.

It's done this way for backward compatibility. These arguments weren't in the original clone() system call, but were added in later Linux versions. They're optional so that older code will continue to compile.

Who sets the RIP register when you call the clone syscall?

Normally the way it works is that, when the computer boots, Linux sets up a MSR (Model Specific Register) to work with the assembly instruction syscall. The assembly instruction syscall will make the RIP register jump to the address specified in the MSR to enter kernel mode. As stated in 64-ia-32-architectures-software-developer-vol-2b-manual from Intel:

SYSCALL invokes an OS system-call handler at privilege level 0.
It does so by loading RIP from the IA32_LSTAR MSR

Once in kernel mode, the kernel will look at the arguments passed into conventional registers (RAX, RBX etc.) to determine what the syscall is asking. Then the kernel will invoke one of the sys_XXX functions whose prototypes are in linux/syscalls.h (https://elixir.bootlin.com/linux/latest/source/include/linux/syscalls.h#L217). The definition of sys_clone is in kernel/fork.c.

SYSCALL_DEFINE5(clone, unsigned long, clone_flags, unsigned long, newsp,
         int __user *, parent_tidptr,
         int __user *, child_tidptr,
         unsigned long, tls)
#endif
{
    return _do_fork(clone_flags, newsp, 0, parent_tidptr, child_tidptr, tls);
}

The SYSCALLDEFINE5 macro takes the first argument and prefixes sys_ to it. This function is actually sys_clone and it calls _do_fork.

It means there really isn't a clone() function which is invoked by glibc to call into the kernel. The kernel is called with the syscall instruction, it jumps to an address specified in the MSR and then it invokes one of the syscalls in the sys_call_table.

The entry point to the kernel for x86 is here: https://github.com/torvalds/linux/blob/16f73eb02d7e1765ccab3d2018e0bd98eb93d973/arch/x86/entry/entry_64.S. If you scroll down you'll see the line: call *sys_call_table(, %rax, 8). Basically, call one of the functions of the sys_call_table. The implementation of the sys_call_table is here: https://elixir.bootlin.com/linux/latest/source/arch/x86/entry/syscall_64.c#L20.

// SPDX-License-Identifier: GPL-2.0
/* System call table for x86-64. */

#include <linux/linkage.h>
#include <linux/sys.h>
#include <linux/cache.h>
#include <linux/syscalls.h>
#include <asm/unistd.h>
#include <asm/syscall.h>

#define __SYSCALL_X32(nr, sym)
#define __SYSCALL_COMMON(nr, sym) __SYSCALL_64(nr, sym)

#define __SYSCALL_64(nr, sym) extern long __x64_##sym(const struct pt_regs *);
#include <asm/syscalls_64.h>
#undef __SYSCALL_64

#define __SYSCALL_64(nr, sym) [nr] = __x64_##sym,

asmlinkage const sys_call_ptr_t sys_call_table[__NR_syscall_max+1] = {
    /*
     * Smells like a compiler bug -- it doesn't work
     * when the & below is removed.
     */
    [0 ... __NR_syscall_max] = &__x64_sys_ni_syscall,
#include <asm/syscalls_64.h>
};

I recommend you read the following: https://0xax.gitbooks.io/linux-insides/content/SysCall/linux-syscall-2.html. On this website is stated that

As you can see, we include the asm/syscalls_64.h header at the end of the array. This header file is generated by the special script at arch/x86/entry/syscalls/syscalltbl.sh and generates our header file from the syscall table (https://github.com/torvalds/linux/blob/16f73eb02d7e1765ccab3d2018e0bd98eb93d973/arch/x86/entry/syscalls/syscall_64.tbl).

...

...

So, after this, our sys_call_table takes the following form:

asmlinkage const sys_call_ptr_t sys_call_table[__NR_syscall_max+1] = {
   [0 ... __NR_syscall_max] = &sys_ni_syscall,
   [0] = sys_read,
   [1] = sys_write,
   [2] = sys_open,
   ...
   ...
   ...
};

Once you have the table generated, one of its entries is being jumped to when you use the syscall assembly instruction. For clone() it will call sys_clone() which itself calls _do_fork(). Which is defined as such:

long _do_fork(unsigned long clone_flags,
          unsigned long stack_start,
          unsigned long stack_size,
          int __user *parent_tidptr,
          int __user *child_tidptr,
          unsigned long tls)
{
    struct task_struct *p;
    int trace = 0;
    long nr;

    /*
     * Determine whether and which event to report to ptracer.  When
     * called from kernel_thread or CLONE_UNTRACED is explicitly
     * requested, no event is reported; otherwise, report if the event
     * for the type of forking is enabled.
     */
    if (!(clone_flags & CLONE_UNTRACED)) {
        if (clone_flags & CLONE_VFORK)
            trace = PTRACE_EVENT_VFORK;
        else if ((clone_flags & CSIGNAL) != SIGCHLD)
            trace = PTRACE_EVENT_CLONE;
        else
            trace = PTRACE_EVENT_FORK;

        if (likely(!ptrace_event_enabled(current, trace)))
            trace = 0;
    }

    p = copy_process(clone_flags, stack_start, stack_size,
             child_tidptr, NULL, trace, tls);
    /*
     * Do this prior waking up the new thread - the thread pointer
     * might get invalid after that point, if the thread exits quickly.
     */
    if (!IS_ERR(p)) {
        struct completion vfork;
        struct pid *pid;

        trace_sched_process_fork(current, p);

        pid = get_task_pid(p, PIDTYPE_PID);
        nr = pid_vnr(pid);

        if (clone_flags & CLONE_PARENT_SETTID)
            put_user(nr, parent_tidptr);

        if (clone_flags & CLONE_VFORK) {
            p->vfork_done = &vfork;
            init_completion(&vfork);
            get_task_struct(p);
        }

        wake_up_new_task(p);

        /* forking complete and child started to run, tell ptracer */
        if (unlikely(trace))
            ptrace_event_pid(trace, pid);

        if (clone_flags & CLONE_VFORK) {
            if (!wait_for_vfork_done(p, &vfork))
                ptrace_event_pid(PTRACE_EVENT_VFORK_DONE, pid);
        }

        put_pid(pid);
    } else {
        nr = PTR_ERR(p);
    }
    return nr;
}

It calls wake_up_new_task() which puts the task on the runqueue and wakes it. I'm surprised it even wakes the task immediatly. I would have guessed that the scheduler would have done it instead and that it would have been given a high priority to run as soon as possible. In itself, the kernel doesn't have to receive a function pointer because as stated on the manpage for clone():

The raw clone() system call corresponds more closely to fork(2)
in that execution in the child continues from the point of the
call. As such, the fn and arg arguments of the clone() wrapper
function are omitted.

The child continues execution where the syscall was made. I don't understand exactly the mechanism but in the end the child will continue execution in a new thread. The parent thread (which created the new child thread) returns and the child thread jumps to the specified function instead.

I think it works with the following lines (on the link you provided):

testq   %rax,%rax
jl  SYSCALL_ERROR_LABEL
jz  L(thread_start) //Child jumps to thread_start

ret //Parent returns to where it was

Because rax is a 64 bits register, they use the 'q' version of the GNU syntax assembly instruction test. They test if rax is zero. If it is less than zero then there was an error. If it is zero then jump to thread_start. If it is not zero nor negative (in the case of the parent thread), continue execution and return. The new thread is created with rax as 0. It allows to diffenrentiate between the parent and the child thread.

EDIT

As stated on the link you provided,

The parameters are passed in register and on the stack from userland:
rdi: fn
rsi: child_stack
rdx: flags
rcx: arg
r8d: TID field in parent
r9d: thread pointer

So when your program executes the following lines:

/* Insert the argument onto the new stack.  */
subq    $16,%rsi
movq    %rcx,8(%rsi)

/* Save the function pointer.  It will be popped off in the
      child in the ebx frobbing below.  */
movq    %rdi,0(%rsi)

it inserts the function pointer and arguments onto the new stack. Then it calls the kernel which itself doesn't have to push anything onto the stack. It just receives the new stack as an argument and then makes the child's thread RSP register point to it. I would guess this happens in the copy_process() function (called from fork()) along the lines of:

retval = copy_thread_tls(clone_flags, stack_start, stack_size, p, tls);
if (retval)
    goto bad_fork_cleanup_io;

It seems to be done in the copy_thread_tls() function which itself calls copy_thread(). copy_thread() has its prototype in include/linux/sched.h and it is defined based on the architecture. I'm not sure where it is defined for x86.

How to access errno after clone (or: How to set errno location)

According to the glibc source code, errno is defined as a thread-local variable. Unfortunately, this requires significant C library support. Any threads created using pthread_create() will be made aware of thread-local variables. I would not even bother trying to get glibc to accept your foreign threads.

An alternative would be to use a different libc implementation that may allow you to extract some of its internal structures and manually set the thread control block if errno is part of it. This would be incredibly hacky and unreliable. I doubt you'll find anything like __set_errno_location(), but rather something like __set_tcb().

#include <bits/some_hidden_file.h>

void init_errno(void)
{
    struct __tcb* tcb;

    /* allocate a dummy thread control block (malloc may set errno
     * so might have to store the tcb on stack or allocate it in the
     * parent) */
    tcb = malloc(sizeof(struct __tcb));

    /* initialize errno */
    tcb->errno = 0;

    /* set pointer to thread control block (x86) */
    arch_prctl(ARCH_SET_FS, tcb);
}

This assumes that the errno macro expands to something like: ((struct __tcb*)__read_fs())->errno.

Of course, there's always the option of implementing an extremely small subset of libc yourself. Or you could write your own implementation of the write() system call with a custom stub to handle errno and have it co-exist with the chosen libc implementation.

#define my_errno /* errno variable stored at some known location */

ssize_t my_write(int fd, const void* buf, size_t len)
{
    ssize_t ret;

    __asm__ (
        /* set system call number */
        /* set up parameters */
        /* make the call */
        /* retrieve return value in c variable */
    );

    if (ret >= -4096 && ret < 0) {
        my_errno = -ret;
        return -1;
    }

    return ret;
}

I don't remember the exact details of GCC inline assembly and the system call invocation details vary depending on platform.

Personally, I'd just implement a very small subset of libc, which would just consist of a little assembler and a few constants. This is remarkably simple with so much reference code available out there, although it may be overambitious.

---

Related Topics