How to Add a Custom System Call on X86 Ubuntu Linux

Unable to add a custom system call on x86 ubuntu linux

Okay, after hours of trial and error, I have finally found the problem. From linux kernel v4.17 onwards, x86_64 system calls may begin with "__x64_sys".

So, instead of using 548 64 printmsg sys_printmsg, I changed it to 548 64 printmsg __x64_sys_printmsg. Then everything works.

Hoped this helped everyone that might have this problem.

How to write system calls on debian/ubuntu

This is just example how to write a simple kernel system call.
Consider the following C function system_strcpy() that simply copies one string into another: similar to what strcpy() does.

#include<stdio.h>

long system_strcpy(char* dest, const char* src)
{
   int i=0;
   while(src[i]!=0)
      dest[i]=src[i++];

   dest[i]=0;
   return i;
}

Before writing, get a kernel source tar and untar it to get a linux-x.x.x directory.

File 1: linux-x.x.x/test/system_strcpy.c
Create a directory within the linux-x.x.x, named test and save this code as file system_strcpy.c in it.

#include<linux/linkage.h>
#include<linux/kernel.h>
asmlinkage long system_strcpy(char*dest, const char* src)
{
   int i=0;
   while(src[i]!=0)
      dest[i]=src[i++];

   dest[i]=0;
   return i;
}

File 2: linux-x.x.x/test/Makefile
Create a Makefile within the same test directory you created above and put this line in it:

obj-y := system_strcpy.o

File 3: linux-x.x.x/arch/x86/kernel/syscall_table_32.S
Now, you have to add your system call to the system call table.
Append to the file the following line:

.long system_strcpy

NOTE: For Kernel 3.3 and higher versions.

*Refer:linux-3.3.xx/arch/x86/syscalls/syscall_64.tbl*

And in there, now add at the end of the following series of lines:

310 64 process_vm_readv sys_process_vm_readv

311 64 process_vm_writev sys_process_vm_writev

312 64 kcmp sys_kcmp

313 64 system_strcpy system_strcpy

The format for the 3.3 version is in:
number abi name entry point

File 4: linux-x.x.x/arch/x86/include/asm/unistd_32.h

NOTE: This section is redundant for 3.3 and higher kernel versions

In this file, the names of all the system calls will be associated with a unique number. After the last system call-number pair, add a line

#define __NR_system_strcpy 338

(if 337 was the number associated with the last system call in the system call-number pair).

Then replace NR_syscalls value, stating total number of system calls with (the existing number incremented by 1) i.e. in this case the NR_syscalls should've been 338 and the new value is 339.

#define NR_syscalls 339

File 5: linux-x.x.x/include/linux/syscalls.h

Append to the file the prototype of our function.

asmlinkage long system_strcpy(char *dest,char *src);

just before the #endif line in the file.

File 6: Makefile at the root of source directory.

Open Makefile and find the line where core-y is defined and add the directory test to the end of that line.

core-y += kernel/ mm/ fs/ test/

Now compile the kernel. Issue:
make bzImage -j4

Install the kernel by executing the following command as root(or with root permissions):
make install

Reboot the system.

To use the recently created system call use:

syscall(338,dest,src); (or syscall(313,dest,src); for kernel 3.3+) instead of the regular strcpy library function.

#include "unistd.h"
#include "sys/syscall.h"
int main()
{
 char *dest=NULL,*src="Hello";
 dest=(char*)malloc(strlen(src)+1);
 syscall(338,dest,src);//syscall(313,dest,src); for kernel 3.3+
 printf("%s \n %s\n",src,dest);
 return 0;
}

Instead of numbers like 313,etc in syscall, you can also directly use __NR_system_strcpy

This is a generic example. You will need to do a little experimentation to see what works for your specific kernel version.

How to define a system call in Linux with non-default return type?

Returning long is the only proper way for a system call in Linux.

There is no way for a system call to return a value, which size differs from the size of long.

Do you expect ssize_t to have the same size as long on all platforms? If yes (this is a correct expectation), then there is no reason to prefer ssize_t over long. If you are not sure that your return type will fit to long on every platform, then you simply cannot use this return type.

For example, from the C standard you knows, that read function returns ssize_t. But read system call has long as return type (it is defined using DEFINE_SYSCALL macro):

SYSCALL_DEFINE3(read, unsigned int, fd, char __user *, buf, size_t, count)
{
    struct fd f = fdget(fd);
    ssize_t ret = -EBADF;

    if (f.file) {
        loff_t pos = file_pos_read(f.file);
        ret = vfs_read(f.file, buf, count, &pos);
        file_pos_write(f.file, pos);
        fdput(f);
    }
    return ret;
}

Note, that despite on long being return type of the system call, the above implementation returns a value of type ssize_t.

Adding new System Call to Linux Kernel 3.13 on 64 bit system

The problem was from step 6 to last step (Compile Kernel).

After step 5, we have to do following steps :

6- Compiling this kernel on my system

To configure the kernel I tried the following command :

# make menuconfig

After above command a pop up window came up and I made sure that ext4 was selected and then save.

Then to create DEB file from new kernel we have to :

# make -j 5 KDEB_PKGVERSION=1.arbitrary-name deb-pkg

It will create some deb files in /usr/src/.

After that we need to install them :

# dpkg -i linux*.deb

It will install new kernel on your system.

Now, reboot your system. After system rebooted you can find out whether new kernel is installed or not :

$ uname -r

And if you want to know your new System Call added to kernel or not just type :

$ cat /proc/kallsyms | grep <system call name>

In my case :

$ cat /proc/kallsyms | grep hello

Following output indicates that your System Call successfully added to the Kernel :

0000000000000000 T sys_hello

---

Related Topics