How to Use Qemu for Learning Arm Linux Kernel Development

Linux kernel development using Qemu?

Download Linux kernel source code from kernel.org, then modify, configure, build your kernel. After building you'll get a kernel image usually located at arch/x86/boot/bzImage (if your arch is x86, look at appropriate location for your arch), and this is what you need to test your modified kernel on Qemu. You'll also need a rootfs (use busybox) to run it properly. Use qemu's '-kernel' parameter to pass the bzImage. An example:

 `qemu-system-x86_64 -m 1024 -smp 4 -kernel arch/x86/boot/bzImage -initrd initramfs.gz -append "rdinit=/ root=/dev/ram`

it tells qemu to use 1024 of ram, use 4 cpu.

Playing/Learning -- QEMU (for ARM), Angstrom Linux (or Debian)

The beagleboard, hawkboard, open-rd sites all tend to lead to their distros being built on qemu (arm), and from there there is no reason why you cannot just continue to keep running on the simulation instead of heading for hardware.

This is an example of how to do it with ubuntu.
https://wiki.edubuntu.org/ARM/RootfsFromScratch

Yes it is also possible to cross compile everything as well, I would start with wiki pages that hand hold you through all of the steps. Or as with the hawkboard or beagleboard get a pre-built binary (kernel and root file system) and just boot it and run on that environment and not mess with building everything.

x86 linux kernel development in qemu

WordPress typography features strike again! You're using the wrong kind of quotation marks:

-append “root=/dev/ram rdinit=/sbin/init”
        ^                               ^

The indicated characters are, respectively, U+201C LEFT DOUBLE QUOTATION MARK and U+201D RIGHT DOUBLE QUOTATION MARK (i.e, "curly quotes"), not normal quotation marks.

Retype the command line by hand (or, at least, those two characters) and it'll work correctly.

How to compile baremetal hello_world.c and run it on qemu-system-aarch64?

You are right that you can use qemu-system-aarch64 for achieving your goal.
Depending on what you exactly want to do, you have multiple options:

1. use the semihosting mode of qemu, along with the gcc --specs=rdimon.specs with newlib, or using another semihosting library, such as the one available in the Arm Trusted Firmware source code - the example hereafter uses this approach.

2. provide your own syscalls.c , and use the --specs=nosys.specs ld option, so that you can use newlib in baremetal programs: I would suggest to read the excellent article from Francesco Balducci on the Balau blog - the example hereafter uses this approach.

3. use a more baremetal-like approach, such as the one described hereafter: it does use sprintf() and the pl011 UART of the qemu-virt machine for displaying the resulting string.

gcc_arm64_ram.ld:

/******************************************************************************
 * @file     gcc_arm32.ld
 * @brief    GNU Linker Script for Cortex-M based device
 * @version  V2.0.0
 * @date     21. May 2019
 ******************************************************************************/
/*
 * Copyright (c) 2009-2019 Arm Limited. All rights reserved.
 *
 * SPDX-License-Identifier: Apache-2.0
 *
 * Licensed under the Apache License, Version 2.0 (the License); you may
 * not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 * www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an AS IS BASIS, WITHOUT
 * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

MEMORY
{
  RAM   (rwx) : ORIGIN = __RAM_BASE, LENGTH = __RAM_SIZE
}

/* Linker script to place sections and symbol values. Should be used together
 * with other linker script that defines memory regions FLASH and RAM.
 * It references following symbols, which must be defined in code:
 *   Reset_Handler : Entry of reset handler
 *
 * It defines following symbols, which code can use without definition:
 *   __exidx_start
 *   __exidx_end
 *   __copy_table_start__
 *   __copy_table_end__
 *   __zero_table_start__
 *   __zero_table_end__
 *   __etext
 *   __data_start__
 *   __preinit_array_start
 *   __preinit_array_end
 *   __init_array_start
 *   __init_array_end
 *   __fini_array_start
 *   __fini_array_end
 *   __data_end__
 *   __bss_start__
 *   __bss_end__
 *   __end__
 *   end
 *   __HeapLimit
 *   __StackLimit
 *   __StackTop
 *   __stack
 */
ENTRY(Reset_Handler)

SECTIONS
{
  .text :
  {
    KEEP(*(.vectors))
    *(.text*)

    KEEP(*(.init))
    KEEP(*(.fini))

    /* .ctors */
    *crtbegin.o(.ctors)
    *crtbegin?.o(.ctors)
    *(EXCLUDE_FILE(*crtend?.o *crtend.o) .ctors)
    *(SORT(.ctors.*))
    *(.ctors)

    /* .dtors */
    *crtbegin.o(.dtors)
    *crtbegin?.o(.dtors)
    *(EXCLUDE_FILE(*crtend?.o *crtend.o) .dtors)
    *(SORT(.dtors.*))
    *(.dtors)

    *(.rodata*)

    KEEP(*(.eh_frame*))
  } > RAM

  /*
   * SG veneers:
   * All SG veneers are placed in the special output section .gnu.sgstubs. Its start address
   * must be set, either with the command line option �--section-start� or in a linker script,
   * to indicate where to place these veneers in memory.
   */
/*
  .gnu.sgstubs :
  {
    . = ALIGN(32);
  } > RAM
*/
  .ARM.extab :
  {
    *(.ARM.extab* .gnu.linkonce.armextab.*)
  } > RAM

  __exidx_start = .;
  .ARM.exidx :
  {
    *(.ARM.exidx* .gnu.linkonce.armexidx.*)
  } > RAM
  __exidx_end = .;

  .copy.table :
  {
    . = ALIGN(16);
    __copy_table_start__ = .;
    LONG (__etext)
    LONG (__data_start__)
    LONG (__data_end__ - __data_start__)
    /* Add each additional data section here */
/*
    LONG (__etext2)
    LONG (__data2_start__)
    LONG (__data2_end__ - __data2_start__)
*/
    __copy_table_end__ = .;
  } > RAM

  .zero.table :
  {
    . = ALIGN(16);
    __zero_table_start__ = .;
    /* Add each additional bss section here */
/*
    LONG (__bss2_start__)
    LONG (__bss2_end__ - __bss2_start__)
*/
    __zero_table_end__ = .;
  } > RAM

  /**
   * Location counter can end up 2byte aligned with narrow Thumb code but
   * __etext is assumed by startup code to be the LMA of a section in RAM
   * which must be 4byte aligned 
   */
  __etext = ALIGN(16);

  .data : AT (__etext)
  {
    __data_start__ = .;
    *(vtable)
    *(.data)
    *(.data.*)

    . = ALIGN(16);
    /* preinit data */
    PROVIDE_HIDDEN (__preinit_array_start = .);
    KEEP(*(.preinit_array))
    PROVIDE_HIDDEN (__preinit_array_end = .);

    . = ALIGN(16);
    /* init data */
    PROVIDE_HIDDEN (__init_array_start = .);
    KEEP(*(SORT(.init_array.*)))
    KEEP(*(.init_array))
    PROVIDE_HIDDEN (__init_array_end = .);

    . = ALIGN(16);
    /* finit data */
    PROVIDE_HIDDEN (__fini_array_start = .);
    KEEP(*(SORT(.fini_array.*)))
    KEEP(*(.fini_array))
    PROVIDE_HIDDEN (__fini_array_end = .);

    KEEP(*(.jcr*))
    . = ALIGN(16);
    /* All data end */
    __data_end__ = .;

  } > RAM

  /*
   * Secondary data section, optional
   *
   * Remember to add each additional data section
   * to the .copy.table above to asure proper
   * initialization during startup.
   */
/*
  __etext2 = ALIGN(16);

  .data2 : AT (__etext2)
  {
    . = ALIGN(16);
    __data2_start__ = .;
    *(.data2)
    *(.data2.*)
    . = ALIGN(16);
    __data2_end__ = .;

  } > RAM2
*/

  .bss :
  {
    . = ALIGN(16);
    __bss_start__ = .;
    *(.bss)
    *(.bss.*)
    *(COMMON)
    . = ALIGN(16);
    __bss_end__ = .;
  } > RAM AT > RAM

  /*
   * Secondary bss section, optional
   *
   * Remember to add each additional bss section
   * to the .zero.table above to asure proper
   * initialization during startup.
   */
/*
  .bss2 :
  {
    . = ALIGN(16);
    __bss2_start__ = .;
    *(.bss2)
    *(.bss2.*)
    . = ALIGN(16);
    __bss2_end__ = .;
  } > RAM2 AT > RAM2
*/

  .heap (COPY) :
  {
    . = ALIGN(16);
    __end__ = .;
    PROVIDE(end = .);
    . = . + __HEAP_SIZE;
    . = ALIGN(16);
    __HeapLimit = .;
  } > RAM

  .stack (ORIGIN(RAM) + LENGTH(RAM) - __STACK_SIZE) (COPY) :
  {
    . = ALIGN(16);
    __StackLimit = .;
    . = . + __STACK_SIZE;
    . = ALIGN(16);
    __StackTop = .;
  } > RAM
  PROVIDE(__stack = __StackTop);

  /* Check if data + heap + stack exceeds RAM limit */
  ASSERT(__StackLimit >= __HeapLimit, "region RAM overflowed with stack")
}

qemu-virt-aarch64.ld:

__RAM_BASE = 0x40000000;
__RAM_SIZE =  0x08000000;
__STACK_SIZE = 0x00100000;
__HEAP_SIZE  =  0x00100000;
INCLUDE gcc_arm64_ram.ld

startup.s:

                .title startup64.s
                .arch armv8-a
                .text
                .section .text.startup,"ax"    
                .globl Reset_Handler
Reset_Handler:
                ldr x0, =__StackTop
                mov sp, x0
                bl  main
wait:           wfe
                b wait
               .end

pl011.c:

#include <stdint.h>

static volatile unsigned int * const UART0DR = ( unsigned int * ) ( uintptr_t * ) 0x9000000;

int putchar(int c)
{
    *UART0DR = c; /* Transmit char */
     return c;
}

void putchar_uart0( int c )
{
    *UART0DR = c; /* Transmit char */
}

void putc_uart0( int c )
{
    *UART0DR = c; /* Transmit char */
}

void print_uart0( const char * s )
{
    while( *s != '\0' )                     /* Loop until end of string */
    {
        *UART0DR = ( unsigned int ) ( *s ); /* Transmit char */
        s++;                                /* Next char */
    }
}

void puts_uart0( const char * s )
{
    while( *s != '\0' )                     /* Loop until end of string */
    {
        *UART0DR = ( unsigned int ) ( *s ); /* Transmit char */
        if (*s == '\n') {
           *UART0DR = ( unsigned int ) ( '\r' );
        } 
        s++;                                /* Next char */
    }
}

pl011.h:

#pragma once

#ifdef __cplusplus
extern "C" {
#endif

void putchar_uart0( int c );
void print_uart0( const char * s );
void putc_uart0( int c );
void puts_uart0( const char * s );

#ifdef __cplusplus
}
#endif

qemu-virt-aarch64.c:

#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <errno.h>

#include "pl011.h"

// angel/semihosting interface
#define SYS_WRITE0                       0x04 
static uint64_t semihosting_call(uint32_t operation, uint64_t parameter)
{
    __asm("HLT #0xF000");
}

// syscall stubs
int _close (int fd)
{
    errno = EBADF;
    return -1;
}

int _isatty (int fd)
{
    return 1;
}

int _fstat (int fd, struct stat * st)
{
    errno = EBADF;
    return -1;
}

off_t _lseek (int fd, off_t ptr, int dir)
{
    errno = EBADF;
    return (off_t) -1;
}

int _read (int fd, void *ptr, size_t len)
{
    errno = EBADF;
    return -1;
}

int _write (int fd, const char *ptr, size_t len)
{
    for (size_t i = 0; i < len; i++) {
        putchar_uart0(ptr[i]);
    }
    return len;
}

void main()
{
   char buffer[BUFSIZ];
   uint64_t regCurrentEL;

   __asm volatile ("mrs %0, CurrentEL" : "=r" (regCurrentEL));

   // UART0
   sprintf(buffer, "Hello EL%d World!\n", (regCurrentEL >> 2) & 0b11);
   puts_uart0(buffer);

   // angel/semihosting interface
   sprintf(buffer, "Hello semi-hosted EL%d World!\n", (regCurrentEL >> 2) & 0b11);
   semihosting_call(SYS_WRITE0, (uint64_t) (uintptr_t)  buffer);

   // newlib -  custom syscalls.c, with _write() using UART0
   printf("Hello EL%d World! (syscalls version)\n", (regCurrentEL >> 2) & 0b11);
}

Please note that the code responsible for initializing the .bss section was omitted.

Compiling:

/opt/arm/9/gcc-arm-9.2-2019.12-x86_64-aarch64-none-elf/bin/aarch64-none-elf-gcc -I. -O0 -ggdb -mtune=cortex-a53 -nostartfiles -ffreestanding --specs=nosys.specs -L. -Wl,-T,qemu-virt-aarch64.ld -o virt.elf startup.s  pl011.c qemu-virt-aarch64.c 

Running:

/opt/qemu-5.2.0/bin/qemu-system-aarch64 -semihosting -m 128M -nographic  -monitor none -serial stdio  -machine virt,gic-version=2,secure=on,virtualization=on -cpu cortex-a53 -kernel virt.elf
Hello EL3 World!
Hello semi-hosted EL3 World!
Hello EL3 World! (syscalls version)

Qemu arm Linux kernel boot debug, no source code

The Linux kernel uses a 2-step booting processing (and this does not include any boot loader like u-Boot ...). You can better understand this especially by looking into 2 .lds files (detailed below) for linking:

1. arch/arm/boot/compressed/vmlinux.lds.in, which generates arch/arm/boot/compressed/vmlinux.lds.

   Along with other .o files in arch/arm/boot/compressed, a vmlinux is generated inside this folder.

   You can use arm-none-eabi-nm -a -n arch/arm/boot/compressed/vmlinux to see the symbols for this stage. All addresses are physical addresses.

   These symbols are NOT included in System.map

2. The second vmlinux is generated by kernel .o files and arch/arm/kernel/vmlinux.lds (note: the path is different)

I hope this explains why you can not see the booting source code in Eclipse.

---

Related Topics