What Is the Purpose of 'Kernel'

What is an OS kernel ? How does it differ from an operating system?

The technical definition of an operating system is "a platform that consists of specific set of libraries and infrastructure for applications to be built upon and interact with each other". A kernel is an operating system in that sense.

The end-user definition is usually something around "a software package that provides a desktop, shortcuts to applications, a web browser and a media player". A kernel doesn't match that definition.

So for an end-user a Linux distribution (say Ubuntu) is an Operating System while for a programmer the Linux kernel itself is a perfectly valid OS depending on what you're trying to achieve. For instance embedded systems are mostly just kernel with very small number of specialized processes running on top of them. In that case the kernel itself becomes the OS itself.

I think you can draw the line at what the majority of the applications running on top of that OS do require. If most of them require only kernel, the kernel is the OS, if most of them require X Window System running, then your OS becomes X + kernel.

what is left in operating system if we remove kernel?

A kernel is the part of the operating system that mediates access to system resources. It's responsible for enabling multiple applications to effectively share the hardware by controlling access to CPU, memory, disk I/O, and networking.

An operating system is the kernel plus applications that enable users to get something done (i.e compiler, text editor, window manager, etc).

But,these applications cannot be used if the kernel doesn't exist,because basic functions of OS are provided by the kernel.
Refer the diagram HERE . If the kernel is removed,you will have applications remaining,but you wouldn't be able to use them.

So to conclude,

The kernel is part of the operating system and closer to the hardware it provides low level services like:

device driver
process management
memory management
system calls

An operating system also includes applications like the user interface (shell,gui,tools and services).

What is the difference between user and kernel modes in operating systems?

Kernel Mode

In Kernel mode, the executing code has complete and unrestricted
access to the underlying hardware. It
can execute any CPU instruction and
reference any memory address. Kernel
mode is generally reserved for the
lowest-level, most trusted functions
of the operating system. Crashes in
kernel mode are catastrophic; they
will halt the entire PC.

User Mode

In User mode, the executing code has no ability to directly access
hardware or reference memory. Code
running in user mode must delegate to
system APIs to access hardware or
memory. Due to the protection afforded
by this sort of isolation, crashes in
user mode are always recoverable. Most
of the code running on your computer
will execute in user mode.

Understanding User and Kernel Mode

What is the role of kernel's top level Makefile while compiling a new module?

While kernel module is compiled using the same compiler (gcc) as user-space programs, it is required to pass many special parameters to the compiler for create a module. Moreover, some of these parameters depends on the kernel, against which you build the module.

Yes, you may create the Makefile, which passes proper parameters to the compiler and creates a module.

However, it is much simpler to use existing kernel's Makefile which cares about these parameters. In that case, your makefile needs to provide only source files you wish to compile into the module.

Also, building the kernel module is not just a compiling it. Asides from invoking the compiler, some additional actions should be taken for prepare the module.

E.g., one need to prepropcess Modules.symvers file for extract information of kernel core functions which are used in the module. Without that step the module created cannot be inserted into the kernel.

Operating system kernel and processes in main memory

Question is, first is this simple model correct?

Your model is extremely simplified but essentially correct - note that the last two parts of your model aren't really considered to be part of the boot process, and the kernel isn't a process. It can be useful to visualize it as one, but it doesn't fit the definition of a process and it doesn't behave like one.

Second, how is the executable program made aware of its virtual stack?
Is it the OS job to calculate the virtual stack pointer and place it
in the relevant CPU register? Is the rest of the stack bookkeeping
done by CPU pop and push commands?

An executable C program doesn't have to be "aware of its virtual stack." When a C program is compiled into an executable, local variables are usually referenced in relative to the stack pointer - for example, [ebp - 4].

When Linux loads a new program for execution, it uses the start_thread macro (which is called from load_elf_binary) to initialize the CPU's registers. The macro contains the following line:

regs->esp = new_esp;

which will initialize the CPU's stack pointer register to the virtual address that the OS has assigned to the thread's stack.

As you said, once the stack pointer is loaded, assembly commands such as pop and push will change its value. The operating system is responsible for making sure that there are physical pages that correspond to the virtual stack addresses - in programs that use a lot of stack memory, the number of physical pages will grow as the program continues its execution. There is a limit for each process that you can find by using the ulimit -a command (on my machine the maximum stack size is 8MB, or 2KB pages).

Does the kernel itself have its own main stack and heap?

This is where visualizing the kernel as a process can become confusing. First of all, threads in Linux have a user stack and a kernel stack. They're essentially the same, differing only in protections and location (kernel stack is used when executing in Kernel Mode, and user stack when executing in User Mode).

The kernel itself does not have its own stack. Kernel code is always executed in the context of some thread, and each thread has its own fixed-size (usually 8KB) kernel stack. When a thread moves from User Mode to Kernel Mode, the CPU's stack pointer is updated accordingly. So when kernel code uses local variables, they are stored on the kernel stack of the thread in which they are executing.

During system startup, the start_kernel function initializes the kernel init thread, which will then create other kernel threads and begin initializing user programs. So after system startup the CPU's stack pointer will be initialized to point to init's kernel stack.

As far as the heap goes, you can dynamically allocate memory in the kernel using kmalloc, which will try to find a free page in memory - its internal implementation uses get_zeroed_page.

What is an OS' HAL?

I believe it's safe to think of the HAL roughly as an I/O driver for your CPU. In any reasonable circumstance, you should expect the manufacturer of an I/O device to provide you with its driver. The same applies to a CPU and its HAL.

You asked:

Or, whether it is not separate from the OS, then, how does the OS developer know all the particularities of all the processors from the entire world and what is its purpose if the OS developer has to write a HAL for every OS?

This is actually what the HAL is intended to address. At one point, the OS was roughly expected to handle everything. But, as hardware became more complex, an additional abstraction layer was introduced.