prevent linux thread from being interrupted by scheduler
How do you tell the thread scheduler in linux to not interrupt your thread for any reason?
Can't really be done, you need a real time system for that. The closes thing you'll get with linux is to
set the scheduling policy to a realtime scheduler, e.g. SCHED_FIFO, and also set the PTHREAD_EXPLICIT_SCHED attribute. See e.g. here , even now though, e.g. irq handlers and other other stuff will interrupt your thread and run.
However, if you only care about the threads in your own process not being able to do anything, then yes, having them block on a mutex your running thread holds is sufficient.
The hard part is to coordinate all the other threads to grab that mutex whenever your thread needs to do its thing.
How can I turn off interrupt that switching threads in OS, on short time, for my critical part of code?
You can't turn off the timer interrupts for the OS; they're
managed within the OS itself, and are necessary for its correct
functionning. Under most Unix, you can use sched_setscheduler
and/or pthread_setscheduler
to set the scheduling policy toSCHED_FIFO
, and sched_setparam
/pthread_setschedprio
to set
the priority; you will then be guaranteed to not be interrupted
by anything unless it has an even higher priority.
Under Linux, at least (and also under other Unix I've worked on),
it is also possible to lock pages into real memory, so you can't
get a page fault on them (which would cause you to wait for the
disk access). The function in Linux is mlock
.
Note that you'll probably need special privileges to change the
global scheduler or to use mlock
.
Prevent a thread from being unscheduled
Threads are scheduled OS wide and not only for one process. You can change the prio to 31. This will be the highest possible... but if other threads also runs with prio 31, your thread will also interrupted... Also all hardware-events (IRQs) will have higher prio than 31...
If you just want to have an "Process-wide" solution, I suggest to use Fibers. With fibers, you can do your own scheduling and also "prevent" other "Fibers" to "interrupt" your task. Of course, it might still be "interrupted" by an other thread. But if you run your thread with prio 31, this should be no big problem....
With this scenario, we run very high responsive (almost) realtime systems with Profibus connections... Also SQL server is using the fiber mode for optimized and very high performance mode.
Alternatively you can modify (restrict) the process affinity mask for all other processes (SetProcessAffinityMask)! To modify all processes you need to start your process as SYSTEM (for example via a service).
If you just care about your own process you can modify (restrict) the thread affinity of all other threads (SetThreadAffinityMask) in your own process!
Be aware, that it makes no sense to restrict your "important" thread to a specific CPU! This will only restrict your thread. But you must restrict all other threads/processes in order to get your "important" thread the best performance!
Also be aware that you never should consider using SuspendThread
! This might lead to dead-locks... see also: http://blog.kalmbachnet.de/?postid=6 / http://blog.kalmbachnet.de/?postid=16
how to shield a cpu from the linux scheduler (prevent it scheduling threads onto that cpu)?
The answer is to use cpusets. The python cpuset utility makes it easy to configure them.
Basic concepts
3 cpusets
root
: present in all configurations and contains all cpus (unshielded)system
: contains cpus used for system tasks - the ones which need to run but aren't "important" (unshielded)user
: contains cpus used for "important" tasks - the ones we want to run in "realtime" mode (shielded)
The shield
command manages these 3 cpusets.
During setup it moves all movable tasks into the unshielded cpuset (system
) and during teardown it moves all movable tasks into the root
cpuset.
After setup, the subcommand lets you move tasks into the shield (user
) cpuset, and additionally, to move special tasks (kernel threads) from root
to system
(and therefore out of the user
cpuset).
Commands:
First we create a shield. Naturally the layout of the shield will be machine/task dependent. For example, say we have a 4-core non-NUMA machine: we want to dedicate 3 cores to the shield, and leave 1 core for unimportant tasks; since it is non-NUMA we don't need to specify any memory node parameters, and we leave the kernel threads running in the root
cpuset (ie: across all cpus)
$ cset shield --cpu 1-3
Some kernel threads (those which aren't bound to specific cpus) can be moved into the system
cpuset. (In general it is not a good idea to move kernel threads which have been bound to a specific cpu)
$ cset shield --kthread on
Now let's list what's running in the shield (user
) or unshielded (system
) cpusets: (-v
for verbose, which will list the process names) (add a 2nd -v
to display more than 80 characters)
$ cset shield --shield -v
$ cset shield --unshield -v -v
If we want to stop the shield (teardown)
$ cset shield --reset
Now let's execute a process in the shield (commands following '--'
are passed to the command to be executed, not to cset
)
$ cset shield --exec mycommand -- -arg1 -arg2
If we already have a running process which we want to move into the shield (note we can move multiple processes by passing a comma separated list, or ranges (any process in the range will be moved, even if there are gaps))
$ cset shield --shield --pid 1234
$ cset shield --shield --pid 1234,1236
$ cset shield --shield --pid 1234,1237,1238-1240
Advanced concepts
cset set/proc
- these give you finer control of cpusets
Set
Create, adjust, rename, move and destroy cpusets
Commands
Create a cpuset, using cpus 1-3, use NUMA node 1 and call it "my_cpuset1"
$ cset set --cpu=1-3 --mem=1 --set=my_cpuset1
Change "my_cpuset1" to only use cpus 1 and 3
$ cset set --cpu=1,3 --mem=1 --set=my_cpuset1
Destroy a cpuset
$ cset set --destroy --set=my_cpuset1
Rename an existing cpuset
$ cset set --set=my_cpuset1 --newname=your_cpuset1
Create a hierarchical cpuset
$ cset set --cpu=3 --mem=1 --set=my_cpuset1/my_subset1
List existing cpusets (depth of level 1)
$ cset set --list
List existing cpuset and its children
$ cset set --list --set=my_cpuset1
List all existing cpusets
$ cset set --list --recurse
Proc
Manage threads and processes
Commands
List tasks running in a cpuset
$ cset proc --list --set=my_cpuset1 --verbose
Execute a task in a cpuset
$ cset proc --set=my_cpuset1 --exec myApp -- --arg1 --arg2
Moving a task
$ cset proc --toset=my_cpuset1 --move --pid 1234
$ cset proc --toset=my_cpuset1 --move --pid 1234,1236
$ cset proc --toset=my_cpuset1 --move --pid 1238-1340
Moving a task and all its siblings
$ cset proc --move --toset=my_cpuset1 --pid 1234 --threads
Move all tasks from one cpuset to another
$ cset proc --move --fromset=my_cpuset1 --toset=system
Move unpinned kernel threads into a cpuset
$ cset proc --kthread --fromset=root --toset=system
Forcibly move kernel threads (including those that are pinned to a specific cpu) into a cpuset (note: this may have dire consequences for the system - make sure you know what you're doing)
$ cset proc --kthread --fromset=root --toset=system --force
Hierarchy example
We can use hierarchical cpusets to create prioritised groupings
- Create a
system
cpuset with 1 cpu (0) - Create a
prio_low
cpuset with 1 cpu (1) - Create a
prio_met
cpuset with 2 cpus (1-2) - Create a
prio_high
cpuset with 3 cpus (1-3) - Create a
prio_all
cpuset with all 4 cpus (0-3) (note this the same as root; it is considered good practice to keep a separation from root)
To achieve the above you create prio_all, and then create subset prio_high under prio_all, etc
$ cset set --cpu=0 --set=system
$ cset set --cpu=0-3 --set=prio_all
$ cset set --cpu=1-3 --set=/prio_all/prio_high
$ cset set --cpu=1-2 --set=/prio_all/prio_high/prio_med
$ cset set --cpu=1 --set=/prio_all/prio_high/prio_med/prio_low
How does the scheduler know that a thread is blocked waiting for input?
Be careful not to confuse preemption with the ability of a process to sleep. Processes can sleep even with a non-preempting scheduler. This is what happens when a process is waiting for I/O. The process makes a system call such as read()
and the device determines no data is available. It then internally puts the process to sleep by updating a data structure used by the scheduler. The scheduler then executes other processes until an interrupt or some other event occurs that wakes the original process. The awoken process then becomes eligible again for scheduling.
On the other hand preemption is the ability of an architecture's scheduler to stop execution of a process without its cooperation. The interruption can occur anywhere in the program's instruction stream. Control returns to the scheduler which can then execute other processes and return to the interrupted (preempted) process later. Most schedulers allocate time slices where a process is allowed to run for up to a predetermined amount of time, after which it is preempted if higher-priority processes need time slices.
Unless you're writing drivers or kernel code, you don't need to worry about the underlying mechanisms too much. When writing user-space applications the key concepts are (1) that some system calls may block which means your process is put to sleep until an event occurs, and (2) on preemptible systems (all mainstream modern operating systems) your program may be preempted at any time so that other processes can run.
* Note that in some platforms, such as Linux, a thread is really just another process which shares its virtual address space with another process. Processes and threads are therefore treated exactly the same by the scheduler.
What is the mechanism that allows the scheduler to switch which threads are executing?
It's the timer interrupt.
Interrupts are one of those things that are beyond Turing-complete. A Turing-complete machine/language does not need to implement interrupts. But without interrupts you will have a hard time implementing a time-sliced or preemptive multitasking OS.
Without interrupts
It is still possible to implement a multitasking OS on a system without interrupts - you use Cooperative Multitasking. The old MacOS (before OSX) did just this. You just tell app developers that every now and then they must hand over execution to the OS by calling something like yield()
. This "yield" function is actually the OS itself. Cooperative multitasking is not ideal of course - you can imagine a program crashing will cause the OS to never execute thus take down the entire machine. The advantage of cooperative multitasking is that you can do it on a very simple CPU with no interrupt support. It also typically requires a lot less RAM and CPU resources. And, as a programmer you have more control over the CPU - for example if you need to really use 100% of the CPU you can prevent the OS from taking any CPU time.
Interrupts
An interrupt is just a feature of the CPU where you can configure the CPU to call a function (called the interrupt handler) when something happens. For OS programmers, the most useful is the timer interrupt - where you can configure a timer to trigger an interrupt. The OS gets to run when the interrupt is triggered and at the end of execution the OS will simply schedule another timer interrupt - this is time-sliced multitasking.
Some OSes like Linux or Real-time Windows allow you to configure this timer. Linux call it jiffy
, some OSes call it tick
.
If you have done any javascript programming this will feel familiar to you - this is almost like how setTimeout()
appears to behave to a programmer.
I/O
Another important type of interrupt is I/O interrupt. Your keyboard actually works with an I/O interrupt. A normal PC does not scan the keyboard at all. Instead the I/O controller (these days it's usually the USB controller) queries the keyboard and if there is a key press will send an interrupt signal to the CPU. This will trigger the OS and the OS will check which process the key belongs to and will switch to the process to allow it to receive the input - this is preemptive multitasking. Obviously, preemptive multitasking uses a time-slicer in the background in case there are long periods of no I/O activity.
Related Topics
Efficiently Test If a Port Is Open on Linux
Best Way to Find Os Name and Version in Unix/Linux Platform
Replace Whitespace with a Comma in a Text File in Linux
How to Install Svn Post-Commit Hook
Bash: Set Array Env Variable and De-Referencing It from Any Shell Script Fails
Cross-Compile a Rust Application from Linux to Windows
What's the Purpose of Each of the Different Uids a Process Can Have
Bluetooth Le Signal Strength Linux
Insert Multiple Lines into a File After Specified Pattern Using Shell Script
How to Pass a Wildcard Parameter to a Bash File
How to Automatically Add User Account and Password with a Bash Script
How to Compile/Install Node.Js(Could Not Configure a Cxx Compiler!) (Ubuntu)
How to Set a Custom Baud Rate on Linux
How to Create a Dynamic Variable and Assign Value to It