How Are Interrupts Handled on Smp

How are interrupts handled on SMP?

On multicore/multiprocessor architectures, an APIC is used to route interrupts to cores/processors. As the name implies, APICs can be programmed to do the routing as desired.

Regarding the synchronization of the kernel: This depends on the kernel/OS. You can either use a scheme with locking (although IPIs might be necessary on non-cachecoherent architectures) or you can also use your suggested approach of running a kernel on every core and use some kind of explicit inter-kernel communication.

Barrelfish is an example of an OS running multiple kernels. If you are interested in that kind of architecture, you might want to read the paper "The Multikernel: A new OS architecture
for scalable multicore systems"

Interrupt handling on an SMP ARM system with a GIC

The GIC is divided into two sections. The first is called the distributor. This is global to the system. It has several interrupt sources physically routed to it; although it maybe within an SOC package. The second section is replicated per-CPU and it called the cpu interface. The distributor has logic on how to distribute the shared peripheral interrupts or SPI. These are the type of interrupt your question is asking about. They are global hardware interrupts.

In the context of Linux, this is implemented in irq-gic.c. There is some documentation in gic.txt. Of specific interest,

• reg : Specifies base physical address(s) and size of the GIC registers. The
  first region is the GIC distributor register base and size. The 2nd region is
  the GIC cpu interface register base and size.

The distributor must be accessed globally, so care must be taken to manage it's registers. The CPU interface has the same physical address for each CPU, but each CPU has a separate implementation. The distributor can be set up to route interrupts to specific CPUs (including multiples). See: gic_set_affinity() for example. It is also possible for any CPU to handle the interrupt. The ACK register will allocate IRQ; the first CPU to read it, gets the interrupt. If multiple IRQs pend and there are two ACK reads from different CPUs, then each will get a different interrupt. A third CPU reading would get a spurious IRQ.

As well, each CPU interface has some private interrupt sources, that are used for CPU-to-CPU interrupts as well as private timers and the like. But I believe the focus of the question is how a physical peripheral (unique to a system) gets routed to a CPU in an SMP system.

Implementing SMP properly on a Linux/MIPS platform

The bug was ultimately worked out to not assigning the IRQ numbers to their correct handler. I was initially assigning ALL 64 IRQs to use handle_level_irq, which is incorrect for SMP interprocessor interrupts (IPIs). The fix turned out to assign the 8 CPU-specific interrupts, 42-45 and 46-49, to handle_percpu_irq instead.

How do interrupts work in multi-core system?

your quote from google looks quite generic or perhaps even leaning on the size of x86, but doesnt really matter if that were the case.

I sure hope that you would be able to control interrupts per cpu such that you can have one type go to one and another to another.

Likewise that there is a choice to have all of them interrupted in case you want that.

Interrupts are irrelevant to shared resources, you have to handle shared resources whether you are in an ISR or not, so the interrupt doesnt matter you have to deal with it. Having the ability to isolate interrupts from one peripheral to one cpu could make the sharing easier in that you could have one cpu own a resource and other cpus make requests to the cpu that owns it for example.

Dual, Quad, etc cores doesnt matter, treat each core as a single cpu, which it is, and solve the interrupt problems as you would for a single cpu. Again shared resources are shared resources, during interrupts or not during interrupts. Solve the problem for one cpu then deal with any sharing.

Being an ARM each chip vendors implementation can vary from another, so there cannot be one universal answer, you have to read the arm docs for the arm core (and if possible the specific version as they can/do vary) as well as the chip vendors docs for whatever they have around the arm core. Being a Broadcom in this case, good luck with chip vendor docs. They are at best limited, esp with the raspi2. You might have to dig through the linux sources. No matter what, arm, x86, mips, etc, you have to just read the documentation and do some experiments. Start off by treating each core as a standalone cpu, then deal with sharing of resources if required.

If I remember right the default case is to have just the first core running the kernel7.img off the sd card, the other three are spinning in a loop waiting for an address (each has its own) to be written to get them to jump to that and start doing something else. So you quite literally can just start off with a single cpu, no sharing, and figure that out, if you choose to not have code on the other cpus that touch that resource, done. if you do THEN figure out how to share a resource.

Can an interrupt handler be preempted by the same interrupt handler?

x86 disables all local interrupts (except NMI of course) before jumping to the interrupt vector. Linux normally masks the specific interrupt and re-enables the rest of the interrupts (which aren't masked), unless a specific flags is passed to the interrupt handler registration.

Note that while this means your interrupt handler will not race with itself on the same CPU, it can and will race with itself running on other CPUs in an SMP / SMT system.

---

Related Topics