How Can the Linux Kernel Be Forced to Enumerate the Pci-E Bus

How can the linux kernel be forced to enumerate the PCI-e bus?

I wonder what platform you are on: A work around (aka hack) for this that works on x86 systems is to have the BIOS basically statically configure a PCI device at whatever bus, device, function the FPGA normally lands on, then the OS will enumerate the device and reserve the PCI space for it (even though the device isn't really there). Then in your device driver you will have to do some extra things like setup the BARs and int lines manually after the fpga has been programmed. Of course this requires modifying the BIOS, which if you are working with a BIOS vendor you can contract them to make this change for you, if you are not working with a BIOS vendor then it will be much harder... Also keep in mind that I was working on VxWorks on x86, and we had a AMI make a custom BIOS for our boards...

If you don't have a BIOS, then consider programming it in the bootloader, there you already have the ability to read from disk, and adding GPIO capabilities probably isn't too difficult (assuming you are using jtag and GPIOs?), in fact depending on what bootloader you use it might already be able to do GPIO?

The issues with modifying the kernel to do this is that you have to find the sweet spot where you can read the bitfile, before the PCI enumeration... If for example the disk device drivers are initialized after PCI, then obviously you must do some radical changes to the kernel just to read the bitfile prior to PCI enumeration, which might cause other annoying problems...

One other option which you may have already discovered, and which is really only ok for development time: Power up the system, program the fpga board, then do a reset (without power cycle, for example: sudo reboot now), the FPGA should keep its configuration, and linux should enumerate it...

How does Linux kernel discover PCI devices?

Actually, BIOS need not be involved.

PCI standardizes a certain procedure for discovery of devices on the bus. This procedure can be triggered at any time (not exclusively on boot) by hotplug controller or even manually, via /sys/bus/pci/rescan (see pci_rescan_bus).

The scan will proceed recursively, traversing the bridges as discovered and reading the configuration space data off each device encountered (see PCI configuration space).

For each device found, if not yet active, the kernel will look for an instance of pci_driver object with a matching pci_device_id. Then it will call the probe method of that object (the rest is driver implementation specific).

If appropriate pci_driver instance is not found, kernel will emit an event to an user space daemon (udev or hotpug or whatever), which may load an appropriate module and create the necessary pci_driver object.

How does PCI/PCIe devices init/register themselves in the Linux kernel?

Since the question is partially incomplete, but comments are too small to give a answer I'll try to mix this in a bit.

So the kernel tries to abstract the physical implementation of the PCI(e) bus from the driver developer. Hence a PCI bus on an NVidia Tegra is different to a PCI bus on a Freescale ARM and a x86_64 PCI bus, but it should be possible to register devices against them regardless of the real bus implementation.

The structure pci_root_buses is a list of abstract PCI buses, where the implementation could be different.

You can see this in the bus type structure, where function pointers are defined to allow each real bus to have a different implementation how to treat a device. I think it's best if you read the PCI chapter in LDD3 and have a special look at Boot Time.

Also look at Configuration Time to see how the Kernel matches drivers to hardware. The rough idea of PCI is that the kernel can discover the bus and map memory to each physical PCI device allowing access to the PCI configuration space of the device. The driver developer registers a device class by calling pci_register_driver and therefore telling the kernel which driver functions to use for certain vendor ids.

Looking at LDD3 again it seems the missing mapping that you might be looking for is the probe function:

int (*probe) (struct pci_dev *dev, const struct pci_device_id *id);

Pointer to the probe function in the PCI driver. This function is called by the PCI core when it has a struct pci_dev that it thinks this driver wants to control. A pointer to the struct pci_device_id that the PCI core used to make this decision is also passed to this function. If the PCI driver claims the struct pci_dev that is passed to it, it should initialize the device properly and return 0. If the driver does not want to claim the device, or an error occurs, it should return a negative error value. More details about this function follow later in this chapter.

Kernel data structures

Bus type

How Can the Linux Kernel Be Forced to Enumerate the Pci-E Bus