I am extremely confused about the exact series of steps involved in having the CPU write a value into a PCIe card's memory. It's very difficult to understand the precise meaning of stuff you read on the internet, so I'm hoping someone can read my theory of what's happening and point out any mistakes.
Suppose I have a PCIe card with some memory on it. For the sake of discussion, assume the following concrete setup:
Let's also settle on terminology:
pci_devstructure filled by the kernel
At boot time, Linux starts probing different addresses to see if there's anything there.
pci_devstruct with other information in the configuration space
0x5550FFFFon the system bus.
0x55500000in its base address field (why bother?)
0x5550FFFFare "caught" by the root complex, and issued out to the PCIe card
0x55501230on the system bus, and that the root complex sent out the packet to the PCIe card, the card receives the packet and writes
0x01230in its local 4 MB memory
So: what parts of this are right and which are wrong?
My experience is with Intel processors, so some of the details below may be specific to Intel processors, but it is mostly general. Also I don't know the details of how Linux identifies the driver to load for each device, so I skipped that question.
Modern CPUs don't have a system bus (except hidden within the CPU itself). They have memory channels, PCIe root ports, and a DMI port that connects to the chipset (also called the peripheral controller hub, or PCH). The PCH contains additional devices and may have additional root ports. The root complex comprises circuitry integrated into both the CPU and PCH. (Some CPU SoCs don't have DMI or a PCH, and all the root complex circuitry is within the CPU SoC.)
Even if your card is the only PCIe card in the whole system, there are other PCIe devices integrated into the root complex (called RCIEP or root-complex integrated endpoints). These may be within the CPU or in the PCH.
Your device, connected to a PCIe root port, will be configured as device 0 on some non-zero bus number. The bus number is dependent on the PCIe root port (i.e., slot) that the device is connected to and the way in which the BIOS configures the PCIe bus. (The same slot will generally have the same bus number, but it may not, depending on what is connected to the other PCIe root ports.)
The rest of your assumptions and terminology are fine.
Software accesses PCI config space either by using in/out instructions to the I/O ports 0xcf8 and 0xcfc, or by using the memory-mapped config space. The memory address range of PCI config space is set up by the BIOS. Software finds out the address by looking at the ACPI tables. The mechanism by which these I/O or memory accesses are converted to PCIe signals is totally within the root complex hardware.
The offset into the PCI config space address range controls which device/register software is accessing. For example, an access to MMCFG + 0 accesses register offset 0 of device 0:0.0. An access to MMCFG + 0x1000 accesses register offset 0 of device 0:0.1, and an access to MMCFG + 0x102000 accesses register offset 0 of device 1:0.2.
Software reads the vendor id/device id registers at offset 3:0 of each device address to detect whether a device exists at that device address. If no device is present, the PCI controller returns 0xffffffff. If a device is present, the device returns the vendor id and device id, allowing software to determine the type of device.
Each device has 6 BAR registers, at offsets 0x10, 0x14, ... 0x24. If a device supports 64-bit BARs, two adjacent BAR registers are used to configure a single region. Normally the BIOS configures the BARs of every device and also configures other (hidden) registers within the root complex to enable it to route memory accesses to the proper device. Software normally only writes to the BAR registers to detect the region size and then restores the values that were set by the BIOS. Depending on the root complex hardware, software may or may not be able to change the BAR values and still have accesses work properly.
User contributions licensed under CC BY-SA 3.0