Xen SATA PCI Passthrough to Ubutnu DomU not recognising disks

I have a Xen 4.1 server with a Debian Wheezy 7.4 Dom0 (hmpyxen01). It has a SATA PCI card that I am passing through to an Ubuntu Precise 12.04 DomU (hmvmtmp01), installed via xen-tools which in this case is just a wrapper for debootstrap so the DomU is very minimal (tools such as lshw and lspci have to be install manually after booting the DomU).

The passthrough itself seems to work as the storage controller uses the pciback module in the Dom0:

root@hmpyxen01:~# lspci -k
...snip...
00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller
        Subsystem: ASUSTeK Computer Inc. P5Q Deluxe Motherboard
        Kernel driver in use: pciback
...snip...

and appears in the DomU:

root@hmvmtmp01:~# lspci -k
00:00.2 SATA controller: Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller
        Subsystem: ASUSTeK Computer Inc. P5Q Deluxe Motherboard
        Kernel driver in use: ahci
root@hmvmtmp01:~# lshw -class storage
  *-storage
       description: SATA controller
       product: 82801JI (ICH10 Family) SATA AHCI Controller
       vendor: Intel Corporation
       physical id: 0.2
       bus info: pci@0000:00:00.2
       version: 00
       width: 32 bits
       clock: 66MHz
       capabilities: storage msi pm ahci_1.0 bus_master cap_list
       configuration: driver=ahci latency=0
       resources: irq:29 ioport:9c00(size=8) ioport:9880(size=4) ioport:9800(size=8) ioport:9480(size=4) ioport:9400(size=32) memory:f9cfc000-f9cfc7ff

Unfortunately, none of the attached disks seem to load in the DomU, only the root/swap Xen drives set up by the hypervisor.

root@hmvmtmp01:~# lsblk
NAME  MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda2 202:2    0  10G  0 disk /
xvda1 202:1    0   1G  0 disk [SWAP]
root@hmvmtmp01:~# lshw -class disk
root@hmvmtmp01:~# 

It looks as if dmesg finds the controller & disks, but something fails:

root@hmvmtmp01:~# dmesg
...snip...
[    4.192202] pcifront pci-0: Installing PCI frontend
[    4.192263] pcifront pci-0: Creating PCI Frontend Bus 0000:00
[    4.192636] pci 0000:00:00.2: [8086:3a22] type 0 class 0x000106
[    4.192810] pci 0000:00:00.2: reg 10: [io  0x9c00-0x9c07]
[    4.192886] pci 0000:00:00.2: reg 14: [io  0x9880-0x9883]
[    4.192959] pci 0000:00:00.2: reg 18: [io  0x9800-0x9807]
[    4.193033] pci 0000:00:00.2: reg 1c: [io  0x9480-0x9483]
[    4.193132] pci 0000:00:00.2: reg 20: [io  0x9400-0x941f]
[    4.193198] pci 0000:00:00.2: reg 24: [mem 0xf9cfc000-0xf9cfc7ff]
[    4.193718] pcifront pci-0: New device on 0000:00:00.02 found.
[    4.195283] pcifront pci-0: claiming resource 0000:00:00.2/0
[    4.195286] pcifront pci-0: claiming resource 0000:00:00.2/1
[    4.195287] pcifront pci-0: claiming resource 0000:00:00.2/2
[    4.195289] pcifront pci-0: claiming resource 0000:00:00.2/3
[    4.195290] pcifront pci-0: claiming resource 0000:00:00.2/4
[    4.195292] pcifront pci-0: claiming resource 0000:00:00.2/5
[    4.195325] ahci 0000:00:00.2: version 3.0
[    4.195352] ahci 0000:00:00.2: enabling device (0000 -> 0003)
[    4.195421] ahci 0000:00:00.2: Xen PCI mapped GSI20 to IRQ28
[    4.195650] ahci: SSS flag set, parallel bus scan disabled
[    4.195702] ahci 0000:00:00.2: AHCI 0001.0200 32 slots 6 ports 3 Gbps 0x3f impl SATA mode
[    4.195705] ahci 0000:00:00.2: flags: 64bit ncq sntf stag pm led clo pmp pio slum part ccc ems sxs
[    4.195752] ahci 0000:00:00.2: setting latency timer to 64
[    4.235098] scsi0 : ahci
[    4.236153] scsi1 : ahci
[    4.238341] scsi2 : ahci
[    4.241175] scsi3 : ahci
[    4.242971] scsi4 : ahci
[    4.245287] scsi5 : ahci
[    4.245331] ata1: SATA max UDMA/133 abar m2048@0xf9cfc000 port 0xf9cfc100 irq 29
[    4.245333] ata2: SATA max UDMA/133 abar m2048@0xf9cfc000 port 0xf9cfc180 irq 29
[    4.245336] ata3: SATA max UDMA/133 abar m2048@0xf9cfc000 port 0xf9cfc200 irq 29
[    4.245338] ata4: SATA max UDMA/133 abar m2048@0xf9cfc000 port 0xf9cfc280 irq 29
[    4.245340] ata5: SATA max UDMA/133 abar m2048@0xf9cfc000 port 0xf9cfc300 irq 29
[    4.245342] ata6: SATA max UDMA/133 abar m2048@0xf9cfc000 port 0xf9cfc380 irq 29
[    6.427298] init: failsafe main process (429) killed by TERM signal
[    6.460121] init: rc-sysinit main process (550) killed by TERM signal
[   14.244071] ata1: softreset failed (1st FIS failed)
[   14.480058] eth0: no IPv6 routers present
[   24.244072] ata1: softreset failed (1st FIS failed)
[   59.244072] ata1: softreset failed (1st FIS failed)
[   59.244087] ata1: limiting SATA link speed to 1.5 Gbps
[   64.244070] ata1: softreset failed (1st FIS failed)
[   64.244082] ata1: reset failed, giving up
[   74.244070] ata2: softreset failed (1st FIS failed)
[   84.244071] ata2: softreset failed (1st FIS failed)
[  119.244072] ata2: softreset failed (1st FIS failed)
[  119.244084] ata2: limiting SATA link speed to 1.5 Gbps
[  124.244071] ata2: softreset failed (1st FIS failed)
[  124.244083] ata2: reset failed, giving up
[  124.372116] init: udevtrigger post-stop process (335) terminated with status 1
[  124.390542] init: udev-fallback-graphics main process (674) terminated with status 1
[  124.407645] init: plymouth-splash main process (682) terminated with status 1
[  134.244071] ata3: softreset failed (1st FIS failed)
[  144.244071] ata3: softreset failed (1st FIS failed)
[  179.244070] ata3: softreset failed (1st FIS failed)
[  179.244083] ata3: limiting SATA link speed to 1.5 Gbps
[  184.244071] ata3: softreset failed (1st FIS failed)
[  184.244083] ata3: reset failed, giving up
[  194.244071] ata4: softreset failed (1st FIS failed)
[  204.244071] ata4: softreset failed (1st FIS failed)
[  239.244071] ata4: softreset failed (1st FIS failed)
[  239.244083] ata4: limiting SATA link speed to 1.5 Gbps
[  244.244070] ata4: softreset failed (1st FIS failed)
[  244.244082] ata4: reset failed, giving up
[  244.564074] ata5: SATA link down (SStatus 0 SControl 300)
[  244.884073] ata6: SATA link down (SStatus 0 SControl 300)

As I mentioned before, I have a feeling that it is something to do with the install method of xen-tools and the DomU may be missing something simple, but I don't know what...

Any help is very much appreciated.

EDIT: Added information RE hotswap

So I restarted the DomU machine without the drives attached and inserted them in to an already-running system. Here is the output form that:

[  305.424655] ata3: exception Emask 0x10 SAct 0x0 SErr 0x4040000 action 0xe frozen
[  305.424671] ata3: irq_stat 0x00000040, connection status changed
[  305.424681] ata3: SError: { CommWake DevExch }
[  305.424688] ata3: hard resetting link
[  308.353316] ata4: exception Emask 0x10 SAct 0x0 SErr 0x4040000 action 0xe frozen
[  308.353328] ata4: irq_stat 0x00000040, connection status changed
[  308.353337] ata4: SError: { CommWake DevExch }
[  308.353346] ata4: hard resetting link
[  310.651627] ata1: exception Emask 0x10 SAct 0x0 SErr 0x4040000 action 0xe frozen
[  310.651639] ata1: irq_stat 0x00000040, connection status changed
[  310.651648] ata1: SError: { CommWake DevExch }
[  310.651661] ata1: hard resetting link
[  313.120770] ata2: exception Emask 0x10 SAct 0x0 SErr 0x4040000 action 0xe frozen
[  313.120783] ata2: irq_stat 0x00000040, connection status changed
[  313.120791] ata2: SError: { CommWake DevExch }
[  313.120804] ata2: hard resetting link
[  315.428073] ata3: softreset failed (1st FIS failed)
[  315.428083] ata3: hard resetting link
[  318.356070] ata4: softreset failed (1st FIS failed)
[  318.356080] ata4: hard resetting link

The drives all definitely work as without xen-passthrough enabled, the drives are all accessible & readable from the Dom0.

Wow, 3 days of pulling my hair out...

I don't know if it is because my motherboard chipset doesn't support VT-d (only VT-x), but in the DomU I have to add a simple iommu=soft to the kernel boot parameters in /boot/grub/menu.lst and it solved everything!!

It's always the simple things in the end.