Suse 10.1 to 10.3 upgrade Episode 1

So I’ve been preparing a test server for the past couple of days, so that I can test the Suse upgrade path from 10.1 to 10.3. I’ve got a spare Dell 2650 machine with a PERC3/i card and two 37 GB scsi drives in raid-1 configuration.

To get the test going I installed SuSE 10.1 from the install DVD and proceeded to choose pretty much all the options for the packages thinking that if the upgrade worked with (almost) all the installable packages selected, then it would almost certainly work on the real server. That was I believe my first mistake, by choosing (almost) all the options, I had also included XEN virtualization. This forces the 10.1 installer to install the XENified version of the linux kernel off the 10.1 DVD. This was fine and dandy as long as I was using 10.1, after doing all the online updates for 10.1 and installing 10.3 (and updating it), I rebooted and was greeted by the following kernel messages:
Mar 2 19:15:21 xxxxxx kernel: SCSI subsystem initialized
Mar 2 19:15:21 xxxxxx kernel: Adaptec aacraid driver 1.1-5[2437]-mh4
Mar 2 19:15:21 xxxxxx kernel: ACPI: PCI Interrupt 0000:04:08.1[A] -> GSI 30 (level, low) -> IRQ 16
Mar 2 19:15:21 xxxxxx kernel: AAC0: kernel 2.7-1[3170]
Mar 2 19:15:21 xxxxxx kernel: AAC0: monitor 2.7-1[3170]
Mar 2 19:15:21 xxxxxx kernel: AAC0: bios 2.7-1[3170]
Mar 2 19:15:21 xxxxxx kernel: AAC0: serial 5C8881D3
Mar 2 19:15:21 xxxxxx kernel: AAC0: 64bit support enabled.
Mar 2 19:15:21 xxxxxx kernel: AAC0: 64 Bit DAC enabled
Mar 2 19:15:21 xxxxxx kernel: scsi0 : percraid
Mar 2 19:15:21 xxxxxx kernel: Vendor: DELL Model: Lombok Rev: V1.0
Mar 2 19:15:21 xxxxxx kernel: Type: Direct-Access ANSI SCSI revision: 02
Mar 2 19:15:21 xxxxxx kernel: SCSI device sda: 574210048 512-byte hdwr sectors (293996 MB)
Mar 2 19:15:21 xxxxxx kernel: sda: Write Protect is off
Mar 2 19:15:21 xxxxxx kernel: SCSI device sda: drive cache: write back
Mar 2 19:15:21 xxxxxx kernel: SCSI device sda: 574210048 512-byte hdwr sectors (293996 MB)
Mar 2 19:15:21 xxxxxx kernel: sda: Write Protect is off
Mar 2 19:15:21 xxxxxx kernel: SCSI device sda: drive cache: write back
Mar 2 19:15:21 xxxxxx kernel: sda: sda1 sda2
Mar 2 19:15:21 xxxxxx rpc.statd[2231]: Version 1.0.9 Starting
Mar 2 19:15:21 xxxxxx kernel: sd 0:0:0:0: Attached scsi removable disk sda
Mar 2 19:15:21 xxxxxx kernel: AAC:AAC received an unrecognized command [601].
Mar 2 19:15:21 xxxxxx kernel: aacraid: Host adapter abort request (0,1,0,0)
Mar 2 19:15:21 xxxxxx kernel: aacraid: Host adapter abort request (0,1,0,0)
Mar 2 19:15:21 xxxxxx rpc.statd[2231]: statd running as root. chown /var/lib/nfs/statd/sm to choose different user
Mar 2 19:15:21 xxxxxx kernel: aacraid: Host adapter reset request. SCSI hang ?
Mar 2 19:15:21 xxxxxx kernel: aacraid: Host adapter abort request (0,1,0,0)
Mar 2 19:15:21 xxxxxx kernel: scsi 0:1:0:0: scsi: Device offlined - not ready after error recovery
Mar 2 19:15:21 xxxxxx kernel: aacraid: Host adapter abort request (0,1,1,0)
Mar 2 19:15:21 xxxxxx kernel: aacraid: Host adapter abort request (0,1,1,0)
Mar 2 19:15:21 xxxxxx kernel: aacraid: Host adapter reset request. SCSI hang ?
Mar 2 19:15:21 xxxxxx kernel: aacraid: Host adapter abort request (0,1,1,0)
Mar 2 19:15:21 xxxxxx kernel: scsi 0:1:1:0: scsi: Device offlined - not ready after error recovery
Mar 2 19:15:21 xxxxxx kernel: aacraid: Host adapter abort request (0,1,2,0)
Mar 2 19:15:21 xxxxxx kernel: aacraid: Host adapter abort request (0,1,2,0)
Mar 2 19:15:21 xxxxxx kernel: aacraid: Host adapter reset request. SCSI hang ?
Mar 2 19:15:21 xxxxxx kernel: aacraid: Host adapter abort request (0,1,2,0)
Mar 2 19:15:22 xxxxxx kernel: scsi 0:1:2:0: scsi: Device offlined - not ready after error recovery
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,3,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,3,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter reset request. SCSI hang ?
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,3,0)
Mar 2 19:15:22 xxxxxx kernel: scsi 0:1:3:0: scsi: Device offlined - not ready after error recovery
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,4,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,4,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter reset request. SCSI hang ?
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,4,0)
Mar 2 19:15:22 xxxxxx kernel: scsi 0:1:4:0: scsi: Device offlined - not ready after error recovery
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,5,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,5,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter reset request. SCSI hang ?
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,5,0)
Mar 2 19:15:22 xxxxxx kernel: scsi 0:1:5:0: scsi: Device offlined - not ready after error recovery
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,6,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,6,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter reset request. SCSI hang ?
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,6,0)
Mar 2 19:15:22 xxxxxx kernel: scsi 0:1:6:0: scsi: Device offlined - not ready after error recovery
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,7,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,7,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter reset request. SCSI hang ?
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,7,0)
Mar 2 19:15:22 xxxxxx kernel: scsi 0:1:7:0: scsi: Device offlined - not ready after error recovery
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,8,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,8,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter reset request. SCSI hang ?
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,8,0)
Mar 2 19:15:22 xxxxxx kernel: scsi 0:1:8:0: scsi: Device offlined - not ready after error recovery
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,9,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,9,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter reset request. SCSI hang ?
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,9,0)
Mar 2 19:15:22 xxxxxx kernel: scsi 0:1:9:0: scsi: Device offlined - not ready after error recovery
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,10,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,10,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter reset request. SCSI hang ?
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,10,0)
Mar 2 19:15:22 xxxxxx kernel: scsi 0:1:10:0: scsi: Device offlined - not ready after error recovery
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,11,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,11,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter reset request. SCSI hang ?
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,11,0)
Mar 2 19:15:22 xxxxxx kernel: scsi 0:1:11:0: scsi: Device offlined - not ready after error recovery
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,12,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,12,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter reset request. SCSI hang ?
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,12,0)
Mar 2 19:15:22 xxxxxx kernel: scsi 0:1:12:0: scsi: Device offlined - not ready after error recovery
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,13,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,13,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter reset request. SCSI hang ?
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,13,0)
Mar 2 19:15:22 xxxxxx kernel: scsi 0:1:13:0: scsi: Device offlined - not ready after error recovery
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,14,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,14,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter reset request. SCSI hang ?
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,14,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter reset request. SCSI hang ?
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,1,14,0)
Mar 2 19:15:22 xxxxxx kernel: scsi 0:1:14:0: scsi: Device offlined - not ready after error recovery
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,2,0,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,2,0,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter reset request. SCSI hang ?
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,2,0,0)
Mar 2 19:15:22 xxxxxx kernel: scsi 0:2:0:0: scsi: Device offlined - not ready after error recovery
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,2,1,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,2,1,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter reset request. SCSI hang ?
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,2,1,0)
Mar 2 19:15:22 xxxxxx kernel: scsi 0:2:1:0: scsi: Device offlined - not ready after error recovery
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,2,2,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,2,2,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter reset request. SCSI hang ?
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,2,2,0)
Mar 2 19:15:22 xxxxxx kernel: scsi 0:2:2:0: scsi: Device offlined - not ready after error recovery
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,2,3,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,2,3,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter reset request. SCSI hang ?
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,2,3,0)
Mar 2 19:15:22 xxxxxx kernel: scsi 0:2:3:0: scsi: Device offlined - not ready after error recovery
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,2,4,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,2,4,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter reset request. SCSI hang ?
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,2,4,0)
Mar 2 19:15:22 xxxxxx kernel: scsi 0:2:4:0: scsi: Device offlined - not ready after error recovery
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,2,5,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter abort request (0,2,5,0)
Mar 2 19:15:22 xxxxxx kernel: aacraid: Host adapter reset request. SCSI hang ?

ETC,ETC,ETC,.......

So what’s the deal. Looks like the driver is poking the scsi bus for every single ID on both buses. This literally takes 20-30 minutes, but the machine DOES boot up. Of course this is not acceptable, so after poking around I found out that It’s the XENified Kernel that’s causing this.

I’m testing this right now by reinstalling 10.1 from DVD WITHOUT XEN Virtualization. I will then apply the updates and upgrade (and update again) to 10.3 and report back in Episode 2……hopefully everything will work…….famous last word :-).