I have a Dell server with a Dell storage attached to it with multipath set tup. Dell technicians left the server with Red Hat 7 installed in an internal disk (/dev/sda) and with a mapped drive to the storage in /dev/mapper/DB_DATA01.

Everything was working fine, except by the 2 paths to the external storage being also viewed as /dev/sdb and /dev/sdc.

Then I commited a mistake: runned parted to create a gpt table into /dev/sdb instead of /dev/mapper/DB_DATA01. After I did this, I have correctly created and formatted a xfs partition inside /dev/mapper/DB_DATA01, correctly setup a mount instruction into /dev/fstab. Then I rebooted.

Now, everytime I reboot the server it asks for maintenance mode saying it was unable to mount /dev/mapper/DB_DATA01 into /db2 mount point. But, if I press CTRL +D the server starts OK and everything works...

How can I fix that mess and make the server starts without asking for maintenance?

I know that there is a missconfiguration in someplace. By the way, these are the outputs I have for now:

[root@db2server etc]# cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Mon Oct 17 16:29:48 2016
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=ad5ce7b2-4517-42a9-b558-afc40f5950b4 /                       xfs     defaults        0 0
UUID=ad193bfa-b17d-49da-92b5-53295f5e33b2 swap                    swap    defaults        0 0
UUID=f0241ef6-f167-4edb-9a48-4bda7433eec4 /db2                    xfs     noatime,nobarrier,allocsize=4k,largeio 0 2

[root@db2server etc]# ls -lha /dev/mapper/

total 0
drwxr-xr-x.  2 root root     140 Oct 31 17:27 .
drwxr-xr-x. 20 root root    3.5K Oct 31 17:27 ..
crw-------.  1 root root 10, 236 Oct 31 16:57 control
lrwxrwxrwx.  1 root root       7 Oct 31 17:27 DB_DATA01 -> ../dm-0
lrwxrwxrwx.  1 root root       7 Oct 31 17:27 DB_DATA01p1 -> ../dm-1
lrwxrwxrwx.  1 root root       7 Oct 31 17:27 DB_DATA01p2 -> ../dm-2
lrwxrwxrwx.  1 root root       7 Oct 31 17:27 DB_DATA01p3 -> ../dm-3

[root@db2server etc]# fdisk -l

Disk /dev/sda: 299.4 GB, 299439751168 bytes, 584843264 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x57b68016

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1              63       80324       40131   de  Dell Utility
/dev/sda2   *       81920     4276223     2097152    c  W95 FAT32 (LBA)
/dev/sda3         4276224    66775039    31249408   82  Linux swap / Solaris
/dev/sda4        66775040   584843263   259034112    5  Extended
/dev/sda5        66777088   584843263   259033088   83  Linux

Disk /dev/sdc: 3793.3 GB, 3793315115520 bytes, 7408818585 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 524288 bytes / 524288 bytes
Disk label type: dos
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1  4294967295  2147483647+  ee  GPT
Partition 1 does not start on physical sector boundary.
fdisk: cannot open /dev/sdb: Input/output error

[root@db2server etc]# multipath -l

DB_DATA01 (36000d310034940000000000000000003) dm-0 COMPELNT,Compellent Vol
size=3.5T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=0 status=active
  |- 1:0:0:1  sdb 8:16 active undef  running
  `- 1:0:1:1  sdc 8:32 active undef  running

[root@db2server etc]# multipath -ll

DB_DATA01 (36000d310034940000000000000000003) dm-0 COMPELNT,Compellent Vol
size=3.5T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  |- 1:0:0:1  sdb 8:16 active ready  running
  `- 1:0:1:1  sdc 8:32 active ready  running

[root@db2server etc]# multipath -v2

*No data returned*

[root@db2server etc]# multipath -v3

Oct 31 17:40:16 | loading /lib64/multipath/libcheckdirectio.so checker
Oct 31 17:40:16 | loading /lib64/multipath/libprioconst.so prioritizer
Oct 31 17:40:16 | sda: device node name blacklisted
Oct 31 17:40:16 | sdb: not found in pathvec
Oct 31 17:40:16 | sdb: mask = 0x3f
Oct 31 17:40:16 | sdb: dev_t = 8:16
Oct 31 17:40:16 | sdb: size = 7408818585
Oct 31 17:40:16 | sdb: vendor = COMPELNT
Oct 31 17:40:16 | sdb: product = Compellent Vol
Oct 31 17:40:16 | sdb: rev = 0606
Oct 31 17:40:16 | sdb: h:b:t:l = 1:0:0:1
Oct 31 17:40:16 | sdb: tgt_node_name = 0x5000d31003494015
Oct 31 17:40:16 | sdb: path state = running

Oct 31 17:40:16 | sdb: 2425 cyl, 255 heads, 63 sectors/track, start at 0
Oct 31 17:40:16 | sdb: serial = 00034940-00000003
Oct 31 17:40:16 | sdb: get_state
Oct 31 17:40:16 | loading /lib64/multipath/libchecktur.so checker
Oct 31 17:40:16 | sdb: path checker = tur (controller setting)
Oct 31 17:40:16 | sdb: checker timeout = 30000 ms (sysfs setting)
Oct 31 17:40:16 | sdb: state = up
Oct 31 17:40:16 | sdb: uid_attribute = ID_SERIAL (internal default)
Oct 31 17:40:16 | sdb: uid = 36000d310034940000000000000000003 (udev)
Oct 31 17:40:16 | sdb: detect_prio = 1 (config file default)
Oct 31 17:40:16 | sdb: prio = const (controller setting)
Oct 31 17:40:16 | sdb: prio args = (null) (controller setting)
Oct 31 17:40:16 | sdb: const prio = 1
Oct 31 17:40:16 | sdc: not found in pathvec
Oct 31 17:40:16 | sdc: mask = 0x3f
Oct 31 17:40:16 | sdc: dev_t = 8:32
Oct 31 17:40:16 | sdc: size = 7408818585
Oct 31 17:40:16 | sdc: vendor = COMPELNT
Oct 31 17:40:16 | sdc: product = Compellent Vol
Oct 31 17:40:16 | sdc: rev = 0606
Oct 31 17:40:16 | sdc: h:b:t:l = 1:0:1:1
Oct 31 17:40:16 | sdc: tgt_node_name = 0x5000d31003494008
Oct 31 17:40:16 | sdc: path state = running

Oct 31 17:40:16 | sdc: 2425 cyl, 255 heads, 63 sectors/track, start at 0
Oct 31 17:40:16 | sdc: serial = 00034940-00000003
Oct 31 17:40:16 | sdc: get_state
Oct 31 17:40:16 | sdc: path checker = tur (controller setting)
Oct 31 17:40:16 | sdc: checker timeout = 30000 ms (sysfs setting)
Oct 31 17:40:16 | sdc: state = up
Oct 31 17:40:16 | sdc: uid_attribute = ID_SERIAL (internal default)
Oct 31 17:40:16 | sdc: uid = 36000d310034940000000000000000003 (udev)
Oct 31 17:40:16 | sdc: detect_prio = 1 (config file default)
Oct 31 17:40:16 | sdc: prio = const (controller setting)
Oct 31 17:40:16 | sdc: prio args = (null) (controller setting)
Oct 31 17:40:16 | sdc: const prio = 1
Oct 31 17:40:16 | sr0: device node name blacklisted
Oct 31 17:40:16 | sdd: not found in pathvec
Oct 31 17:40:16 | sdd: mask = 0x3f
Oct 31 17:40:16 | sdd: dev_t = 8:48
Oct 31 17:40:16 | sdd: size = 0
Oct 31 17:40:16 | sdd: vendor = iDRAC
Oct 31 17:40:16 | sdd: product = Virtual Floppy
Oct 31 17:40:16 | sdd: rev = 0329
Oct 31 17:40:16 | sdd: h:b:t:l = 12:0:0:1
Oct 31 17:40:16 | sdd: path state = running

Oct 31 17:40:16 | sr1: device node name blacklisted
Oct 31 17:40:16 | dm-0: device node name blacklisted
Oct 31 17:40:16 | dm-1: device node name blacklisted
Oct 31 17:40:16 | dm-2: device node name blacklisted
Oct 31 17:40:16 | dm-3: device node name blacklisted
===== paths list =====
uuid                              hcil     dev dev_t pri dm_st chk_st vend/pro
36000d310034940000000000000000003 1:0:0:1  sdb 8:16  1   undef ready  COMPELNT
36000d310034940000000000000000003 1:0:1:1  sdc 8:32  1   undef ready  COMPELNT
                                  12:0:0:1 sdd 8:48  -1  undef faulty iDRAC
Oct 31 17:40:16 | params = 1 queue_if_no_path 0 1 1 service-time 0 2 2 8:16 1 1 8:32 1 1
Oct 31 17:40:16 | status = 2 0 0 0 1 1 A 0 2 2 8:16 A 1 0 1 8:32 A 0 0 1
Oct 31 17:40:16 | DB_DATA01: disassemble map [1 queue_if_no_path 0 1 1 service-time 0 2 2 8:16 1 1 8:32 1 1 ]
Oct 31 17:40:16 | DB_DATA01: disassemble status [2 0 0 0 1 1 A 0 2 2 8:16 A 1 0 1 8:32 A 0 0 1 ]
Oct 31 17:40:16 | tur checker refcount 2
Oct 31 17:40:16 | const prioritizer refcount 2
Oct 31 17:40:16 | tur checker refcount 1
Oct 31 17:40:16 | const prioritizer refcount 1
Oct 31 17:40:16 | unloading const prioritizer
Oct 31 17:40:16 | unloading tur checker
Oct 31 17:40:16 | unloading directio checker

Well... after an extensive research on the Web I found a solution "zeroing" the wrong GPT partition created inside the /dev/sdb. After that, all I needed to do was create another gpt partition table on /dev/mapper/DB_DATA01 (if you notice there were 3 partitions before).

I used a small python here How to remove GPT from HDD to drop all partition tables from /dev/sdb.

Then I used gdisk utility to create a new GPT table and new partition using all space in the mapped device.

After that, juste used mkfs.xfs again to format and got ID with blkid. Then replaced the old ID in /etc/fstab and rebooted.

Voilà! Everything worked again!