raid5 disk degraded partition detached

1

I'm having a problem with my raid5 disk. I already had disk failures and replaced disks without any problem, but this time I'm having a hard time fixing it.

Here is the situation : I am running Ubuntu 12.04. I have 3x2TB disks. I have 2 raid5 disks md0 and md1. md0 is working fine. I'm having problem with md1 that is working now in degraded mode as sdc2 in not part of the array anymore. But sdc is not dead as sdc1 is part of md0 and works fine.

$ cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md1 : active raid5 sdb2[3] sdd2[2]
      409336832 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [_UU]

md0 : active raid5 sdc1[4] sdb1[5] sdd1[3]
      3497163776 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]

unused devices: <none>

Details about /dev/md1 are as follow :

$ sudo mdadm --detail /dev/md1
/dev/md1:
        Version : 1.2
  Creation Time : Mon Aug 22 14:17:57 2016
     Raid Level : raid5
     Array Size : 409336832 (390.37 GiB 419.16 GB)
  Used Dev Size : 204668416 (195.19 GiB 209.58 GB)
   Raid Devices : 3
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Wed Dec 28 08:17:51 2016
          State : clean, degraded 
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : serv1:1  (local to host serv1)
           UUID : bf2095af:69c02451:1f31ee06:93b92c8b
         Events : 844

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       3       8       18        1      active sync   /dev/sdb2
       2       8       50        2      active sync   /dev/sdd2

Trying to remove /dev/sdc2 from /dev/md1 with sudo mdadm /dev/md1 -r /dev/sdc2 give me the following mdadm: hot remove failed for /dev/sdc2: No such device or address which is fine as it means that /dev/sdc2 no longer belongs to the array.

But when trying to add /dev/sdc2 to the array with sudo mdadm /dev/md1 -a /dev/sdc2 it gives me the following error mdadm: add new device failed for /dev/sdc2 as 4: Invalid argument. I noticed that when trying to add sdc2, I have a bunch of errors like these logged in /var/log/syslog :

ata3.00: exception Emask 0x0 SAct 0x4000000 SErr 0x0 action 0x0
ata3.00: irq_stat 0x40000008
ata3.00: failed command: READ FPDMA QUEUED
ata3.00: cmd 60/08:d0:08:88:76/00:00:d0:00:00/40 tag 26 ncq 4096 in
         res 41/40:00:09:88:76/00:00:d0:00:00/40 Emask 0x409 (media error) <F>
ata3.00: status: { DRDY ERR }
ata3.00: error: { UNC }
ata3.00: configured for UDMA/133
ata3: EH complete
ata3.00: exception Emask 0x0 SAct 0x8000000 SErr 0x0 action 0x0
ata3.00: irq_stat 0x40000008
ata3.00: failed command: READ FPDMA QUEUED
ata3.00: cmd 60/08:d8:08:88:76/00:00:d0:00:00/40 tag 27 ncq 4096 in
         res 41/40:00:09:88:76/00:00:d0:00:00/40 Emask 0x409 (media error) <F>
ata3.00: status: { DRDY ERR }
ata3.00: error: { UNC }
ata3.00: configured for UDMA/133
ata3: EH complete
ata3.00: exception Emask 0x0 SAct 0x10000000 SErr 0x0 action 0x0
ata3.00: irq_stat 0x40000008
ata3.00: failed command: READ FPDMA QUEUED
ata3.00: cmd 60/08:e0:08:88:76/00:00:d0:00:00/40 tag 28 ncq 4096 in
         res 41/40:00:09:88:76/00:00:d0:00:00/40 Emask 0x409 (media error) <F>
ata3.00: status: { DRDY ERR }
ata3.00: error: { UNC }
ata3.00: configured for UDMA/133
sd 2:0:0:0: [sdc] Unhandled sense code
sd 2:0:0:0: [sdc]  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 2:0:0:0: [sdc]  Sense Key : Medium Error [current] [descriptor]
Descriptor sense data with sense descriptors (in hex):
        72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
        d0 76 88 09 
sd 2:0:0:0: [sdc]  Add. Sense: Unrecovered read error - auto reallocate failed
sd 2:0:0:0: [sdc] CDB: Read(10): 28 00 d0 76 88 08 00 00 08 00
end_request: I/O error, dev sdc, sector 3497429001
Buffer I/O error on device sdc2, logical block 1
ata3: EH complete

I am not understanding what I need to do. As it looks that my sdc disk is dead whereas I have absolutely no problem with /dev/md0 wich uses /dev/sdc1. I already tried to stop md1 then assemble it with sudo mdadm --assemble /dev/md1 /dev/sdb2 /dev/sdd2. But adding sdc2 would always give the same problem.

Here is what I get with sudo smartctl -a /dev/sdc2 :

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   188   188   051    Pre-fail  Always       -       81654
  3 Spin_Up_Time            0x0027   175   174   021    Pre-fail  Always       -       4250
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       33
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   086   086   000    Old_age   Always       -       10611
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       33
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       22
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       19
194 Temperature_Celsius     0x0022   119   106   000    Old_age   Always       -       28
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

And here is what I obtain with sudo badblocks /dev/sdc2 :

4
5
6
7
ubuntu
mdadm
raid5
asked on Server Fault Dec 28, 2016 by Hedi

0 Answers

Nobody has answered this question yet.


User contributions licensed under CC BY-SA 3.0