I'm having a problem with my raid5 disk. I already had disk failures and replaced disks without any problem, but this time I'm having a hard time fixing it.
Here is the situation : I am running Ubuntu 12.04. I have 3x2TB disks. I have 2 raid5 disks md0 and md1. md0 is working fine. I'm having problem with md1 that is working now in degraded mode as sdc2 in not part of the array anymore. But sdc is not dead as sdc1 is part of md0 and works fine.
$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md1 : active raid5 sdb2[3] sdd2[2]
409336832 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [_UU]
md0 : active raid5 sdc1[4] sdb1[5] sdd1[3]
3497163776 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
unused devices: <none>
Details about /dev/md1 are as follow :
$ sudo mdadm --detail /dev/md1
/dev/md1:
Version : 1.2
Creation Time : Mon Aug 22 14:17:57 2016
Raid Level : raid5
Array Size : 409336832 (390.37 GiB 419.16 GB)
Used Dev Size : 204668416 (195.19 GiB 209.58 GB)
Raid Devices : 3
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Wed Dec 28 08:17:51 2016
State : clean, degraded
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Name : serv1:1 (local to host serv1)
UUID : bf2095af:69c02451:1f31ee06:93b92c8b
Events : 844
Number Major Minor RaidDevice State
0 0 0 0 removed
3 8 18 1 active sync /dev/sdb2
2 8 50 2 active sync /dev/sdd2
Trying to remove /dev/sdc2 from /dev/md1 with sudo mdadm /dev/md1 -r /dev/sdc2
give me the following mdadm: hot remove failed for /dev/sdc2: No such device or address
which is fine as it means that /dev/sdc2 no longer belongs to the array.
But when trying to add /dev/sdc2 to the array with sudo mdadm /dev/md1 -a /dev/sdc2
it gives me the following error mdadm: add new device failed for /dev/sdc2 as 4: Invalid argument
. I noticed that when trying to add sdc2, I have a bunch of errors like these logged in /var/log/syslog
:
ata3.00: exception Emask 0x0 SAct 0x4000000 SErr 0x0 action 0x0
ata3.00: irq_stat 0x40000008
ata3.00: failed command: READ FPDMA QUEUED
ata3.00: cmd 60/08:d0:08:88:76/00:00:d0:00:00/40 tag 26 ncq 4096 in
res 41/40:00:09:88:76/00:00:d0:00:00/40 Emask 0x409 (media error) <F>
ata3.00: status: { DRDY ERR }
ata3.00: error: { UNC }
ata3.00: configured for UDMA/133
ata3: EH complete
ata3.00: exception Emask 0x0 SAct 0x8000000 SErr 0x0 action 0x0
ata3.00: irq_stat 0x40000008
ata3.00: failed command: READ FPDMA QUEUED
ata3.00: cmd 60/08:d8:08:88:76/00:00:d0:00:00/40 tag 27 ncq 4096 in
res 41/40:00:09:88:76/00:00:d0:00:00/40 Emask 0x409 (media error) <F>
ata3.00: status: { DRDY ERR }
ata3.00: error: { UNC }
ata3.00: configured for UDMA/133
ata3: EH complete
ata3.00: exception Emask 0x0 SAct 0x10000000 SErr 0x0 action 0x0
ata3.00: irq_stat 0x40000008
ata3.00: failed command: READ FPDMA QUEUED
ata3.00: cmd 60/08:e0:08:88:76/00:00:d0:00:00/40 tag 28 ncq 4096 in
res 41/40:00:09:88:76/00:00:d0:00:00/40 Emask 0x409 (media error) <F>
ata3.00: status: { DRDY ERR }
ata3.00: error: { UNC }
ata3.00: configured for UDMA/133
sd 2:0:0:0: [sdc] Unhandled sense code
sd 2:0:0:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 2:0:0:0: [sdc] Sense Key : Medium Error [current] [descriptor]
Descriptor sense data with sense descriptors (in hex):
72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
d0 76 88 09
sd 2:0:0:0: [sdc] Add. Sense: Unrecovered read error - auto reallocate failed
sd 2:0:0:0: [sdc] CDB: Read(10): 28 00 d0 76 88 08 00 00 08 00
end_request: I/O error, dev sdc, sector 3497429001
Buffer I/O error on device sdc2, logical block 1
ata3: EH complete
I am not understanding what I need to do. As it looks that my sdc disk is dead whereas I have absolutely no problem with /dev/md0
wich uses /dev/sdc1
.
I already tried to stop md1 then assemble it with sudo mdadm --assemble /dev/md1 /dev/sdb2 /dev/sdd2
. But adding sdc2 would always give the same problem.
Here is what I get with sudo smartctl -a /dev/sdc2
:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 188 188 051 Pre-fail Always - 81654
3 Spin_Up_Time 0x0027 175 174 021 Pre-fail Always - 4250
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 33
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 086 086 000 Old_age Always - 10611
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 33
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 22
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 19
194 Temperature_Celsius 0x0022 119 106 000 Old_age Always - 28
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline - 0
And here is what I obtain with sudo badblocks /dev/sdc2
:
4
5
6
7
User contributions licensed under CC BY-SA 3.0