Testing faulty disk

0

I have a degraded RAID-1 setup and I'm trying to troubleshoot the problem whether it's completely dead (and need to send the drive back to the vendor) or it can be brought back.

So I ran smartctl:

root@linux:~# smartctl -l selftest /dev/sda
smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-176-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      1930         -
# 2  Short offline       Completed without error       00%      1930         -
# 3  Extended offline    Completed without error       00%      1930         -

Looks good. Let's take a look at the other stuff:

root@linux:~# smartctl --all /dev/sda
smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-176-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD40EFRX-68N32N0
Serial Number:    WD-WCC7K3NU4V6D
LU WWN Device Id: 5 0014ee 211e15108
Firmware Version: 82.00A82
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sun Mar 29 23:04:32 2020 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                    without error or no self-test has ever 
                    been run.
Total time to complete Offline 
data collection:        (44340) seconds.
Offline data collection
capabilities:            (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:    (   2) minutes.
Extended self-test routine
recommended polling time:    ( 470) minutes.
Conveyance self-test routine
recommended polling time:    (   5) minutes.
SCT capabilities:          (0x303d) SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   100   253   021    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       3
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   098   098   000    Old_age   Always       -       1930
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       3
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       0
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       1286
194 Temperature_Celsius     0x0022   117   114   000    Old_age   Always       -       33
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
ATA Error Count: 1757 (device log contains only the most recent five errors)
    CR = Command Register [HEX]
    FR = Features Register [HEX]
    SC = Sector Count Register [HEX]
    SN = Sector Number Register [HEX]
    CL = Cylinder Low Register [HEX]
    CH = Cylinder High Register [HEX]
    DH = Device/Head Register [HEX]
    DC = Device Command Register [HEX]
    ER = Error register [HEX]
    ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 1757 occurred at disk power-on lifetime: 1930 hours (80 days + 10 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  04 61 02 00 00 00 a0  Device Fault; Error: ABRT

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  ef 10 02 00 00 00 a0 08  11d+12:18:30.077  SET FEATURES [Enable SATA feature]
  ec 00 00 00 00 00 a0 08  11d+12:18:30.077  IDENTIFY DEVICE
  ef 03 46 00 00 00 a0 08  11d+12:18:30.077  SET FEATURES [Set transfer mode]
  ef 10 02 00 00 00 a0 08  11d+12:18:30.073  SET FEATURES [Enable SATA feature]
  ec 00 00 00 00 00 a0 08  11d+12:18:30.073  IDENTIFY DEVICE

Error 1756 occurred at disk power-on lifetime: 1930 hours (80 days + 10 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  04 61 46 00 00 00 a0  Device Fault; Error: ABRT

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  ef 03 46 00 00 00 a0 08  11d+12:18:30.077  SET FEATURES [Set transfer mode]
  ef 10 02 00 00 00 a0 08  11d+12:18:30.073  SET FEATURES [Enable SATA feature]
  ec 00 00 00 00 00 a0 08  11d+12:18:30.073  IDENTIFY DEVICE
  c8 00 08 08 00 00 e0 08  11d+12:18:30.051  READ DMA
  ef 10 02 00 00 00 a0 08  11d+12:18:30.041  SET FEATURES [Enable SATA feature]

Error 1755 occurred at disk power-on lifetime: 1930 hours (80 days + 10 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  04 61 02 00 00 00 a0  Device Fault; Error: ABRT

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  ef 10 02 00 00 00 a0 08  11d+12:18:30.073  SET FEATURES [Enable SATA feature]
  ec 00 00 00 00 00 a0 08  11d+12:18:30.073  IDENTIFY DEVICE
  c8 00 08 08 00 00 e0 08  11d+12:18:30.051  READ DMA
  ef 10 02 00 00 00 a0 08  11d+12:18:30.041  SET FEATURES [Enable SATA feature]
  ec 00 00 00 00 00 a0 08  11d+12:18:30.040  IDENTIFY DEVICE

Error 1754 occurred at disk power-on lifetime: 1930 hours (80 days + 10 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  04 61 08 08 00 00 e0  Device Fault; Error: ABRT 8 sectors at LBA = 0x00000008 = 8

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 08 08 00 00 e0 08  11d+12:18:30.051  READ DMA
  ef 10 02 00 00 00 a0 08  11d+12:18:30.041  SET FEATURES [Enable SATA feature]
  ec 00 00 00 00 00 a0 08  11d+12:18:30.040  IDENTIFY DEVICE
  ef 03 46 00 00 00 a0 08  11d+12:18:30.040  SET FEATURES [Set transfer mode]
  ef 10 02 00 00 00 a0 08  11d+12:18:30.037  SET FEATURES [Enable SATA feature]

Error 1753 occurred at disk power-on lifetime: 1930 hours (80 days + 10 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  04 61 02 00 00 00 a0  Device Fault; Error: ABRT

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  ef 10 02 00 00 00 a0 08  11d+12:18:30.041  SET FEATURES [Enable SATA feature]
  ec 00 00 00 00 00 a0 08  11d+12:18:30.040  IDENTIFY DEVICE
  ef 03 46 00 00 00 a0 08  11d+12:18:30.040  SET FEATURES [Set transfer mode]
  ef 10 02 00 00 00 a0 08  11d+12:18:30.037  SET FEATURES [Enable SATA feature]
  ec 00 00 00 00 00 a0 08  11d+12:18:30.037  IDENTIFY DEVICE

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      1930         -
# 2  Short offline       Completed without error       00%      1930         -
# 3  Extended offline    Completed without error       00%      1930         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

I tried to look through the output but unfortunately I'm not an expert in analyzing smartctl output. It said the drive passed the tests:

SMART overall-health self-assessment test result: PASSED

In the tresholds section everything looks okay (this is a rather new WD Red HDD, a few months old).

At the end of the output there are tons of errors though. I tried to look them up with no luck.

On the other hand, I cannot use the disk at all:

root@idealib:~# fdisk /dev/sda

Welcome to fdisk (util-linux 2.27.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

fdisk: cannot open /dev/sda: Input/output error

dmesg looks bad as well:

[ 1708.769491] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1708.771763] ata1.00: irq_stat 0x40000001
[ 1708.773710] ata1.00: failed command: READ DMA
[ 1708.775667] ata1.00: cmd c8/00:08:08:00:00/00:00:00:00:00/e0 tag 15 dma 4096 in
                        res 61/04:08:08:00:00/00:00:00:00:00/e0 Emask 0x1 (device error)
[ 1708.779619] ata1.00: status: { DRDY DF ERR }
[ 1708.781613] ata1.00: error: { ABRT }
[ 1708.784344] ata1.00: failed to enable AA (error_mask=0x1)
[ 1708.787355] ata1.00: failed to enable AA (error_mask=0x1)
[ 1708.789183] ata1.00: configured for UDMA/133 (device error ignored)
[ 1708.796930] ata1: EH complete

What other test should I run before I can be 100% sure that the disk is faulty and I need replace it? Or it's dead and I should just accept it? :)

linux
hard-drive
smartctl
asked on Server Fault Mar 29, 2020 by riska

1 Answer

1

I can't help with the SMART data (I'm not that smart!) but my default tool to poke at the drive is badblocks.

Is a tool for finding bad blocks on a hard drive. If you're okay deleting data, it's worth running it in destructive mode - note this will take some time to run!:

badblocks -wsv /dev/sda

There's actually pretty good information on the Arch wiki (https://wiki.archlinux.org/index.php/badblocks) about the tool and how to use it.

Another tool you could try is Spinrite, which operates from a freedos environment at boot. It's a little old, but does go quite low-level on the disk. I'd suggest a level 4 scan if you use it. https://www.grc.com/sr/spinrite.htm

I'm sure you've already done so, but I always suggest checking/changing the SATA cable, port on the motherboard, etc. I've been caught out by this kind of problem more times than I care to admit.

answered on Server Fault Mar 29, 2020 by SamR.

User contributions licensed under CC BY-SA 3.0