miscellaneous

Monitor your hard drive health with SMART daemon

In the last post we covered some possible options that we have to backup our data. It all boils down to providing redundancy in case something fails, which allows us to have some peace of mind in the event of hard drive failure.

But having copies of our data doesn’t mean that we are just going to sit and wait until the hard drive fails. What if the secondary backup drive fails? What if we don’t realize until it is too late and our main drive fails as well?

Drives will fail sooner or later, the question is… when will this happen?

 

Introducing S.M.A.R.T

S.M.A.R.T, which is the acronym for Self-Monitoring, Analysis and Reporting Technology is a technology that appeared in 2004 for PATA IDE devices and now has spread to all kinds of storage devices such as HDD, SDD, or eMMC and technologies such as ATA, SCSI and NVMe. SMART is included inside any hard drive that we can buy nowadays and its mission is to provide the storage device with the capability of monitoring its own health status, and report it to the operating system.

In other words, our hard drives are aware of how old they are getting and they can detect signs of wear that are a strong prediction of imminent failure, even before the drive actually fails. According to Seagate, 60% of spinning drive failures are of mechanical nature and can be predicted before they happen, which gives us an opportunity of saving our data and replacing them before it is too late.

In Linux we can query the SMART system through the tools included in smartmontools.

 

How is my hard drive doing?

 

SMART predictions will be based on physical statistics, such as temperature, read errors,  seek errors and such. Bad performance on these indicators is a strong red flag, and studying them allows us to assess how worn out our hardware is getting. You can see some descriptions in this list.

In order to check these counters, first make sure that SMART is enabled in the drive

# smartctl --smart=on /dev/sdX

, then you can get all available information from

# smartctl -a /dev/sdX

=== START OF INFORMATION SECTION ===                                                                                                                                                                                                                                                                                           
Device Model:     WDC WD8003FRYZ-01JPDB1                                                                                                                                                                                                                                                                                       
Serial Number:    7SJ008RW                                                                                                                                                                                                                                                                                                     
LU WWN Device Id: 5 000cca 252dc3010                                                                                                                                                                                                                                                                                           
Firmware Version: 01.01H02                                                                                                                                                                                                                                                                                                     
User Capacity:    8,001,563,222,016 bytes [8.00 TB]                                                                                                                                                                                                                                                                            
Sector Sizes:     512 bytes logical, 4096 bytes physical                                                                                                                                                                                                                                                                       
Rotation Rate:    7200 rpm                                                                                                                                                                                                                                                                                                     
Form Factor:      3.5 inches                                                                                                                                                                                                                                                                                                   
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sat Nov 10 15:51:28 2018 MST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (   93) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        (1057) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   131   131   054    Pre-fail  Offline      -       104
  3 Spin_Up_Time            0x0007   153   153   024    Pre-fail  Always       -       430 (Average 429)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       29
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   128   128   020    Pre-fail  Offline      -       18
  9 Power_On_Hours          0x0012   100   100   000    Old_age   Always       -       477
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       29
 22 Unknown_Attribute       0x0023   100   100   025    Pre-fail  Always       -       100
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       121
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       121
194 Temperature_Celsius     0x0002   157   157   000    Old_age   Always       -       38 (Min/Max 20/39)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     32086         -
# 2  Short offline       Completed without error       00%     32062         -
# 3  Short offline       Completed without error       00%     32038         -
# 4  Short offline       Completed without error       00%     32014         -
# 5  Short offline       Completed without error       00%     31990         -
# 6  Short offline       Completed without error       00%     31966         -
# 7  Short offline       Completed without error       00%     31956         -
# 8  Short offline       Completed without error       00%     31929         -
# 9  Short offline       Completed without error       00%     31929         -
#10  Short offline       Aborted by host               40%     31929         -
#11  Short offline       Aborted by host               80%     31929         -
#12  Short offline       Completed without error       00%     31929         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

We might have noticed this section

Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     32086         -
# 2  Short offline       Completed without error       00%     32062         -
# 3  Short offline       Completed without error       00%     32038         -
# 4  Short offline       Completed without error       00%     32014         -
# 5  Short offline       Completed without error       00%     31990         -
# 6  Short offline       Completed without error       00%     31966         -
# 7  Short offline       Completed without error       00%     31956         -
# 8  Short offline       Completed without error       00%     31929         -
# 9  Short offline       Completed without error       00%     31929         -
#10  Short offline       Aborted by host               40%     31929         -
#11  Short offline       Aborted by host               80%     31929         -
#12  Short offline       Completed without error       00%     31929         -

As we can infer from it, SMART enabled drives are able to perform an offline self test on themselves in order to assess the health of the drive. We have mainly two kinds of self test

  • Short: uses basic heuristics that can normally predict failure. They take typically less than five minutes.
  • Long: includes a surface scan and can take hours, but the results will be more accurate.

It goes without saying that the performance of the drive will go down while busy running the tests.

In order to launch a short test just

# smartctl -t short /dev/sdX

, or a for long one

# smartctl -t long /dev/sdX

The test can be cancelled with

# smartctl -X /dev/sdX

The test results can be obtained with smartctl -a  or smartctl -l selftest.

We can compare some stats for a drive in a good state

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   150   140   021    Pre-fail  Always       -       9491
  4 Start_Stop_Count        0x0032   076   076   000    Old_age   Always       -       24699
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   037   037   000    Old_age   Always       -       46078
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       667
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       30
193 Load_Cycle_Count        0x0032   095   095   000    Old_age   Always       -       317796
194 Temperature_Celsius     0x0022   115   088   000    Old_age   Always       -       37
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

, with a drive that is starting to fail

=== START OF READ SMART DATA SECTION ===                                                                                                                                                                                                                                                                                       
SMART Attributes Data Structure revision number: 16                                                                                                                                                                                                                                                                            
Vendor Specific SMART Attributes with Thresholds:                                                                                                                                                                                                                                                                              
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE                                                                                                                                                                                                                               
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       54                                                                                                                                                                                                                                      
  3 Spin_Up_Time            0x0027   205   173   021    Pre-fail  Always       -       4716                                                                                                                                                                                                                                    
  4 Start_Stop_Count        0x0032   099   099   000    Old_age   Always       -       1463                                                                                                                                                                                                                                    
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0                                                                                                                                                                                                                                       
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0                                                                                                                                                                                                                                       
  9 Power_On_Hours          0x0032   062   062   000    Old_age   Always       -       28284                                                                                                                                                                                                                                   
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0                                                                                                                                                                                                                                       
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0                                                                                                                                                                                                                                       
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       778                                                                                                                                                                                                                                     
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       544                                                                                                                                                                                                                                     
193 Load_Cycle_Count        0x0032   001   001   000    Old_age   Always       -       2033894                                                                                                                                                                                                                                 
194 Temperature_Celsius     0x0022   113   092   000    Old_age   Always       -       37                                                                                                                                                                                                                                      
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0                                                                                                                                                                                                                                       
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       1                                                                                                                                                                                                                                       
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       1                                                                                                                                                                                                                                       
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0                                                                                                                                                                                                                                       
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       1

The following one is already failing hard, we can even see the reason

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%     16492         584440551
# 2  Short offline       Completed: read failure       90%     16491         584440551

* Health Check for sda

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
Please note the following marginal Attributes:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
190 Airflow_Temperature_Cel 0x0022   070   030   045    Old_age   Always   In_the_past 30 (255 255 43 30 0)

* Stats for sda

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   094   094   006    Pre-fail  Always       -       97132869
  3 Spin_Up_Time            0x0003   091   090   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   087   087   020    Old_age   Always       -       13813
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       6
  7 Seek_Error_Rate         0x000f   091   060   030    Pre-fail  Always       -       5727032525
  9 Power_On_Hours          0x0032   082   082   000    Old_age   Always       -       16492
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   098   098   020    Old_age   Always       -       2554
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   083   083   000    Old_age   Always       -       17
188 Command_Timeout         0x0032   100   098   000    Old_age   Always       -       240521838652
189 High_Fly_Writes         0x003a   099   099   000    Old_age   Always       -       1
190 Airflow_Temperature_Cel 0x0022   070   030   045    Old_age   Always   In_the_past 30 (255 255 43 30 0)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       300
193 Load_Cycle_Count        0x0032   086   086   000    Old_age   Always       -       29035
194 Temperature_Celsius     0x0022   030   070   000    Old_age   Always       -       30 (0 13 0 0 0)
195 Hardware_ECC_Recovered  0x001a   020   003   000    Old_age   Always       -       97132869
197 Current_Pending_Sector  0x0012   100   095   000    Old_age   Always       -       87
198 Offline_Uncorrectable   0x0010   100   095   000    Old_age   Offline      -       87
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       10159 (199 92 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       13982601688
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       16679969184

You can see some good examples on how to react when bad things happens here.

While some stats will be more significant than others depending on the vendor, the following seem to be good indicators across the board

SMART 5           Reallocated Sectors Count
SMART 187         Reported Uncorrectable Errors
SMART 188         Command Timeout
SMART 197         Current Pending Sector Count
SMART 198         Uncorrectable Sector Count

 

Monitoring drives automatically

 

smartmontools also come with a the smartd daemon. This service will monitor the device status and notify us through a configurable action.

Enable and start the service

# systemctl enable --now smartd

, and edit smartd.conf according to your needs.

You can shedule long and short tests during the week, for instance the following will do a short scan every day at 2 am, and a long one on Sundays at 3 am. See the man page for details

DEVICESCAN -s (S/../.././02|L/../../6/03)

You can also specify whether you want your hard drive to be woken up for the test if it is idle or sleeping, see the powermode options here.

Adding -m my@email.com will result in an email being sent when problems are detected. Also we can have our own scripts called with the -M  switch, in which case we have the diagnostics information available as variables such as  SMARTD_MESSAGE and SMARTD_FAILTYPE. See the man page for the full list.

For instance this simple script will warn all logged in users in the terminal

#!/bin/sh

# Warn all users of a problem     
wall <<EOF
Problem detected with disk: $SMARTD_DEVICESTRING
Warning message from smartd is: $SMARTD_MESSAGE
Shutting down machine in 30 seconds...
EOF

 

For more details and examples, check the Arch Wiki, and /usr/share/doc/smartmontools/examples.

 

Resources

 

https://www.smartmontools.org/wiki

https://linux.die.net/man/8/smartd

https://linux.die.net/man/5/smartd.conf

Five stats to predict hard drive failure

What stats indicate hard drive failure

Author: nachoparker

Humbly sharing things that I find useful [ github dockerhub ]

2 Comments on “Monitor your hard drive health with SMART daemon

Leave a Reply

Your email address will not be published. Required fields are marked *