In the last post we covered some possible options that we have to backup our data. It all boils down to providing redundancy in case something fails, which allows us to have some peace of mind in the event of hard drive failure.
But having copies of our data doesn’t mean that we are just going to sit and wait until the hard drive fails. What if the secondary backup drive fails? What if we don’t realize until it is too late and our main drive fails as well?
Drives will fail sooner or later, the question is… when will this happen?
Introducing S.M.A.R.T
S.M.A.R.T, which is the acronym for Self-Monitoring, Analysis and Reporting Technology is a technology that appeared in 2004 for PATA IDE devices and now has spread to all kinds of storage devices such as HDD, SDD, or eMMC and technologies such as ATA, SCSI and NVMe. SMART is included inside any hard drive that we can buy nowadays and its mission is to provide the storage device with the capability of monitoring its own health status, and report it to the operating system.
In other words, our hard drives are aware of how old they are getting and they can detect signs of wear that are a strong prediction of imminent failure, even before the drive actually fails. According to Seagate, 60% of spinning drive failures are of mechanical nature and can be predicted before they happen, which gives us an opportunity of saving our data and replacing them before it is too late.
In Linux we can query the SMART system through the tools included in smartmontools.
How is my hard drive doing?
SMART predictions will be based on physical statistics, such as temperature, read errors, seek errors and such. Bad performance on these indicators is a strong red flag, and studying them allows us to assess how worn out our hardware is getting. You can see some descriptions in this list.
In order to check these counters, first make sure that SMART is enabled in the drive
# smartctl --smart=on /dev/sdX
, then you can get all available information from
# smartctl -a /dev/sdX
=== START OF INFORMATION SECTION ===
Device Model: WDC WD8003FRYZ-01JPDB1
Serial Number: 7SJ008RW
LU WWN Device Id: 5 000cca 252dc3010
Firmware Version: 01.01H02
User Capacity: 8,001,563,222,016 bytes [8.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Nov 10 15:51:28 2018 MST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 93) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: (1057) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always - 0
2 Throughput_Performance 0x0005 131 131 054 Pre-fail Offline - 104
3 Spin_Up_Time 0x0007 153 153 024 Pre-fail Always - 430 (Average 429)
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 29
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 128 128 020 Pre-fail Offline - 18
9 Power_On_Hours 0x0012 100 100 000 Old_age Always - 477
10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 29
22 Unknown_Attribute 0x0023 100 100 025 Pre-fail Always - 100
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 121
193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always - 121
194 Temperature_Celsius 0x0002 157 157 000 Old_age Always - 38 (Min/Max 20/39)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 32086 -
# 2 Short offline Completed without error 00% 32062 -
# 3 Short offline Completed without error 00% 32038 -
# 4 Short offline Completed without error 00% 32014 -
# 5 Short offline Completed without error 00% 31990 -
# 6 Short offline Completed without error 00% 31966 -
# 7 Short offline Completed without error 00% 31956 -
# 8 Short offline Completed without error 00% 31929 -
# 9 Short offline Completed without error 00% 31929 -
#10 Short offline Aborted by host 40% 31929 -
#11 Short offline Aborted by host 80% 31929 -
#12 Short offline Completed without error 00% 31929 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
We might have noticed this section
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 32086 - # 2 Short offline Completed without error 00% 32062 - # 3 Short offline Completed without error 00% 32038 - # 4 Short offline Completed without error 00% 32014 - # 5 Short offline Completed without error 00% 31990 - # 6 Short offline Completed without error 00% 31966 - # 7 Short offline Completed without error 00% 31956 - # 8 Short offline Completed without error 00% 31929 - # 9 Short offline Completed without error 00% 31929 - #10 Short offline Aborted by host 40% 31929 - #11 Short offline Aborted by host 80% 31929 - #12 Short offline Completed without error 00% 31929 -
As we can infer from it, SMART enabled drives are able to perform an offline self test on themselves in order to assess the health of the drive. We have mainly two kinds of self test
- Short: uses basic heuristics that can normally predict failure. They take typically less than five minutes.
- Long: includes a surface scan and can take hours, but the results will be more accurate.
It goes without saying that the performance of the drive will go down while busy running the tests.
In order to launch a short test just
# smartctl -t short /dev/sdX
, or a for long one
# smartctl -t long /dev/sdX
The test can be cancelled with
# smartctl -X /dev/sdX
The test results can be obtained with smartctl -a or smartctl -l selftest.
We can compare some stats for a drive in a good state
=== START OF READ SMART DATA SECTION === SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 150 140 021 Pre-fail Always - 9491 4 Start_Stop_Count 0x0032 076 076 000 Old_age Always - 24699 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 037 037 000 Old_age Always - 46078 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 667 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 30 193 Load_Cycle_Count 0x0032 095 095 000 Old_age Always - 317796 194 Temperature_Celsius 0x0022 115 088 000 Old_age Always - 37 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
, with a drive that is starting to fail
=== START OF READ SMART DATA SECTION === SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 54 3 Spin_Up_Time 0x0027 205 173 021 Pre-fail Always - 4716 4 Start_Stop_Count 0x0032 099 099 000 Old_age Always - 1463 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0 9 Power_On_Hours 0x0032 062 062 000 Old_age Always - 28284 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 778 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 544 193 Load_Cycle_Count 0x0032 001 001 000 Old_age Always - 2033894 194 Temperature_Celsius 0x0022 113 092 000 Old_age Always - 37 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 1 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 1 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 1
The following one is already failing hard, we can even see the reason
=== START OF READ SMART DATA SECTION === SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed: read failure 90% 16492 584440551 # 2 Short offline Completed: read failure 90% 16491 584440551 * Health Check for sda === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED Please note the following marginal Attributes: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 190 Airflow_Temperature_Cel 0x0022 070 030 045 Old_age Always In_the_past 30 (255 255 43 30 0) * Stats for sda === START OF READ SMART DATA SECTION === SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 094 094 006 Pre-fail Always - 97132869 3 Spin_Up_Time 0x0003 091 090 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 087 087 020 Old_age Always - 13813 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 6 7 Seek_Error_Rate 0x000f 091 060 030 Pre-fail Always - 5727032525 9 Power_On_Hours 0x0032 082 082 000 Old_age Always - 16492 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 098 098 020 Old_age Always - 2554 183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 083 083 000 Old_age Always - 17 188 Command_Timeout 0x0032 100 098 000 Old_age Always - 240521838652 189 High_Fly_Writes 0x003a 099 099 000 Old_age Always - 1 190 Airflow_Temperature_Cel 0x0022 070 030 045 Old_age Always In_the_past 30 (255 255 43 30 0) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 300 193 Load_Cycle_Count 0x0032 086 086 000 Old_age Always - 29035 194 Temperature_Celsius 0x0022 030 070 000 Old_age Always - 30 (0 13 0 0 0) 195 Hardware_ECC_Recovered 0x001a 020 003 000 Old_age Always - 97132869 197 Current_Pending_Sector 0x0012 100 095 000 Old_age Always - 87 198 Offline_Uncorrectable 0x0010 100 095 000 Old_age Offline - 87 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 10159 (199 92 0) 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 13982601688 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 16679969184
You can see some good examples on how to react when bad things happens here.
While some stats will be more significant than others depending on the vendor, the following seem to be good indicators across the board
SMART 5 Reallocated Sectors Count SMART 187 Reported Uncorrectable Errors SMART 188 Command Timeout SMART 197 Current Pending Sector Count SMART 198 Uncorrectable Sector Count
Monitoring drives automatically
smartmontools also come with a the smartd daemon. This service will monitor the device status and notify us through a configurable action.
Enable and start the service
# systemctl enable --now smartd
, and edit smartd.conf according to your needs.
You can shedule long and short tests during the week, for instance the following will do a short scan every day at 2 am, and a long one on Sundays at 3 am. See the man page for details
DEVICESCAN -s (S/../.././02|L/../../6/03)
You can also specify whether you want your hard drive to be woken up for the test if it is idle or sleeping, see the powermode options here.
Adding -m my@email.com will result in an email being sent when problems are detected. Also we can have our own scripts called with the -M switch, in which case we have the diagnostics information available as variables such as SMARTD_MESSAGE and SMARTD_FAILTYPE. See the man page for the full list.
For instance this simple script will warn all logged in users in the terminal
#!/bin/sh # Warn all users of a problem wall <<EOF Problem detected with disk: $SMARTD_DEVICESTRING Warning message from smartd is: $SMARTD_MESSAGE Shutting down machine in 30 seconds... EOF
For more details and examples, check the Arch Wiki, and /usr/share/doc/smartmontools/examples.
Resources
https://www.smartmontools.org/wiki
https://linux.die.net/man/8/smartd
https://linux.die.net/man/5/smartd.conf
2 Comments