Troubleshooting a “Dead Timeout” NVMe Device on ESX

I’ve been spending some time working with VMware Cloud Foundation (VCF) 9.1 in my lab environment. The lab consists of several nested ESXi hosts that use NFS storage provided by a virtual machine. The VM providing the NFS storage, in turn, uses a consumer-grade NVMe SSD for its storage.

Recently, I ran into an interesting problem where the datastore backing this NFS appliance would occasionally become unavailable.

An NVMe Controller Failure

The first time I encountered the problem, I started looking through the ESXi logs to see what had happened. In vmkwarning.log, I found several messages related to the NVMe controller:

WARNING: NVMEDEV:9464 Controller 257 failed to recover after multiple attempts, switch to periodical recovery. 
WARNING: NVMEPSA:217 Complete vmkNvmeCmd: 0x45d9a265b8c0, vmkPsaCmd: 0x45d9c7dd5fc0, cmdId.initiator=0x45395c79ba58, CmdSN: 0x0, status: 0x80d 
WARNING: NvmeDevice: 573: Get identify namespace data failed for device Status %s. 
WARNING: NvmeUtil: 151: Error on Cmd(0x45d9c7dd5fc0) 0x6, CmdSN 0x0 from world 0 to component "nqn.1994-11.com.samsung:nvme:990PRO:M.2:<REDACTED>" H:0xe D:0x0 P:0x0 
WARNING: NVMEPSA:217 Complete vmkNvmeCmd: 0x45d9a26d4ac0, vmkPsaCmd: 0x45d9c7dd5fc0, cmdId.initiator=0x45395c79ba58, CmdSN: 0x0, status: 0x80d 
WARNING: NVMEPSA:217 Complete vmkNvmeCmd: 0x45d9a27100c0, vmkPsaCmd: 0x45d9c7dd5fc0, cmdId.initiator=0x45395c79ba58, CmdSN: 0x0, status: 0x80d

The first line was particularly interesting:

Controller 257 failed to recover after 21 attempts

My interpretation of these messages is that ESX detected a problem communicating with the NVMe controller and attempted to recover it. After 21 unsuccessful recovery attempts, ESX stopped trying to recover the controller normally and switched to periodic recovery.

The subsequent messages indicate that ESXi was unable to retrieve information from the NVMe namespace and was encountering command failures.

At this point, the underlying NVMe device was effectively unavailable to ESX, which explained why the VMFS datastore disappeared, which in turn made the guest hosted NFS mount disappear.

Checking the NVMe Device

A reboot of the ESX host brought the device back online and the datastore became available again. However, a few days later the error returned. This time I saw similar vmkwarning.log entries, but I wanted to take a closer look at the NVMe device. I used the following command:

esxcli storage core device list | grep -i "samsung" -B 1 -A 12

The output included the following:

t10.NVMe____Samsung_SSD_990_PRO_with_Heatsink_4TB___<redacted>
   Display Name: Local NVMe Disk (t10.NVMe____Samsung_SSD_990_PRO_with_Heatsink_4TB___<redacted>)
   Has Settable Display Name: true
   Size: 3815447
   Device Type: Direct-Access
   Multipath Plugin: HPP
   Devfs Path:
   Vendor: NVMe
   Model: Samsung SSD 990 PRO with Heatsink 4TB
   Revision: 4B2QJXD7
   SCSI Level: 0
   Is Pseudo: false
   Status: dead timeout
   Is RDM Capable: false
   Is Local: true
   Is Removable: false
   Is SSD: true
   Is VVOL PE: false
   Is Offline: false
   Is Perennially Reserved: false
   Queue Full Sample Size: 0

There were two things that caught my attention here.

First, the device status was:

Status: dead timeout

That was consistent with what I had seen in vmkwarning.log.

Second, the firmware revision was:

Revision: 4B2QJXD7

Checking the Samsung Firmware

I then checked Samsung’s support site for firmware updates for the 990 PRO.

Samsung had newer firmware available for the drive, including firmware versions that addressed issues involving intermittent drive recognition and blue-screen problems.

That seemed particularly interesting given the behavior I was seeing. The drive wasn’t permanently failing; it would occasionally stop responding to the host and then become available again after a reboot.

At that point, I decided that updating the firmware was worth trying.

I updated the drive to the latest firmware available at the time, 8B2QJXD7, using the ISO image provided by Samsung.

Has the Firmware Update Fixed It?

The real test was simply to wait and see what happened.

I’ve continued using the NVMe drive in the lab for roughly a month since updating the firmware. During that time, I’ve performed a number of tests and continued using the drive in the same general configuration.

So far, the device has remained stable.

I haven’t seen the NVMe controller recovery messages return, and I haven’t had another occurrence where the device went into a dead timeout state and caused the NFS datastore to become unavailable.

While that doesn’t prove conclusively that the firmware was the root cause, the combination of the symptoms, the older firmware version, Samsung’s firmware notes, and the fact that the problem has not returned after the update makes the firmware a strong suspect.

Conclusion

This was a good reminder that when an ESX storage device suddenly disappears, it is worth looking beyond the datastore and filesystem layers.

In my case, the NFS datastore was only the symptom. The underlying problem was an NVMe device that had stopped responding to ESX.

The vmkwarning.log messages showed ESX repeatedly attempting to recover the NVMe controller, while esxcli storage core device list subsequently showed the device in a dead timeout state.

Checking the firmware version ultimately revealed that the Samsung 990 PRO was running an older firmware revision. Updating it to 8B2QJXD7 appears to have resolved the intermittent failures in my lab.

If you are seeing similar NVMe controller recovery messages or dead timeout states on ESXi, checking the firmware version of the affected drive would be a worthwhile troubleshooting step.