{"id":2429,"date":"2026-08-13T15:29:45","date_gmt":"2026-08-13T19:29:45","guid":{"rendered":"https:\/\/enterpriseadmins.org\/blog\/?p=2429"},"modified":"2026-08-13T15:29:45","modified_gmt":"2026-08-13T19:29:45","slug":"troubleshooting-a-dead-timeout-nvme-device-on-esx","status":"publish","type":"post","link":"https:\/\/enterpriseadmins.org\/blog\/virtualization\/troubleshooting-a-dead-timeout-nvme-device-on-esx\/","title":{"rendered":"Troubleshooting a \u201cDead Timeout\u201d NVMe Device on ESX"},"content":{"rendered":"\n

I’ve been spending some time working with VMware Cloud Foundation (VCF) 9.1 in my lab environment. The lab consists of several nested ESXi hosts that use NFS storage provided by a virtual machine. The VM providing the NFS storage, in turn, uses a consumer-grade NVMe SSD for its storage.<\/p>\n\n\n\n

Recently, I ran into an interesting problem where the datastore backing this NFS appliance would occasionally become unavailable.<\/p>\n\n\n\n

An NVMe Controller Failure<\/h2>\n\n\n\n

The first time I encountered the problem, I started looking through the ESXi logs to see what had happened. In vmkwarning.log<\/code>, I found several messages related to the NVMe controller:<\/p>\n\n\n\n

WARNING: NVMEDEV:9464 Controller 257 failed to recover after multiple attempts, switch to periodical recovery. \nWARNING: NVMEPSA:217 Complete vmkNvmeCmd: 0x45d9a265b8c0, vmkPsaCmd: 0x45d9c7dd5fc0, cmdId.initiator=0x45395c79ba58, CmdSN: 0x0, status: 0x80d \nWARNING: NvmeDevice: 573: Get identify namespace data failed for device Status %s. \nWARNING: NvmeUtil: 151: Error on Cmd(0x45d9c7dd5fc0) 0x6, CmdSN 0x0 from world 0 to component \"nqn.1994-11.com.samsung:nvme:990PRO:M.2:<REDACTED>\" H:0xe D:0x0 P:0x0 \nWARNING: NVMEPSA:217 Complete vmkNvmeCmd: 0x45d9a26d4ac0, vmkPsaCmd: 0x45d9c7dd5fc0, cmdId.initiator=0x45395c79ba58, CmdSN: 0x0, status: 0x80d \nWARNING: NVMEPSA:217 Complete vmkNvmeCmd: 0x45d9a27100c0, vmkPsaCmd: 0x45d9c7dd5fc0, cmdId.initiator=0x45395c79ba58, CmdSN: 0x0, status: 0x80d<\/code><\/pre>\n\n\n\n
The first line was particularly interesting:<\/p>\n\n\n\n
Controller 257 failed to recover after 21 attempts<\/code><\/pre>\n\n\n\n
My interpretation of these messages is that ESX detected a problem communicating with the NVMe controller and attempted to recover it. After 21 unsuccessful recovery attempts, ESX stopped trying to recover the controller normally and switched to periodic recovery.<\/p>\n\n\n\n
The subsequent messages indicate that ESXi was unable to retrieve information from the NVMe namespace and was encountering command failures.<\/p>\n\n\n\n
At this point, the underlying NVMe device was effectively unavailable to ESX, which explained why the VMFS datastore disappeared, which in turn made the guest hosted NFS mount disappear.<\/p>\n\n\n\n
Checking the NVMe Device<\/h2>\n\n\n\n
A reboot of the ESX host brought the device back online and the datastore became available again.  However, a few days later the error returned.  This time I saw similar vmkwarning.log<\/code> entries, but I wanted to take a closer look at the NVMe device. I used the following command:<\/p>\n\n\n\n
esxcli storage core device list | grep -i \"samsung\" -B 1 -A 12<\/code><\/pre>\n\n\n\n
The output included the following:<\/p>\n\n\n\n
t10.NVMe____Samsung_SSD_990_PRO_with_Heatsink_4TB___<redacted>\n   Display Name: Local NVMe Disk (t10.NVMe____Samsung_SSD_990_PRO_with_Heatsink_4TB___<redacted>)\n   Has Settable Display Name: true\n   Size: 3815447\n   Device Type: Direct-Access\n   Multipath Plugin: HPP\n   Devfs Path:\n   Vendor: NVMe\n   Model: Samsung SSD 990 PRO with Heatsink 4TB\n   Revision: 4B2QJXD7\n   SCSI Level: 0\n   Is Pseudo: false\n   Status: dead timeout\n   Is RDM Capable: false\n   Is Local: true\n   Is Removable: false\n   Is SSD: true\n   Is VVOL PE: false\n   Is Offline: false\n   Is Perennially Reserved: false\n   Queue Full Sample Size: 0<\/code><\/pre>\n\n\n\n
There were two things that caught my attention here.<\/p>\n\n\n\n
First, the device status was:<\/p>\n\n\n\n
Status: dead timeout<\/code><\/pre>\n\n\n\n
That was consistent with what I had seen in vmkwarning.log<\/code>.<\/p>\n\n\n\n
Second, the firmware revision was:<\/p>\n\n\n\n
Revision: 4B2QJXD7<\/code><\/pre>\n\n\n\n
Checking the Samsung Firmware<\/h2>\n\n\n\n
I then checked Samsung’s support site for firmware updates for the 990 PRO.<\/p>\n\n\n\n
Samsung had newer firmware available for the drive, including firmware versions that addressed issues involving intermittent drive recognition and blue-screen problems.<\/p>\n\n\n\n
That seemed particularly interesting given the behavior I was seeing. The drive wasn’t permanently failing; it would occasionally stop responding to the host and then become available again after a reboot.<\/p>\n\n\n\n
At that point, I decided that updating the firmware was worth trying.<\/p>\n\n\n\n
I updated the drive to the latest firmware available at the time, 8B2QJXD7<\/strong>, using the ISO image provided by Samsung.<\/p>\n\n\n\n
Has the Firmware Update Fixed It?<\/h2>\n\n\n\n
The real test was simply to wait and see what happened.<\/p>\n\n\n\n
I’ve continued using the NVMe drive in the lab for roughly a month since updating the firmware. During that time, I’ve performed a number of tests and continued using the drive in the same general configuration.<\/p>\n\n\n\n
So far, the device has remained stable.<\/p>\n\n\n\n
I haven’t seen the NVMe controller recovery messages return, and I haven’t had another occurrence where the device went into a dead timeout<\/code> state and caused the NFS datastore to become unavailable.<\/p>\n\n\n\n
While that doesn’t prove conclusively that the firmware was the root cause, the combination of the symptoms, the older firmware version, Samsung’s firmware notes, and the fact that the problem has not returned after the update makes the firmware a strong suspect.<\/p>\n\n\n\n
Conclusion<\/h2>\n\n\n\n
This was a good reminder that when an ESX storage device suddenly disappears, it is worth looking beyond the datastore and filesystem layers.<\/p>\n\n\n\n
In my case, the NFS datastore was only the symptom. The underlying problem was an NVMe device that had stopped responding to ESX.<\/p>\n\n\n\n
The vmkwarning.log<\/code> messages showed ESX repeatedly attempting to recover the NVMe controller, while esxcli storage core device list<\/code> subsequently showed the device in a dead timeout<\/code> state.<\/p>\n\n\n\n
Checking the firmware version ultimately revealed that the Samsung 990 PRO was running an older firmware revision. Updating it to 8B2QJXD7 appears to have resolved the intermittent failures in my lab.<\/p>\n\n\n\n
If you are seeing similar NVMe controller recovery messages or dead timeout<\/code> states on ESXi, checking the firmware version of the affected drive would be a worthwhile troubleshooting step.<\/p>\n\n\n\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
I’ve been spending some time working with VMware Cloud Foundation (VCF) 9.1 in my lab environment. The lab consists of several nested ESXi hosts that use NFS storage provided by a virtual machine. The VM providing the NFS storage, in … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":6,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"footnotes":""},"categories":[9,4],"tags":[],"class_list":["post-2429","post","type-post","status-publish","format-standard","hentry","category-lab-infrastructure","category-virtualization"],"_links":{"self":[{"href":"https:\/\/enterpriseadmins.org\/blog\/wp-json\/wp\/v2\/posts\/2429","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/enterpriseadmins.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/enterpriseadmins.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/enterpriseadmins.org\/blog\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/enterpriseadmins.org\/blog\/wp-json\/wp\/v2\/comments?post=2429"}],"version-history":[{"count":1,"href":"https:\/\/enterpriseadmins.org\/blog\/wp-json\/wp\/v2\/posts\/2429\/revisions"}],"predecessor-version":[{"id":2430,"href":"https:\/\/enterpriseadmins.org\/blog\/wp-json\/wp\/v2\/posts\/2429\/revisions\/2430"}],"wp:attachment":[{"href":"https:\/\/enterpriseadmins.org\/blog\/wp-json\/wp\/v2\/media?parent=2429"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/enterpriseadmins.org\/blog\/wp-json\/wp\/v2\/categories?post=2429"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/enterpriseadmins.org\/blog\/wp-json\/wp\/v2\/tags?post=2429"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}