PVE 搭配西数 SN580 2TB， FLUSH 死锁排查

# PVE 搭配西数 SN580 2TB， FLUSH 死锁排查

我跑生产用的一台 UM580D（R7 5800H AMD Cezanne 平台mini主机）系统盘用的西数 SN580 2TB 固态，跑了几年了一直没出过问题，就在刚刚，直接卡死了，虚拟机全部断连（在这块固态上的，包括我的blog）然后从日志服务器以及`journalctl`中发现，pve 的`pveproxy`管理接口连接超时，从日志服务器看到，最先超时阻塞的是**FLUSH 刷盘指令**，内核尝试中止卡死 IO 请求，但 ssd的主控没在响应后续命令，nvme控制器无法执行重置，IO 队列冻结，最终导致整个系统阻塞。

![](https://img.wanghaoyu.com.cn/Picture/img/20260722231029651.png)

我重启了下机器，但是重启后硬盘 SMART 健康检测全部正常？？？没坏块？？？没io错误，而且我用fio测了下io是没问题的

```bash
fio --name=bandwidth-test --filename=/tmp/fio-test.dat --rw=randread --ioengine=libaio --direct=1 --bs=128k --numjobs=4 --iodepth=32 --size=1G --runtime=30
bandwidth-test: (g=0): rw=randread, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=32
...
fio-3.39
Starting 4 processes

bandwidth-test: (groupid=0, jobs=1): err= 0: pid=94569: Wed Jul 22 22:53:18 2026
  read: IOPS=46.3k, BW=5785MiB/s (6066MB/s)(1024MiB/177msec)
    slat (nsec): min=10510, max=97348, avg=20412.35, stdev=5365.29
    clat (nsec): min=1462, max=2188.5k, avg=664500.76, stdev=112376.69
     lat (usec): min=12, max=2278, avg=684.91, stdev=116.23
    clat percentiles (usec):
     |  1.00th=[  396],  5.00th=[  441], 10.00th=[  603], 20.00th=[  619],
     | 30.00th=[  627], 40.00th=[  644], 50.00th=[  660], 60.00th=[  676],
     | 70.00th=[  693], 80.00th=[  725], 90.00th=[  758], 95.00th=[  799],
     | 99.00th=[  938], 99.50th=[ 1045], 99.90th=[ 1893], 99.95th=[ 2040],
     | 99.99th=[ 2180]
  lat (usec)   : 2=0.01%, 20=0.01%, 50=0.04%, 100=0.05%, 250=0.15%
  lat (usec)   : 500=5.74%, 750=81.53%, 1000=11.68%
  lat (msec)   : 2=0.72%, 4=0.07%
  cpu          : usr=1.14%, sys=98.30%, ctx=1, majf=0, minf=1034
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=99.6%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=8192,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
bandwidth-test: (groupid=0, jobs=1): err= 0: pid=94570: Wed Jul 22 22:53:18 2026
  read: IOPS=46.3k, BW=5785MiB/s (6066MB/s)(1024MiB/177msec)
    slat (nsec): min=10601, max=93892, avg=20401.49, stdev=5324.06
    clat (nsec): min=1453, max=2204.9k, avg=664785.63, stdev=113507.63
     lat (usec): min=13, max=2294, avg=685.19, stdev=117.45
    clat percentiles (usec):
     |  1.00th=[  383],  5.00th=[  441], 10.00th=[  611], 20.00th=[  627],
     | 30.00th=[  644], 40.00th=[  652], 50.00th=[  660], 60.00th=[  676],
     | 70.00th=[  693], 80.00th=[  709], 90.00th=[  742], 95.00th=[  783],
     | 99.00th=[  898], 99.50th=[ 1106], 99.90th=[ 2024], 99.95th=[ 2057],
     | 99.99th=[ 2212]
  lat (usec)   : 2=0.01%, 20=0.01%, 50=0.02%, 100=0.06%, 250=0.15%
  lat (usec)   : 500=5.73%, 750=84.64%, 1000=8.74%
  lat (msec)   : 2=0.51%, 4=0.12%
  cpu          : usr=2.84%, sys=96.59%, ctx=0, majf=0, minf=1035
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=99.6%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=8192,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
bandwidth-test: (groupid=0, jobs=1): err= 0: pid=94571: Wed Jul 22 22:53:18 2026
  read: IOPS=47.9k, BW=5988MiB/s (6279MB/s)(1024MiB/171msec)
    slat (nsec): min=13316, max=93821, avg=19627.63, stdev=4743.79
    clat (usec): min=2, max=2223, avg=639.80, stdev=96.79
     lat (usec): min=17, max=2308, avg=659.43, stdev=100.29
    clat percentiles (usec):
     |  1.00th=[  553],  5.00th=[  570], 10.00th=[  578], 20.00th=[  594],
     | 30.00th=[  603], 40.00th=[  611], 50.00th=[  619], 60.00th=[  627],
     | 70.00th=[  652], 80.00th=[  676], 90.00th=[  725], 95.00th=[  766],
     | 99.00th=[  881], 99.50th=[ 1074], 99.90th=[ 1975], 99.95th=[ 2089],
     | 99.99th=[ 2212]
  lat (usec)   : 4=0.01%, 20=0.01%, 50=0.02%, 100=0.04%, 250=0.10%
  lat (usec)   : 500=0.16%, 750=93.46%, 1000=5.57%
  lat (msec)   : 2=0.54%, 4=0.10%
  cpu          : usr=3.53%, sys=95.29%, ctx=5, majf=0, minf=1033
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=99.6%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=8192,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
bandwidth-test: (groupid=0, jobs=1): err= 0: pid=94572: Wed Jul 22 22:53:18 2026
  read: IOPS=48.2k, BW=6024MiB/s (6316MB/s)(1024MiB/170msec)
    slat (nsec): min=13556, max=88551, avg=19605.10, stdev=4725.53
    clat (usec): min=2, max=2191, avg=638.42, stdev=95.92
     lat (usec): min=20, max=2278, avg=658.03, stdev=99.41
    clat percentiles (usec):
     |  1.00th=[  553],  5.00th=[  570], 10.00th=[  578], 20.00th=[  586],
     | 30.00th=[  603], 40.00th=[  611], 50.00th=[  619], 60.00th=[  635],
     | 70.00th=[  652], 80.00th=[  676], 90.00th=[  709], 95.00th=[  750],
     | 99.00th=[  914], 99.50th=[ 1172], 99.90th=[ 1909], 99.95th=[ 2040],
     | 99.99th=[ 2180]
  lat (usec)   : 4=0.01%, 50=0.02%, 100=0.04%, 250=0.09%, 500=0.16%
  lat (usec)   : 750=94.59%, 1000=4.15%
  lat (msec)   : 2=0.87%, 4=0.07%
  cpu          : usr=1.18%, sys=97.63%, ctx=2, majf=0, minf=1032
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=99.6%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=8192,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=22.6GiB/s (24.3GB/s), 5785MiB/s-6024MiB/s (6066MB/s-6316MB/s), io=4096MiB (4295MB), run=170-177msec
root@pve:~#
```

这就有点奇怪了，，io没问题，而且我还有一块ssd，在另一块ssd上的虚拟机是正常运行的。

然后我在网上找了一些相关文章，然后又一篇文章说说wd官方最初针对 Windows11 24H2 推送固件更新`281050WD`，公开说明旧固件 HMB 内存分配逻辑异常，会触发系统蓝屏，虽然官方没有单独标注 Linux环境的 FLUSH 死锁问题，但底层大概率都是 HMB 资源竞争缺陷，Windows 环境的蓝屏、Linux 环境的 NVMe IO 死锁，本质是同一个固件底层漏洞在不同操作系统下的不同表现，我的处理器是 R7 5800H（Cezanne 架构），pcIe4 链路的 ASPM 链路休眠，唤醒时序，跟我的 SN580 旧固件适配性很差，pciIe 链路如果切入低功耗，ssd 切入 apst深度休眠，两类休眠切换时序叠加，主机如果这个时候下发 flush 指令，容易触发主控资源抢占冲突？如果将 pcie 从 4.0 强制降级到 3.0、bios 关闭 aspm是不是能够降低故障概率？，ssd自身的 apst 休眠是不是还会触发？

[scode type="yellow"]
PS：这个问题目前没解决，但是我在bios侧以及kernel侧关闭了PCIE的链路休眠，以及配置了`nvme_core.default_ps_max_latency_us=0`禁止 SSD 进入低功耗状态，感觉不是闪存颗粒的问题，先这样跑着试试吧，反正我有备份，，不慌，后面再更新，下次再出现这种问题更新固件试下[/scode]

# PVE 搭配西数 SN580 2TB， FLUSH 死锁排查

![](https://img.wanghaoyu.com.cn/Picture/img/20260722231029651.png)

我重启了下机器，但是重启后硬盘 SMART 健康检测全部正常？？？没坏块？？？没io错误，而且我用fio测了下io是没问题的

Run status group 0 (all jobs):
   READ: bw=22.6GiB/s (24.3GB/s), 5785MiB/s-6024MiB/s (6066MB/s-6316MB/s), io=4096MiB (4295MB), run=170-177msec
root@pve:~#
```

这就有点奇怪了，，io没问题，而且我还有一块ssd，在另一块ssd上的虚拟机是正常运行的。

PS：这个问题目前没解决，但是我在bios侧以及kernel侧关闭了PCIE的链路休眠，以及配置了`nvme_core.default_ps_max_latency_us=0`禁止 SSD 进入低功耗状态，感觉不是闪存颗粒的问题，先这样跑着试试吧，反正我有备份，，不慌，后面再更新，下次再出现这种问题更新固件试下
</div>

Last modification：July 22, 2026