CVE-2025-68310
Unknown Unknown - Not Provided
BaseFortify

Publication date: 2025-12-16

Last updated on: 2025-12-18

Assigner: kernel.org

Description
In the Linux kernel, the following vulnerability has been resolved: s390/pci: Avoid deadlock between PCI error recovery and mlx5 crdump Do not block PCI config accesses through pci_cfg_access_lock() when executing the s390 variant of PCI error recovery: Acquire just device_lock() instead of pci_dev_lock() as powerpc's EEH and generig PCI AER processing do. During error recovery testing a pair of tasks was reported to be hung: mlx5_core 0000:00:00.1: mlx5_health_try_recover:338:(pid 5553): health recovery flow aborted, PCI reads still not working INFO: task kmcheck:72 blocked for more than 122 seconds. Not tainted 5.14.0-570.12.1.bringup7.el9.s390x #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:kmcheck state:D stack:0 pid:72 tgid:72 ppid:2 flags:0x00000000 Call Trace: [<000000065256f030>] __schedule+0x2a0/0x590 [<000000065256f356>] schedule+0x36/0xe0 [<000000065256f572>] schedule_preempt_disabled+0x22/0x30 [<0000000652570a94>] __mutex_lock.constprop.0+0x484/0x8a8 [<000003ff800673a4>] mlx5_unload_one+0x34/0x58 [mlx5_core] [<000003ff8006745c>] mlx5_pci_err_detected+0x94/0x140 [mlx5_core] [<0000000652556c5a>] zpci_event_attempt_error_recovery+0xf2/0x398 [<0000000651b9184a>] __zpci_event_error+0x23a/0x2c0 INFO: task kworker/u1664:6:1514 blocked for more than 122 seconds. Not tainted 5.14.0-570.12.1.bringup7.el9.s390x #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:kworker/u1664:6 state:D stack:0 pid:1514 tgid:1514 ppid:2 flags:0x00000000 Workqueue: mlx5_health0000:00:00.0 mlx5_fw_fatal_reporter_err_work [mlx5_core] Call Trace: [<000000065256f030>] __schedule+0x2a0/0x590 [<000000065256f356>] schedule+0x36/0xe0 [<0000000652172e28>] pci_wait_cfg+0x80/0xe8 [<0000000652172f94>] pci_cfg_access_lock+0x74/0x88 [<000003ff800916b6>] mlx5_vsc_gw_lock+0x36/0x178 [mlx5_core] [<000003ff80098824>] mlx5_crdump_collect+0x34/0x1c8 [mlx5_core] [<000003ff80074b62>] mlx5_fw_fatal_reporter_dump+0x6a/0xe8 [mlx5_core] [<0000000652512242>] devlink_health_do_dump.part.0+0x82/0x168 [<0000000652513212>] devlink_health_report+0x19a/0x230 [<000003ff80075a12>] mlx5_fw_fatal_reporter_err_work+0xba/0x1b0 [mlx5_core] No kernel log of the exact same error with an upstream kernel is available - but the very same deadlock situation can be constructed there, too: - task: kmcheck mlx5_unload_one() tries to acquire devlink lock while the PCI error recovery code has set pdev->block_cfg_access by way of pci_cfg_access_lock() - task: kworker mlx5_crdump_collect() tries to set block_cfg_access through pci_cfg_access_lock() while devlink_health_report() had acquired the devlink lock. A similar deadlock situation can be reproduced by requesting a crdump with > devlink health dump show pci/<BDF> reporter fw_fatal while PCI error recovery is executed on the same <BDF> physical function by mlx5_core's pci_error_handlers. On s390 this can be injected with > zpcictl --reset-fw <BDF> Tests with this patch failed to reproduce that second deadlock situation, the devlink command is rejected with "kernel answers: Permission denied" - and we get a kernel log message of: mlx5_core 1ed0:00:00.1: mlx5_crdump_collect:50:(pid 254382): crdump: failed to lock vsc gw err -5 because the config read of VSC_SEMAPHORE is rejected by the underlying hardware. Two prior attempts to address this issue have been discussed and ultimately rejected [see link], with the primary argument that s390's implementation of PCI error recovery is imposing restrictions that neither powerpc's EEH nor PCI AER handling need. Tests show that PCI error recovery on s390 is running to completion even without blocking access to PCI config space.
CVSS Scores
EPSS Scores
Probability:
Percentile:
Meta Information
Published
2025-12-16
Last Modified
2025-12-18
Generated
2026-05-07
AI Q&A
2025-12-16
EPSS Evaluated
2026-05-05
NVD
CVE-2025-68310
EUVD
EUVD-2025-203759
Affected Vendors & Products
Showing 1 associated CPE
Vendor Product Version / Range
mlx5 mlx5_core *
Helpful Resources
Exploitability
CWE
CWE Icon
KEV
KEV Icon
CWE ID Description
CWE-UNKNOWN
Attack-Flow Graph
AI Powered Q&A
Can you explain this vulnerability to me?

This vulnerability involves a deadlock situation in the Linux kernel on the s390 architecture between PCI error recovery and the mlx5 core driver's crash dump (crdump) process. Specifically, during PCI error recovery, the code blocks PCI configuration accesses using pci_cfg_access_lock(), which can cause tasks related to mlx5 health recovery and error reporting to hang indefinitely. The deadlock occurs because mlx5_unload_one() tries to acquire a devlink lock while PCI error recovery has blocked config access, and simultaneously mlx5_crdump_collect() tries to set block_cfg_access while devlink_health_report() holds the devlink lock. This results in tasks being blocked for extended periods, causing system instability. The fix avoids blocking PCI config accesses by acquiring a less restrictive lock (device_lock() instead of pci_dev_lock()) during error recovery on s390, preventing the deadlock.


How can this vulnerability impact me? :

This vulnerability can cause system tasks related to PCI error recovery and mlx5 device health reporting to become hung or blocked for extended periods, leading to potential system instability or degraded performance. Specifically, it can cause deadlocks that prevent proper error recovery and health reporting for mlx5 devices, which may impact the reliability and availability of systems using these devices on the s390 architecture.


How can this vulnerability be detected on my network or system? Can you suggest some commands?

This vulnerability can be detected by observing hung tasks related to mlx5_core and PCI error recovery in the kernel logs. Specifically, look for messages indicating tasks blocked for more than 122 seconds, such as 'task kmcheck blocked' or 'task kworker blocked', and call traces involving mlx5_health_try_recover, mlx5_unload_one, mlx5_pci_err_detected, and pci_cfg_access_lock. Additionally, you can attempt to reproduce the deadlock by requesting a crdump with the command: devlink health dump show pci/<BDF> reporter fw_fatal, and on s390 systems, inject PCI error recovery with: zpcictl --reset-fw <BDF>.


What immediate steps should I take to mitigate this vulnerability?

Immediate mitigation involves applying the patch that avoids blocking PCI config accesses through pci_cfg_access_lock() during s390 PCI error recovery by acquiring device_lock() instead of pci_dev_lock(). This prevents the deadlock between PCI error recovery and mlx5 crdump. Until patched, avoid triggering devlink health dump commands on affected devices during PCI error recovery, and monitor for hung tasks in kernel logs to detect deadlock conditions.


Ask Our AI Assistant
Need more information? Ask your question to get an AI reply (Powered by our expertise)
0/70
EPSS Chart