Speculative Decoding Engine Crash in vLLM

Executive Summary

This vulnerability exists in vLLM, a high-throughput and memory-efficient inference and serving engine for large language models (LLMs). Prior to version 0.24.0, a specific multi-request speculative decoding workload can cause the engine's rejection sampler to produce an invalid token value equal to the model vocabulary size boundary. This invalid token is then converted to negative one when selecting the next token, which is written back into the drafter's input IDs. When this out-of-vocabulary token is later processed by the model's embedding and attention mechanisms, it triggers a GPU device-side assertion failure that crashes the engine worker.

Because the triggering request sequence can be sent through the public gRPC Generate and Abort endpoints, a remote client able to send generation requests can cause the shared engine worker to crash. This aborts concurrent requests and results in a service-wide denial of service until the worker is restarted.

This issue was fixed in version 0.24.0 of vLLM.

Useful 0 Not Useful 0

Detection Guidance

This vulnerability manifests as a remote denial-of-service (DoS) condition where the vLLM engine worker crashes due to a GPU device-side assertion failure triggered by invalid token values during speculative decoding.

Detection involves monitoring the vLLM service for unexpected crashes or worker restarts, especially following gRPC Generate and Abort requests.

Since the vulnerability is triggered by specific overlapping gRPC generation and abort requests, network traffic inspection for unusual or repeated gRPC calls to these endpoints may help identify exploitation attempts.

There are no explicit commands provided in the resources for direct detection of this vulnerability.

Suggested general approaches include:

• Monitor vLLM logs for GPU assertion failures or worker crashes.
• Use network monitoring tools (e.g., tcpdump, Wireshark) to capture and analyze gRPC traffic targeting the Generate and Abort endpoints for suspicious patterns.
• Check systemd or service manager logs for frequent restarts of the vLLM worker process.

Useful 0 Not Useful 0

Impact Analysis

The vulnerability can be exploited remotely by a client sending specially crafted generation requests to the vLLM engine. This causes the engine worker to crash due to a GPU assertion failure.

As a result, all concurrent requests handled by that worker are aborted, leading to a denial of service (DoS) condition for all users relying on that shared engine worker.

The service remains unavailable until the worker is manually or automatically restarted, potentially causing downtime and disruption of service.

Useful 0 Not Useful 0

Compliance Impact

The vulnerability described is a remote denial-of-service (DoS) issue affecting availability of the vLLM engine worker. It does not involve unauthorized access, data leakage, or modification of sensitive information.

Since the impact is limited to availability and does not affect confidentiality or integrity of data, it does not directly affect compliance with data protection regulations such as GDPR or HIPAA, which primarily focus on protecting personal data privacy and security.

However, service availability is a component of operational security and resilience, which can be relevant under some regulatory frameworks. Organizations using affected versions should consider the risk of service disruption and apply the patch to maintain reliable service.

Useful 0 Not Useful 0

Mitigation Strategies

To mitigate this vulnerability, you should upgrade vLLM to version 0.24.0 or later, where the issue is fixed.

Until the upgrade is applied, be aware that remote clients able to send generation requests via the public gRPC Generate and Abort endpoints can crash the engine worker, causing a denial of service.

Useful 0 Not Useful 0