Trace API Authorization Bypass in MLflow

Executive Summary

In MLflow versions prior to 3.14.0, when authentication is enabled, the trace API endpoints do not have proper authorization validators. This means that any authenticated user can bypass experiment-level authorization controls and perform trace operations such as reading, deleting, or modifying traces on experiments they are not authorized to access.

The root cause is that the internal handler responsible for processing requests (_before_request) does not register authorization validators for trace endpoints, allowing requests to proceed without proper permission checks.

This vulnerability exposes sensitive data, allows destruction of audit logs, and permits unauthorized modifications to trace data.

Useful 0 Not Useful 0

Impact Analysis

This vulnerability can have serious impacts including exposure of sensitive data related to experiments, unauthorized deletion or modification of trace information, and destruction of audit logs.

Because any authenticated user can bypass authorization controls on trace operations, attackers or unauthorized users could manipulate or access data they should not have access to, potentially compromising the integrity and confidentiality of your MLflow experiments.

Useful 0 Not Useful 0

Detection Guidance

This vulnerability involves unauthorized access to MLflow trace API endpoints due to missing authorization validators. Detection can focus on monitoring API requests to trace endpoints for unauthorized or unexpected access patterns.

You can detect potential exploitation by inspecting logs or network traffic for calls to trace-related endpoints such as BatchGetTraceInfos, DeleteTraces, GetTrace, SearchTraces, StartTrace, EndTrace, SetTraceTag, DeleteTraceTag, LinkTracesToRun, and others.

Suggested commands include using network monitoring or log analysis tools to filter requests to these endpoints. For example, using grep on server logs:

• grep -E 'BatchGetTraceInfos|DeleteTraces|GetTrace|SearchTraces|StartTrace|EndTrace|SetTraceTag|DeleteTraceTag|LinkTracesToRun' /path/to/mlflow/server/logs
• Using curl or similar tools to test authorization enforcement on trace endpoints by making authenticated requests and verifying if unauthorized access is possible.

Since the vulnerability allows any authenticated user to bypass experiment-level authorization, testing with different user credentials to access trace endpoints can help detect the issue.

Useful 0 Not Useful 0

Mitigation Strategies

The primary mitigation step is to upgrade MLflow to version 3.14.0 or later, where the vulnerability has been fixed by adding proper authorization validators to all trace API endpoints.

Until the upgrade can be performed, restrict access to MLflow trace API endpoints to trusted users only, and monitor usage closely for suspicious activity.

Additionally, review and tighten authentication and authorization configurations to ensure that only authorized users have access to sensitive API operations.

Implement network-level controls such as firewall rules or API gateways to limit access to trace endpoints.

Useful 0 Not Useful 0

Compliance Impact

This vulnerability allows any authenticated user to bypass experiment-level authorization controls on trace operations, including reading, deleting, and modifying traces on experiments they do not have permission to access.

Such unauthorized access and modification can expose sensitive data and destroy audit logs, which are critical for maintaining data integrity and accountability.

Exposure of sensitive data and loss of audit logs can lead to non-compliance with common standards and regulations like GDPR and HIPAA, which require strict access controls and auditability to protect personal and sensitive information.

Useful 0 Not Useful 0