Algorithmic Complexity Flaw in NLTK TreebankWordTokenizer

Executive Summary

This vulnerability affects NLTK versions 3.2.5 through 3.10.3. It involves a flaw in TreebankWordTokenizer.span_tokenize where processing long sequences of double-quote characters causes quadratic CPU consumption due to repeated list.pop(0) calls. Each call shifts remaining list elements, making total work O(n^2). Attackers can exploit this to saturate a CPU core and degrade service availability.

Useful 0 Not Useful 0

Detection Guidance

To detect this vulnerability, monitor CPU usage spikes when processing text with long sequences of double-quote characters. Check NLTK versions in use (3.2.5 to 3.10.3) and inspect services using TreebankWordTokenizer.span_tokenize on untrusted input. Look for O(n^2) processing time growth with input size.

Useful 0 Not Useful 0

Impact Analysis

If you use NLTK versions 3.2.5 to 3.10.3 for processing untrusted text, an attacker could send crafted input with many double quotes to consume excessive CPU resources. This may cause denial of service, slow response times, or system unavailability for services relying on span tokenization of untrusted data.

Useful 0 Not Useful 0

Mitigation Strategies

Upgrade NLTK to version 3.10.4 or later if available. If upgrading is not possible, implement input length limits for text processing and replace list.pop(0) with an O(1) consumption strategy like iterators or collections.deque in TreebankWordTokenizer.span_tokenize.

Useful 0 Not Useful 0