OpenSSL Silently Fixes ‘HollowByte’ DoS Vulnerability
Okta’s red team discovered "HollowByte," a denial-of-service vulnerability in OpenSSL allowing memory exhaustion via an 11-byte malicious payload. The exploit leverages pre-allocation of receive buffers based on untrusted handshake headers, causing worker threads to block indefinitely while glibc retains memory. Attackers can fragment and freeze significant portions of server RAM (e.g., 25% on a 16GB system) without exceeding connection limits, bypassing standard defenses. Patches were silently
Analysis
TL;DR
- Okta’s red team discovered "HollowByte," a denial-of-service vulnerability in OpenSSL allowing memory exhaustion via an 11-byte malicious payload.
- The exploit leverages pre-allocation of receive buffers based on untrusted handshake headers, causing worker threads to block indefinitely while glibc retains memory.
- Attackers can fragment and freeze significant portions of server RAM (e.g., 25% on a 16GB system) without exceeding connection limits, bypassing standard defenses.
- Patches were silently included in OpenSSL 4.0.1 and backported to versions 3.6.3, 3.5.7, 3.4.6, and 3.0.21, changing buffer growth logic to trust actual data arrival over headers.
Why It Matters
This vulnerability highlights critical risks in widely used cryptographic libraries like OpenSSL, affecting a vast ecosystem of servers, databases, and runtimes including Apache, NGINX, and Node.js. It demonstrates how subtle interactions between application-layer protocols and system-level memory allocators (glibc) can be exploited for efficient denial-of-service attacks that evade traditional connection-rate limiting. Practitioners must verify their OpenSSL versions and apply patches immediately to prevent service degradation or outages.
Technical Details
- Mechanism: Older OpenSSL versions pre-allocated receive buffer sizes based on the 4-byte header in the handshake message, allocating up to 131 KB before any data arrived.
- Exploit Chain: An attacker sends an 11-byte payload declaring a larger body, triggering unvalidated pre-allocation. The worker thread blocks waiting for non-existent data, while glibc retains the allocated memory chunks upon connection drop.
- Memory Exhaustion: By sending waves of connections with randomized claimed sizes, attackers prevent memory reuse, leading to permanent fragmentation and freezing of server memory even after disconnection.
- Impact Scope: Tested on 1GB and 16GB systems, showing the ability to lock up substantial memory (547MB on 1GB, 25% on 16GB) while staying under connection ceilings.
- Remediation: OpenSSL now dynamically increases buffer sizes as bytes actually land, removing trust in the handshake header for buffer growth. Patches are available in 3.0.21, 3.4.6, 3.5.7, 3.6.3, and 4.0.1.
Industry Insight
- Patch Verification: Since patches were silently backported, organizations must actively audit their OpenSSL versions across all infrastructure components, as many may still be running vulnerable legacy versions unaware of the fix.
- Defense-in-Depth Limitations: This attack vector proves that standard connection-limiting strategies are insufficient against sophisticated memory-exhaustion exploits; network-level monitoring and application-layer validation are crucial.
- Supply Chain Vigilance: The silent patching strategy underscores the need for proactive dependency management and regular security audits of third-party libraries to ensure alignment with latest security updates.
Disclaimer: The above content is generated by AI and is for reference only.