Where Sandbox Ingress Speed Actually Comes From
Tensorlake rebuilt its sandbox ingress path by replacing an L7 reverse proxy hop with an L4 forwarder using kernel TLS (kTLS) and splice(2), achieving significant CPU and throughput gains Most performance improvement came from eliminating the application-layer request parsing overhead, not from kTLS itself as initially expected kTLS enabled a zero-copy path by moving TLS record processing into the kernel, adding a smaller but meaningful throughput gain on top of the L4 transition The edge gatewa
Analysis
TL;DR
- Tensorlake rebuilt its sandbox ingress path by replacing an L7 reverse proxy hop with an L4 forwarder using kernel TLS (kTLS) and splice(2), achieving significant CPU and throughput gains
- Most performance improvement came from eliminating the application-layer request parsing overhead, not from kTLS itself as initially expected
- kTLS enabled a zero-copy path by moving TLS record processing into the kernel, adding a smaller but meaningful throughput gain on top of the L4 transition
- The edge gateway retained its L7 role because it performs protocol-aware routing, authentication, and request transformation that require application-layer visibility
- L4 forwarding became viable for the dataplane hop because routing decisions were already made upstream, and missing L7 capabilities (routing, liveness) were rebuilt using connection-level mechanisms
Why It Matters
This case study provides a concrete, measured example of the L7-vs-L4 tradeoff that every AI infrastructure team faces when designing proxy chains for sandboxed or multi-tenant workloads. It challenges the common assumption that kernel TLS is the primary performance driver, showing instead that removing unnecessary parsing is often the bigger win—critical insight for practitioners optimizing ingress paths.
Technical Details
- Architecture shift: The old path used two L7 hops (edge gateway with TLS + auth, followed by an L7 reverse proxy with mTLS and HTTP/2). The new path replaces the second hop with an L4 forwarder using kTLS and splice(2), forwarding plaintext to the sandbox app
- Routing preamble: Since L4 cannot parse application protocols, routing information now arrives as a short, length-bounded preamble before tenant data, containing the sandbox ID and private target address (ip:port), learned from the scheduler
- Liveness reconstruction: The idle timer mechanism was rebuilt by metering bytes per connection and reporting counts to the dataplane, which resets the sandbox idle timer as long as traffic flows
- kTLS implementation: Uses Linux 6.x with CONFIG_TLS, software-based kTLS (CPU cryptography, no NIC offload), AES-256-GCM at ~7.9 GB/s per core, fail-closed design that refuses to start if kernel TLS ULP attachment fails
- Measured performance: Plain L4 userspace-copy stage achieved 2.07 GB/s and 0.50 CPU-seconds per GB; kTLS added further gain with crypto accounting for ~26% of forwarder CPU, remainder from TCP handling and syscalls
Industry Insight
- When evaluating performance optimizations in proxy chains, isolate each change incrementally rather than comparing only old vs. new systems; attributing gains to the wrong component (e.g., kTLS vs. removed parsing) leads to incorrect architectural decisions
- L4 forwarding is a strong candidate for hops where upstream layers have already made protocol-aware decisions, but teams must budget engineering effort to rebuild routing, liveness, and error-handling capabilities at the connection level
- Portability should constrain low-level networking choices; Tensorlake rejected VXLAN, eBPF, and CNI plugins in favor of application-layer L4 forwarding to maintain cross-cloud compatibility across AWS, GCP, and GPU neoclouds
Disclaimer: The above content is generated by AI and is for reference only.