Every Layer Counts: An Exponential $L_2$ Depth Hierarchy for ReLU Networks
Proves an exponential depth hierarchy for ReLU networks where each additional layer can exponentially reduce the number of required neurons For every depth ℓ ≥ 3, a 1-Lipschitz function can be realized by a depth-ℓ network of width O(d⁴), while any depth-(ℓ-1) network with width at most 2^d/[2d(ℓ-2)] incurs squared L₂ error of at least 1/24 This is the first exponential separation between two fixed ReLU depths where the shallower depth is at least 3, and the first across all adjacent fixed depth
Analysis
TL;DR
- Proves an exponential depth hierarchy for ReLU networks where each additional layer can exponentially reduce the number of required neurons
- For every depth ℓ ≥ 3, a 1-Lipschitz function can be realized by a depth-ℓ network of width O(d⁴), while any depth-(ℓ-1) network with width at most 2^d/[2d(ℓ-2)] incurs squared L₂ error of at least 1/24
- This is the first exponential separation between two fixed ReLU depths where the shallower depth is at least 3, and the first across all adjacent fixed depths
- Resolves an open question from Safran, Eldan, and Shamir (2019) by establishing a compactly supported depth-3 versus depth-2 separation with unrestricted shallow-network weights
- Also proves an exact separation for a more benign function: polynomial-width depth-4 suffices while depth-3 requires exponentially many neurons in its first hidden layer
Why It Matters
This result provides rigorous theoretical justification for the empirical observation that deeper networks are more parameter-efficient, establishing that depth genuinely confers exponential representational advantages in ReLU networks. For researchers studying expressivity and efficiency trade-offs, it closes a significant open question in the theoretical foundations of deep learning and sets a benchmark for future depth hierarchy analyses.
Technical Details
- The main theorem shows that for any ℓ ≥ 3, there exists a globally [0,1]-valued, 1-Lipschitz function computable by a depth-ℓ ReLU network with width O(d⁴), while any depth-(ℓ-1) network with width bounded by 2^d/[2d(ℓ-2)] has squared L₂ error ≥ 1/24 under an absolutely continuous distribution
- The depth-3 versus depth-2 case (ℓ=3) yields a compactly supported separation, answering the 2019 open question, though the distribution has all its mass at exponential radius, placing it outside the regularity regime that would connect to major threshold-circuit lower bounds
- A secondary exact separation result demonstrates a function computable by polynomial-width depth-4 networks where any depth-3 network agreeing on the unit hypercube requires exponentially many neurons in its first hidden layer; this function is O(√d)-Lipschitz and maps the unit hypercube onto [0,1]
- The lower bound technique leverages the geometric complexity of ReLU piecewise-linear regions, showing that shallower networks cannot partition the input space finely enough to approximate the target function within bounded error
Industry Insight
- Theoretical results of this nature reinforce the importance of depth in architecture design, suggesting that investing in deeper (rather than merely wider) networks is not just empirically effective but theoretically optimal for representational efficiency
- Practitioners should be aware that the exponential advantage of depth comes with distributions that may concentrate at large radii, implying that real-world data distributions (which are typically well-behaved) may exhibit weaker but still meaningful depth benefits
- This work opens the door to tighter bounds on the relationship between depth, width, and approximation error, which could inform future architectural search strategies and resource allocation in model design
Disclaimer: The above content is generated by AI and is for reference only.