The Boolean Power of ReLU
ReLU-MPLang expresses strictly more Boolean queries than any Σ-MPLang using eventually constant activation functions on finite simple undirected graphs with Boolean node features This settles an open problem confirming that ReLU-MPLang is more powerful than trReLU-MPLang for Boolean queries ReLU-GNNs are strictly more expressive than {TrReLU, id}-GNNs on Boolean-featured graphs The result holds with arbitrary real coefficients, strengthening the separation The finding bridges theoretical logic i
Analysis
TL;DR
- ReLU-MPLang expresses strictly more Boolean queries than any Σ-MPLang using eventually constant activation functions on finite simple undirected graphs with Boolean node features
- This settles an open problem confirming that ReLU-MPLang is more powerful than trReLU-MPLang for Boolean queries
- ReLU-GNNs are strictly more expressive than {TrReLU, id}-GNNs on Boolean-featured graphs
- The result holds with arbitrary real coefficients, strengthening the separation
- The finding bridges theoretical logic in computer science and practical machine learning regarding activation function expressiveness
Why It Matters
This result has direct implications for practitioners choosing activation functions in graph neural networks, as it formally establishes that ReLU provides strictly greater expressive power for Boolean query tasks compared to truncated ReLU or identity activations. For researchers, it resolves a long-standing open question about the hierarchy of neural network expressiveness, providing a rigorous theoretical foundation for why ReLU remains the dominant activation function in deep learning practice.
Technical Details
- The paper works within the framework of MPLang (Message Passing Language), a logical formalism for characterizing the expressive power of graph neural networks
- The authors consider finite simple undirected graphs equipped with a single Boolean node feature as the domain of analysis
- Σ-MPLang refers to message passing languages using any collection Σ of eventually constant activation functions with arbitrary real coefficients
- TrReLU (truncated ReLU) and identity are specific examples of eventually constant activation functions that fall within the weaker class
- The proof establishes a strict hierarchy: Boolean queries expressible in ReLU-MPLang properly contain those expressible in any Σ-MPLang with eventually constant activations
Industry Insight
- Practitioners should recognize that ReLU's dominance is not merely empirical but theoretically grounded—its non-eventually-constant nature provides genuine expressive advantages for graph-based Boolean reasoning tasks
- When designing GNN architectures for classification or query-based tasks on graph-structured data, switching from ReLU to truncated variants may incur a measurable expressiveness loss, particularly for discrete/Boolean features
- Future theoretical work should explore whether similar separations exist for other non-eventually-constant activations (e.g., Swish, GELU) to guide activation function selection beyond ReLU in production systems
Disclaimer: The above content is generated by AI and is for reference only.