Structure of the Circular-Dyadic Convolution Error
Substituting the Discrete Fourier Transform (DFT) with the Hadamard Transform for convolution introduces structured algebraic error despite both operating in O(N log N) time. The error operator is nearly full rank with a logarithmic-dimensional null space, meaning most filter configurations incur significant distortion. Two specific input and output positions are universally error-free, but this cancellation cannot be achieved through simple output reordering. The expected substitution error is
Analysis
TL;DR
- Substituting the Discrete Fourier Transform (DFT) with the Hadamard Transform for convolution introduces structured algebraic error despite both operating in O(N log N) time.
- The error operator is nearly full rank with a logarithmic-dimensional null space, meaning most filter configurations incur significant distortion.
- Two specific input and output positions are universally error-free, but this cancellation cannot be achieved through simple output reordering.
- The expected substitution error is governed by a single alignment scalar, asymptotically doubling output energy for general filters unless they reside in a zero-error subspace.
Why It Matters
This research provides critical theoretical bounds for practitioners considering faster, real-valued alternatives to FFT-based convolutions in signal processing and neural network layers. Understanding the precise structure and magnitude of this error helps engineers decide when the computational savings of Hadamard transforms justify the potential loss in numerical accuracy.
Technical Details
- Computational Context: Both Dyadic convolution (using Hadamard transform) and circular convolution (using FFT/DFT) achieve O(N log N) complexity, but the Hadamard transform uses real-valued sign flips instead of complex arithmetic.
- Error Characterization: The study identifies that replacing DFT with Hadamard transforms creates an algebraic error that is structured rather than random noise.
- Null Space Properties: The error operator has a null space of only logarithmic dimension relative to N, indicating that very few filters avoid this error entirely.
- Alignment Scalar: A closed-form expression for the expected error is derived based on an alignment scalar, averaged over random filters, showing that error magnitude is predictable.
Industry Insight
- Optimization Trade-offs: Developers optimizing for speed in resource-constrained environments should quantify the "alignment scalar" for their specific filters before swapping FFTs for Hadamard transforms.
- Filter Design: To minimize error without changing the algorithm, filter weights can be projected into the identified universal zero-error subspace, though this subspace is extremely small.
- Hybrid Approaches: Given that two positions are universally error-free, hybrid methods might leverage these stable points for control signals or metadata while using standard FFT for the bulk of data processing.
Disclaimer: The above content is generated by AI and is for reference only.