Measuring the Dependency Gap: Diagnosing Inter-Column Fidelity in Tabular Generative Models
Standard metrics for evaluating synthetic tabular data (e.g., logistic-regression C2ST, pairwise Trend score) are largely blind to inter-column dependencies, failing to distinguish between models that preserve dependencies and those that destroy them. The authors introduce a dependency-aware fidelity diagnostic that decomposes a strong classifier two-sample test (XGB-C2ST) into marginal, dependency, and numerical-categorical cross components, anchored between a worst-case fully-factorized refere
Analysis
TL;DR
- Standard metrics for evaluating synthetic tabular data (e.g., logistic-regression C2ST, pairwise Trend score) are largely blind to inter-column dependencies, failing to distinguish between models that preserve dependencies and those that destroy them.
- The authors introduce a dependency-aware fidelity diagnostic that decomposes a strong classifier two-sample test (XGB-C2ST) into marginal, dependency, and numerical-categorical cross components, anchored between a worst-case fully-factorized reference and a best-case real-data oracle.
- Applying this diagnostic to TabbyFlow/EF-VFM reveals a real dependency gap that standard metrics miss; destroying dependency outright collapses minority-class utility, and the generator's residual gap carries a smaller, consistent utility cost.
- The dependency gap is not due to a structural limitation of mean-field generative objectives (which are asymptotically exact) but rather due to the absence of direct dependency supervision; increasing model capacity by 16x does not close the gap.
- No cheap intervention (in-model dependency mechanism, post-hoc copula correction, or increased capacity) closes the residual dependency gap, highlighting the need for more sophisticated approaches to capture inter-column dependencies in tabular generative models.
Why It Matters
This work is highly relevant to AI practitioners and researchers working with synthetic tabular data, as it exposes a critical flaw in current evaluation metrics that can lead to overconfidence in the quality of generated data. By demonstrating that standard metrics fail to capture inter-column dependencies, the authors emphasize the importance of developing more robust evaluation methods that account for these dependencies, especially in applications where minority-class discrimination is crucial, such as fraud detection and clinical risk assessment.
Technical Details
- Dependency-Aware Fidelity Diagnostic: The authors propose a diagnostic that decomposes the XGB-C2ST into marginal, dependency, and numerical-categorical cross components. This diagnostic is anchored between a worst-case fully-factorized reference (where all dependencies are destroyed) and a best-case real-data oracle.
- TabbyFlow/EF-VFM: A state-of-the-art flow-matching generator is evaluated using the proposed diagnostic. The results reveal a significant dependency gap that standard metrics fail to detect.
- Impact of Dependency Destruction: Destroying inter-column dependencies in the synthetic data leads to a collapse in minority-class utility, highlighting the importance of preserving these dependencies for downstream tasks.
- Model Capacity and Dependency Gap: Increasing the model capacity of TabbyFlow/EF-VFM by 16x does not close the dependency gap, suggesting that the issue is not due to model capacity but rather the absence of direct dependency supervision.
- Interventions: Various interventions, including in-model dependency mechanisms, post-hoc copula correction, and increased capacity, are tested but fail to close the residual dependency gap, indicating the need for more advanced approaches.
Industry Insight
- Need for Better Evaluation Metrics: The industry should prioritize the development and adoption of evaluation metrics that are sensitive to inter-column dependencies in synthetic tabular data. This will help ensure that generated data is suitable for critical applications where minority-class discrimination is essential.
- Focus on Direct Dependency Supervision: Generative models for tabular data should incorporate direct supervision mechanisms to capture inter-column dependencies, rather than relying solely on marginal distribution matching or increased model capacity.
- Caution Against Simple Fixes: The findings caution against assuming that simple fixes, such as increasing model capacity or applying post-hoc corrections, will adequately address dependency gaps. More sophisticated approaches are needed to ensure the fidelity of synthetic tabular data.
Disclaimer: The above content is generated by AI and is for reference only.