The Leaderboard Model You Cannot Actually Deploy
Tabular foundation models (TabFM, TabICL, TabDPT) lead on small datasets under 1,000 rows but the accuracy advantage over tree-based methods is narrow (~2 AUC points) and often not statistically significant When tree-based algorithms (LightGBM, XGBoost, CatBoost) are given full training data, they match or exceed foundation model performance across four real-world datasets, with all differences falling below significance thresholds Foundation models carry massive serving costs: prediction times
Analysis
TL;DR
- Tabular foundation models (TabFM, TabICL, TabDPT) lead on small datasets under 1,000 rows but the accuracy advantage over tree-based methods is narrow (~2 AUC points) and often not statistically significant
- When tree-based algorithms (LightGBM, XGBoost, CatBoost) are given full training data, they match or exceed foundation model performance across four real-world datasets, with all differences falling below significance thresholds
- Foundation models carry massive serving costs: prediction times range from over an hour vs. ~0.01 seconds for trees, with memory usage 10-15GB vs. under 1GB, and the top-performing model (TabFM) carries a non-commercial license
- Development time is the genuine advantage of foundation models—no hyperparameter tuning, feature engineering, or retraining pipelines—making them attractive for solo practitioners with tight deadlines
- TabDPT emerges as the most practical foundation model for production: commercial license, low memory footprint (~1GB), fast inference via k-NN lookup, and competitive accuracy
Why It Matters
This analysis directly challenges the marketing narrative around tabular foundation models by introducing prediction cost and full-data performance into the evaluation, two dimensions consistently absent from leaderboards. For AI practitioners, it provides a data-driven decision framework: foundation models are viable only in the narrow regime of very small datasets with no existing ML pipeline, while traditional tree-based methods dominate once sufficient training data is available.
Technical Details
- Datasets: Four real-world OpenML datasets from TabArena—German credit risk, FICO home equity lines, Portuguese bank telemarketing, and Orange telecom customer churn—covering credit, lending, marketing, and churn prediction domains
- Models evaluated: Three foundation models (TabFM, TabICL, TabDPT) and four classical approaches (XGBoost, LightGBM, CatBoost, logistic regression), all tested on identical train/test splits with paired bootstrap (5,000 resamples) and Holm correction for statistical rigor
- Cost methodology: Prediction latency (seconds per 1,000 rows), peak memory usage, and serving cost logged as first-class metrics on identical hardware, revealing a log-scale cost gap between foundation models and trees
- Key finding on full data: When LightGBM was trained on all available training records (removing the 700-1,000 row cap imposed by foundation model cost constraints), it matched TabFM to the fourth decimal on bank marketing and exceeded it on telecom, with paired bootstrap confirming zero statistically significant differences across all four datasets
- Architectural difference: TabFM processes the entire training table on every prediction (costly but accurate), while TabDPT uses k-NN lookup of similar rows (fast, lightweight, commercially licensed), explaining the dramatic serving time disparity (7+ hours vs. ~0.67 seconds for 3,000 telecom customers)
Industry Insight
- The "tabular foundation model" label masks significant heterogeneity: the three evaluated models differ more from each other (0.13 AUC spread on telecom) than from the best tree-based methods, suggesting the category is not yet coherent and individual model evaluation remains essential
- Leaderboard accuracy figures are misleading when serving cost, licensing restrictions, and full-data performance are ignored; practitioners should demand cost-adjusted evaluations before adopting foundation models for production tabular workloads
- The practical recommendation creates a clear decision boundary: under 1,000 rows with no existing pipeline, use TabDPT or TabICL for rapid deployment; at 10,000+ rows, train a tree-based model first as it will match or beat foundation models at negligible cost and with full commercial usability
Disclaimer: The above content is generated by AI and is for reference only.