Class-Structure Preservation Beats Diversity: A Comprehensive Benchmark of Text Augmentation Methods for Imbalanced Text Classification
LLM-based text augmentation methods are statistically equivalent or inferior to embedding-space SMOTE-style retrieval (EmbSMOTE) across seven public datasets The performance gap between LLM-based methods and EmbSMOTE widens monotonically as class imbalance increases, reaching ~0.063 macro F1 delta on GoEmotions-28 Surface-level diversity/uniqueness of augmented text has negligible correlation with downstream classification performance LLM-specific artifacts—text elongation and label-distribution
Analysis
TL;DR
- LLM-based text augmentation methods are statistically equivalent or inferior to embedding-space SMOTE-style retrieval (EmbSMOTE) across seven public datasets
- The performance gap between LLM-based methods and EmbSMOTE widens monotonically as class imbalance increases, reaching ~0.063 macro F1 delta on GoEmotions-28
- Surface-level diversity/uniqueness of augmented text has negligible correlation with downstream classification performance
- LLM-specific artifacts—text elongation and label-distribution uniformization—are negatively associated with classification accuracy
- Class-conditional structural fidelity, not diversity, is the critical variable determining augmentation effectiveness for imbalanced text classification
Why It Matters
This study directly challenges the prevailing assumption that generative LLM-based augmentation is superior for handling imbalanced text classification, a common pain point in NLP pipelines. For practitioners deploying data augmentation strategies, it provides empirical evidence that simpler, retrieval-based approaches may outperform more complex and costly LLM-generated alternatives, especially under severe class imbalance.
Technical Details
- Benchmark covers 11 augmentation methods across three categories: classical perturbation, embedding-space retrieval (EmbSMOTE), and LLM-based generation, evaluated on seven public text classification datasets
- Datasets span class counts K=2–28 and imbalance ratios from 1.1 to over 500, with five random seeds per experimental condition
- Evaluation metrics include macro F1, Welch's t-tests for statistical significance, five distributional metrics, and an LLM-family sensitivity analysis using Qwen3-8B
- Key finding: class-conditional structural fidelity—the degree to which augmented samples preserve the class-conditioned geometry of the training distribution—is identified as the effective variable, not surface-level uniqueness
- LLM artifacts such as text elongation and artificial label-distribution uniformization were found to degrade classification accuracy
Industry Insight
- Retrieval-based oversampling (EmbSMOTE) should be adopted as the default strategy for imbalanced multi-class text classification before investing in LLM-based augmentation pipelines
- Organizations relying on LLM-generated synthetic data for imbalanced classification should re-evaluate their approach, as the added computational cost may not translate to performance gains and could even harm results
- Future augmentation research should prioritize structural fidelity metrics over diversity-oriented benchmarks, and practitioners should monitor for LLM-specific artifacts like text elongation when deploying generative augmentation in production
Disclaimer: The above content is generated by AI and is for reference only.