When Audit Quality Fails to Predict Downstream Utility: Synthetic-Data Selection for Low-Resource Classification

Abstract

Quality-aware synthetic-data selection assumes that examples rated highly by an LLM judge will help downstream learning. We test this assumption in a controlled replay across two low-resource African-language classification tasks, four languages, and seven selectors. Six selectors retain 64 examples per task-language cell. CoSDA-V2 yields the highest judged label correctness (0.904 versus 0.767 for naive) and the lowest measured shortcut score, but AlpaGasus leads CoSDA-V2 on downstream Macro-F1 in both the main replay (0.202 versus 0.163) and the three-seed refit (0.193 versus 0.157). Adding the audit’s strict gate to AlpaGasus reduces Macro-F1 by 0.029, driven by one cell where only 31 of 64 candidates survive. Because the gate changes pool composition and training-set size, this comparison measures their joint effect. LESS also falls below naive when unconstrained top-k collapses label balance, but recovers with quotas. At this budget, retention constraints account for more of the observed differences than a consistent ranking advantage. In this replay, audit scores do not reliably predict downstream utility. We therefore recommend reporting both metrics for the same retained sets. Keywords: Synthetic data selection, Data-centric NLP, Counterfactual audit, Low-resource NLP

Cite this paper

@inproceedings{le2026audit,
  author    = {Le, Xuan-Bach and Tran-Truong, Phat T. and Ha Xuan, Son},
  title     = {When Audit Quality Fails to Predict Downstream Utility: Synthetic-Data Selection for Low-Resource Classification},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
  year      = {2026},
  address   = {Budapest, Hungary}
}