Abstract
Quality-aware synthetic-data selection rests on a proxy: examples that an LLM judge rates as good should also help a downstream model learn. We show that this proxy breaks in low-resource African-language classification. Across four languages (Amharic, Hausa, Swahili, Yoruba), two classification tasks (MasakhaNEWS, AfriSenti), and five matched-budget selectors, audit rankings and downstream rankings diverge systematically. Within each cell, the Spearman correlation between judged label correctness and Macro-F1 across selectors has mean ρ = 0.04 (median 0.00), showing that the mismatch is not an aggregation artifact. CoSDA-V2, our counterfactual audit framework, produces the cleanest selected pool on three audit channels at once: highest judged label correctness (0.904 vs. 0.767 for naive, a 17.9% relative gain), lowest shortcut score, and a hard-reject rate of 0.162 vs. 0.486 for naive. AlpaGasus nevertheless leads downstream Macro-F1 (0.202 vs. 0.163 for CoSDA-V2), and the inversion persists on the five non-degenerate cells. The lesson is methodological: audit quality is a property of the selected pool, not a guarantee of downstream utility. Synthetic-data evaluation should therefore report audit and downstream metrics on the same retained sets. We release the audit tables and a claim ledger that links every reported number to its source row. Keywords: Synthetic data selection, Data-centric NLP, Counterfactual audit, Low-resource NLP