Abstract
Open-vocabulary segmenters use text to recognize classes without segmentation training labels. A small head can adapt their predictions to labeled (seen) classes, but it can also suppress held-out (novel) classes even when the backbone remains frozen. We study this tension on region-based and dense vision-language segmenters under matched data and supervision. On ADE20K, an unconstrained head with class-dependent outputs reduces novel-class intersection over union (IoU) from 15.1 to 0.1. Training the head to stay close to frozen class probabilities restores novel-class IoU to 15.3 and raises mean IoU (mIoU) from 14.5 to 15.8. To characterize this change, we measure alignment drift , the mean cosine distance between corrected and frozen probability vectors. Across 38 class-specific settings (32 distinct corrections) in 13 families, drift tracks novel-class retention (Pearson correlation r=−0.90). Corrections that apply a shared rule across classes retain novel-class IoU throughout their observed drift range, establishing that drift must be interpreted within the correction family. IoU recovery leaves a coverage gap. Anchored heads predict novel classes over a smaller part of the image, while improved confidence calibration alone can conceal novel-class loss. Dividing all class scores by a shared temperature gives the lowest expected calibration error among the tested in-domain corrections retaining novel-class IoU (27.2→8.7). Our results support anchoring class-specific heads to frozen predictions and evaluating novel-class accuracy, prediction coverage and confidence calibration together.