Answer Prior Collapse Control Through Counterbalanced Curricula for Robust Medical Vision Language Binary Inference Across Rare Findings

• Rafael Mendoza

  Western Mindanao State University, College of Computing Studies, Department of Information Systems, Normal Road, Baliwasan, Zamboanga City 7000, Philippines

  Author

• Joshua Villanueva

  Bohol Island State University, College of Computer Studies, Department of Information Technology, CPG Avenue, Tagbilaran City 6300, Bohol, Philippines

  Author

Medical vision-language models are  evaluated through binary questions that appear simple on the surface. In practice, these questions can expose a model’s tendency to rely on answer frequency, diagnostic prevalence, or prompt regularities instead of image-conditioned reasoning. Rare findings are especially vulnerable because the training stream may contain many more negative than positive cases, and a model can obtain acceptable average accuracy by leaning toward the majority answer. This paper presents an empirical study of answer-prior collapse in binary medical vision-language inference and evaluates counterbalanced curricula designed to reduce it. We construct a multi-domain dataset of 24,000 image-question examples covering chest radiography, retinal imaging, dermatology, and musculoskeletal radiography. The training distribution is deliberately skewed across finding prevalence, while the test distribution includes balanced, natural-prevalence, and rare-positive evaluation panels. We compare standard empirical-risk training with class reweighting, focal training, oversampling, hard-positive replay, and a proposed counterbalanced curriculum that alternates prevalence-realistic batches with balanced diagnostic contrast batches. The proposed curriculum reduces rare-positive miss rate from 38.7% to 19.4%, lowers answer-prior index from 0.42 to 0.17, and improves macro-balanced accuracy by 7.8 percentage points while preserving performance on common negative cases. The results show that curriculum structure, not only loss weighting, materially affects whether a model learns image-conditioned binary decisions for rare clinical findings.

• PDF