A classifier produces a score, but a decision needs a threshold, and the threshold depends on the cost of each kind of error, on how common the positive class is, and on how much labelled data is available to set it. We study these three dependences in a binormal model, where every quantity has a closed form. The optimal threshold is ?? = d/2 ? ln ?/d, with ? the cost-weighted prior odds. The excess cost of a threshold error ?? is ½ dA ??² to second order. The default rule that predicts the positive class when the posterior exceeds 0.5 is off by exactly ln r/d in score units, where r is the ratio of the two error costs, whatever the prevalence. A change of prevalence from ? to ?? moves the optimum by ?[logit ?? ? logit ?]/d. A threshold chosen by minimising the empirical cost on n labelled points has excess cost of order n?2/3, against n?1 for a plug-in estimate that uses the Gaussian model. In simulation the excess-cost formulas matched within 0.3 cost units per 1,000 cases for the cost curve and 0.004 for the default rule, and fitted slopes were -0.69 and -1.01. When prevalence fell from 0.3 to 0.01, keeping the old threshold cost 5.4 times as much as always predicting the negative class. On the breast cancer and digits datasets, a shift-corrected Bayes threshold was best or near best in all four settings, and tuning a threshold on up to 160 labelled points did not beat it. Synthetic results are simulations; the real data are two small benchmark sets with resampled prevalence.
Turn pages by dragging a corner, clicking a page, or with the ← → keys.
Loading article…
Open the PDF instead
DOI: https://doi.org/10.63328/IJRDES-V8RI6P2
OpenYou've reached the end of this article.
Mahender Reddy Sheri , Vijayalakshmi Kakulapati, “ Robust Threshold Selection for Cost-Sensitive Classification under Changing Prevalence and Limited Validation Data ", International Journal of Research and Development in Engineering Sciences(IJRDES), Volume 8 , Issue 6 , pp 9-19, November 2026. CrossRef DOI: https://doi.org/10.63328/ijrdes-v8ri6p2
If an AI agent’s actions are logged, the log is only useful if nobody can quietly change it afterw...