When the term moved online: detecting and correcting dataset shift in early-warning models

In March 2020 every college in India stopped teaching in rooms. When the new academic session opened online, the early-warning models built on the previous three years of records were still running, and nobody could say whether they still worked. The honest answer, for most deployments, was that nobody knew, and that the tools to find out quickly were not part of the standard kit.

This paper is our attempt to build that kit before the labels arrive. A persistence model only learns whether it was right at the end of a term, which is far too late. What can be known in the first weeks is the distribution of the features, a small number of early outcomes, and some theory about what kind of change has happened. We use simulation to ask, for each plausible kind of change, which detectors notice it, which corrections fix it, and which corrections quietly make it worse.

Five ways the world can change

The machine-learning literature distinguishes shifts by which part of the joint distribution of features x and outcome y moves. Writing the training distribution as p and the deployment distribution as q:

We add a fourth case that the taxonomy tends to leave implicit and that we think is the most likely after March 2020: the outcome becomes more common for reasons that leave no trace in the recorded features. A family loses its income; the student’s attendance, submissions and fees all look the same until the day they stop. Formally this is concept shift confined to the intercept with q(x) = p(x), and we call it hidden shift because, as we will show, no test on the features alone can see it.

Detecting shift before the labels arrive

We test three detectors on unlabelled data from the new term, each at four sample sizes corresponding roughly to the first one to four weeks of enrolment data.

• Population stability index (PSI), the industry default: for each feature, Σ (q_b − p_b) log(q_b / p_b) over ten quantile bins of the training data, with the conventional alarm at 0.25 on any feature.
• A domain classifier in the sense of Rabanser, Günnemann and Lipton: train a model to tell old students from new ones and test its cross-validated AUC against a null distribution from two samples of the training population.
• A kernel two-sample test (MMD) on the model’s own scores, following Gretton and colleagues, with a permutation p-value. Testing the score rather than the raw features asks the narrower question that matters: has the population the model sees changed in a way the model can register?

Three results stand out. First, covariate and concept shift are trivially detectable: the domain classifier separates old and new students almost perfectly (AUC 0.999), the attendance PSI is 0.53 and the LMS-activity PSI is 2.0. When the meaning of a feature changes, it changes its distribution too, and any detector will notice.

Second, the hidden shift is invisible to every detector, as it must be: the features are drawn from exactly the same distribution as before, so anything a detector reports is a false alarm (the domain classifier fired in 15% of deployments at 250 students against 5% with no shift, a difference within the noise of 20 repetitions). Its AUC is 0.504. A college that relied on drift monitoring would be told nothing had changed while departures rose by more than half.

Third, PSI’s rule of thumb is a large-sample rule. With 100 students and ten bins, the maximum PSI across seven features exceeded 0.25 in 65% of deployments in which nothing had changed at all. It never fired falsely at 250 or more. The threshold is fine; the sample sizes in a college’s first fortnight are not.

Drift monitoring only sees shifts that move the features. The shift we were most worried about in 2020, departures rising for reasons the record does not contain, does not. There is no unlabelled substitute for a small sample of outcomes.

Correcting without labels

If the shift is pure label shift, the posterior can be corrected without any new labels. Bayes’ rule gives the adjusted probability for a new prior π_t from a model trained at prior π_s:

and π_t itself can be estimated from unlabelled data, either by iterating (2) to a fixed point in the EM procedure of Saerens, Latinne and Decaestecker, or by black-box shift estimation (BBSE), which inverts the confusion matrix of the thresholded classifier measured on held-out training data. Both are elegant. Both rest entirely on the assumption that q(x|y) = p(x|y).

Under true label shift, EM moves the estimate from 8.8% to 14.4% against a truth of 13.0% and cuts expected calibration error from 0.042 to 0.016. BBSE, using a 0.5 threshold on a model whose scores rarely cross it, is badly conditioned and stops at 9.8%. Under the hidden shift the assumption is violated in a mild way and neither method helps much (9.5% and 6.8% against 13.3%; BBSE moves in the wrong direction).

Under covariate and concept shift the assumption is violated badly and both estimators collapse: the estimated departure rate goes to 0.0% and 0.1%, and calibration error rises from 0.085 to 0.133 under covariate shift. The reason is mechanical. When online attendance is inflated for everyone, every student looks like a stayer by the old standard, so the procedure concludes there are no leavers. An unlabelled correction applied without checking its assumption is worse than no correction.

Correcting with a few labels

The alternative is to collect a small labelled sample from the new term as early as possible: students who have formally withdrawn, or who have failed to register for internal assessments, confirmed by an advisor. We then compare four updates in the order of how much they change, following the clinical-model updating hierarchy of Steyerberg and of Janssen and colleagues.

• Intercept recalibration: add a single constant to the model’s logit, estimated by maximum likelihood on the new labels. One parameter.
• Platt recalibration: refit intercept and slope on the logit. Two parameters.
• Refit: a regularised logistic model on the new labels alone.
• Stacked refit: a regularised logistic model on the new labels with the old model’s logit as an additional feature.

Intercept recalibration is the dominant first move. With 400 labels it reduces calibration error in every regime (from 0.050 to 0.015 under hidden shift, 0.085 to 0.021 under covariate shift, 0.130 to 0.019 under concept shift) and, because a monotone transform cannot change a ranking, it never costs precision. Under no shift at all it is nearly harmless (calibration error 0.011 against 0.004).

Refitting is the only update that recovers ranking when the features have changed meaning. Under covariate shift, precision at a 15% flag rate rises from 0.333 for the frozen model to 0.391 with 400 labels and 0.399 with 1,000, against an oracle ceiling of 0.406. Under concept shift it rises from 0.467 to 0.481 and 0.492. But refitting on a small sample is costly when nothing has changed: with 150 labels and no shift, precision falls from 0.240 to 0.217, and it takes about 3,000 labels before the refit matches the model it replaced. Stacking the old score as a feature did not change this materially.

A decision rule for the first six weeks

Putting the detection and correction results together gives a policy that is simple enough to run in a college with one analyst:

• Week 1 to 2. Run the domain classifier and an MMD test on scores with at least 250 students; ignore PSI on smaller samples. Do not apply any unlabelled prior correction unless the detectors are silent and there is an independent reason to believe the shift is pure label shift.
• Week 2 to 6. Collect early outcomes from a random sample of at least 400 students, not from the students the model flagged, which would bias the update. Recalibrate the intercept as soon as the sample is complete, whatever the detectors said. It is cheap, it is safe, and it is the only thing that catches hidden shift.
• If a detector fired, refit using the new labels once there are around 1,000, and keep the frozen model running in parallel until the refit beats it on a held-out slice of the new term.
• If no detector fired, do not refit on a small sample. The model is very likely fine apart from its intercept, and a refit will make it worse.

Method

Limitations

The regimes are stylised, and real change after March 2020 will be a mixture: some label shift, some hidden shift, some features changing meaning and some not. Our detectors were evaluated at modest repetition counts, and the domain classifier’s null threshold, estimated from 40 draws, is too coarse at 1,000 students, where it fired falsely in a quarter of no-shift deployments; we report the 250-student results for that reason. The ‘early outcomes’ that make label-based updating possible are themselves a proxy for end-of-term departure, and a proxy that changes meaning online (a student may stop logging in without having left) would reintroduce exactly the problem we are trying to solve. Finally, a single deployment term per regime means the regime-level numbers carry sampling noise of a point or so of precision.

Simulated data only. The generator, the five regimes and the detector harness are published so that a college can substitute its own assumptions about what changed. Write to team@eduplatter.com.

Related work

The shift taxonomy follows Quiñonero-Candela and colleagues [1] and Moreno-Torres and colleagues [2]; Gama and colleagues [3] survey adaptation under concept drift. The unlabelled prior correction is Saerens, Latinne and Decaestecker [4], and black-box shift estimation is Lipton, Wang and Smola [5], whose identifiability conditions are exactly the ones our covariate-shift regime violates. Rabanser, Günnemann and Lipton [6] compared shift detectors empirically and found domain classifiers and score-level two-sample tests competitive, which our results reproduce in a small-sample tabular setting; the kernel test is Gretton and colleagues [7]. Shimodaira [8] gives importance weighting for covariate shift, which we did not use because the shifted features change meaning rather than frequency. The updating hierarchy is from clinical prediction [9, 10]. The pre-2020 model being corrected is the hazard formulation in EPR-2019-01 [11], simplified here to a single term.

Artefacts

Everything below is published or available on request. A number nobody can reproduce is an advertisement, not a result. Real institutional records are never shareable under our processor obligations, so where that applies we release a simulated corpus that reproduces the qualitative finding.

References

1. Quiñonero-Candela, J., Sugiyama, M., Schwaighofer, A., & Lawrence, N. D. (Eds.) (2009). Dataset Shift in Machine Learning. MIT Press.
2. Moreno-Torres, J. G., Raeder, T., Alaiz-Rodríguez, R., Chawla, N. V., & Herrera, F. (2012). A Unifying View on Dataset Shift in Classification. Pattern Recognition, 45(1), 521–530.
3. Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014). A Survey on Concept Drift Adaptation. ACM Computing Surveys, 46(4).
4. Saerens, M., Latinne, P., & Decaestecker, C. (2002). Adjusting the Outputs of a Classifier to New a Priori Probabilities: A Simple Procedure. Neural Computation, 14(1), 21–41.
5. Lipton, Z. C., Wang, Y.-X., & Smola, A. J. (2018). Detecting and Correcting for Label Shift with Black Box Predictors. ICML.
6. Rabanser, S., Günnemann, S., & Lipton, Z. C. (2019). Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift. NeurIPS.
7. Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B., & Smola, A. (2012). A Kernel Two-Sample Test. Journal of Machine Learning Research, 13, 723–773.
8. Shimodaira, H. (2000). Improving Predictive Inference under Covariate Shift by Weighting the Log-Likelihood Function. Journal of Statistical Planning and Inference, 90(2), 227–244.
9. Steyerberg, E. W. (2009). Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. Springer.
10. Janssen, K. J. M., Moons, K. G. M., Kalkman, C. J., Grobbee, D. E., & Vergouwe, Y. (2008). Updating Methods Improved the Performance of a Clinical Prediction Model in New Patients. Journal of Clinical Epidemiology, 61(1), 76–86.
11. Bhardwaj, M. (2019). Time to departure: discrete-time hazard models for student attrition. EduPlatter Research, EPR-2019-01.
12. University Grants Commission (2020). Guidelines on Examinations and Academic Calendar for the Universities in View of COVID-19 Pandemic and Subsequent Lockdown. 29 April 2020.