Early-warning model calibration across 14 institutional contexts

Almost every analytics vendor in higher education sells the same model to every institution. It is a good business: fit once, deploy many. The question nobody in a procurement meeting asks is whether the model that worked at a large public university in one country tells you anything useful about a 4,000-student autonomous college in another.

We were asked this by a prospective partner who had been burned by exactly that, and we did not have a defensible answer. So we built one.

What we mean by a context

A context here is one institution and one programme family — for example, undergraduate engineering at a single campus. We use that granularity rather than “institution” because within a large university, engineering and humanities differ from each other about as much as two separate colleges do. Fourteen contexts came from nine partner institutions who consented to de-identified use for methodological work.

The gap is not subtle. Within-context cross-validation gives a median precision of 0.68. Transfer from a single other context drops that to 0.49 — worse than a simple attendance-and-assessment heuristic in four of the fourteen cases. A pooled model trained on the other thirteen contexts does better at 0.59, but it never catches the local fit.

Sample size is not the explanation

Our first hypothesis was the obvious one: small contexts have too little data, and pooling should help them most. It does help them most, but pooling does not close the gap even for the largest contexts, which have plenty of data of their own.

Two mechanisms account for most of it, and both are about meaning rather than volume.

• Base rates differ enormously. Term-to-term non-persistence in our contexts ranges from 3.1% to 21.6%. A model calibrated at one base rate is badly calibrated at the other even when its ranking is fine, and a flag rate tuned for one produces either a flood or a trickle in the other.
• The same variable means different things. “Attendance” is biometric in three contexts, faculty-entered in eight, and self-reported in three. “Assessment submitted” includes formative work in some and only summative in others. A coefficient learned on one definition is not portable to another.
• Calendar structure varies. Semester, trimester and annual-exam contexts put the decision point in a different place relative to the signal, which changes which features are even available at prediction time.

In four of fourteen contexts, a transferred model performed worse than two hand-written rules. A model can be sophisticated and still be worse than nothing, and the institution buying it has no way to tell without a local holdout.

What recovers the gap

Warm-starting from a pooled model and fine-tuning on local data is the practical answer, and it works faster than we expected. With one term of local history a warm-started model reaches 0.62; with two it reaches 0.66, within two points of a fully local fit. This is now how we onboard an institution with insufficient history for a cold local fit — and we tell them explicitly that the first term’s predictions are the weakest they will see.

Method

Limitations

Nine institutions is not a sample of higher education. Ours skew Indian, they skew toward institutions willing to work with an analytics vendor on methodology, and none of them is a very large open-enrolment system where the base rate and the data shape differ again. The de-identification required for cross-context work removed some features we would normally use, so absolute precision here is a little below what a live deployment sees. And precision at a fixed flag rate is a proxy: what matters is whether the flagged student was helped, which is the subject of a different paper.

The evaluation harness and a simulated corpus that reproduces the qualitative result are available for replication. Real institutional data is not shareable under our processor obligations. Write to team@eduplatter.com.

Related work

Bird et al. [1] compared predictive modelling methods across institutions and found method choice mattered less than practitioners assume — a result our data-quality paper extends. Jayaprakash et al. [2] showed an early-alert model could be ported between institutions with retuning, which is often cited as evidence that transfer works; our reading is that their retuning step is doing most of the work and is not optional. The distinctive contribution here is quantifying the untuned transfer penalty across fourteen contexts, and showing that it is driven by base rate and variable semantics rather than sample size.

Artefacts

Everything below is published or available on request. A number nobody can reproduce is an advertisement, not a result. Real institutional records are never shareable under our processor obligations, so where that applies we release a simulated corpus that reproduces the qualitative finding.

References

1. Bird, K. A., Castleman, B. L., Mabel, Z., & Song, Y. (2021). Bringing Transparency to Predictive Analytics: A Systematic Comparison of Predictive Modeling Methods in Higher Education. AERA Open.
2. Jayaprakash, S. M., Moody, E. W., Lauría, E. J. M., Regan, J. R., & Baron, J. D. (2014). Early Alert of Academically At-Risk Students: An Open Source Analytics Initiative. Journal of Learning Analytics, 1(1).
3. Tinto, V. (1993). Leaving College: Rethinking the Causes and Cures of Student Attrition. University of Chicago Press.
4. Chen, T., & Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System. ACM KDD.
5. Ekowo, M., & Palmer, I. (2016). The Promise and Peril of Predictive Analytics in Higher Education. New America.
6. Bhardwaj, M. (2025). Data quality is the binding constraint, not model choice. EduPlatter Research, EPR-2025-02.