A single model deployed to every client is the ideal software business: fit once, sell many, margin improves with every customer. Almost every analytics vendor in higher education runs this way. We do not, and this is the measurement that decided it.
We took de-identified records from fourteen institutional contexts — 186,000 student-terms — and asked what happens when you fit a persistence model on one context and predict in another.
Within-context, precision was 0.68. Transferred from a single other context, 0.49. In four of the fourteen, the transferred model performed worse than two hand-written rules about attendance and missed assessments.
Sample size is not the problem
Our first assumption was that small institutions lack data and pooling should help. Pooling does help them most, but it does not close the gap even for the largest contexts, which have plenty of their own data. The cause is elsewhere.
- Base rates differ enormously. Non-persistence ranged from 3.1% to 21.6% across our contexts. A flag threshold tuned for one produces a flood or a trickle in the other.
- The same variable means different things. Attendance was biometric in three contexts, faculty-entered in eight and self-reported in three. A coefficient learned on one definition does not travel.
- Calendar structure varies. Semester, trimester and annual-exam institutions put the decision point in a different place relative to the signal.
A model can be sophisticated and still be worse than two rules, and the institution buying it has no way to tell without a local holdout.
What we do instead
We fit on your data. Where an institution has too little history for a cold fit, we warm-start from a pooled model and fine-tune — which reaches within two points of a full local fit after two terms. We tell those institutions explicitly that their first term of predictions is the weakest they will see.
The practical question for a buyer
Ask any vendor for precision on your held-out data before you deploy, and ask what it is compared against. If the comparison is not a simple heuristic you could implement yourself, the number does not mean much. That single question would have saved several institutions we have spoken to a great deal of money.
Full study: EPR-2026-04, including the warm-start analysis and per-context results.
