I spent nine years in institutional research before joining EduPlatter, and I want to be clear that this finding is about the sector, not about anyone’s competence. Including mine.
Over three years our partner institutions nominated 220 student success initiatives for proper assessment. We used matched comparison — students who received the intervention against comparable students who did not — rather than the usual year-on-year comparison, which cannot distinguish a programme from a cohort.
103 of the 220 produced an effect distinguishable from zero. Fourteen had a negative point estimate with a confidence interval excluding zero, which is worth saying plainly: a few things institutions do appear to make outcomes slightly worse, usually by displacing something that was working.
Category barely mattered
I expected tutoring and structured advising to dominate and orientation to trail. The differences by category were real but modest — roughly 38% to 56% success across seven categories. Something else was doing the work.
Three characteristics did
Whether participation was tracked at individual level: initiatives that tracked it were 3.1 times more likely to show an effect. Whether there was a single named owner: 2.1 times. Whether it started before week four of term: 1.7 times.
Budget was almost uncorrelated. Involving new software was mildly negative.
Whether one person owned the initiative and knew who turned up mattered several times more than what it cost.
I think most of the tracking effect is measurement rather than magic — an initiative that does not know who attended cannot be evaluated well, so its true effect is likelier to be missed. But part is real. Tracking participation is a proxy for a programme somebody is actually running rather than merely offering.
The effect decays and nobody re-checks
Among the 61 initiatives we could follow for three years or more, median effect fell from +3.1 points in year one to +0.8 by year three. Almost none had been re-evaluated by the institution after its first year.
So the practical advice is uncomfortable but simple. Pick your three largest initiatives. Find out whether you know who participated. If you do not, fix that before you spend anything else — and re-evaluate whatever you justified with a 2019 study.
Full study: EPR-2026-03. Every institution received its own initiative-level results, including the null and negative ones.
