Only 47% of student success initiatives do anything measurable.
We assessed 220 initiatives across partner institutions using matched comparison rather than year-on-year change. This is what the data says about which ones work, for which students, and why — including the fourteen that appear to have made outcomes slightly worse.
What the data actually showed.
Category barely predicts success
The spread between initiative types — tutoring, advising, mentoring, orientation — ran from 38% to 56%. Far narrower than most sector guidance implies, and swamped by something else.
Tracking participation matters most
Initiatives that recorded who actually attended were 3.1 times more likely to show an effect. Partly measurement, partly a proxy for a programme somebody is really running.
Budget was almost irrelevant
Spending above the sector median gave a 1.1× ratio — effectively nothing. A single named owner gave 2.1×. Starting before week four gave 1.7×.
The effect decays and nobody re-checks
Median effect fell from +3.1 points in year one to +0.8 by year three across 61 initiatives we could follow. Almost none had been re-evaluated after year one.
Why this is not another benchmark report.
Most sector reports compare institutions on outcomes and call the differences insight. We did something narrower and harder: for each individual initiative, we matched participating students to comparable non-participating students within the same institution and term, and estimated an effect with a confidence interval.
Matched comparison, not pre/post
Year-on-year change cannot separate a programme from its cohort, the economy, or a change in how the registrar counts.
Outcome fixed before analysis
For each initiative the outcome measure and design were recorded before we looked, so a disappointing result could not become a different question.
Underpowered studies flagged, not counted as null
A programme serving 60 students cannot show a 2-point effect. Those are reported separately rather than folded into a failure count.
Bias runs in the flattering direction
Where take-up correlated with motivation our covariates missed, we over-estimated. The true success rate may be below 47%.
What would this look like for you?
We can run the same assessment on your three largest initiatives. Most institutions find at least one they were funding on the strength of a study that no longer holds.
The full report is available as a PDF on request