Doubly robust estimates of programme effects when uptake clusters by college

Picture a university that offers a structured advising programme to all of its affiliated colleges. Participation is voluntary for students, enthusiastic at some colleges and lukewarm at others. The raw comparison between participants and non-participants shows a modest gain in persistence. The adjusted comparison, controlling for everything known about the students, shows a gain more than twice as large. The natural reading is that adjustment removed noise. In the setting simulated here, it adds bias.

This paper explains how, using simulation so that the right answer is known. It is a methods paper, but the problem it addresses is ordinary: almost every programme an Indian university evaluates is delivered through colleges, and colleges differ in ways that affect both who takes part and how students fare. The National Education Policy’s push toward larger multi-college clusters will make that structure more common, not less.

The estimand and the three standard estimators

Write A_i = 1 if student i took part, Y_i = 1 if they persisted to the next year, and Y_i(1), Y_i(0) for the potential outcomes with and without the programme. The target is the average treatment effect across all students in the university:

Identification requires that, given the covariates we condition on, uptake is as good as random (no unmeasured confounding) and that every kind of student had some chance of taking part (positivity). Three estimators then follow from two nuisance functions, the propensity score e(x) = Pr(A = 1 | x) and the outcome regressions m_a(x) = E[Y | A = a, x]:

and the augmented estimator of Robins, Rotnitzky and Zhao, which combines both and is consistent if either nuisance model is correct:

The terms φ_i are estimated values of the efficient influence function, and their sample variance divided by n gives a standard error. That standard error assumes the students are independent. In a university of colleges they are not.

Where the college comes in

Suppose each college c has a characteristic U_c that nobody records: the energy of its principal, the culture of its faculty, the strength of its alumni network. It makes students more likely to take up a new programme and also more likely to persist regardless. Student covariates do not capture it. The assumption of no unmeasured confounding given student covariates is then false, and no amount of modelling flexibility in x will rescue it.

But U_c is constant within a college, so conditioning on college identity removes it entirely. Adding college fixed effects, a dummy for every college, to both the propensity and the outcome model restores identification under the weaker assumption that uptake is as good as random given student covariates within a college:

Li, Zaslavsky and Landrum analysed exactly this choice for propensity weighting with multilevel data and showed that the fixed-effects propensity model protects against cluster-level confounding where a pooled model does not. What we add is the doubly robust version, a machine-learning version, and a demonstration of how badly the pooled versions fail at the cluster counts universities actually have.

The simulation

Result 1: adjusting for students alone makes it worse

Without college effects, every adjusted estimator is biased upward by about 5.1 points: the analysis reports an effect of nearly 9 points when the truth is under 4. Coverage of a nominal 95% interval is zero in all four scenarios. Getting the student-level models exactly right does not help, because the problem is not in the student-level models.

The unadjusted difference in means is biased by only 1.4 points in the same scenario, which looks like a point in its favour and is not. The programme recruits struggling students (biasing the raw comparison down) inside thriving colleges (biasing it up), and in this scenario the two roughly cancel. Adjusting for student risk removes the first bias and leaves the second fully exposed. This is the bias-amplification phenomenon described by Pearl and by Middleton and colleagues: conditioning on covariates that explain treatment but not the unmeasured confounder can increase, not reduce, the bias that confounder causes. Change the selection pattern (scenario D) and the naive estimate swings to −6.7 points. It is not a robust fallback; it is luck.

When adjustment makes an effect estimate larger, and the unit of delivery is a college, the first hypothesis should be confounding at the college level, not a programme that works better than the raw numbers suggested.

Result 2: college fixed effects restore double robustness

With college fixed effects in both nuisance models, AIPW is unbiased to within a tenth of a point whenever at least one of the two models is correctly specified, exactly as the theory says it should be. Root-mean-square error falls from about 5.3 points to 0.73. Outcome regression and IPW with fixed effects are nearly as good in scenarios A to C, which is a reminder that double robustness is insurance, not a performance guarantee: its value appears when you do not know which model is wrong.

That is scenario D, where both are. Parametric AIPW with fixed effects then carries a bias of −0.61 points, a sixth of the effect, and its interval covers the truth 88% of the time. Cross-fitted AIPW, with gradient-boosted trees for both nuisance functions and the college passed as a categorical feature so the trees can learn college-specific intercepts, is unbiased (−0.06) with 97% coverage, at the price of slightly higher variance in the easy scenarios. Following Chernozhukov and colleagues, the cross-fitting (fitting nuisance models on two folds and evaluating φ on the third) is what makes the flexible fits safe to plug into (3).

Result 3: the standard error tells you something is wrong

One diagnostic proved more useful than any other. Without college effects, the cluster-robust standard error of AIPW, computed by summing the influence-function values within each college,

averaged 1.38 points against 0.59 for the naive version, a ratio of 2.3. Once college effects were included, the two agreed to within 7% (0.77 against 0.72). The cluster correction is not itself the fix: with the bias in place, even the wider interval covered the truth in at most 3.5% of universities. But a large gap between the two standard errors means the influence-function values are strongly correlated within colleges, which means something about the college is driving outcomes that the model has not absorbed. We now compute both on every evaluation and investigate any ratio above 1.5 before reporting anything.

Positivity, and what fixed effects cost

Fixed effects are not free. Within-college comparison needs both participants and non-participants in every college; a college where everyone or no one took part contributes nothing, and a college where almost everyone did contributes enormous weights. In our simulation the median maximum inverse-propensity weight was 15 with correct models and 22 to 24 when the propensity model was misspecified, after truncating estimated propensities to [0.01, 0.99]. That is manageable at 30 colleges with uptake around 31%; a programme that is near-universal in some colleges and absent in others is not evaluable this way, and no estimator can make it so. The right response is to say so, and to design the next rollout with some within-college variation in timing.

Recommendations

• Condition on the unit of delivery. If a programme is offered through colleges, include college fixed effects in both the propensity and the outcome model. If there are too few students per college for that, the programme cannot be evaluated at the student level and the analysis should move to the college.
• Use AIPW, cross-fitted with flexible learners when n allows. At 15,000 students, three folds and boosted trees cost seconds and remove the dependence on getting a functional form right.
• Report both standard errors. A cluster-robust SE far larger than the naive one is a symptom of an unmodelled cluster effect, not a technicality to be corrected.
• Distrust an adjustment that grows the effect. It can be right. It is more often the signature of offsetting biases coming apart.
• Check positivity by college and publish the weight distribution alongside the estimate.

Limitations

Fixed effects remove confounding that is constant within a college. Anything that varies within a college and drives both uptake and persistence, such as a particular teacher’s encouragement, remains a threat that this design does not address. The simulation also has no interference between students, whereas advising programmes plausibly spill over to friends of participants, which violates the stable-unit assumption on which (1) depends. Thirty colleges is enough for the cluster-robust variance to behave; with fewer than about fifteen, small-sample corrections or a wild cluster bootstrap are needed. The cross-fitted estimator was run on 60 rather than 200 universities per scenario because of its cost, so its coverage figures carry a Monte Carlo standard error of about 2.8 points.

Simulated data only. The simulation, every estimator and both variance formulas are published as a single script. Write to team@eduplatter.com.

Related work

The propensity score is Rosenbaum and Rubin [1]; the augmented estimator is Robins, Rotnitzky and Zhao [2], and its double robustness is set out by Bang and Robins [3]. Kang and Schafer [4] showed how doubly robust estimators can misbehave when both models are wrong and weights are extreme, which our scenario D reproduces in miniature. Cross-fitting with machine-learning nuisance models follows Chernozhukov and colleagues [5]. For clustered data, Arpino and Mealli [6] and Li, Zaslavsky and Landrum [7] analyse propensity specification with multilevel structure; our fixed-effects AIPW extends the latter. Cameron and Miller [8] is the practical reference on cluster-robust inference. The bias-amplification mechanism is described by Pearl [9] and examined empirically by Middleton and colleagues [10]. Imbens and Rubin [11] supply the potential-outcomes framing throughout.

Artefacts

Everything below is published or available on request. A number nobody can reproduce is an advertisement, not a result. Real institutional records are never shareable under our processor obligations, so where that applies we release a simulated corpus that reproduces the qualitative finding.

References

1. Rosenbaum, P. R., & Rubin, D. B. (1983). The Central Role of the Propensity Score in Observational Studies for Causal Effects. Biometrika, 70(1), 41–55.
2. Robins, J. M., Rotnitzky, A., & Zhao, L. P. (1994). Estimation of Regression Coefficients When Some Regressors Are Not Always Observed. Journal of the American Statistical Association, 89(427), 846–866.
3. Bang, H., & Robins, J. M. (2005). Doubly Robust Estimation in Missing Data and Causal Inference Models. Biometrics, 61(4), 962–973.
4. Kang, J. D. Y., & Schafer, J. L. (2007). Demystifying Double Robustness: A Comparison of Alternative Strategies for Estimating a Population Mean from Incomplete Data. Statistical Science, 22(4), 523–539.
5. Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., & Robins, J. (2018). Double/Debiased Machine Learning for Treatment and Structural Parameters. The Econometrics Journal, 21(1), C1–C68.
6. Arpino, B., & Mealli, F. (2011). The Specification of the Propensity Score in Multilevel Observational Studies. Computational Statistics & Data Analysis, 55(4), 1770–1780.
7. Li, F., Zaslavsky, A. M., & Landrum, M. B. (2013). Propensity Score Weighting with Multilevel Data. Statistics in Medicine, 32(19), 3373–3387.
8. Cameron, A. C., & Miller, D. L. (2015). A Practitioner’s Guide to Cluster-Robust Inference. Journal of Human Resources, 50(2), 317–372.
9. Pearl, J. (2010). On a Class of Bias-Amplifying Variables that Endanger Effect Estimates. Proceedings of the 26th Conference on Uncertainty in Artificial Intelligence.
10. Middleton, J. A., Scott, M. A., Diakow, R., & Hill, J. L. (2016). Bias Amplification and Bias Unmasking. Political Analysis, 24(3), 307–323.
11. Imbens, G. W., & Rubin, D. B. (2015). Causal Inference for Statistics, Social, and Biomedical Sciences. Cambridge University Press.
12. Bhardwaj, M. (2020). When the term moved online: detecting and correcting dataset shift in early-warning models. EduPlatter Research, EPR-2020-01.