A repository of papers that bring machine learning into economics and econometrics curated by John Coglianese, William Murdock, Ashesh Rambachan, Jonathan Roth, Elizabeth Santorella, and Jann Spiess
Athey and Imbens (2016) and Wager and Athey (2017) introduced causal trees and causal forests as new methods for identifying treatment heterogeneity that have potential gains over traditional methods. This paper applies the causal forest method to data from two randomized experiments that evaluated the impact of a summer jobs program on disadvantaged youth in Chicago.
In this paper, Davis and Heller use a causal forest to predict which participants of a summer jobs program had positive or negative conditional average treatment effects (CATE). They split the data in half in order to investigate how the causal forest algorithm compares in- and out-of-sample — I.e., they run the causal forest procedure in one sample and use the trees generated in that sample to predict across both samples. They find that the two subgroups created by this procedure had statistically different impacts for the in sample — the youth predicted to have positive CATEs were indeed found to have reduced violent crime rates and increased employment relative to the youth predicted to have negative CATEs. However, the differences were severely attenuated for the out sample — the difference for violent crime rates across the predicted subgroups in this sample was not statistically different (p-value of 0.77). Indeed, Davis and Heller report that they can reject the null-hypothesis that the differences were the same across the in and out samples (p-value < 0.01). An additional analysis offers evidence that the difference between the in- and out-of-sample results appears to be overfitting caused by the causal forest algorithm predicting an individual’s outcome using trees where that individual’s outcome was already used to grow the tree. When they implement a procedure that separates estimation and prediction, the results are similar across the in and out samples. However, they note that this is an ad hoc solution and it may invalidate the theoretical justifications of the causal forest method. Regardless, the authors point out that standard interaction methods with multiple hypothesis testing corrections would not have uncovered the heterogenous treatment effects that the causal forest method found.
Highlight: Section III of the paper provides researchers with a step-by-step road map of the causal forest algorithm. Applied researchers should be able to follow these steps to implement a similar procedure in their own research. Note that Wager and Athey (2017) have also provided statistical packages to aid researchers with implementation.
causal forests heterogeneous treatment effects causal inference
Reviewed by William on . Suggested by Ashesh.