This is my short response paper on mediation analysis and some related readings. I wrote this memo as a graduate student, so don't rely on this note in lieu of reading the papers.


Though researchers care about causal mechanisms as well as causal effects, randomized experiments often fail to identify the causal mechanism and only illustrate whether the treatment causes changes in the outcome. In other words, experimental studies have rarely shown how or why the treatment changes the outcome, leaving the causal mechanism as a black box.

Mediation analysis has been a point of debate in response to this criticism that experiments often do not show causal mechanisms. The goal of mediation analysis is to decompose total effects into direct and indirect effects. Incumbency advantage, which has been one of the most studied topics in political science, can be a good example to understand what a mediator is. It had been a consensus in the 1980s in the incumbency effects literature that incumbency advantage is positive and growing in magnitude. Later, the question about where incumbency advantage comes from has received more attention. One of the most convincing mediators is the quality of challengers – incumbent candidates have a higher ability to deter high-quality challengers from entering the race (Levitt and Wolfram 1997). In this example, the mediator is quality of challenger with incumbency status and election outcome as the independent variable and the dependent variable, respectively.

Baron and Kenny (1986) propose a test for mediation with the framework of linear structural equation models [1]. The test consists of three regression equations. First, regressing the mediator (M) on the independent variable (X) and showing a significant relationship; second, regressing the dependent variable (Y) on the treatment (T) and showing a significant relationship between the two; and third, regressing the dependent variable (Y) on the mediator (M) and the independent variable (X) and showing a significant relationship between the dependent variable (Y) and the mediator (M). In other words, we can decompose the total effect of X on Y, into the direct effect of X on Y without going through M, and the indirect effect of X on Y through M (Gerber and Green 2012, 322).

Baron and Kenny (1986) argue that if there is a valid mediator, there should be statistical significance in the predicted direction in all the three equations. In addition, the effect of the independent variable on the dependent variable must be smaller in the third equation than in the second one. In this test, perfect mediation shows no statistically significant effect of the independent variable when the mediator is controlled in the third equation. This method to test mediation assumes two things: first, there has to be no measurement error in the mediator, since the presence of measurement error would cause overestimation of the effect of the independent variable on the dependent variable and underestimation of the mediator. Second, the mediator has to cause the dependent variable, not the other direction.

Gerber and Green (2012), however, point out potential biases of this test. They argue that randomizing the treatment alone without randomizing mediator cannot prevent the problem of omitted variable bias, which means that there might be another factor that influences both the mediator and the dependent variable. The authors take an example of Bhavnani’s study of local government representation in India (2009) to explain this problem. In this study, the treatment is randomly assigned reserved seats in 1997, the dependent variable is the election of a female representative in the ensuing 2002 election, and the mediator is the number of women candidates running for office in 2002. The problem is that there might be other unmeasured factors that cause female candidates to run for office. For example, women in more egalitarian constituencies are more likely to run for office, which also influences the election of female representatives.

The potential biases can be understood in potential outcomes framework as well. In the incumbency advantage example, we only observe electoral outcomes in which an incumbent faces a challenger with lower quality. Though we want to observe electoral outcomes where an incumbent with the quality of challenger that this incumbent politician would face if he or she is not an incumbent, this rarely happens in reality.

To overcome the potential biases, one might envision an experiment where both the treatment and the mediator are randomly assigned. However, in reality, directly randomizing the mediator is also very difficult to conduct. For example, thinking back to Bhavanani’s study, directly randomizing the number of women candidates running for office in 2002 seems to be almost impossible.

Though direct manipulation of mediator is difficult in many cases, researchers can use encouragement design by introducing a random encouragement (for encouragement design, see Gerber and Green 2012, chapter 6). The advantage of this encouragement design is that it allows for unobserved confounders between the mediator and the outcome by taking an instrumental variable approach. For example, if a mediator in an experiment is about belief, it is really difficult to think about practical strategies to randomly manipulate people’s beliefs. In encouragement design, an instrumental variable can be used to lead to different types of beliefs or different intensity levels of beliefs.

Crossover design can be an another alternative when mediator is not directly randomized. This design consists of two steps. In the first step, simply conduct a standard experiment. In the second step, the treatment is changed to the opposite status with the mediator fixed to the value observed in the first experiment. This design could be very powerful in that it can identify mediation effects for each subject. However, it assumes no carryover effect, which means that the first experiment must not affect the second one, which is a strong assumption.

Though both Gerber and Green (2012) and Baron and Kenny (1986) provide not only comprehensive but also in-depth discussions on mediation analysis, there is insufficient discussion on the multiple mediators. Multiple mediators are often implicitly assumed to be independent in many experimental studies, but multiple mediators need additional care especially when they are correlated with each other. Druckman and Nelson (2003), for instance, examine the effects of reading news articles on a proposed election campaign finance reform, emphasizing either its positive or negative aspects, on support for the proposed reform. In this study, mediators are belief importance and belief content. Belief importance means how importantly participants rated the key ideas of “protecting free speech rights of individuals and groups” and “protecting government from excessive influence by special interests.” Belief content measures ask participants if they thought the impact of reform would have a positive or negative effect on “free-speech rights” and “limiting special interest influence.” Their results present that belief importance is significant, while belief content is not.

The authors implicitly assume the two mediators, belief importance and belief content, are independent on each other. However, belief importance and belief content might be related and unable to be isolated individually. If they are correlated, we might need to include an interaction term between the two mediators to better capture the effects of multiple mediators on the outcome or additional methods to examine whether there is a causal relationship between the two mediators. Without these additional considerations the effects of each mediator might be mistakenly estimated. Imai and Yamamoto (2013) develop a sensitivity analysis to examine the robustness of empirical findings of experimental studies which might potentially violate the independence assumption. Although there is no clear cutoff or limit in the sensitivity analysis to say that the violation of the assumption makes the findings no longer valid, it seems to worth checking how empirical findings change and how sensitive they are based on whether the key assumption holds.

Mediation analysis is important in that it can unpack black boxes of causal. However, even randomized experiments often fail to show the mechanisms because of several problems such as difficulties of randomizing mediators and potential violation of key assumptions. This does not necessarily mean that these problems will never be solved; clever or creative experimental designs with additional care can make it possible to do more credible and valid inferences.

[1] In this paper, the authors distinguish a moderator and a mediator. A moderator variable affects the direction and/or the strength of the treatment effect, while a mediator is a mechanism explaining how the treatment affects the outcome.

Discussion Questions
1. How can we examine mechanisms of causal relationships rather than simply showing whether the treatment causes changes in the outcome?
2. What are the differences between a moderator and a mediator?
3. What experimental designs make it possible to randomly manipulate the mediator as well as the treatment? Is it possible to randomly manipulate the mediator? What are the differences between randomization of the treatment and that of the mediator? Why randomization of the mediator is important to estimate the treatment effects?
4. What additional considerations are required when there are multiple mediators?

References
Baron, R.M. and Kenny, D.A., 1986. The moderator–mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of personality and social psychology, 51(6), p.1173.
Bhavnani, R.R., 2009. Do electoral quotas work after they are withdrawn? Evidence from a natural experiment in India. American Political Science Review, 103(1), pp.23-35.
Druckman, J.N. and Nelson, K.R., 2003. Framing and deliberation: How citizens' conversations limit elite influence. American Journal of Political Science, 47(4), pp.729-745.
Gerber, A.S. and Green, D.P., 2012. Field experiments: Design, analysis, and interpretation. WW Norton.
Imai, K. and Yamamoto, T., 2013. Identification and sensitivity analysis for multiple causal mechanisms: Revisiting evidence from framing experiments. Political Analysis, 21(2), pp.141-171.
Levitt, S.D. and Wolfram, C.D., 1997. Decomposing the sources of incumbency advantage in the US House. Legislative Studies Quarterly, pp.45-60.