Page 362 -

C h a p t e r Assessing Studies Based

9 on Multiple Regression

T he preceding five chapters explain how to use multiple regression to analyze
the relationship among variables in a data set. In this chapter, we step back and
ask, What makes a study that uses multiple regression reliable or unreliable? We
focus on statistical studies that have the objective of estimating the causal effect of
a change in some independent variable, such as class size, on a dependent variable,
such as test scores. For such studies, when will multiple regression provide a useful
estimate of the causal effect, and, just as importantly, when will it fail to do so?

To answer these questions, this chapter presents a framework for assessing
statistical studies in general, whether or not they use regression analysis. This
framework relies on the concepts of internal and external validity. A study is internally
valid if its statistical inferences about causal effects are valid for the population and
setting studied; it is externally valid if its inferences can be generalized to other
populations and settings. In Sections 9.1 and 9.2, we discuss internal and external
validity, list a variety of possible threats to internal and external validity, and discuss
how to identify those threats in practice. The discussion in Sections 9.1 and 9.2
focuses on the estimation of causal effects from observational data. Section 9.3
discusses a different use of regression models—forecasting—and provides an
introduction to the threats to the validity of forecasts made using regression models.

As an illustration of the framework of internal and external validity, in Section
9.4 we assess the internal and external validity of the study of the effect on test
scores of cutting the student–teacher ratio presented in Chapters 4 through 8.

9.1 Internal and External Validity

The concepts of internal and external validity, defined in Key Concept 9.1, pro-
vide a framework for evaluating whether a statistical or econometric study is use-
ful for answering a specific question of interest.

Internal and external validity distinguish between the population and setting
studied and the population and setting to which the results are generalized. The
population studied is the population of entities—people, companies, school dis-
tricts, and so forth—from which the sample was drawn. The population to which

361