By Design: Planning Research on Higher Education

Citation: Richard J. Light, Judith D. Singer, John B. Willett (1990) By Design: Planning Research on Higher Education.
Internet Archive Scholar (search for fulltext): By Design: Planning Research on Higher Education
Download: http://www.hup.harvard.edu/catalog.php?isbn=9780674089310
Tagged: NatematiasGenerals (RSS), experiments (RSS), design (RSS), sampling (RSS), methods (RSS), education (RSS), research (RSS)

Summary

How can we plan studies to answer important questions? In By Design, Richard Light, Judith Singer, and John Willett introduce methods for planning studies, using higher education as the common thread, focusing on:

• descriptive studies that characterize how things currently are without answering why questions
• relational studies that "examine relationships [or associations] between two or more factors"
• experimental studies that establish a causal link between an intervention and outcomes in a randomized trial.

The book, which is focused on design rather than mathematics, is designed to be accessible, organized in response to common questions asked of the authors. They especially encourage readers to process it with a research question in mind. The book is rich with examples and reviews of published studies, inviting readers to reason the choices behind them.

this summary currently covers only the first half of the book. Expect the second half later in the Spring of 2015! Natematias (talk) 18:54, 10 March 2015 (UTC)

What Are Your Questions?

When people start a research project, they're often starting with a theme-- a dilemma to investigate--rather than a specific research question. In this chapter, the authors describe the process of working from a theme to a question that can guide "systematic planning of research."

• articulate a set of specific research questions
• understand the link between research questions and methodology
• learn from the work of others and refine your research questions accordingly

A good resrearch question will:

• identify the target population
• determine the appropriate level of aggregation (individuals, classes, schools?)
• identify outcome variables (perception, objective measures, behavioural data...etc)
• identify the key predictors of that outcome
• determine how much research control is required, and what kind of study is appropriate (e.g. descriptive, relaitonal, experimental)
• identify background characteristics that might be related to the outcome (factors to control for, sources of bias, etc)
• raise challenges for measurement and data collection (what standard measures exist, and what will required to carry them out)

In brief, "research questions determine every facet of research design... today's naive choices may have dire consequences tomorrow."

The chapter includes an extremely helpful extended section on how to do a literature review for the methods and findings of other studies, whether doing a narrative review or a meta-analysis, producing quantitative results from a sample of papers addressing a question.

The chapter also includes a helpful section on understanding causal versus correlational claims. To establish a causal link, one must show:

• a change in the predictor produces a change in the outcome
• there is no plausible alternative explanation (no rival predictor)
• offer a mechanism to explain the change, grounded in theory
• replicate the findings

What Groups Do You Want to Study?

This chapter offers a practical guide to defining the population and sample for a study, in a way that allows the findings to be generalized to a "wide range of persons, places, and times" while still staying focused on the substance of the research:

• inclusion criteria: a rationale for identifying individuals eligible to be included
• exclusion criteria: a rationale for who to exclude
• (sometimes) expected effect size, sacrificing some generalizability in favor of choosing a population with greater need or potential outcomes
• feasibility of working with the target population
• multi-site or single-site

After defining these qualities, the chapter describes the process of selecting a sample from the population, from simple random samples to stratified samples (grouping by a particular characteristic that you want equal or proportional representation from). It describes linked and unlinked samples in cases where you might wish to study both students and teachers in a particular context. The chapter concludes with methods to overcome "nonresponse bias" introduced by the varied response rates among people in the population.

What Predictors Do You Want to Study?

This chapter explains why we might want to look at "the connection between a particular predictor (gender, school grades, membership in a mentoring programand an outcome (success in math, satisfaction with college, graduation)." The chapter aims to support studies that:

• "Acknowledge that different types of predictors require different strategies for detecting their effects"
• "Rule out rival explanations for observed relationships between predictors and outcomes"
• "Maximize variation in the predictors. The greater the variation in the predictors, the more likely that your study will be able to detect a relationship between predictor and outcome"
• "Consider statistical interactions among predictors"

The chapter offers a taxonomy of predictor types, whether "membership in an innovation" (did you get the treatment) or attributes of a respondent, among continuous and categorical predictors. The idea of covariate predictors is introduced: "when you include covariates in your analysis, you are trying to distinguish the variation in the outcome that is attributable to the covariates from the variation that is explainable by the key substantive predictors."

Later in the chapter, the authors discuss the importance of ensuring the integrity or standardization of the treatment in a randomized experiment and practices that can be used to monitor the implementaiton of the treatment.

Creating Comparison Groups

In this chapter, the authors outline basic principles of randomized experiments:

• "Why you need comparison groups"
• "The advantages of randomizatoin"
• Choosing alternatives when randomization is not possible

The chapter starts out by arguing for the value of comparison group studies where some participants receive a treatment and others do not. They argue that "because studies with no comparison groups are so weak, you should avoid conducting them at all costs."

The chapter includes an extended discussion of randomization, explaining why randomization is used to create comparison groups, whether random assignment is actually feasible, and common types of random assignment. Here, the focus of the book on higher education is extremely revealing, since it introduces the challenges of the level of randomization (within a class, within a major, within a university?), selection bias (where the students who sign up for a study might be similar in a way that weakens the study), and others. In general, the authors advise to "randomize at the lowest possible level... people, not intact groups."

The chapter also mentions challenges of bias in studies, including volunteer biases, sampling challenges introduced by requiring informed consent, and biases associated with working with groups rather than individuals.

The chapter includes a set of suggestions for creating comparison groups without random assignment, such as historical comparisons (looking at previous years), random samples of nonparticipants, stratified samples of nonparticipants, matched comparison groups, or people who volunteered but were not selected. The chapter concludes by discussing "retrospective case-control studies" that look at infrequent occurrences after they happen. The authors discourage this kind of research in favor of studies that follow a large sample over many years and wait for those rare occurrences to happen.

Theoretical and Practical Relevance

This book was profoundly helpful to me when I was moving from work as a software engineer and computer scientist towards more work that focused on experimental and statistical research. I highly recommend it as a companion to a methods or statistics class." Natematias (talk) 18:31, 10 March 2015 (UTC)