The Experimenting Society

Citation: Donald Campbell (1998) The Experimenting Society. The Experimenting Society (RSS)
Internet Archive Scholar (search for fulltext): The Experimenting Society
Download: https://books.google.com/books?hl=en&lr=&id=TDuXdlxjdSsC&oi=fnd&pg=PA35&dq=campbell+the+experimenting+society&ots=dKg7GSD3KR&sig=Za7G69VLohZb3mUQrEVr4pZXo4Q#v=onepage&q=campbell%20the%20experimenting%20society&f=false
Tagged: Sociology (RSS) NatematiasGenerals (RSS), Evaluation (RSS), Political Philosophy (RSS), Policy Evaluation (RSS), Experimental Social Science (RSS), Karl Popper (RSS), Democracy (RSS)

Summary

In "The Experimenting Society," Donald Campbell asks, "Can the open society be an experimenting society?" Writing the first version of this essay in 1971 and updating it over time as people photocopied and distributed it, Campbell describes an experimenting society as one that would "vigorously try out possible solutions to recurrent problems and would make hard-headed, multidimensional evaluations of outcomes, and when the evaluation of one reform showed it to have been ineffective or harmful, would move on to try other alternatives."

The article sets up a series of reasons why the experimenting society and Popper's idea of the open society are not compatible. Then, Campbell discusses possible political and methodological ways to bring them together.

The Ideology of the Experimenting Society

Campbell outlines the main characteristics of an experimenting society:

• "It will be an active society preferring exploratory innovation to inaction" (38). Campbell argues that "it will be committed to action research, to action as research rather than research as a postponement of action" (39) Here, Campbell cites Lewin, Etzioni, and Dunn, among others.
• "It will be a scientific society" characterized by "scientific values of honesty, open criticism, experimentation, willingness to change once-advanced theories in the face of experimental and other evidence" (41). Campbell contrasts this attitude from the use of scientific findings to govern in ways that are "dogmatic, non-experimental" and "non-scientific" (41). Campbell complains that "economists, operations researchers, and mathematical decision theorists trustingly extrapolate from past science and conjecture, but in general fail to use the implemented decisions to correct or expand that knowledge" (42).
• "It will be an accountable, challengeable, due-process society. Campbell advocates for open access to evaluation records and "recounts, audits, reanalyses, and reinterpretations of results" through "competitive criticism possible at the level of social experimentation." He also imagines that "citizens not a part of the governmental bureaucracy will have the means to communicate with their fellow citizens disagreements with official analyses and to propose alternative experiments," characteristics that make the experimenting society an open society in Popper's sense (42).
• "It will be a decentralized society." Campbell suggests that "through autonomy or deliberate diversification, different administrative units will try out different ameliorative innovations and will cross-validate those discoveries as they borrow from others" (42). This semi-competitive situation with independent groups will allow replication and verification of findings elsewhere, Campbell argues.
• "It will be a society committed to means-idealism as well as ends-idealism." By this, Campbell suggests that society committed to improvement will accept that it never reaches the "ends," and that all societal states are just "transitional steps" (42).
• "It will be a popularly responsive society, whose goals and means are determined by collective good and popular preference." Campbell argues that "it will be a voluntaristic society, providing for individual participation and consent at all decision levels possible" (42)

The Social Scientist as Servant of the Experimenting Society

In this section, Campbell returns to ideas he set out in the 1970s (Campbell 1973), arguing that social scientists should take a passive role in politics, committing themselves to the "modesty of the physical sciences" and "more often say that we can't know until we've tried." Campbell encourages social scientists to "avoid cloaking their recommendations in a specious pseudoscientific certainty and instead acknowledge their advice as consisting of wise conjectures that need to be tested in implementation" (46).

Methodological Problems

In this section, Campbell describes the problem of administrators who closely align their political careers with the success or failure of a single idea. To avoid supporting this state of affairs, Campbell strongly encourages researchers to focus on evaluating programs rather than people and to test policies as a way to guide decisions elsewhere rather than evaluate that particular policy implementation.

The Use of Experimentation in the Experimenting Society

In this section, Campbell outlines the value of causal inference to policy evaluation, focusing on randomized trials. He argues that experiments "present special moral problems":

• experiments select people from a population and often force participation in the treatment. Consequently, they might be accused as "an authoritarian, paternalistic imposition, treating citizens as passive recipients rather than co-agents directing their own society... rather than as collegial agents of the experiment" (Campbell, 49, summarizing Janousek, 1970)
• "The enforcement of assigned treatments also violates the egalitarian and voluntaristic ideals of the experimenting society. The disguised experiment violates these too and, in addition, the values of openness, honesty, and accountability" (49).

Campbell considers several ways to address these wide-ranging problems:

• informing control and treatment groups about an experiment in advance
• adopting quasi-experimental designs
• mutual criticism and "competitive cross-validation" between different evaluation groups, in several ways: (citing Campbell, 1986)
  • "contagious cross-validation" where local communities test their own ideas and also evaluate other communities' ideas when they consider adopting them. Campbell imagines a network of local communities who might be able to organize to generate "100 locally interpretable experiments" in 5 years, alongside o"a community of applied social scientists familiar with them all, that had cross-examined each others' data, suggested and done reanalyses" (52). Campbell suggests that "from the consensus of this mutually monitoring research community we would advise government and potential adopters" (52).
  • Competitive replication within national pilot studies could be done by splitting policy evaluation in to smaller units, including "adversarial stake-holder participation in the design of each pilot experiment or program evaluation and again in the interpretation of results" (Krause and Howard 1976, Bryk 1983). Campbell suggests that results be subject to "competitive reanalysis," and that "dissenting-opinion research reports" be encouraged and supported through free access to experimental data and community norms that welcome re-analysis. Campbell suggests that the results of experiments should be communicated back to participants, who he describes as "co-owners" of the findings, and who "would be allowed to use these results in political debates" (55).
• Using different measures to evaluate programs as the ones used to evaluate people, since institutions will often adjust their behavior (including reporting behavior) to meet those targets: "nail factories that overproduced large spikes when the quota was set by tonnage, and overproduced small nails when the quota was set by number of items turned out" (55). Campbell also argues that the US military metrics of body counts "created an immoral and irrelevant military goal in Vietnam" (56). Campbell expects that in an experimenting society "social indicators will be used more than they are at present, and the corruption pressures will thus be greater" (56). For this reason, Campbell distances himself from the "accountability movement", writing that "I end up opposing the use of quantitative indicators for achieving managerial control" since it can "create more evils than it cures." In its place, Campbell suggests "the temporary use of quantitative measures in evaluating alternative programs" (56) and using "multiple indicators of the same problem, each of the indicators being recognized as imperfect, but so chosen as to have different imperfections" (56).
• "Legitimating and facilitating evaluation by nonprofessional participants and observers" (57). Campbell does not expect that these methods will not be statistical. When results disagree, "we should remember that the statistical analyses involve simplifying assumptions that may be seriously in error" (57). Campbell argues that "it is those who have situation-specific information who make the best critics, and the best judges, of the plausibility of most of the rival hypotheses in their specific setting... in this process we must provide these nonprofessional observers with the self-confidence and opportunity to publicly disagree with the conclusions of the professional applied social scientists" (58).
• "Long-term followup" to study the eventual outcomes of an intervention. Campbell offers a review of the methodological, administrative, and political challenges of setting up longitudinal research.
• Grappling with the fact of experimentation as "Normal rather than extraordinary or revolutionary science" (63). Experiments are tools for incremental change rather than revolutionary change. Furthermore, revolutions tend to destroy the systems of measurement that make comparison possible. But Campbell does suggest that revolutionary governments might wish to test their own policies once a revolution has concluded.

Campbell concludes by suggesting "organized skepticism" toward the idea of an experimenting society, encouraging others to address the problems he listed "before we can wholeheartedly advocate for an experimenting society" (65).

References of Note

• Bryk, A. S. (Ed.). (1983). Stakeholder-based evaluation. Jossey-Bass.
• Campbell, D. T. (1973). The social scientist as methodological servant of the experimenting society. Policy Studies Journal, 2(1), 72-75.
• Campbell, D. T. (1986). Science’s social system of validity-enhancing collective belief change and the problems of the social sciences. Metatheory in social science: Pluralisms and subjectivities, 108-135. University of Chicago Press
• Campbell, D. T. (1974). Qualitative knowing in action research. In The Social Context of Method, edited by M. Brenner, P. marsh, and M Brenner, 184-209. London: Croom Helm.
• Krause, M. S., & Howard, K. I. (1976). Program evaluation in the public interest: A new research methodology. Community Mental Health Journal, 12(3), 291-300.
• Janoušek, J. (1970). Comments on Campbell’s“ Reforms as experiments.” American Psychologist. 25(2), 191-93.
• Popper, K. S. (2012). The open society and its enemies. Routledge.
• Etzioni, A. (1968). The active society. Free Press.
• Dunn, E. S., & Dunn, E. S. (1971). Economic and social development; a process of social learning (No. 04; HM106, D8.).

Theoretical and Practical Relevance

In The Experimenting Society, Donald Campbell asks if a society that conducts social experiments to test policy can be an "open society" rather than a totalitarian one. In the article, he outlines the values of an "experimenting society" and offers a list of challenges for experimenters to address before it will be possible to evaluate the potential of an open, experimenting society. This article is a classic of policy evaluation, and the issues in the article continue to be important as measurement and experimentation become more common in society.