Learning from Mistakes: Howard S. Becker’s “Evidence”

But works that represent themselves as instructional texts serve as a means for notable scholars to take positions in important disciplinary debates; often, the position taken in such a book is a notably clear, short, and constructive statement of the approach that has informed the author’s own distinguished career. Evidence is Becker’s fifth book on the conduct of social research. For Becker’s exemplars, the moment of confrontation with error is succeeded by a readiness to do whatever is necessary to get better evidence. Unsurprisingly, Wallin and Waldo found that children gave incomplete or confused answers to this question. Evidence is, in many respects, a backward-looking work, which is both a weakness and a strength. I suspect these works also command the interest of scholars at all stages of their careers. On the surface, Becker’s examples illustrate creative approaches to evidentiary problems. Becker has undoubtedly played a significant role in giving this literature a humane bent. In fairness, one probably shouldn’t look to an 89-year-old scholar for a comprehensive overview of the contemporary state of the field, but for wisdom and a sense of disciplinary history, and these Becker has in spades. In the pages of current sociology journals, one may find many illustrations of the thoughtful concern with evidence that Becker counsels: how to study the gaps between what people do and what they tell researchers they do; how to tell the difference between routine and deliberate forms of cognition; how to produce numerical evidence when engaged in fieldwork or analysis of large bodies of written text; how to assure that quantitative studies report accurate findings that can be verified or replicated by other researchers; how to draw conclusions from research participants recruited from services like Mechanical Turk; and how to make ethically responsible and social scientifically valuable use of the great mass of user-generated and network information commonly called “big data.” Evidence mentions none of this. JUNE 25, 2017
EVERY ACADEMIC DISCIPLINE creates a body of didactic literature: handbooks and textbooks devoted to expressing first principles and recommending practical methodologies. Indeed, he is most critical when discussing his own work, recounting how his failure to scrutinize his own political beliefs and professional experiences blinded him to important questions arising during his research: “possessed by my prejudices, I didn’t pursue those paths.” This self-criticism is also offered in support of his view of social science as a collective enterprise, for he discusses other researchers who eventually asked and answered the questions he missed. The question built in what now appear as very troublesome assumptions about gender and household structure; it further supposed that middle-schoolers were willing and able to give accurate descriptions of the world of adult work, and that researchers could use a simple job title as a proxy for complex matters such as household income and parental educational attainments. Other recent books in the theoretical-didactic genre address themselves wholly to the problem of making valid inferences in qualitative research, or how sociological ideas are formed. Although it is true that census responses about race and ethnicity simplify (and in some ways oversimplify) a complex reality, I am not certain this means “we can’t use data consisting of someone’s ethnic or racial self-identification […] as evidence of anything else about them.” Similarly, medical examiners’ reticence to classify police- or physician-involved deaths as homicides is a matter of both sociological significance and serious political concern, but this does not obviously imply that “this specialized subgroup of deaths reveals what makes the medical examiners’ official conclusions useless for research purposes.” But these are extreme formulations of generally sound advice: infer with caution. The examples rarely suggest that getting better evidence is easy. The book opens with a discussion of how, in the 1960s, the Stanford sociologists Paul Wallin and Leslie Waldo confronted problems arising from what now seems a glaring example of bad evidence: the once-common device of measuring the social class of children by asking them what job their “real father” did. As a junior faculty member, I find myself reading these books more often than I did in graduate school, when an earful of advice was always readily available. The book proceeds through many other examples: police reports of drug crimes, accusations of witchcraft in Oaxaca, exaggerated attendance of union meetings, manifestations of social status in tuberculosis sanitaria, and evolving preferences in baby names, to name only a few. His most-cited work, Outsiders, is at once a sympathetic study of groups labeled as “deviant” in the mid-20th-century United States, and an adept demonstration that deviant groups operate in much the same way as other groups. This is perhaps the most important piece of advice offered by the book, and one at the core of Becker’s own work: by remaining mindful of how data is produced, it is possible to “turn ‘technical problems’ into researchable questions.”
Evidence is structured around real examples from sociological research. He counsels researchers not to lie to themselves about the nature of their evidence, and to remain mindful of the possibility that they are relying on information that confuses labels or interpretations of things for things themselves. Surprisingly, they found that nobody else had written about problems with this frequently used survey question. If read in isolation, one could easily fail to be aware that Evidence is as much a description of an emerging consensus as it is a proposal for disciplinary change. Deviance, Becker emphasized, was not a fact of the activities he studied, but an artifact of what others called it. His cycle of books has treated social research in much the same way as he treats the rest of the world: as a process, in this case one involving the continuous interplay of data (observations about the world), evidence (data transformed or deployed to support a claim about the world), and ideas (a term that covers a great deal, including general social theories, formal hypotheses, researchers’ intuitions, and pointed questions). After recognizing the problem, they openly acknowledge it, itself a step that many researchers refuse to take, instead preferring to ignore the problem, exorcise it through methodological rites, or take quiet comfort in the knowledge that many other researchers are committing the same error. Though he himself is a living, active, and much-read scholar, many of his peers have left behind unjustly neglected work which he generously recovers here. In particular he singles out for praise the work of many talented women from midcentury who have received less credit than they deserve, including Helen McGill Hughes, who was an uncredited co-author in Everett Hughes’s study of industrialization in Quebec; Blanche Geer’s undervalued role as Becker’s co-investigator in research on student culture; and Lois DeFleur’s remarkable multi-method study of racial bias in drug arrests by the Chicago Police Department. Contemporary advice for qualitative researchers often urges them to think about problems of sampling, not because researchers should ritually emulate statistical work, but because quantitative research design has valuable lessons to impart. In sociology, Howard S. Becker presents weaknesses or errors in evidence, and the ways that researchers honestly acknowledged these issues, then found something interesting by making the evidentiary problem the object of study. To describe his orientation crudely, Becker is inclined to think about the social world in terms of processes rather than outcomes, and is ever mindful of the difference between what things are and what labels are appended to them. His frequent use of examples from the 1940s and 1950s (his own formative years in the discipline) also serves as a welcome reminder that a comfortable intermingling of methods is not a novelty for sociology. Becker’s proposed approach to the problem is a mix of curiosity and vigilance. His examples tell a similar story over and over. Such work is often nominally addressed to graduate students in the dismal intermediate years of doctoral education. Becker urges researchers to steer directly into sources of error: matters that first present themselves as data limitations — ambiguous survey questions, subjects’ unreliable recollections of their behavior, self-serving practices of bureaucratic recordkeeping — are, for the creative researcher, likely to be unusually interesting and fruitful topics for study. At times, Becker’s conclusions about data limitations are put rather too strongly. First, researchers recognize an error or weakness in their work, or in an entire program of research. That book offers a short, much-read example of his approach to the social world as a process: his study of how one learns to enjoy smoking marijuana, in his view an acquired and essentially social skill rather than a physiological state. The book addresses itself to topics that are part of active and notably productive debates in sociology, but less than a 10th of the bibliography refers to English-language scholarship published in the last decade. In many cases, the solution is to spend substantially more time on the research, use an entirely different method, or enlist the help of others grounded in different research traditions: “we get further, collectively,” he writes, “by recognizing the multiple ways we can advance knowledge in our field.”
Becker’s discussion takes a constructive rather than negative approach: he spends much more time lauding thoughtful research than criticizing scholars for failures of attention. “Mixed methods” is now a stock phrase in sociology, and many new scholars are encouraged to learn and use a variety of research techniques so that their training will not limit their ability to answer important questions. Evidence, as the title suggests, concerns itself with just one segment of the social research process: the hazardous business of turning observations of the world into claims about the world. At a deeper level, though, they are homilies: the discussion of how researchers addressed unexpected problems is an argument for personal virtue as much as it is for intellectual rigor. As a researcher, he has produced an enormous body of carefully observed, undogmatic, and sometimes startling work. Rather, it is a book about how to think about evidence as one is producing it. The book’s position is that the responsible use of evidence is a permanent problem for social science, one that demands the constant attention of researchers; the end of developing good evidence can never be achieved purely by procedural or technical means, but requires that we “use the deficiencies in our own way of working as sources of ideas about how to improve.” Evidence is thus not a methods book, and indeed rejects the very idea that there is a useful distinction between qualitative and quantitative methods. As the discipline works past the methodological divisions of the latter part of the 20th century, attending to the discipline’s history may help sociologists heed another of Becker’s maxims: “don’t make the same mistake twice.”
Ben Merriman — a sociologist by training — is an assistant professor at the School of Public Affairs & Administration at the University of Kansas.