The bitter partisanship of the Obama era leaves some pundits and commentators longing for political transcendence, for a land of harmony and cooperation in which Americans work together to achieve common purposes. Such longings for political and cultural unity surface whenever partisanship threatens to paralyze government and bickering brushes aside reasonable discourse and debate. Just as James Davison Hunter's analysis of America's deep divides, Culture Wars (1991), provoked a range of counter-arguments asserting that America was "one nation after all," so the current drumbeat of political gridlock and paralysis kindles hopes that it just isn't so, that most Americans quietly inhabit a middle-American cultural space between small but vocal extremes.
NBC News and Esquire Magazine are only the latest in a long line of purveyors of this hope, a hope that Washington’s bitter polarization bubbles up from an American melting pot much sweeter to the taste. They ground this hope not in America’s history of economic, technological, and military accomplishment, nor in convictions about American pragmatism or exceptionalism. Instead, it rests on their faith in the tools of survey research, specifically in a survey conducted on their behalf by the Benenson Strategy Group. This is how NBC’s Today Show host Matt Lauer and NBC Political Director Chuck Todd described their survey’s “startling” discovery on NBC’s Today Show:
Matt Lauer: ... We often talk about the political center, and the common wisdom has been that there is 40 percent on the right, 40 percent on the left, 20 percent somewhere in the middle, and maybe they're wishy washy about things. We did a survey with Esquire Magazine — we meaning NBC News — and it’s startling what they found.Chuck Todd: It is. A majority of the country is in the middle. Fifty-one percent is in the middle. The wings—the dominant part in Washington—is in between the 25-yard lines (21 percent on the left, 28 percent on the right), but the vast majority of the country—a majority of the country—is in between the 25-yard lines.
Matt Lauer: And when you look at the people in this 51 percent... It is the center, and they’re not wishy-washy on politics. They have very strong views.
Chuck Todd: ... Some of them are supposedly “liberal.” Some of them are supposedly “conservative.” You know what they are? They’re pragmatic. This is people who live their everyday lives and look at politics through their own lives. They don’t look at it with a blue jersey or a red jersey.
NBC News Senior Staff Writer Tony Dokoupil summarizes the survey findings this way: “At the Center of national sentiment, there’s no longer a chasm but a common ground where a diverse and growing majority—51 percent—is bound by a surprising set of shared ideas.”
Discounting the fact that only a slim majority of this New American Center self-identifies as political moderates (the others calling themselves either liberals or conservatives), Dokoupil assures us that they nonetheless inhabit the center.
This raises a big question: How did the Benenson Strategy Group (BSR) decide that respondents who don’t claim to be moderate in fact are? The answer lies in a statistical technique called cluster analysis, which is commonly employed by survey researchers to identify systematic patterns of difference among groups that were only vaguely imagined prior to the survey. While cluster analysis can sometimes identify sub-groups that have tremendous interpretive value, in most cases the survey researcher must specify the number of buckets (or groups) in advance before the cluster analysis can sort respondents into them. That is, had BSR specified 2 buckets, they would have found that Americans are sorted into 2 contrasting clusters with no one in the center. Had they specified 5 buckets, they might have concluded that only 20 percent were in the center (one of the five buckets), or perhaps 55 to 65 percent (three of the five). Instead, they settled on an 8-bucket solution with approximately equal numbers of Americans (ranging from 10 to 14 percent) in each bucket. Had respondents been distributed evenly among the eight, which could easily have been the case, Chuck Todd would have reported that half (50 percent) of Americans, rather than the “vast majority” (51percent) are in the center. And if even 1 percent fewer, 49 percent, had fallen into the central four buckets, he might have concluded that the vast majority were at the extremes. In a word, a completely different conclusion would have resulted from a 2 percent shift.
But such was not the case. They did find 51 percent in the central four buckets, didn’t they? Yes, but to give this any interpretive weight is to place greater faith in cluster analysis than would statisticians themselves. Statisticians realize that even a slight modification in the survey questions employed could yield different distributions into the eight buckets, easily shifting the 51 percent in the purported “center” to 44 percent or 58 percent. What is more, skilled data analysts know that if you specify 8 buckets and feed the cluster analysis purely random data generated by cats tiptoeing on computer keyboards—where there are no meaningful clusters at all—it will still sort “respondents” fairly evenly into eight buckets, just as they were in this research.
So how might the Benenson Group (and NBC News / Esquire), with the same data, have discovered a majority at the extremes instead?
1. Re-run the analysis, reporting three-, five-, or seven-group cluster solutions rather than eight.
2. Re-run the analysis, basing the cluster solution upon a different subset of questions from the same survey.
3. Keep the same analysis, but reclassify one of their existing groups—“Minivan Moderates,” for example—as liberal. After all, two-thirds of Minivan Moderates voted for Obama, 41 percent say they are Democrats, 68 percent say they are pro-choice, and 70 percent strongly agree that gay people should be able to marry just like anyone else. Had Minivan Moderates been classified as liberals, NBC/Esquire would have reported that only 38 percent of Americans, slightly over a third, are in the Center. The important point here is that Minivan Moderates are in the Center because the Benenson Strategy Group says they are, not because Minivan Moderates make that claim.
Beyond the exaggerated claims for their cluster analysis solution, there are problems with the survey itself. Rather than using a standard probability sampling procedure, for example, in which Americans from all walks of life are systematically sampled to achieve representative results, Benenson employed an opt-in approach, drawing from lists supplied by professional sampling organizations. No margins of error are reported for the New American Center study because their sampling procedures fail to meet the criteria for such claims. Even if I accepted their cluster analysis at face value, which I do not, the study’s authors would not be in a position to say whether 51 plus or minus 3 percent are the center, or 51 plus or minus twenty. What is more, since the survey was privately commissioned by the very media outlets that are reporting it, we are told that the raw data are private and unavailable, so there can be no testing or replication of their analysis by independent scholars. Additionally, rather than deciding in advance which survey questions were most meaningful for sorting Americans in groups, they used nearly every question in the survey.
Bottom line? The lauded American Center recently touted as a startling discovery by NBC News was based on an opt-in sample for which margins of error cannot be calculated. It used a shotgun— “throw everything in and see what comes out”—approach to question selection. And it relied on a trial-balloon approach involving alternative solutions—of seven, eight, nine buckets, etc.—to find one that seemed most interpretable. Based on the result, a claim was made that the “vast majority” of Americans (51 percent) now constitutes an emerging Center. Because the data are private, independent scholars cannot verify their claim.
As a university-based scholar who strongly believes in the interpretative value of cluster analysis and also in the value of national surveys conforming to the standards of the Council of American Survey Research Organization (CASRO) and the American Association of Public Opinion Researchers (AAPOR), I am disappointed to see the depths to which NBC News, and survey research in particular, has fallen in this case. The study should have been called the “Imagined American Center” or “Desired American Center” rather than the “New American Center.” It would have been less misleading.
Cultural and political polarization are complex phenomena that do not emerge overnight. At the Institute for Advanced Studies in Culture, we have explored such realities for decades. (See for example this monograph from 2006 and this Hedgehog article from 2010.) Other university-based and independent survey organizations (most notably the Pew Research Center for the People and the Press) have also monitored the ebb and flow of American "centers" and ""extremes." NBC News and Esquire Magazine would do well to study such sources seriously before giving so much air-play to a study with so many methodological limitations. And consumers of such reports should stop listening the moment they hear 51 percent referred to as a "vast majority."