Our Polling Director on the Pentagon Survey – and Criticism Of It
EmailI asked our Polling Director, Gary Langer, to weigh in on the Pentagon survey of troops about gays and lesbians serving openly in the military – and the criticism of it from the advocacy group Servicemembers United. (See earlier blog post today HERE.)
Langer says that “the use of agree/disagree questions is poor practice given the well-documented problems of acquiescence bias. Some other questions are hypothetical or ask respondents to guess what others think or would do, which likewise is not a good way to go. Other than these, the rest seem reasonable; phrasing is neutral and response categories are balanced. Characterizations of policy are always open to debate.”
Langer says that it is interesting that the Pentagon uses “gay and lesbian” and “homosexual” interchangeably. “Given questions and concerns about this language, which are hardly new, it would have been best if they had half-sampled each question.”
He disagrees with Servicemembers United, however, in its assertion that it’s a “well-established fact that the use of the term ‘homosexual” induces bias in survey research.”
Not true, Langer says. “It is a subject of discussion and current investigation in the field, with the most recent research indicating no effect, or no substantial effect. The subject was treated at length in a recent paper by a pair of graduate students in sociology at New York University: ‘We report no difference in support when respondents are asked about “gay,” “same-sex” or “homosexual” marriage and civil unions.’ They did find substantial differences in the strength, rather than overall level of such views; and said this effect was concentrated among middle-aged and older respondents.”
ABC News’ test of the language, last year, Langer says, “found perhaps a slight effect, but within the margin of sampling error for our survey, with a more clearly negative reaction to the term ‘homosexual’ among seniors. In spring ’09, we asked a random half of our respondents if it should be legal or illegal for ‘homosexual couples’ to get married; they divided 46-48 percent.
“We asked the other half about ‘gay or lesbian couples’; they divided 51-44 percent. Given our sample sizes the differences are not significant at the usual 95 percent confidence level.
“Directionally they do suggest that ‘homosexual’ could be somewhat pejorative, though certainly not broadly so; people asked about ‘gays and lesbians’ were 5 points more apt to support their being able to marry. Since this wasn’t replicated in studies cited by the NYU paper, it needs further study.”
- Jake Tapper
Biden Reflects on Grief, Suicidal Thoughts 
Is Congress Sounding Dumber?
There was a copy of the Pentagon survey yesterday with this report, that has been curiously removed. I skimmed through it and found the question about showering to be the most inflammatory one. So as usual, a snippet of news gets emphasized for sensationalism.
Posted by: jane | July 10, 2010, 11:52 am 11:52 am
I answered this in my comment on the post below. I believe the point SU raised here is unwarranted. Further, the question about the showers is only “inflammatory” to people who have no connection to the military. These are real situations, where real humans have to make decisions, large and small, in the best interests of their units.
The survey is worded to gather information, and was written to a certain group — military servicemen, many of whom have spent an entire decade of their lives living in cramped quarters with no facilities, fighting, dying and depending on their brothers to stay alive. It was not worded to appease or soothe the presumptive prejudices of a political action group who’s greatest concern is their own funding.
To a soldier, these questions are blunt, no-nonsense requests for information based on real, every day military activities. This survey was written to elicit accurate and meaningful information from SOLDIERS, not political advocates and ideological agitators with the trembling sensitivities of a teacup poodle.
Posted by: phalanx | July 10, 2010, 12:40 pm 12:40 pm
Jake – You really need to dig deeper here. The leadership at the Pentagon is counting on the mainstream media to do as little investigation into their polling methods and allow this whole scheme to follow through to its completion.
It shouldn’t be a surprise to anyone that the troops survey falls short of the mark required to elicit any substantive responses helpful in the implementation of DADT repeal. Tony Perkins, the noted anti-gay religious activist, has made public statements on his Family Research Council blog that he is participating in discussions at the Pentagon with the Defense Secretary’s Comprehensive Review Working Group and helping that group uncover “problems” with open service. Meanwhile, none of the advocates for repeal have a seat at the table with the DOD’s working group, and they have every right to be indignant when details of the survey show complete lack of thought towards any bias or unscientific approaches in the polling methods.
The recently leaked DOD survey is only one of many insults that gay and lesbian servicemembers have faced during this review process. If the mainstream media looks the other way while the forces of intolerance continue to pull the levers inside DOD, then tens of thousands of gay and lesbian servicemembers will continue to serve without the dignity that should be afforded anyone who is willing to sacrifice their life to serve their country.
Among other questions I would be asking about this survey:
1) Why the even distribution among surveys sent to active duty and reservists? While the two components are nearly equal in strength, the reserve component doesn’t have nearly the same career experiences as the active duty component. Yet, demographically, the reserve component skews much higher in terms of age and are located in more rural parts of the country where intolerance may be more common. Does giving the reserve component and equal voice in the survey allow the survey to skew its demographics to an older and more conservative age group?
2) Why was the survey only distributed via e-mail? Access to computers is not universal in the military and the youngest and most junior members of the service are not likely to have the same access to an online survey as those with more age and seniority. Bandwith constraints for those serving overseas complicate the issue of getting survey results backs as well. By choosing an electronic-only method of delivery, the Pentagon guarantees a survey sample that skews to an older demographic — one that is probably less comfortable with policy changes in general.
3) The Pentagon’s press release on the day of the survey start indicated that they used a methodology to pick the survey respondents — in other words, it was not a random sampling. What methodology was used? What criteria where used to determing this non-random sample?
4) Why did the Pentagon survey preface nearly every question with “If Don’t Ask Don’t Tell is repealed . . ” when the Defense Secretary has maintained for quite some time now that the purpose of the study group is to determine how to implement DADT repeal “when” and not “if?”
5) Whey does the basic premise of the survey assume that gay and lesbian service results in some necessary reaction from servicemembers? Every question about DADT repeal is phrased in a way to illicit some sort of response about how servicemembers will change their behavior or react to open gays and lesbians. How is this a neutral position for the survey? When foreign militaries lifted their gay bans, they didn’t experience any noticeable change at all in the behavior of their forces. Yet, our DOD produces a survey that presumes all sorts of uncomfortableness and reactions from heterosexual servicemembers reacting to gay and lesbian servicemembers. This is the most biased aspect of the survey by far.
6) Did anyone at DOD perform the basic test of replacing the terms gay, lesbian, or homosexual with an equivalent description of another group subject to discrimination (black, jew, women, latino, etc.) in order to at least test their questions to a standard that would not be inflammatory to groups subject to these sorts of indignities? Clearly, they did not. imagine if we polled servicemembers on how they would feel if a Jewish couple moved in next door to them in base housing what response they would have if an interracial couple began attending military social functions.
Posted by: Rich | July 10, 2010, 11:04 pm 11:04 pm
6) Did anyone at DOD perform the basic test of replacing the terms gay, lesbian, or homosexual with an equivalent description of another group subject to discrimination (black, jew, women, latino, etc.) in order to at least test their questions to a standard that would not be inflammatory to groups subject to these sorts of indignities?
===========
I said this on the other thread, but the reason the *question* singles out gays and lesbians in a way no other group would be is because the *policy* singles out gays and lesbians in a way no other group would be.
As discriminatory as it sounds when compared to other groups, it is the law of the land to single out gay people for different treatment in the US military.
It has been believed that this is best for the morale of the military, though I disagree with that.
However, it has not been convincing to the US military to simply assert it. They think the law is as it is for a reason other than to be discriminatory. They are concerned about what repeal of it will do to the military.
And so they are trying to find out.
Posted by: MayBee | July 11, 2010, 12:57 am 12:57 am
I’m not aware of a purpose having been clearly stated for the survey. It appears that it could either be used to determine if the military is ready for the repeal, or as a guide in implementing the new policy. I hope it’s not going to serve as the basis for whether the DADT repeal is acceptable or not by the Pentagon.
Posted by: jane | July 11, 2010, 12:37 pm 12:37 pm