Selection Bias in Online Polls: Twitter vs. Cognitive Daily

There was an uproarious SmackDown on Twitter between @brucewagner and @davemunger over the issue of polling the percentage of gay/homosexual tweeters/blog readers.

You see, there's this small issue called selection bias that prevents reputable pollsters from asking a question in this way: How gay is Twitter? ;) Are you gay?

selection bias

Selection bias comes in two flavors: (1) self-selection of individuals to participate in an activity or survey, or as a subject in an experimental study; (2) selection of samples or studies by researchers to support a particular hypothesis.

Was Bruce actually serious that his Are you gay? poll was completely unbiased?



Yes, that's right, only 54% of all Twitterers self-identified as straight in this fabulous poll, with 28% calling themselves gay and the rest in between. Off just a little?

Sexual Behavior and Selected Health Measures: Men and Women 15-44 Years of Age, United States, 2002

Sexual orientation

In response to a question that asked, “Do you think of yourself as heterosexual, homosexual, bisexual, or something else?” 90 percent of men 18-44 years of age responded that they think of themselves as heterosexual, 2.3 percent of men answered homosexual, 1.8 percent bisexual, 3.9 percent “something else,” and 1.8 percent did not answer the question. Percents for women were similar.

[See Alkaline Earth for more.]

Given the wildly inaccurate Twitter numbers, Dave Munger wanted to prove a point by posting his own Are you homosexual? poll to the readers of Cognitive Daily.


He then compared these results to an earlier "Casual Friday" poll, which embedded a query about same- vs. opposite-sex partners in a series of questions about romantic gifts:

Selection bias and homosexuality

A couple hours ago I posted a quick poll, in what might be construed as an unbiased fashion. I simply asked respondents for their sexual orientation, offering a wide array of choices ranging from "straight" to "mostly gay" to "gay" to "other."

In fact, my poll was biased -- not because the question itself was slanted, but because of the way respondents were recruited: I titled the post "Are you homosexual?" Potential respondents who are homosexual or who don't have traditional sexual preferences are more likely to be interested in the question, and therefore more likely to respond. How do I know this biased the sample? Because I collected similar data last week in the Casual Fridays survey about romantic gifts. In that survey, women reported same-gender partners 5.7 percent of the time, and men reported same-gender partners 3.7 percent of the time.

However, some commenters noted that the Romantic Gift poll was not free of bias, either. Heterosexual partnered individuals were perhaps more likely to answer questions on whether jewelry, perfume, chocolate, flowers, lingerie, etc. constituted romantic gifts or not.

The moral of the story?



Do not give sexy nightwear or sex toys to women on the first date...