Turned up to eleven: Fair and Balanced

Wednesday, February 13, 2002

The discussion I have been having on Megan McArdle's site has sparked some thoughts on another difficulty of social science research. As mentioned below, methodology poses a substantial challenge in this type of research, but another, more insidious problem emerges in discussion of results. The biases of any observer are much stronger on social issues than, say, "hard science". Certainly any scientist who has spent years in the field will have a favorite theory about how it works, and some will be incredibly dogmatic in making sure that their theory comes out on top, regardless of the evidence. Yet, in most cases, flatly contradictory evidence will result in the theory being discarded (no one rejects Einstein in favor of Newton, even though Newton's laws are still used and relied upon in many, many instances, but everyone acknowledges that Einstein was correct, because the physical evidence is simply overwhelming). Social science issues are more "fuzzy", however. Not only that, but virtually everyone involved has a deeply held opinion, based on their own (anecdotal) experience. My discussion on Megan's site has illustrated that. Many people pointed out their own personal experience, citing that to back up their claim that even though the Horowitz poll was bad (as I have discussed here before), the results were still true. Funny, I think that I have heard this before, and I like Instapundit's response. In either case (guns or academic bias), there are two ways to approach the question. One is "I know I am right, but I need some data to prove it." This is very, very dangerous, as the temptation to "fudge" may be too great for most to withstand. The other is "I think I know what is going on, but I want to know for sure." There is still the danger that the investigator, or reader, may be to beholden to self interest or previous opinion, but I think this is the most plausible route to good social science. The danger of reader bias, however, is just a great. As I mentioned above, my experience on Megan's blog illustrates to me that people are loathe to give up ideas that they feel comfortable with, and quick to embrace shoddy data that agrees with their experience. I hope that, collectively, we can learn from each other.