Andrew Gelman links to this nice paper by Nosek, Spies and Motel, about an exciting “result” in psychological research: instead of rushing to publish, they scrupulously rushed to replicate, and the result disappeared. The fairy tale ending is that they got a nice publication from using this experience to tell us what we already know – that “significant” results obtained from small, ad hoc experimental samples are pretty much worthless. Continue reading
You have to love that title, which comes from a paper by Christopher Ferguson and Moritz Heene, which the excellent Andrew Gelman parses, and passes on to the rest of us. Any field that uses statistics is susceptible to publication bias (i.e., not publishing statistical analyses that find “no effect”). It is notorious in pharmaceutical research, where money talks shouts. I am guessing that the reason psychology gets a particularly bad reputation for publication bias, compared with other social sciences, is that it deals with a lot of small experimental data sets – so you really do have a situation where nearly identical experiments can be run twenty times by different researchers, and the one that gets a significant effect gets published. Statistical work in economics and political science tends to keep re-using a small number of mostly public data sets, so the problems are different.
Frederick Guy 