How reading Discworld makes you a better researcher

At what point does bad research turn into propaganda? Over the last week I have have had the misfortune of reading two pieces of work which might qualify as such. One was a piece of research which purported to show differences in outcomes and rates of improvement of outcomes for patients belonging to different ethnic groups. The design was flawed and it was poorly written up, but more fundamentally the conclusion, that people from minority ethnic groups did worse than whites, was flatly contradicted by the results. In fact, the results showed little variation, with what variation there was being greater between minority ethnic groups than between BME groups and whites. Indeed, the frustrating thing was that there were some generally interesting anomalies which were worthy of further consideration which were missed because of the authors' desire to show institutional racism.

A still worse piece of research was lauded by a representative of the US right on another blog which quoted the Commonwealth Fund's figures on international expenditure of health services. This was produced by Heritage Foundation, a well known US conservative think tank. Its thesis was that multi-purchaser health system (such as the US one) were inherently superior to single purchaser ones such as Canada and the UK.

I need to declare an interest and state that I don't think that this is true: notably because increased transaction costs seem inevitable, and because it is harder to maintain a public health focus in commissioning decisions in a multi-payer system. However, my problem was with the approach taken in arguing. In short there was neither a coherent conceptual framework nor a convincing evidence based comparison presented to justify the position. What was available was a list of failings noted in single payer systems, with no consideration of either how these were caused by the single payer system or that they were not present in multi-payer systems. Of course without this the research becomes little more than a write up of a google search for “problems single payer health systems”.

Why does this sort of stuff get produced and published? It's tempting to allege incompetence and dishonesty. But I tend to a more subtle explanation. I think what is going on is that the researcher holds mental model which acts as a prism through which evidence is interpreted, and crucially which discourages against objective consideration of the evidence. Hence, “health services are institutionally racist and here is evidence that proves it”, or “socialized medicine is inferior – look at all the problems it has”. To an extent we all hold mental models and interpret what we see in the light of them, but the key to avoiding seems to me bound up in the old quote from (I think) Keynes,
“When the facts change I change my mind. What do you do?”

Or more pertinently from Sam Vimes,
"Never overlook the possibility that you might be completely wrong"