2015-02-10

Rough Guide to Spotting Bad Science

A ROUGH GUIDE TO SPOTTING BAD SCIENCE 1. SENSATIONALISED HEADLINES Headlines of articles are commonly designed to entice viewers into clicking on and reading the article. At best, they over-simplify the findings of research. At worst, they sensationalise and misrepresent them. 7. UNREPRESENTATIVE SAMPLES In human trials, researchers will try to select individuals that are representative of a larger population. If the sample is different from the population as a whole, then the conclusions may well also be different. 2. MISINTERPRETED RESULTS News articles sometimes distort or misinterpret the findings of research for the sake of a good story, intentionally or otherwise. If possible, try to read the original research, rather than relying on the article based on it for information. 3. CONFLICT OF INTERESTS Many companies employ scientists to carry out and publish research - whilst this does not necessarily invalidate research, it should be analysed with this in mind. Research can also be misrepresented for personal or financial gain. 8. NO CONTROL GROUP USED In clinical trials, results from test subjects should be compared to a 'control group' not given the substance being tested. Groups should also be allocated randomly. In general experiments, a control test should be used where all variables are controlled. 9. NO BLIND TESTING USED To prevent any bias, subjects should not know if they are in the test or the control group. In double-blind testing, even researchers don't know which group subjects are in until after testing. Note, blind testing isn't always feasible, or ethical. 10. 'CHERRY-PICKED' RESULTS This involves selecting data from experiments which supports the conclusion of the research, whilst ignoring those that do not. If a research paper draws conclusions from a selection of its results, not all, it may be cherry-picking 4. CORRELATION & CAUSATION Be wary of confusion of correlation & causation. Correlation between two variables doesn't automatically mean one causes the other. Global warming has increased since the 1800s, and pirate numbers decreased, but lack of pirates doesn't cause global warming. 5. SPECULATIVE LANGUAGE Speculations from research are just that speculation. Be on the look out for words such as 'may, 'could', 'might, and others, as it is unlikely the research provides hard evidence for any conclusions they precede. 6. SAMPLE SIZE TOO SMALL In trials, the smaller a sample size, the lower the confidence in the results from that sample. Conclusions drawn should be considered with this in mind, though in some cases small samples are unavoidable. It may be cause for suspicion if a large sample was possible but avoided. 11. UNREPLICABLE RESULTS Results should be replicable by independent research, and tested over a wide range conditions (where possible) to ensure they are generalisable. Extraordinary claims require extraordinary evidence - that is, much more than one independent study! 12. JOURNALS & CITATIONS Research published to major journals will have undergone a review process, but can still be flawed, so should still be evaluated with these points in mind. Similarly, large numbers of citations do not always indicate that research is highly regarded. 2014 COMPOUND INTEREST - WWW.COMPOUNDCHEM.COM

Click to enbiggen

A Rough Guide to Spotting Bad Science:

The vast majority of people will get their science news from online news site articles, and rarely delve into the research that the article is based on. Personally, I think it’s therefore important that people are capable of spotting bad scientific methods, or realising when articles are being economical with the conclusions drawn from research, and that’s what this graphic aims to do. Note that this is not a comprehensive overview, nor is it implied that the presence of one of the points noted automatically means that the research should be disregarded. This is merely intended to provide a rough guide to things to be alert to when either reading science articles or evaluating research.

I don't have comments but I love email or you can find me on Mastodon.

Last updated 9/2/2024
Site Design by Beardy Guy Creative