It’s almost flu season, which means flu-shot season. The New York Times suggests that a good mood can improve your immune response to the shot, giving you one more reason to be happy to head to the pharmacy. Too bad this feel-good headline is only backed up by some sketchy statistics.
The Headline: An Upbeat Mood May Boost Your Flu Shot’s Effectiveness
The Story: Flu shots are not 100% effective—though let me say this up top: if you are able to get one, you absolutely should—and scientists don’t know all of the variables that influence that efficacy. They do know that age is a factor, for example: for young adults, the flu vaccine is 70 to 90 percent effective, but for older adults it drops to 17 to 53 percent. So the authors of this study decided to test the impact of a host of behavioral and psychological factors on the vaccine’s effectiveness in older patients.
In this study, 122 older adults recorded their diets, sleep patterns, exercise habits, stress, and mood for the two weeks before and 16 weeks after they got their flu shots. They were tested for flu antibodies at four and 16 weeks after vaccination, as a measure of the vaccine’s effectiveness.
This is where the statistical jiu-jitsu comes into play. The researchers had a heap of data to compare—on the one hand, the vaccine’s effectiveness at two time points; on the other, allllllll of those lifestyle factors—to look for trends and correlations. There’s lots of math under the hood here to identify results that are “statistically significant,” meaning they don’t look like the result of randomness or luck.
The researchers found that the correlation between positive mood and vaccine effectiveness was statistically significant. However, they didn’t correct for the fact that they were testing almost a dozen possible influences on the outcome. That’s a big problem for their findings. The more comparisons you make, the more likely it is that one will register as statistically significant just by random sampling error.
Rebecca Goldin, Professor of Mathematical Sciences at George Mason University, and Director of STATS at Sense About Science, USA, put it this way:
In this study, the authors conducted so many comparisons that some kind of smoking gun was likely to appear, regardless of whether there’s an actual relationship or not. […] This study does not provide statistical evidence either for, or against, the scientific hypothesis that mood impacts vaccine efficacy.
The authors acknowledge this in the study: “Due to the exploratory nature of these analyses and a desire not to increase type-2 errors, no corrections for multiple comparisons were applied.” They’re right that if they’d corrected for multiple comparisons, they risked type-2 errors, or false negatives—but the compromise is that their method allowed for false positives. There was no way to come to a solid conclusion from a study with so many comparisons. Without a more focused follow-up, we really don’t know anything at all.
The Takeaway: The authors of this study told the New York Times that “they were not able to control for all possible variables, and that their observational study does not prove cause and effect.” Yeah, it really doesn’t prove anything. Being in a good mood is great, and so are flu shots, but as far as a connection between the two, that’s about as far as it goes based on current data.