Is a statistical test letting significance slip through the cracks?
Every scientist is familiar with the p-value: it’s one of the most commonly used metrics in statistics to evaluate the likeliness that an observed relationship is due to chance. Typically, a cutoff is set at p=0.05, such that any p-value of greater than 0.05 means the result is deemed “not statistically significant”—a heartbreaking outcome for so many researchers.
The p-value has a massive impact on how data is interpreted, from whether others think following up on a result is worthwhile to whether it’s published at all—so it’s vital to get it right. In some areas of genetics, the Sequence Kernel Association Test (SKAT) is commonly used to calculate p-values. The test’s low computational cost feeds into its popularity, but it can’t be used in all situations: SKAT is unreliable for small datasets, and correcting for this issue is computationally demanding. In work reported in GENETICS, researchers developed a new, faster method to solve that problem and, along the way, identified previously unknown conditions under which the test breaks down.
The group found that SKAT can even fail when applied to some large datasets, resulting in extremely skewed p-values that could lead researchers to falsely conclude that their findings are not statistically significant. Using a large dataset from previous research on relationships between blood lipids and chemical modifications to DNA, they found that their new test, RL-SKAT, identified almost 40 times more statistically significant relationships than the traditional SKAT analysis did.
These potential issues with typical SKAT analysis could mean important associations are being ignored. The researchers have made the code for RL-SKAT freely available online so others can investigate this issue further—and maybe even find out whether they have some interesting results collecting dust.
Schweiger, R.; Weissbrod, O.; Rahmani, E.; Müller-Nurasyid, M.; Kunze, S.; Gieger, C.; Waldenberger, M.; Rosset, S.; Halperin, E. RL-SKAT: An Exact and Efficient Score Test for Heritability and Set Tests.
GENETICS, 207(4), 1275-1283.