The p-value (probability value) is a central concept in the inferential statistical analysis of clinical trials. It indicates the probability with which the observed data, or results even more extreme, would occur if the null hypothesis were true. The p-value is therefore not a measure of the probability that the null hypothesis is correct, nor is it a measure of the clinical importance of an effect. These misunderstandings are widespread in clinical research and can lead to erroneous conclusions. A low p-value indicates that the observed data are unlikely under the null hypothesis. The p-value is not a direct measure of the magnitude of an effect and says nothing about whether a finding is practically relevant. These limitations are fundamental and must be considered in every interpretation.
Definition and interpretation
In classical frequentist statistics, the p-value is calculated within the framework of a significance test. A threshold of 0.05 is generally used as the significance level in clinical research. If the p-value lies below this threshold, the result is considered statistically significant and the null hypothesis is rejected. If it lies above the threshold, the null hypothesis cannot be rejected. A p-value of, for example, 0.03 means: if the null hypothesis were true, data like those observed would occur in 3 out of 100 random samples.
It is essential that statistical significance is not equivalent to clinical relevance. A trial with a very large sample size can produce statistically significant results even though the observed effect is clinically meaningless. Conversely, a clinically important difference may fail to reach statistical significance if the trial is underpowered. The p-value should therefore always be interpreted together with the confidence interval and the clinical context.
P-value and confidence interval
Confidence intervals provide more information than the p-value alone, as they make the precision of an estimate visible. A 95% confidence interval that does not include the null value corresponds to a two-sided p-value below 0.05. Regulators and scientific assessors generally expect both figures to be reported. The ICH E9 guideline on statistical principles for clinical trials explicitly emphasises that confidence intervals should preferably be reported alongside or instead of p-values.
More recently, the scientific literature has increasingly discussed the sole use of the p-value as a decision criterion critically. The American Statistical Association published statements in 2016 and 2019 calling for cautious interpretation of the p-value and recommending that statistical significance not be used as the sole criterion for scientific findings. This discussion has also found its way into regulatory guidelines.
Multiple testing problem
When several hypotheses are tested simultaneously in a trial, the probability of obtaining at least one false-positive result increases (alpha error inflation). This problem occurs in clinical trials with multiple primary endpoints, multiple treatment arms or subgroup analyses. To control alpha error inflation, correction procedures such as the Bonferroni correction, the Holm method or hierarchical testing procedures are used.
Regulators require that the handling of multiple testing be defined prospectively in the statistical analysis plan. This requirement applies both to pivotal trials and to studies for benefit assessment after market authorisation. Subsequent adjustments to the significance level or the testing procedure are considered a serious methodological flaw and can lead to the rejection of a marketing authorisation application. Full-service CROs such as mediconomics support sponsors in statistical study planning, the development of the statistical analysis plan, and regulatory communication on the testing procedure.
P-value in regulatory assessment
In authorisation procedures, the p-value of the primary endpoint is the central criterion for demonstrating efficacy. The EMA and FDA require a predefined primary endpoint, a predefined significance level, and a confirmatory analysis in accordance with the statistical analysis plan. Exploratory analyses and subgroup analyses provide valuable hypotheses for future trials, but are not considered proof of efficacy. The clear distinction between confirmatory and exploratory analyses is a fundamental prerequisite for the regulatory acceptance of study results.
In practice, it is important not to consider the p-value in isolation. It only provides a meaningful statement if the trial was correctly planned and conducted, the sample size was adequate, and the predefined question was analysed without subsequent adjustments. Trials conducted solely in search of a significant p-value (so-called p-hacking) do not provide reliable results. Reproducibility and transparency in data analysis are therefore essential prerequisites for a valid interpretation of the p-value. Pre-registration of trials and analysis plans in public databases is an important step in strengthening confidence in scientific results and safeguarding the integrity of clinical research.
Frequently asked questions (FAQ)
What does p < 0.05 mean in concrete terms?
A p-value below 0.05 means that, assuming the null hypothesis is true, the observed data would occur with a probability of less than 5%. It does not mean that the alternative hypothesis is true with 95% probability, nor that the effect is clinically meaningful. Interpretation always requires the clinical context and the reporting of effect size and confidence interval.
Can a non-significant p-value mean that no effect is present?
No. A p-value above 0.05 merely means that the data are insufficient to reject the null hypothesis. A true effect may be present that could not be demonstrated because the sample was too small (insufficient power). A lack of statistical significance is not proof of the absence of an effect.