Ask ten trainees what a p value means and you will get ten slightly different answers — which is why statistics questions feel like free marks lost. If you have been searching for what does p value mean statistics, here is the short version: the p value is the probability of observing data at least as extreme as yours, assuming the null hypothesis is true. Getting that one sentence exactly right is the difference between a passed and a failed stats section, and it is what this guide will fix.
We will cover null hypothesis significance testing (NHST) as examiners actually frame it, why p < 0.05 became the default threshold, the things a p value is definitively not, the two error types and statistical power, and how effect sizes and confidence intervals complete the picture — with a mini study you can interpret end-to-end and a table of exam statements corrected.
The NHST framework in one pass
Every significance test starts with two competing hypotheses. The null hypothesis (H0) states there is no effect or no difference — the new antidepressant is no better than placebo. The alternative hypothesis (H1) states there is an effect. You then ask: if H0 were true, how surprising would my observed data be?
The p value answers exactly that question and nothing else: P(data this extreme or more | H0 true). It is a probability about data given a hypothesis, never a probability about the hypothesis given the data. If p lands below your pre-specified alpha (conventionally 0.05), you "reject the null." That threshold is a historical convention — the medical research community settled on 5% in the mid-20th century — not a law of nature, and it is worth remembering that it was chosen to be stringent enough to limit false positives, not because 5% is magically special.
Two practical consequences examiners love. First, the p value depends on sample size: with n = 5,000, a trivial difference becomes "statistically significant," while a large clinically meaningful effect in n = 12 does not. Second, you must decide tails and significance level before seeing the data; choosing the test after inspecting the numbers is how analyses become unreliable.
What does p value mean statistics questions really test?
Most lost marks are not calculation errors — they are misinterpretations. A p value is:
- Not P(H0 is true | data). It does not tell you the probability the null hypothesis is correct. That would require a prior probability on H0, which NHST never uses.
- Not the probability the result is due to chance/luck. It does not say "there is a 3% chance this happened by luck."
- Not an effect size. p tells you nothing about how large or clinically important the difference is. Significance is a function of effect size and sample size together.
- Not a measure of study quality. A p = 0.04 from a badly randomised, unblinded trial is still a badly randomised, unblinded trial.
- Not proof of replication. p just below 0.05 from a single study carries a meaningful false-positive risk; this is part of why replication has been a central concern in the open-science literature (see, e.g., work indexed on PubMed).
Exam trap: "p = 0.03 means a 3% chance due to luck"
This is the single most reliably tested misreading in psychiatry statistics questions. p = 0.03 means: if the null hypothesis were true, data this extreme or more would occur 3% of the time. It is NOT the probability that the null is true, NOT the probability the finding is a fluke, and NOT the probability of replicating. When you see an option phrased as "the treatment worked" or "there is only a 3% chance the result is due to chance," mark it wrong. Correct options are phrased as conditional-on-H0 probability statements about the data.
Type I and type II errors, and power
Because a decision is made on incomplete data, two errors are possible:
- Type I error (α, alpha): rejecting a true null hypothesis — a false positive. You conclude the drug works when it does not. The Type I rate is your alpha level: at α = 0.05, roughly 1 in 20 such tests will be falsely positive.
- Type II error (β, beta): failing to reject a false null hypothesis — a false negative. You conclude there is no difference when there genuinely is one.
Statistical power = 1 − β, the probability of detecting a true effect when one exists. The conventional target is power ≥ 0.80 (a ≤ 20% false-negative chance). Power rises with three things: larger effect, larger sample size, and lower variability. The classic trade-off: lowering alpha to reduce Type I errors increases Type II errors unless you compensate by increasing n. A useful exam heuristic — Type I and Type II errors move in opposite directions when you change the significance threshold or the sample size.
Effect sizes and confidence intervals complete the picture
Significance tells you a difference probably exists; clinical importance needs an effect size — Cohen's d (standardised mean difference: 0.2 small, 0.5 medium, 0.8 large), risk ratios, odds ratios, or number-needed-to-treat. A trial can achieve p < 0.001 for a difference of 0.5 mmHg in blood pressure: real, and irrelevant.
The 95% confidence interval is the more informative quantity and is increasingly what exam questions push you toward. Interpretation: the range of values compatible with the data such that, repeated indefinitely, 95% of such intervals would contain the true parameter. The reading rules worth memorising:
- Interval excludes the null (e.g. does not include 1.0 for a ratio, or 0 for a difference) → statistically significant at the 0.05 level.
- Interval includes the null → not significant, whatever the p value says.
- Wide interval → imprecision, usually a small sample; the study is inconclusive rather than negative.
- Interval that excludes the null but is clinically trivial → statistically significant, clinically unimportant.
Mini study: interpreting all of it together
A trial of a new SSRI versus placebo reports: mean HAM-D reduction 2.1 points greater, 95% CI 0.4 to 3.8, p = 0.016, n = 320, powered at 0.85. Read it properly: the null (no difference) is excluded, so the result is statistically significant at α = 0.05; the effect is small (a 2-point HAM-D difference is generally below the conventional threshold for clinically important change); and the CI, while excluding zero, is fairly wide, so the true benefit could plausibly be modest or near the upper end. "Statistically significant, clinically small, reasonably precise" — that is a four-part interpretation an examiner will expect you to give from just the numbers.
Exam statement check: wrong versus right
Every statement below has appeared in some form in a research-methods paper — the Preppsy concept library walks through the full critical-appraisal framework behind them.
| Statement in a question | Verdict | Correct interpretation |
|---|---|---|
| "p = 0.03, so there is a 3% chance the result is due to chance" | Wrong | P(data this extreme | H0 true) = 0.03; not a probability about H0 |
| "p = 0.03 means the null hypothesis has a 3% probability of being true" | Wrong | NHST never assigns probabilities to hypotheses |
| "p > 0.05 means the two groups are equal" | Wrong | Failure to reject H0 — possibly underpowered, not proof of no difference |
| "p < 0.05 means the finding is clinically important" | Wrong | Clinical importance requires effect size and CI context |
| "The 95% CI for the difference excludes 0, so the result is significant" | Correct | Consistent with p < 0.05; report the interval for precision |
| "A study with power 0.80 has a 20% chance of missing a true effect" | Correct | Power = 1 − β; β = 0.20 is the Type II error rate |
| "A larger sample reduces Type II error but not Type I error" | Correct | α is fixed by convention; power rises with n |
Frequently asked questions
What does p value mean statistics exams actually test?
It means that if the null hypothesis were true, data this extreme or more would occur 5% of the time — which crosses your pre-set threshold, so you reject the null. It is a statement about how surprising the data are under one specific assumption, not a statement that there is a 95% chance the treatment works or only a 5% chance the result is a fluke.
Why is p = 0.06 not meaningfully different from p = 0.04?
Because 0.05 is a convention, not a cliff. Two p values either side of the threshold provide nearly identical evidence; treating one as proof and the other as no effect is the "threshold fallacy." This is one reason journals and statisticians increasingly push reporting of effect sizes and confidence intervals rather than dichotomous significant/non-significant verdicts.
How do you increase the power of a study?
Four levers: increase the sample size, increase the true effect size you are targeting (choose a population where the effect should be larger), reduce measurement variability (tighter protocols, validated scales), and use a more efficient design — paired measurements, covariate adjustment, or reducing attrition. Increasing alpha raises power too, but inflates the Type I error rate, which is rarely acceptable.
What is the difference between statistical significance and clinical significance?
Statistical significance says the observed difference is unlikely under the null hypothesis — it is about compatibility with chance given sample size. Clinical significance says the difference matters to patients: it crosses a meaningful threshold on a validated scale, changes management, or produces a worthwhile number-needed-to-treat. A study can be statistically significant with a clinically trivial effect, and clinically meaningful effects can be non-significant in small studies.
Key takeaways
- p = P(data this extreme or more | H0 true). Memorise the conditional form — it resolves most MCQ distractors.
- A p value is never P(H0 true), never "probability due to chance," and never an effect size.
- Type I error is a false positive at rate α; Type II error is a false negative at rate β; power = 1 − β, conventionally ≥ 0.80.
- Alpha, beta, and power trade against each other — lowering alpha without raising n increases false negatives.
- Always read the confidence interval: if it excludes the null, the result is significant; its width tells you precision.
- Report effect size alongside p — significance is about detection, not importance.
Statistics underpins the whole critical-appraisal component of the exams, so it pays to drill it properly — try a Preppsy mock exam to see how research-methods items are actually phrased.
Cover image: “Twittergrams: Morning Tweet Graph” by blprnt_van (BY) via flickr.
