Game 10 · Testing and estimation
Is this result real or chance?
P-values, confidence intervals and the null hypothesis.
Read with: Hypothesis testing77 DNB questions
Play
Read the result, decide if it is real
Read a confidence interval and say whether the result is significant, which way it points and roughly what p is. Predict how many studies of a useless drug will still be significant, then watch them run. Sort trials by statistical and clinical significance. You have three lives.
Loading the first round…
The p-value
What p = 0.04 tells you, and what it does not
The null hypothesis (H₀) states that there is no difference between the treatments in the population. The p-value is the probability of getting a result at least as extreme as yours if H₀ were true. p = 0.04 means: if the drugs were truly equal, a difference this large or larger would occur in about 4% of trials of this size.
The level of significance (α) is the false-positive rate you accept, fixed in the protocol before the data, usually 0.05. If p ≤ α, the result is called statistically significant.
Myth: p = 0.04 means a 4% chance that the null hypothesis is true.
The p-value is calculated assuming H₀ is true. It cannot give the probability that H₀ is true (ASA 2016, principle 2).
Myth: p = 0.04 means a 4% chance that the result is due to chance.
This is the same error in other words (ASA 2016, principle 2).
Myth: A smaller p-value means a bigger effect.
A p-value does not measure the size of an effect or the importance of a result (ASA 2016, principle 5).
Myth: p > 0.05 proves there is no difference.
Absence of evidence is not evidence of absence (Altman and Bland, 1995). Look at the confidence interval.
The six principles of the American Statistical Association's 2016 statement on p-values are listed in the Hypothesis testing chapter.
Confidence intervals
Read the interval against the line of no effect
A 95% confidence interval is a range calculated from the sample. If the same study were repeated many times, 95% of the intervals calculated this way would include the true value. Values inside the interval are compatible with the data.
| Feature | Difference (mean difference, risk difference) | Ratio (risk ratio, odds ratio, hazard ratio) |
|---|---|---|
| No effect | 0 | 1 |
| Significant at 5% | The 95% CI does not include 0 | The 95% CI does not include 1 |
| Direction | Which side of 0 the whole interval lies | Below 1 means fewer events with the new treatment |
| Scale | Plain | Logarithmic: the interval is symmetric around the estimate on the log scale |
You can work back from the interval to an approximate p-value (Altman and Bland, 2011):
SE = (upper limit − lower limit) ÷ (2 × 1.96) · z = estimate ÷ SE · p = 2 × (1 − Φ(|z|))
- SE
- standard error of the estimate
- z
- how many SEs the estimate lies from no effect
- Φ
- the area under the standard normal curve to the left of z
- ratios
- use the natural logs of the estimate and of both limits
A quick guide: z of 1.96 gives p = 0.05, z of 2.58 gives p = 0.01 and z of 3.29 gives p = 0.001. An interval whose end only just touches the line of no effect has p close to 0.05.
Chance
When nothing is going on, 1 study in 20 is still significant
If H₀ is true, every significant result is a type I error (a false positive). At α = 0.05 that happens in about 5% of studies: about 1 in 20. In the simulation the count of significant studies varies from run to run, but it settles near k × α.
P(at least one false positive in k independent tests) = 1 − (1 − α)k
With 20 independent tests at α = 0.05: 1 − 0.9520 = 0.64. This is why a study that tests many outcomes or subgroups will find something “significant” by chance, and why the primary outcome is fixed in advance. The Multiple comparisons chapter covers the Bonferroni correction and its alternatives.
Clinical significance
Significant is not the same as important
Statistical significance says a difference is unlikely to be due to chance alone. Clinical significance says it is large enough to change treatment. The second is judged by comparing the effect and its CI with the minimal clinically important difference (MCID), fixed before the study.
| Teaching trial (SD 12 mmHg, MCID 5) | Result | Reading |
|---|---|---|
| 5,000 per group | 1.0 (0.5 to 1.5), p < 0.001 | Significant, but too small to matter |
| 100 per group | 10.0 (6.7 to 13.3), p < 0.001 | Significant and clinically important |
| 10 per group | 10.0 (−1.3 to 21.3), p = 0.08 | Not significant: inconclusive |
| 2,000 per group | 0.5 (−0.2 to 1.2), p = 0.19 | Not significant, and rules out 5 mmHg |
The p-value alone cannot separate the first trial from the second, or the third from the fourth. The confidence interval can.
Writing the answer
How to write this in a 10-mark DNB answer
- Define the null and alternative hypotheses with a two-drug example, then define α and the p-value in one sentence each.
- Draw the 2 × 2 table of decision against truth: type I error (α), type II error (β), power (1 − β).
- Interpret one p-value in words, then list what it does not mean, citing the ASA 2016 statement.
- Define the 95% CI, give its formula (estimate ± 1.96 × SE) and draw two intervals against the line of no effect, one crossing it.
- State the link: a 95% CI that excludes 0 (or 1 for a ratio) means p < 0.05, and the CI also shows size and precision.
- End with statistical versus clinical significance, using the MCID and the four-trial example above.
The one thing to remember
A p-value says how surprising the data would be if nothing were going on. The confidence interval says how big the effect could be.
Past papers
DNB questions this game helps you answer
Questions from NBE theory papers for DNB, DrNB and Diploma courses, most often asked first. The tag after each question is the specialty and the year it was set.
- ×2Null hypothesis.
- ×2Confidence Interval.
- ×2Confidence interval
- ×1Chi square test, paired T test, triple blind study, p value. [2+2+2+1]
- ×1Define p-value and confidence interval.
- ×1Discuss about p-value, h index and citation.
- ×1P value and level of significance.
- ×1What is p-value? How do you interpret it?
- ×1What is the significance of P value?
- ×1Confidence interval and Forest Plot.
- ×1Discuss the different statistical measures to observe for test of significance related to parametric and non-parametric data.
- ×1Confidence Intervals.
- ×1Importance of confidence interval in clinical trials.
- ×1a) Confidence interval. b) Tests to evaluate normal distribution of data. c) Sample size calculation.
- ×1Define Null hypothesis and Alternate hypothesis respectively with special reference to their significance in clinical trials. Explain type I and type II errors with examples.
- ×1What is the significance of p value?
- ×195% confidence interval.
- ×1What are confidence intervals? How confidence intervals are useful in calculating equivalence in clinical trials.
- ×1Discuss different types of test of significance in research.
- ×1Tests of Significance.
- ×1Comment on p value.
- ×1Define p-value.
- ×1Describe p value, its calculation and relevance.
- ×1What is the p-value?
- ×1p Value.
- ×1Describe in brief the applications of Confidence interval, Analysis of Variance and Binomial distribution in Biostatistics.
- ×1Explain confidence interval.
- ×1How will confidence interval help to interpret the results?
- ×1Describe about statistical significance versus clinical significance with examples.
- ×1Null Hypothesis.
- ×1Statistical Significance Vs Clinical Significance.
- ×1What is hypothesis testing in research?
- ×1What is normal distribution? Describe the tests of significance which can be applied on normally distributed data.
- ×1What is null hypothesis? Discuss the steps for hypothesis testing.
- ×1Describe p-value, interim analysis and Forrest plot.
- ×1Significance of p-value.
- ×1Confidence Interval (CI) and Hazard Ratio (HR) in biostatistics.
- ×1Define Standard error. Give any 4 examples to illustrate its application in test of significance.
- ×1Describe Tests of statistical significance by examples.
- ×1Null hypothesis and Type I & Type II errors.
- ×1Confidence intervals.
- ×1Clinical studies: 7+3 clinical trials. b) Analysis of variance and statistical significance.
- ×1What are the Type I and Type II error of hypothesis testing?
- ×1What is p value?
- ×1What is confidence interval?
- ×1a) Systematic review. b) Confidence interval.
- ×1Non-Parametric tests of significance.
- ×1Parametric tests of significance.
- ×1Relevance of Null Hypothesis significance testing.
- ×1Test of significance in quantitative studies.
- ×1Tests for statistical significance.
- ×1a) Statistical significance of difference in means. b) The Chi-square test.
- ×1p-value and Null hypothesis- importance and interpretation.
- ×1Define confidence interval.
- ×1Concept of Standard Normal Distribution and its relation to the various tests of significance.
- ×1Confidence interval.
- ×1Null Hypothesis
- ×1Tests of significance.
- ×1Utility and limitations of non-parametric tests of significance.
- ×1Hypothesis tests: Alpha levels and p values
- ×1p-value
- ×1p-value.
- ×1How does confidence interval change ‘the sample size’?
- ×1What do the terms - odds ratio, hazard ratio, relative risk, absolute risk and confidence interval mean and where are these used?
- ×1a) Confidence interval. b) Methods of randomization.
- ×1Define ‘null hypothesis’.
- ×1Differentiate between parametric and non-parametric tests for statistical significance.
- ×1How to test sample size for a research project? How can one derive a value of statistical significance in a research project?
- ×1In a cross sectional study of the relationship between smoking and anxiety, 1000 people were simultaneously classified according to smoking status (500 smokers and 500 non-smokers) and current level of anxiety (high or low). 200 smokers had high anxiety level and 400 non smokers had low anxiety levels. a) Calculate appropriate rates and test of significance. b) Explain whether such association can be considered causal or not.
- ×1With reference to medical Statistics, define:- i) Null hypothesis ii) Power of a study.
- ×1What is the usefulness of confidence interval?
- ×1Differentiate between parametric and non parametric tests of significance.
- ×1Its clinical and statistical significance.
- ×1What is the difference between nominal and ordinal data? Describe student t test and its use. What is a case control study? What does P value signify?
- ×1How is the appropriate test of statistical significance chosen for any data? What statistical analysis is required to evaluate a new diagnostic test?
- ×1What are non-parametric tests of significance? Classify and enumerate these tests and briefly discuss non-parametric tests of significance for nominal data.
- ×1Clinical significance versus statistical significance.
Read next