The p-value stands for probability value. It expresses the probability of getting a statistic at least as extreme as the one we are calculating for. Say we have an assumed mean (µ) of 10 and a sample mean (x̄) of 12. The p-value will express the probability of getting at least 12 assuming that the mean is 10?
The p-value in hypothesis tests
The p-value is typically applied in hypothesis testing (also called significance testing). When conducting a hypothesis test, we pre-define a significance level (α). The p-value is now applied to determine whether to reject or fail to reject the null hypothesis.
Say, we set α = 0.05. Then:
- if we get a p-value below 0.05, we will reject the null hypothesis.
- if we get a p-value greater than or equal to 0.05, we will fail to reject the null hypothesis
- P-value < α => Reject H0
- P-value ≥ α => Fail to reject H0
When the p-value < α
When the p-value is below α, we will state that there is evidence enough to support our alternative hypothesis, and we reject the null hypothesis. The values that return p-values below α are also determined as significant.
Let’s take the example from above with an assumed mean (µ) = 10 and a sample mean of (x̄) 12. Say this returns a p-value of 0.03. This means that there is 3% chance that we would get 12 or more assuming that the mean is 10.
Prior to the conducting of our hypothesis test we had set the significance level (α) to 0.05 and as 0.03 < 0.05, we will reject the null hypothesis. Thus, we conclude that there is sufficient evidence in order to prove our alternative hypothesis stating that the mean (µ) is greater than 10.
We can also say that 12 is significant with a p-value of 0.03, and we therefore reject the null hypothesis.
Had we set α = 0.01 in this case, we would have failed to reject as 0.03 > 0.01. The conclusion would have been that there is sufficient evidence in to support our alternative hypothesis test at α = 0.01.
When the p-value ≥ α
When the p-value is greater than or equal to our significance level (α), we will state that there is no evidence to support our alternative hypothesis, and we reject the null hypothesis. The result is determined as not significant.
Visualizing the p-value
In this graph the p-value is visualized together with the critical x-value (typically a sample mean) and the critical z-value and the alpha:
Debate on the p-value
There is an ongoing debate about the use of p-value and hypothesis testing for inferential statistics.
In hypothesis testing the significance level is a threshold that we set up for ourselves prior to the calculation of the test statistics.
Say that we are about to run a hypothesis test and that we set α = 0.05. We essentially say: “We will reject the null hypothesis for values that return p-values below 0.05”. But, if we get a p-value of 0.051, extremely close to the significance level of 0.05, we will failing to reject H0.
Had the p-value been just 0.002 less, we would have rejected H0. Such an extremely small difference, in this case of 0.002, lead to the opposite desiction. The two only decisions we can take are reject and fail to reject. And they, obviously, lead to opposite conclusions. We determine that a result is not significant, when it is only 0.002 from being “significant”.
So, part of the debate on this lack of nuanced concluding in hypothesis tests. Professionals from scientific environment state the p-value should be the foundation for inference and not only on the hypotheses and the related significance level.
P-value in Excel
Some of my preferred learning materials on the p-value:
- Nic’s Maths & Stats: Understanding the p-value
- Jbstatistics: What is the P-value?
- Khan Academy video (7:57): P-values and significance tests
Freelance Data Analyst
+34 616 71 29 85
Spain: Ctra. 404, km 2, 29100 Coín, Malaga
Denmark: c/o Musvitvej 4, 3660 Stenløse
Drop me a line
What are you working on just now? Can I help you, and can you help me?
Learning statistics. Doing statistics. Freelance since 2005. Dane. Living in Spain. With my Spanish wife and two children.
Connect with me
What they say
20 years in sales, analysis, journalism and startups. See what my customers and partners say about me.