The significance level, denoted by alpha (α), determines the critical level in a hypothesis test. If the test statistics for the new finding falls beyond this critical value, the null hypothesis is rejected, and the finding is thereby qualified as significant.
Communities of statisticians prefer applying estimates as p-values and confidence intervals rather than hypothesis tests with “falsely” set significance levels.
The significance level (α) = the critical value
In statistics the significance level (α) is also called the critical value. It states the limit for where to distinguish whether a new finding can be qualified as significant or not in the density curve.
If the new finding falls beyond the critical value, it is qualified as significant and the null hypothesis can then be rejected:
A p-value of 0.015 can lead to “no-go” (fail to reject), whereas a p-value of 0.014 can lead to a “go” (reject).
Analysts from the statisticians and scientific communities state their reasons why they prefer to make inference based on the p-value and not even apply hypothesis testing or at least not base their conclusion on a “falsely” setup significance level. Confidence intervals can be held up together with the p-value to backup a conclusion.
Risk of error
Statistical analyses typically work with significance levels of 0.1; 0.05 and 0.01. A significance level of 0.05 means that there is a 5% chance (or risk) that the analyst will reject the null hypothesis when it is actually true (a so-called ‘Type I Error’. Ref: Power of test, Type I & II Errors)
So, a significance level of 0.05 gives a 5% risk that the analyst will commit the error of rejecting a true hypothesis.
The lower the significance level, the greater the proof must be in order to support the alternative hypothesis. Because, the lower alpha the more towards the tail of the curve.
As an example, the pharmaceutical industry typically works with α level of 0.01.
Debate & ethics
As described in p-value there is an ongoing debate on how to apply the significance level and the p-value. Summarizing this debate, part of the statistician community state that it gives a more nuanced picture to involve the p-value in the inference and not only reject or fail to reject a hypothesis test.
α is set prior to the calculation
There is an ongoing debate about whether p-value should be applied rather than the significance level (α). The significance level is a threshold that we set up for ourselves prior to the calculation of the test statistics.
Say that we are about to run a hypothesis test and that we set alpha to 0.05. We essentially say: “We will reject the null hypothesis for values with p-values below 0.05”. Now, say that we calculate a p-value of 0.056. This p-value is extremely close to the critical value of 0.05 and yet, it is within the non-rejection area. We, therefore, might feel tempted to raise alpha a bit in order to obtain support for our (and “treasured”) alternative hypothesis.
A p-value of 0.01 can lead to “no-go” (fail to reject), whereas a p-value of 0.001 smaller at 0.009, can lead to a “go” (reject). So small differences that lead to completely opposite decisions.
Therefore, the significance level is set prior to calculating the test statistics. For ethical reasons. To avoid manipulation.
But, isn’t the significance level a “false” threshold and a way of cheating ourselves? We reject or fail to reject based sometimes on extremely small differences down to a millesimal. We are choosing between two opposite conclusions: ‘Reject’ or ‘fail to reject’. We choose between “go” and “no-go” based on a millesimal.
- Khan Academy video: Comparing P-values to different significance levels
Freelance Data Analyst
+34 616 71 29 85
Spain: Ctra. 404, km 2, 29100 Coín, Malaga
Denmark: c/o Musvitvej 4, 3660 Stenløse
Drop me a line
What are you working on just now? Can I help you, and can you help me?
Learning statistics. Doing statistics. Freelance since 2005. Dane. Living in Spain. With my Spanish wife and two children.
Connect with me
What they say
20 years in sales, analysis, journalism and startups. See what my customers and partners say about me.