# Significance level

The significance level, denoted by alpha (α), determines the **critical level in a hypothesis test**. If the test statistics for the new finding falls beyond this critical value, the null hypothesis is rejected, and the finding is thereby qualified as *significant*.

Communities of statisticians prefer applying estimates as p-values and confidence intervals rather than hypothesis tests with “falsely” set significance levels.

## The significance level (α) = the critical value

In statistics the significance level (α) is also called the **critical value**. It states the limit for where to distinguish whether a new finding can be qualified as *significant* or *not* in the density curve.

If the new finding falls beyond the critical value, it is qualified as significant and the null hypothesis can then be **rejected**:

A p-value of 0.015 can lead to “no-go” (fail to reject), whereas a p-value of 0.014 can lead to a “go” (reject).

**Analysts from the statisticians and scientific communities state their reasons** why they prefer to make inference based on the p-value and not even apply hypothesis testing or at least not base their conclusion on a “falsely” setup significance level. Confidence intervals can be held up together with the p-value to backup a conclusion.

## Risk of error

Statistical analyses typically work with significance levels of 0.1; 0.05 and 0.01. A significance level of 0.05 means that there is a 5% chance (or risk) that the analyst will reject the null hypothesis when it is actually true (a so-called *‘Type I Error’*. Ref: Power of test, Type I & II Errors)

So, a significance level of 0.05 gives a **5% risk that the analyst will commit the error of rejecting a true hypothesis**.

**The lower the significance level, the greater the proof must be **in order to support the alternative hypothesis. Because, the lower alpha the more towards the tail of the curve.

**As an example, the pharmaceutical industry** typically works with α level of 0.01.

## Debate & ethics

As described in p-value there is an ongoing debate on how to apply the significance level and the p-value. Summarizing this debate, part of the statistician community state that it gives a more nuanced picture to involve the p-value in the inference and not only reject or fail to reject a hypothesis test.

## α is set prior to the calculation

There is an ongoing debate about whether p-value should be applied rather than the significance level (α). The significance level is a threshold that we set up for ourselves prior to the calculation of the test statistics.

Say that we are about to run a hypothesis test and that we set alpha to 0.05. We essentially say: “We will reject the null hypothesis for values with p-values below 0.05”. Now, say that we calculate a p-value of 0.056. This p-value is extremely close to the critical value of 0.05 and yet, it is within the non-rejection area. We, therefore, might feel tempted to raise alpha a bit in order to obtain support for our (and “treasured”) alternative hypothesis.

A p-value of 0.01 can lead to “no-go” (fail to reject), whereas a p-value of 0.001 smaller at 0.009, can lead to a “go” (reject). So small differences that lead to completely opposite decisions.

Therefore, the significance level is set prior to calculating the test statistics. **For ethical reasons. To avoid manipulation**.

*But, isn’t the significance level a “false” *

**threshold**

*and a way of cheating ourselves?*We reject or fail to reject based sometimes on extremely small differences down to a millesimal. We are choosing between two opposite conclusions: ‘Reject’ or ‘fail to reject’. We choose between “go” and “no-go” based on a millesimal.

## Learning statistics

- Khan Academy video: Comparing P-values to different significance levels

#### Carsten Grube

Freelance Data Analyst

##### Normal distribution

##### Confidence intervals

##### Simple linear regression, fundamentals

##### Two-sample inference

##### ANOVA & the F-distribution

+34 616 71 29 85

Call me

Spain: Ctra. 404, km 2, 29100 Coín, Malaga

...........

Denmark: c/o Musvitvej 4, 3660 Stenløse

**Drop me a line**

*What are you working on just now? Can I help you, and can you help me? *

**About me**

Learning statistics. Doing statistics. Freelance since 2005. Dane. Living in Spain. With my Spanish wife and two children.

**Connect with me**

**What they say**

20 years in sales, analysis, journalism and startups. See what my customers and partners say about me.

## 0 Comments