4. Hypothesis testing: the null, t and the p-value

Level 7: Rigorous Statistical Diagnostics

In practice we do not just want a number: we want answers to concrete questions. Does income affect spending? Do more years of education translate into a higher salary? Does my strategy have positive alpha, or is what I see just noise? Hypothesis tests use the information in the sample to answer yes-or-no questions, though never with absolute certainty.

The null hypothesis. It is denoted H0 and in a regression it is stated as: the coefficient of interest equals a constant c. When we test it we can reject it or fail to reject it. The null is presumed true until the data provide enough evidence against it. And here is the point most abused in reports: failing to reject the null does not mean the null is true. A test does not determine which hypothesis is true or which is more likely; it only evaluates whether the available evidence is enough to reject the null.

The alternative hypothesis. It is denoted H1 and takes three forms: the coefficient is greater than c, smaller than c, or simply different from c. Economic theory usually suggests the sign: if theory says food spending rises when income rises, we test that the income coefficient is zero against the alternative that it is greater than zero. We always speak in terms of the null: we reject or fail to reject; we never accept the null. If we reject it, we adopt the alternative.

The test statistic. It is computed assuming the null is true, because under the null we know its distribution. For the coefficient of a simple regression:

# t = (estimated coefficient - c) / standard error of the coefficient
# Under the null it follows a t distribution with N - 2 degrees of freedom
# (in general, N minus the number of estimated parameters)
t = (model.params['MKT'] - 1) / model.bse['MKT']   # tests beta = 1

If the null is not true, the statistic does not follow that t distribution but a different one, which is why it tends to land on extreme values.

Rejection regions and the significance level. The rejection region is made up of the values that are unlikely to occur when the null is true, and it depends on how we specified the alternative. Its size is set by the significance level alpha, the probability of that unlikely event, usually 0.01, 0.05 or 0.1. If the computed statistic falls in the rejection region, it becomes hard to believe the null holds.

Type I and type II errors. A type I error consists of rejecting a null that is true; its probability is exactly alpha, and therefore we control it. A type II error consists of failing to reject a null that is false; its probability cannot be computed directly because it depends on the unknown parameter, but we know that lowering alpha raises the probability of committing a type II error. Type I errors are considered more serious, hence the recommendation to set a small alpha.

The p-value. It is the probability, under the null, of obtaining a result equal to or more extreme than the one observed. Having it saves you from comparing against tabulated critical values: the rule is simple, if the p-value is less than or equal to alpha we reject the null, and if it is greater we do not.

model.pvalues