QTA 6: Hypothesis Testing
The true values of population parameters sit out of reach, since nobody observes an entire population. A sample is what is available, and it carries real information about those parameters even though it cannot settle them. A hypothesis is a precise statement about one or more population parameters, and hypothesis testing asks whether such a statement sits comfortably alongside the data collected.
Frequentist inference is the framework used throughout, and it is the one most widely applied when relationships between variables are measured. Inside it every test collapses to a single question: if the hypothesis were true, how likely is the data that was observed? A test proves nothing. It weighs evidence against a stated position and reports whether that evidence is strong enough to abandon it.
The six components of any test
Six pieces have to be in place before a conclusion can be drawn, and they are assembled in a fixed order. The null hypothesis fixes the parameter value under examination and is treated as true throughout. The alternative hypothesis marks out the values at which the null should be abandoned. The test statistic summarises the sample and has a known distribution when the null holds. The size of the test sets how large the rejection region is, the critical value is the number the statistic is measured against, and the decision rule puts those two together and returns a verdict.
The null hypothesis, written H0, is a statement about the population value of one or more parameters. It is also called the maintained hypothesis, because it is held true from the first step to the last, and that assumption is load bearing: every probability computed along the way is derived from it.
A good null usually says that nothing remarkable is going on. For an investor weighing up a mutual fund, that position is that the fund produces no abnormal return, which becomes a statement that its expected return matches the expected return on its style-matched benchmark.
The same statement can be written as a difference, which puts the parameter under test at zero.
Turning a vague claim into a testable null
Some claims worth investigating are not statements about parameters at all, and those must be reworked first. Model validation is the standard case. The 95% VaR of a portfolio marks a region inside which the daily return should land 95% of the time, with the excluded region in the left tail so that it captures losses. That such a Value-at-Risk model is correct is not a numerical claim and cannot be tested directly.
What can be tested is a consequence of correctness. A correct 95% VaR should cover the actual loss on 95% of days, leaving it violated on the other 5%. Define an indicator taking the value one when the loss exceeds the VaR and zero otherwise, and call such a day a VaR exceedance. The expected value of that indicator is the parameter under test.
The alternative hypothesis
The alternative hypothesis, written H1, states which parameter values call for rejection, and in most work it is everything the null leaves out. For the mutual fund it is that the two expected returns differ; for the VaR model, that the expected exceedance rate is something other than 5%. Rejection there says only that the model is inaccurate: too few violations means it is too conservative, too many means it is too aggressive.
A two-sided alternative places the values that trigger rejection on both sides of the null, and that is the default, since the natural complement to an equality produces it. Sometimes only one direction is of interest, and the alternative then covers values above the null or below it, but not both. This is the one-sided alternative, which no longer complements the null.
Return to the fund manager, whose null is equal performance against the benchmark. A two-sided alternative tells the investor only that the fund return differs from it. If the question is whether the manager beats the benchmark, the alternative must be written in that direction.
Concentrating the whole rejection region in one tail raises the chance of rejecting a false null whenever the true parameter does lie in the direction the alternative points, and that gain is the reason anyone uses a one-sided test.
The cost of pointing the alternative in one direction
The gain comes with an awkward consequence. Suppose the one-sided upper alternative is in place and the fund in truth underperforms. Neither hypothesis is then true, and because rejection is governed by the alternative alone, however badly the fund lags the null is not rejected. A test that fails to reject has not shown the null to be true, only that the evidence against it was too weak, and the direction the alternative faces helps determine that.
One-sided alternatives are reserved for specific circumstances: when the consequences of an error run heavily in one direction, or when theory or prior experience says where the parameter should sit if the null is wrong. Breaching a VaR limit brings regulatory attention, damage to investor confidence and, at the extreme, insolvency, while sitting below it merely wastes capital. When the errors are that unequal, a test built to detect an excessive number of large losses beats a two-sided test that splits its attention.
A test statistic condenses the sample into one number whose distribution is known when the null holds. Many exist, following many different distributions. This chapter stays with the normally distributed ones, covering every situation where a Central Limit Theorem applies to the estimator: means, regression coefficients and the outputs of econometric models all qualify.
Take the null H0: μ = μ0, where μ0 is the number being tested. When the data are independent and identically distributed, the sample mean estimator is asymptotically normally distributed around the true mean.
If the true mean is the value the null names, this rearranges into a quantity centred at zero with unit variance.
Reading the pieces of the statistic
The numerator measures how far the estimate sits from the value under test. The denominator is the standard error, converting that distance into units of sampling variability. A statistic of three means the estimate lands three standard errors above the null value, a long way for something centred at zero.
The population variance is not known, so the sample variance stands in for it. That substitution is legitimate in reasonably large samples because the sample variance is a consistent estimator of the true variance, and consistency carries the asymptotic argument through. Nothing here is specific to a mean: any asymptotically normal estimator drops into the same expression, which is why one class of test statistic covers so much ground.
Hypotheses concern the population, but only a sample is ever available, and a sample can misrepresent the population it came from. In a perfect world a false null would always be rejected and a true null never would. Real testing forces a trade between those ambitions, since pushing down the chance of one mistake pushes up the other.
Rejecting a null that is in fact true is a Type I error, sometimes called an error of the first kind. The probability of committing one is the size of the test, denoted by α, and the same quantity is called the significance level. It is chosen, not discovered, and the choice expresses the tester’s appetite for discarding a correct null.
Choosing a size
The usual choice is 5%, accepting a 1 in 20 chance that a null thrown out was actually correct. Where a false rejection would be expensive, smaller sizes such as 1% or even 0.1% are used. The size does more than express a preference: it fixes the rejection region, which has to be drawn so that a true null falls inside it with probability exactly α.
One consequence follows immediately. Run many correctly specified tests at a size of 5% where every null is true, and roughly one in twenty will still reject, which is the method working as designed rather than a fault in it.
The critical value separates outcomes consistent with the null from outcomes that are not. It follows from the distribution of the test statistic, taken as normal here, the size of the test, and whether the alternative is one-sided or two-sided. The statistic values that lead to rejection form the rejection region.
With the null H0: μ = μ0 and a two-sided alternative, the rejection region occupies both tails and holds total probability α, each tail taking α/2. A size of 5% puts 2.5% in each tail, leaving the middle 95% where the null survives. A one-sided alternative does not split the region, and the whole of α sits in one tail.
| Size of the test | Two-sided | One-sided lower | One-sided upper |
|---|---|---|---|
| 10% | 1.645 (5% in each tail) | -1.28 | 1.28 |
| 5% | 1.96 (2.5% in each tail) | -1.645 | 1.645 |
| 1% | 2.57 (0.5% in each tail) | -2.32 | 2.32 |
Source: standard normal distribution table.
The decision rule fuses the critical value Cα, the alternative and the test statistic T into one verdict. A two-sided alternative rejects when the absolute value of the statistic exceeds the critical value. A one-sided lower alternative rejects when T falls below the negative critical value, and a one-sided upper alternative when T rises above the positive one. A null rejected at size α is called significant at the α level.
Let X be the annual excess return on the S&P 500 over the window 1951 through 2018, meaning the return above the risk-free rate, proxied by the return on a one-month US government bond. The average excess return is 7.62%, their standard deviation is 16.5%, and n = 68.
The second mistake runs the other way. A Type II error happens when the alternative is true, so the null is wrong, and yet the null is not rejected. Its probability is denoted by β, and the complement, 1 minus β, is the power of the test: the probability that a false null is thrown out.
| Null hypothesis true | Null hypothesis false | |
|---|---|---|
| Fail to reject | Correct decision (1 minus α) | Type II error (β) |
| Reject | Type I error, size (α) | Correct decision, power (1 minus β) |
Source: values in parentheses are the probabilities of reaching each decision, all lying between 0 and 1.
The size of a test is set by whoever runs it. Power is not, since it depends on quantities partly outside anyone’s control: the number of observations, the gap between the value the null names and the value the parameter actually takes, and the size of the test. The middle item is the awkward one, because the true parameter value is exactly what is unknown. Power is therefore computed for an assumed departure from the null, and a claim that a test has 84% power is incomplete until that departure is named.
Each driver can be watched in isolation. Take data generated as Xi distributed N(0.3, 1), so the true mean is 0.3, and test H0: μ = 0 against a two-sided alternative. The sample mean is then normally distributed with expected value 0.3 and variance 1/n.
Compare n = 10 with n = 100. Drawn on the scale of the mean rather than the test statistic, the cutoffs are the normal critical values multiplied by the standard error. At a size of 5% that is 1.96 divided by the square root of the sample size, giving 0.61 in absolute value when n = 10 and 0.196 when n = 100.
Raising the sample size narrows both distributions, so they overlap less and correct rejection becomes more likely. Push it to n = 1000 and the two curves barely touch, leaving the null rejected with probability above 99.999%. Power climbs with n without any change in the size of the test.
Test size works through the critical values. A larger size, 10% rather than 5%, uses cutoffs closer to zero and a wider rejection region, lifting the chance that the sample mean lands inside it. Power rises, and so does the frequency of Type I errors. Plotting power against the true mean for n = 50 at sizes of 1%, 5% and 10% stacks the curves in that order, largest highest.
The third driver is the distance between the truth and the null. When the true mean sits on top of the null value, power equals the size of the test, since a true null is rejected with probability α by construction. As the true mean moves away, power rises steadily, because a badly wrong null leaves more evidence behind.
A group tests whether a set of observations is N(0, 1) by examining the mean of a sample of random draws. The null is that the mean is 0, the test is two-sided at a 90% confidence level, and the sample size is 10. The data in fact come from an N(1, 2.25) population, so the true mean is 1 and the true standard deviation is 1.5.
Critical values so far have been read from the standard normal, a practice resting on the Central Limit Theorem, which applies to averages drawn from almost any distribution once technical conditions hold. Two situations call for the Student’s t instead.
The first is exact rather than approximate. When the variables in the average are iid normal, the test statistic follows a Student’s t distribution with n minus 1 degrees of freedom exactly. Claiming that exactness requires the unbiased estimator of the variance, which divides by n minus 1 rather than by n.
The second concerns the quality of the approximation. In small samples, meaning fewer than 30 observations, the Student’s t describes the behaviour of the statistic better than the normal does, even when the underlying variables are not normally distributed.
What changes when the t is used
A t on n minus 1 degrees of freedom has larger critical values than the normal, so rejection becomes harder, and that extra reluctance is what makes the actual rejection rate line up with the size chosen. The reason sits in the shape of the density. Set the normal against a t on 5 degrees of freedom and a t on 15 degrees of freedom, and the tails of the t are heavier, with the gap widest beyond 1.645 where rejection decisions are made. As degrees of freedom rise the three densities converge.
The working rule follows. With 30 or fewer observations, estimate the variance with s squared and take critical values from the t. In larger samples the difference is negligible, and common practice uses the sample variance with normal critical values.
A test can be reported without naming any size, by quoting a p-value. The p-value is the probability, computed on the assumption that the null hypothesis is true, of observing a test statistic at least as extreme as the one obtained from the data in hand. It folds the statistic, its distribution and the critical values into one number between 0 and 1, and the decision rule becomes one comparison: reject when the p-value is below the size of the test.
An equivalent description is the smallest size at which the null would be rejected: any size above it rejects and any size below it does not, so a p-value of 0.03 rejects at 5% and 10% but not at 1%.
Stating the interpretation precisely
Every probability inside a p-value is conditional on the null being true, and that single fact rules out most of the readings people reach for. A p-value of 0.03 does not mean there is a 3% chance the null is true, nor a 97% chance the alternative is true. Nor is it the probability that a rejection would be a mistake, since that depends on how often nulls of this kind are false to begin with, which the data cannot supply. What it says is narrow: if the null were true, data as unfavourable to it as the data observed, or more so, would turn up 3% of the time. A p-value describes the data given the hypothesis, never the hypothesis given the data.
Calculating it
For a two-sided alternative it doubles the area beyond the absolute value of the statistic, because a statistic that extreme could have landed in either tail.
Taking the absolute value guarantees that the right tail is measured whatever the sign of the statistic. For one-sided alternatives it is dropped, since the sign decides the outcome: a lower-tailed test uses the area below the statistic, an upper-tailed test the area above.
Applying the formula to the S&P 500 test, where the statistic is 3.80, the standard normal probability below 3.80 is 0.99993, so the area above it is 0.00007 and the two-sided p-value is 2 multiplied by 0.00007, which is 0.00014. As a percentage that is 0.014%, far below 5%, so the null is rejected. The matching one-sided upper p-value is half as large, 0.007%.
Both parts test the null H0: μ = μ0 with a test statistic that is standard normal when the null is true.
Comparing two averages is among the most common tasks in applied work. Let Wi = [Xi, Yi] be an iid bivariate random variable, so observations arrive in pairs and the number on each component is the same. Each component is iid on its own, and the two may be contemporaneously correlated, as return series measured over the same dates usually are.
The null of interest is H0: μX = μY, and the problem reduces to one already solved by defining a new variable from the pairs.
The steps then follow the ordinary recipe. Form the difference for each pair. State the null that the mean of the differences is zero, against the two-sided alternative that it is not. Compute the sample mean of those differences and their sample variance. Build the statistic, compare it with the standard normal, and apply the decision rule.
Working with the differences handles any correlation automatically, because the variance of a difference already subtracts twice the covariance. Positive correlation shrinks that variance, since the two series tend to err in the same direction on the same dates and part of the common movement cancels. Negative correlation widens it.
Take average rainfall for City X and City Y, with sample means of 10 cm and 14 cm, variances of 4 and 6, twelve observations on each, and a correlation of 0.30. The variance of the difference is 4 + 6 – 2 x 0.30 x 2 x 2.4495 = 7.0606, and dividing by 12 before taking the square root gives a standard error of 0.7671. For the null to survive a two-sided test at 5%, the gap in means would have to stay under 1.96 x 0.7671, or roughly 1.50 cm. The observed gap of 4 cm is far past that, and the statistic is (10 – 14)/0.7671 = -5.21, with a p-value below 0.01%.
A different statistic applies when the two series are mutually independent rather than paired: sample sizes need not match, and each variance is scaled by its own count rather than by a shared n.
An analyst holds 50 years of paired annual observations on equities and bonds. Equities average 7.3% per year against 2.7% per year for bonds. Volatility runs at 18.4% on the equity series and 5.3% on the bond series, and the two are correlated at -60%.
Multiple testing means running many hypothesis tests on the same body of data, and it distorts results across finance, psychology, biology and medical research alike. The trouble is that the size of a test is a guarantee about one test. Repeat the exercise and the probability that at least one true null is rejected climbs well above the chosen size.
Suppose a set of trading strategies is compared against a buy-and-hold approach, one null per strategy stating that the strategy return equals the buy-and-hold return. If the resulting test statistics are independent and each test is run at 5%, then with 10 strategies the chance that at least one looks significant is 100% – (95%)10 = 40.1%. Push the count to 100 strategies and the same calculation gives 99.4%. Keep testing ideas against one dataset and something will eventually appear to work. That finding is spurious, and it is likely to collapse when the strategy is retested on data that had no hand in selecting it.
Controlling for it
The Bonferroni and Holm-Bonferroni methods adjust the decision rule for the number of tests being run, and they are easy to apply because they need nothing beyond the standard test statistics and their p-values. The price is conservatism: the adjusted rule can be so demanding that genuine effects are missed, which is to say it has low power. Methods that control the False Discovery Rate or limit the Familywise Error Rate recover some of that power, at the cost of being more complex and more data intensive.
Statistical significance versus practical significance
A separate check applies to any result that clears the significance bar. The statistic divides the gap between the estimate and the null value by a standard error that shrinks with the square root of the sample size, so any non-zero discrepancy, however tiny, drives the statistic towards infinity as observations accumulate. Practical significance asks instead how large the estimated effect is in the units of the problem. Imagine a portfolio adjustment that trims annual volatility by four hundredths of a percentage point on a strategy already running at 30% volatility, leaving returns untouched. Given a long history that reduction is statistically significant, and still far too small to matter. A result has to clear both bars before it is worth acting on.