QM 7 Estimation and Hypothesis Testing
Analysts rarely observe an entire population. Instead they work with a sample and use its statistics to infer the parameters of the population behind it. The law of large numbers is what makes this reasonable: it explains why the sample mean tends to settle close to the population mean as the sample grows, and why larger samples generally produce more reliable estimates.
The central limit theorem (CLT) takes this further and describes the shape of the estimate. If random samples are drawn from any distribution that has a finite mean and a finite variance, then the distribution of the sample means moves toward a normal shape as the sample grows. On average the sample mean lands on the population mean, and the spread of that sampling distribution narrows in proportion to the sample size.
Standardizing the sample mean gives a quantity that converges in distribution to the standard normal, written N(0, 1), as n grows without bound.
The remarkable feature is that this holds whatever the underlying distribution looks like, discrete or continuous, symmetric or skewed. That universality is what makes the CLT the engine behind both estimation and hypothesis testing. How quickly the approximation becomes good depends on shape: for symmetric, single-peaked distributions a sample as small as 4 or 5 can be adequate, while skewed distributions usually need at least 25 to 30 observations, and extremely skewed ones need more still.
Properties of a good estimator
An estimator is a rule for computing a parameter estimate from data. The sample mean is an unbiased estimator of the population mean, meaning its expected value equals the parameter it targets. It is also a consistent estimator, converging toward the true value as the sample grows. Beyond these, statisticians rank estimators by efficiency: a minimum-variance unbiased estimator (MVUE) achieves the lowest standard deviation attainable by any unbiased estimator, while a best linear unbiased estimator (BLUE) is the most efficient among estimators built from linear combinations of the data. A robust estimator, by contrast, is designed to stay reliable when outliers or non-normality are present, often by trimming extreme values so they do not dominate the result.
The standard error
The precision of the sample mean is summarized by the standard error of the mean, which is simply the standard deviation of the sampling distribution: it takes the population standard deviation and scales it down by the square root of n. Because that population figure is rarely known, the sample standard deviation, s, usually stands in for it.
A point estimate such as the sample mean is a single best guess. A confidence interval surrounds that guess with a range that is likely to contain the true parameter at a stated level of confidence, 1 minus alpha. For a large sample, the interval for the population mean centers on the sample mean and extends by a multiple of the standard error on each side.
The critical Z-score is the number of standard deviations that captures the chosen central probability. The two most common choices are a 95 percent interval, which uses a Z-score of 1.9600 and leaves 2.5 percent in each tail, and a 99 percent interval, which uses 2.5760 and leaves 0.5 percent in each tail. Raising the confidence level widens the interval.
An analyst estimates the average monthly return on gold in British pounds from 34 years of monthly data, so n is 408. The average monthly return is 0.49 percent and the sample standard deviation is 4.38 percent.
These intervals rely on the CLT assumption of finite variance and limited dispersion. Financial series often violate that assumption. Weekly gold prices over the same history, for example, average GBP867.83 with a standard deviation of GBP566.60 and a bimodal, trending distribution, so a historical mean and its confidence interval carry little predictive power once prices move well outside the observed range. Working with percentage changes rather than price levels mitigates some of this by stabilizing the mean and variance.
Value at risk
An important one-sided application of the same idea is value at risk (VaR), which measures the potential loss on a position over a horizon at a chosen confidence level. VaR sits at the lower bound of a one-sided interval built around the expected value, which is why it captures only the downside. Under the parametric variance-covariance approach, which treats returns as normal and frequently sets the mean return to zero, the estimate is the product of a critical Z-score, the return standard deviation, and the position value.
An analyst holds a spot gold position worth GBP10 million. Based on 44 years of monthly returns, n is 528, the average monthly appreciation is 0.37 percent, and the monthly standard deviation is 5.02 percent. The returns are well approximated by a normal distribution.
The parametric method assumes returns are normal, yet real returns often show skewness and excess kurtosis, which understates the chance of extreme events. It also assumes the return distribution stays in the same family across horizons, leans on historical parameters that may not survive an unprecedented market, and converts exposures into linear relationships, which breaks down for options and other instruments with embedded optionality. VaR is useful, but its limitations matter most in exactly the stressed conditions where risk estimates are needed.
Surveying every member of a population is usually too costly or infeasible, so analysts sample. The methods split into two families. Probability sampling gives every member a known chance of selection and tends to yield representative samples, while non-probability sampling relies on judgment or convenience and risks bias.
Probability sampling
Simple random sampling gives each element an equal chance of selection and works well for homogeneous data, though a biased draw can still produce a biased estimate. Systematic sampling selects every kth member and is likely to be more representative than a simple random draw. Stratified random sampling splits the population into subgroups, the strata, set by representative criteria, then draws from each in proportion and pools them, which sharpens precision when only a small subsample can be taken. Cluster sampling carves the population into small representative groups, picks a random subset of those groups, and then surveys everyone inside the chosen ones, which is fast and cheap for very large populations but can be less accurate.
The CLT interacts with these methods differently. In stratified sampling it applies within each stratum, and the pooled mean approaches normality as the stratum sample sizes grow. In cluster sampling it applies when the size of each cluster is large, not when the number of clusters is large, and higher intra-cluster correlation can mean cluster sampling needs a larger total sample to match the accuracy of simple or stratified sampling.
| Method | How it selects | Key strength |
|---|---|---|
| Simple random | Equal chance for every element | Simple, low bias for homogeneous data |
| Systematic | Every kth member | Representative and easy to implement |
| Stratified random | Proportional draws from each stratum | Greater precision than simple random |
| Cluster | All members of selected clusters | Time and cost efficient at large scale |
Non-probability sampling
Convenience sampling selects accessible elements quickly and cheaply and suits preliminary work. Judgmental sampling picks elements by expertise and can be valuable for targeted tasks such as auditing, though it may inject bias. Because these methods do not give every member a chance of selection, they do not generally produce representative samples, and the CLT does not reliably apply.
Two definitions round out the topic. Sampling error is the difference between an observed sample statistic and the quantity it is meant to estimate, arising because only a subset of the population is used. The sampling distribution of a statistic is the distribution of all values the statistic could take across repeated samples of the same size drawn from the same population.
Hypothesis testing is a structured way to judge a claim about a population parameter using sample data. In finance the claims usually concern returns, variances, or correlations, and the framework proceeds through four steps.
- State the hypotheses. The null hypothesis, written H0, is the default claim about a parameter, assumed true unless the evidence refutes it, and typically represents no effect or no difference. The alternative hypothesis, written HA, is the claim to be supported and is accepted only when the data give sufficient reason to reject H0. The two are mutually exclusive and jointly exhaustive.
- Identify the test statistic and its distribution. The statistic is computed from the sample and, together with a decision rule, drives the reject or fail-to-reject decision.
- Specify the level of significance. The level, alpha, is the probability of falsely rejecting a true null, and its complement, 1 minus alpha, is the confidence level. Smaller alpha demands stronger evidence, so a 1 percent level has a higher critical value than a 5 percent level.
- Establish the decision rule. Compare the calculated statistic with the critical value implied by alpha and the distribution.
Directions and types of tests
Tests take three broad forms: tests against a known value (comparing a sample mean or variance with a benchmark), tests comparing two unknown values (two portfolios or periods), and tests of association (correlation or independence). Each can be framed as a right-tail, left-tail, or two-tailed test depending on the direction of the alternative.
| Test | Null hypothesis | Alternative hypothesis | Rejection region |
|---|---|---|---|
| Right-tail | parameter is at most the value | parameter exceeds the value | Right tail |
| Left-tail | parameter is at least the value | parameter is below the value | Left tail |
| Two-tailed | parameter equals the value | parameter differs from the value | Both tails |
For tests of a mean, the sampling distribution is t-distributed when the population standard deviation is unknown and the population is approximately normal, and Z-distributed when the standard deviation is known. Because the standard deviation is almost always unknown, the t-statistic is the usual choice, and it converges to the normal as n grows. As a practical rule, the Z-distribution suits large samples (n at least 30), a known standard deviation, or a normal population, while the t-distribution suits small samples (n below 30) or an unknown standard deviation. A useful memory aid for the decision is that big is bad for the null: when the absolute value of the test statistic exceeds the critical value, reject H0 and call the result statistically significant, otherwise fail to reject it.
Two kinds of error
Every decision risks one of two mistakes. A Type I error is a false positive: rejecting a true null, which happens with probability alpha. A Type II error is a false negative: failing to reject a false null, which happens with probability beta. The power of a test, 1 minus beta, is the chance of correctly rejecting a false null.
| Decision | H0 is true | H0 is false |
|---|---|---|
| Fail to reject H0 | Correct (confidence level 1 − α) | Type II error (β) |
| Reject H0 | Type I error (α) | Correct (power 1 − β) |
The two risks trade off. Raising alpha lowers the critical value, which lifts the power of the test but also raises the chance of a Type I error, while lowering alpha does the reverse. The way to reduce both at once is to increase the sample size or to enlarge the effect being detected. Notably, increasing the sample size leaves the predetermined Type I rate unchanged while cutting the Type II rate.
The p-value
The p-value is the smallest level of significance at which the null would be rejected, that is, the chance, when H0 holds, of getting a test statistic at least as far out as the one the sample produced. Its form depends on the direction of the test.
The decision rule is direct: reject H0 when the p-value is below alpha, and fail to reject it otherwise. This yields the same conclusion as the critical-value method while also conveying the strength of the evidence, since a smaller p-value argues more forcefully against the null.
Suppose a test returns a p-value of 0.051, just above the 5 percent threshold. An analyst tempted to reach significance might re-run the study on a shorter window, drop volatile months as outliers, or switch benchmarks until one variation prints 0.049 and report only that. This practice, known as p-hacking, inflates the rate of Type I errors and lends statistical support to results that are actually false. In investment management it shows up as selective performance reporting, data mining for strategies, and recalculating metrics such as VaR or the Sharpe ratio on smoothed data. Confirmatory testing, out-of-sample validation, and transparent methods are the defenses.
Parametric tests assume the data follow a known distribution, most often the normal, and require conditions on sample size and variance. To make them concrete, consider two equity strategies: value investing, which targets undervalued companies with stable earnings and dividends, and growth investing, which targets fast-expanding companies that often reinvest rather than pay dividends. The following statistics summarize annual returns from 1979 to 2023 for three MSCI World indexes and are used throughout this section and the next.
| MSCI World | Growth | Value | |
|---|---|---|---|
| Average annual return | 7.91% | 8.81% | 7.17% |
| Sample variance | 0.0381 | 0.0456 | 0.0362 |
| Sample standard deviation | 0.1952 | 0.2134 | 0.1902 |
| Number of observations | 44 | 44 | 44 |
Growth returned more on average than either the broad index or value, and it also carried the greatest variance. Whether those gaps are statistically meaningful, rather than sampling noise, is exactly what a hypothesis test decides.
Test of a single mean
To test whether a population mean equals, exceeds, or falls short of a hypothesized value, the t-statistic is used when the population is roughly normal, the sample is small, and the population variance is not known. It measures how many standard errors separate the sample mean from the hypothesized mean, with n minus 1 degrees of freedom.
Using the Value index data, test whether global value stocks have delivered a positive average annual return, at the 95 percent and 99 percent confidence levels. Here the sample mean is 0.0717, the sample standard deviation is 0.1902, and n is 44, giving 43 degrees of freedom.
The same machinery handles a two-sided question. Testing whether the Growth index mean differs from zero gives a statistic of 2.7373, which is compared against two-tailed critical values of plus or minus 2.0167 at the 5 percent level and plus or minus 2.6951 at the 1 percent level, so the null of a zero mean is rejected at both. A two-sided test is harder to satisfy because each tail receives only half of the rejection area.
Test of a single variance
To test a claim about a population variance, the chi-square statistic is appropriate for a normally distributed population, with n minus 1 degrees of freedom. It scales the sample variance against the hypothesized variance.
Test whether the variance of the Value index returns, 0.0362, is greater than a hypothesized variance of 0.0381, at the 95 percent and 99 percent confidence levels, with n equal to 44.
Analysts frequently compare two datasets: two portfolios, two strategies, or one asset before and after an event. The right test depends on whether the observations are independent or paired, and on whether means, variances, or correlations are at issue.
Difference in two means
For two independent, normally distributed samples with unknown variances, the t-test for the difference in means compares the observed gap with the hypothesized difference, using n1 plus n2 minus 2 degrees of freedom.
Test whether the average annual return on the Growth index exceeds that on the Value index at the 95 percent and 99 percent confidence levels. Let population 1 be Growth, with mean 0.0881 and variance 0.0456, and population 2 be Value, with mean 0.0717 and variance 0.0362. Each has 44 observations, so there are 86 degrees of freedom.
Difference in two variances
To compare two population variances, the F-statistic is the ratio of the sample variances, with the larger placed in the numerator by convention and degrees of freedom of n1 minus 1 and n2 minus 1.
Testing whether the Growth variance exceeds the Value variance gives F equal to 0.0456 divided by 0.0362, which is 1.2595, against critical values of 1.6607 at the 5 percent level and 2.0569 at the 1 percent level. Since 1.2595 falls below both, the null of equal variances is not rejected: the two indexes do not have significantly different return variances.
Paired differences
When observations are naturally paired, such as before-and-after measurements or the same entity in two periods, the paired t-test works on the series of differences directly. It is more powerful than the independent-sample test because forming the paired differences removes shared influences, and its variance comes from the differences themselves rather than a pooled estimate. Here the order of observations matters.
Treating the annual return gaps between Growth and Value as a paired series, the mean difference is 0.0164, the standard deviation of the differences is 0.1062, and n is 44. The statistic is 0.0164 divided by 0.0160, which is 1.0243, well inside the two-tailed critical values of plus or minus 2.0167 and plus or minus 2.6951, so the mean paired difference is not significantly different from zero.
Correlation
To test whether a Pearson correlation differs from zero, the t-statistic uses n minus 2 degrees of freedom.
For a set of four funds and the STOXX50 index over 36 monthly observations, the correlation between Fund 3 and Fund 4 is 0.3102, giving a statistic of 1.903 against two-tailed critical values of plus or minus 2.032 with 34 degrees of freedom, so that single pair is not significant. Every other correlation in the matrix exceeds the critical value and is significant, which points to limited diversification among most of the funds but a genuine opportunity in combining Fund 3 with Fund 4. As the sample grows, the correlation needed to reach significance shrinks, so in very large datasets a zero-correlation null is almost always rejected.
The choice between parametric and non-parametric tests turns on what can be assumed about the data. Parametric tests shine when the distribution is known or the CLT assumptions hold, and they are more powerful and efficient in that case because they lean on parameters such as the mean and variance. When the data have fat tails, contain outliers, or otherwise depart from normality, those approximations break down and can mislead.
Non-parametric tests make no assumption about the underlying distribution, so they are more flexible, suit ordinal data, and stay robust when normality fails. The trade-off is that they usually carry less power and nearly always do a poorer job of catching a real effect when the parametric assumptions actually hold.
Spearman rank correlation
When a population departs meaningfully from normality, the Spearman rank correlation coefficient tests association. Unlike the Pearson coefficient, which measures linear relationships, the Spearman coefficient captures any monotonic relationship. It is computed from the ranks of the two variables after squaring and summing the rank differences.
For large samples the associated statistic can be tested with the t-distribution and n minus 2 degrees of freedom.
Consider whether gold hedges German inflation, using 288 monthly observations from 2000 to 2023. Both series are non-normal: inflation is strongly right-skewed while the gold return is nearly symmetric, which argues for a robust approach. The Pearson correlation is minus 0.0086, whose t-statistic is minus 0.1454, and the Spearman coefficient is 0.0480, whose t-statistic is 0.8127. Both statistics sit well within the two-tailed critical values of plus or minus 1.96, so neither the linear nor the monotonic relationship is significant, and there is insufficient evidence that gold hedges euro inflation over this period.
Test of independence
For categorical data arranged in a contingency table, a chi-square test judges whether two classifications are independent. It compares observed cell counts with the counts expected under independence, which are found from the row and column frequencies and the total.
The table below classifies 500 companies by environmental rating and governance rating. Test whether the two rating dimensions are independent at the 5 percent significance level.
| Environmental rating | GR progressive | GR average | GR poor | Total |
|---|---|---|---|---|
| Progressive | 35 | 40 | 5 | 80 |
| Average | 80 | 130 | 50 | 260 |
| Poor | 40 | 60 | 60 | 160 |
| Total | 155 | 230 | 115 | 500 |
Once independence is rejected, the standardized residual pinpoints where the pattern comes from by scaling each cell deviation by the square root of its expected count.
These residuals roughly follow a standard normal distribution, so values beyond plus or minus 1.96 mark statistically significant departures. In a separate study of 1,594 exchange-traded funds classified by size and style, which returns a chi-square statistic of 32.07 against the same critical value, a residual of 3.69 flags far more medium-size growth funds than independence would predict, while a residual of minus 2.72 flags far fewer large growth funds.