QTA 5: Sample Moments
A random variable carries fixed numerical features: a mean, a variance, a skewness, a kurtosis. These are population moments, belonging to the distribution itself, and nobody ever gets to see them. What is available instead is a finite run of observed outcomes, and a quantity computed from that run is a sample moment.
A sample moment stands in for the population moment it resembles, and the useful question is how good a stand-in it is: whether it lands on the right value on average, whether it tightens up as more data arrive, and what shape its own distribution takes.
An estimator is a rule, an estimate is a number
Two words that sound interchangeable are kept apart in statistics. An estimator is a recipe that turns a collection of random variables into a single value, and an estimate is what comes out once real observed numbers are fed in.
The distinction settles what kind of object each one is. An estimator takes random variables as inputs, so it is itself a random variable, with a distribution, an expectation and a variance of its own, while an estimate is a plain number that does not vary once the data are fixed. Bias, consistency and standard errors are properties of estimators. Both are written with a hat, and the estimators here are point estimators.
The sample analog to the expectation operator
One idea generates almost every estimator here. A population moment is an expectation of some function of the random variable. Replace the expectation operator by an average across the observed data, and the population moment becomes an estimator. That substitution goes by the name sample analog of the expectation operator, and it works at any order of moment and on functions of two variables as readily as one.
The first four standardized moments describe location, dispersion, asymmetry and tail weight, which is why they are reported almost universally in finance and risk management. Quantiles offer a separate route, and they earn their place on financial data because extreme outliers barely disturb them.
The population mean is estimated by the average of the observed data, and this mean estimator is the most heavily used object in applied statistics.
The observations here are treated as independent and identically distributed, so each has expectation equal to the population mean and variance equal to the population variance. Feeding observed values into the formula produces a mean estimate.
The expectation of the mean estimator
Taking expectations through the sum is straightforward, since expectation is a linear operator and passes term by term.
The argument needed very little. Independence never entered it, and neither did any assumption about the shape of the distribution; all that was used is that every observation shares the same expectation. The result therefore holds for time series that are plainly not independent and identically distributed, provided the mean stays constant, which covers the many financial series whose volatility moves around while the average return stays roughly fixed.
What the expectation is taken over matters. It is not an average over the n observations inside one data set, but an average over the whole population of data sets of size n that could have been drawn. The statement says nothing about how close any single estimate happens to be.
A fund reporting nine monthly returns of 4%, -1%, 3%, 2%, -2%, 1%, 3%, -1% and 0% has a sample mean of 1% per month, since the nine values add to 9. That number is the estimate; the rule that produced it is the estimator, and the rule is what has properties worth studying.
Bias puts a number on systematic error: the difference between what an estimator delivers on average and the true value it aims at.
The parameter could be the population mean, the population variance, or anything else being estimated. Applying the definition to the mean estimator is immediate, since its expectation equals the population mean and the difference between a quantity and itself is zero. An estimator with zero bias is called unbiased.
What bias does not tell you
Bias describes the centre of the estimator’s distribution and nothing else. An unbiased estimator is correct on average, which leaves open the possibility that it is wildly wrong in almost every individual sample. Suppose one procedure is centred on the truth but scatters enormously while a rival is off by a small constant amount and barely scatters: the second will typically land closer. An unbiased estimator is not automatically preferable, and the comparison that matters weighs bias against variance together.
Finite sample bias
Some estimators are biased at any fixed sample size but have a bias that vanishes as the sample grows without bound. That is finite sample bias, and such an estimator is asymptotically unbiased. The sample variance estimator introduced next is exactly this case, as are both estimators of the standard deviation.
The population variance is the expected squared deviation of a random variable from its own expectation. The sample analog gives an estimator directly: an average replaces the expectation, and the mean estimator replaces the unknown population mean inside the bracket.
The same substitution generates the third and fourth moment estimators later on. An equivalent route starts from the expectation of the square minus the square of the expectation, and yields the average of the squared observations minus the squared sample mean. The two expressions are numerically identical.
Dividing by n understates the variance
Unlike the mean, this estimator is not unbiased, and its expectation falls short by a factor depending only on the sample size.
The reason sits in the mean estimator inside the sum. Deviations are measured from the sample mean, which is pulled toward whatever data happened to be drawn. Estimating the centre consumes one of the degrees of freedom in the data, and the fitted centre tracks the observed points slightly too closely, so the deviations look smaller than they are.
This is a finite sample bias, since the shortfall goes to zero as the sample grows. With n = 20 the expectation is nineteen twentieths of the population variance, a shortfall of five percent, and with n = 2000 it is one twentieth of one percent.
What repeated sampling shows
Simulation makes the size of the bias visible. Draw many independent samples, each of size 20, from a standard normal distribution and track the running average of the sample variance estimates. The population variance is 1 and the sample size is 20, so the average settles at 0.95. Drawing more samples only pins that average down more precisely at the wrong value.
Because the bias is known exactly, it can be removed exactly: multiplying the sample variance estimator by n divided by n minus 1 scales the expectation back up to the population variance.
Its expectation is the population variance at every sample size, which is where the n minus 1 in every spreadsheet variance function comes from: one degree of freedom was spent estimating the centre, so the divisor drops by one.
Which of the two to use
The choice is genuinely two-sided. The estimator that divides by n has a smaller variance than the one that divides by n minus 1, so removing the bias makes the estimator scatter a little more. Financial data sets are usually large and the two then agree to several decimal places, and the convention is to prefer the version that divides by n once the sample size is above 30.
Estimating the standard deviation
The sample standard deviation is the square root of whichever variance estimator was used. A square root is nonlinear, and unbiasedness does not survive a nonlinear transformation. Both estimators of the standard deviation are therefore biased, though that bias also shrinks with the sample.
Return to the nine monthly returns of 4%, -1%, 3%, 2%, -2%, 1%, 3%, -1% and 0%, whose sample mean is 1%.
The mean estimator is a random variable, so it has a variance, and that variance governs how far a single estimate is likely to sit from the population mean. A sum of random variables carries a variance built from the individual variances added together, plus every covariance.
Independent and identically distributed observations are uncorrelated, so every covariance term is zero and each of the n variances equals the population variance.
Its square root is the standard error of the mean.
What n counts, and what it does not
The n in these formulas is the number of observations inside one sample: the length of the return series being averaged, the count of monthly data points, the number of trades in the book. It is not the number of samples, and not the number of times an experiment is repeated. A data set of 400 daily returns has n = 400, and the standard error of its mean is the standard deviation of those returns divided by 20.
The confusion is worth heading off, because the sampling distribution of the mean is defined by imagining many repeated samples, which tempts a reader to treat n as a count of those repetitions. Each imagined sample has the same fixed size n, and drawing more of them sharpens the picture of the spread without changing it. Only a larger n narrows it, and the square root sets the price: halving the standard error requires four times the data.
The nine monthly returns from Example 1 have a sample mean of 1% and a sample standard deviation estimate of 2.0% using the divisor n.
These two quantities are built from the same raw material and answer different questions. Standard deviation describes the uncertainty in a random variable or the dispersion of a data set, while standard error describes the uncertainty of an estimator, so the dispersion of the mean estimator across samples is a standard error.
The behaviour that separates them is the response to sample size. A standard error declines as the sample size increases, because a longer series locates the parameter more sharply. A standard deviation is a fixed property of the data generating process and is unmoved by how many observations are collected. A daily equity return series with a standard deviation near 1% keeps it whether five years or thirty years of history are used.
Keeping the two questions apart helps: how spread out are tomorrow’s possible returns is a standard deviation question, and how precisely the average return has been pinned down is a standard error question.
Where standard errors are used
Standard errors carry the weight in hypothesis testing, where a test statistic is an estimate divided by its own standard error, so the distance from a hypothesised value is measured in units of precision. They serve the same purpose in Monte Carlo simulation, where the standard error of a simulated quantity reports how much of the answer is simulation noise.
Most assets are priced at least daily, and equities, some sovereign bonds and futures have prices available continuously through the session, while hedge fund returns are often published monthly or quarterly. A statistic computed at one frequency is not comparable with the same statistic computed at another, which is why both are conventionally reported on an annual basis.
Log returns and why they add up
Returns here are differences in log prices, the logarithm of the current price minus the logarithm of the previous one. The convenience is additive: an n-period log return is the sum of n consecutive returns, and simple returns do not behave this way. For returns that are independent and identically distributed, the n-period return has mean n times the single-period mean and variance n times the single-period variance, and the variance result needs only that returns be uncorrelated across time.
The scale factors
Annualising a mean multiplies it by the number of sampling periods in a year: 12 for monthly data, 52 for weekly, 4 for quarterly, and the number of trading days for daily data. That daily count varies by market and by year, with 252 or 260 in common use, and the four-asset study drawn on below uses 245. Standard deviations scale with the square root of the same factor.
Untreated, sampling frequency produces figures that look contradictory. A series sampled over one day might show an average return of 0.1% while the same series sampled weekly shows 0.485%. Both correspond to an annualised mean of 2.52%.
Annualised means and standard deviations computed from daily, weekly and monthly versions of one price series then come out close to one another. Skewness and kurtosis are already scale-free and are never annualised.
Two further moments are routinely reported on financial data. Skewness is the third central moment standardised by the cube of the standard deviation; kurtosis is the fourth central moment standardised by the squared variance. Both are unit-free.
Raising deviations to the third power magnifies large shocks far more aggressively than squaring does, and it keeps the sign of each deviation, so a symmetric variable has a third moment of zero. When the biggest departures from the centre come from the left tail the variable is negatively skewed, and from the right tail positively skewed. The fourth power magnifies outliers harder still and discards sign, so kurtosis measures the relative likelihood of drawing an observation from the tail.
The reference point for kurtosis is the normal distribution, whose kurtosis is 3. A variable with kurtosis above 3 is described as heavy-tailed or fat-tailed. Many packages report excess kurtosis, which is kurtosis minus 3, so a reported value has to be checked against its convention.
The estimators
Average the cubed and the fourth-power deviations from the sample mean, then standardise by the appropriate power of the sample standard deviation.
A ten-observation series of monthly percentage returns reads -2, -1, -1, 0, 0, 1, 1, 2, 2 and 8.
Seeing the shape of a data set
A histogram splits the range of the data into bins and counts the observations in each. A kernel density plot instead evaluates a count at every point on the axis and weights each observation by its distance from that point, with the weight declining as distance grows. The result is a smooth curve, and density plots are generally preferred for visualising a distribution.
Under independent and identically distributed data the sample average earns a title: Best Linear Unbiased Estimator of the population mean, compressed into the acronym BLUE. Unpacking it requires each word.
Linear means a weighted sum of the observations, with weights that do not depend on the data.
Unbiased narrows the field to those linear estimators whose expectation is the population mean, which for constant weights requires the weights to sum to one, and best means smallest variance within that field. The conditions are weaker than full independence: what the result needs is that every observation shares the same mean and the same variance.
What BLUE does not claim
BLUE restricts the field of competitors, and it does not say the sample mean cannot be beaten. Any estimator that beats it must be nonlinear, or biased, or both. Maximum likelihood estimators usually beat the simple average on accuracy, and they escape the theorem by being nonlinear and often biased in finite samples.
Four independent random variables all have mean 1, but their variances are 1/2, 1/2, 2 and 2 respectively. Because the variances differ, the data are not identically distributed.
Unbiasedness and a variance formula describe two features of the mean estimator, and two moments rarely pin down a distribution. If the underlying data are normal then the mean estimator is normal too, but for any other data generating process the exact finite sample distribution is out of reach. Modern econometrics studies the estimator as the sample size grows without bound and uses the limiting result as an approximation.
The Law of Large Numbers is the first of those results. There are many versions, and the simplest for independent and identically distributed variables is the Kolmogorov Strong Law of Large Numbers.
Almost sure convergence is a technical mode of convergence from probability theory, and its practical content is that the probability of the sample mean sitting far from the population mean shrinks to zero.
Consistency
An estimator to which a Law of Large Numbers applies is called consistent. Consistency requires that the estimator be asymptotically unbiased, so any finite sample bias melts away, and that its variance converge to zero, which together collapse the whole distribution onto the population value. When the data are independent and identically distributed and the expectation of the squared variable is finite, the sample variance estimator converges too, so it is consistent despite being biased at any fixed sample size.
What simulation shows
The collapse can be watched directly. Take a log-normal with parameters 0 and 1 and a Poisson with shape parameter equal to 3, both right-skewed. For each, draw 10,000 independent samples at every sample size from 10 up to 640 in multiples of 4, and plot the density of the sample means. That density is not normal, and for the log-normal data it is visibly right-skewed when n is small. As the sample size rises it narrows around the population mean.
Consistency is valuable and insufficient. The difference between the estimate and the population value collapses to zero in the limit, so studying its distribution directly gives nothing. The fix is to rescale that difference so its dispersion stays fixed as the sample grows, and multiplying by the square root of the sample size does exactly that.
The Central Limit Theorem states that the sampling distribution of the mean tends toward a normal distribution centred on the population mean as the sample size tends to infinity. The simplest version for independent and identically distributed data is the Lindeberg-Levy Central Limit Theorem.
Beyond what the Law of Large Numbers needs, this theorem requires one extra assumption, that the population variance is finite; the Law of Large Numbers needs only a finite mean. It requires no knowledge of the distribution that generated the data, which is why it is used so freely on financial returns whose true distribution nobody knows.
In finite samples the result is used as an approximation, with the mean estimator treated as normally distributed around the population mean and variance equal to the population variance divided by n. Once again n counts the observations inside the one sample being analysed, not the number of samples in a simulation and not the number of assets in a portfolio.
How large is large enough
The theorem holds in the limit, so the practical question is whether the sample in hand is long enough. In the log-normal simulation the standardised sample mean is still visibly skewed at n = 10. By n = 40 the skew has largely gone, and at n = 160 very little separates the simulated distribution from a standard normal, so the asymmetry that shows up in the estimator at small n is a finite sample property the theorem washes out. In the Poisson simulation the approximation is accurate at every sample size tried.
The median measures central tendency by a different route from the mean. It is the 50% quantile, the point at which the probability of drawing a value above it equals the probability of drawing one below. For a symmetric distribution it coincides with the mean. For an asymmetric one the two separate: a right-skewed distribution has a median below its mean, because the long right tail drags the average up without moving the middle observation.
Estimating a quantile from data
Estimation begins by sorting the observations from smallest to largest. With an odd sample size the median estimate sits in position n plus 1, all divided by 2, and with an even sample size the estimate averages the two central values.
Any other quantile follows the same recipe, read off the sorted data at position alpha times n, with the two neighbouring observations averaged when that position is not a whole number. Statistical packages differ in how they interpolate, so two programs can return slightly different quartiles from identical data. The gap between the 75% and 25% quantiles is the interquartile range.
Why quantiles suit financial data
Two properties make them attractive. Quantiles carry the same units as the data, so a 25% quantile of a return series is itself a return, and the interquartile range marks a central band holding half the observations. The second is robustness: a contaminated observation far from the rest leaves both the median and the interquartile range untouched, because neither depends on the magnitude of an extreme value, only on its position in the ordering. The mean gives that outlier a weight of one over n, and the variance is more sensitive still.
Use the same ten monthly returns as Example 3, already in ascending order: -2, -1, -1, 0, 0, 1, 1, 2, 2 and 8.
| Asset | 25% quantile | Median | 75% quantile | Range between them |
|---|---|---|---|---|
| Bonds | -0.140% | 0.034% | 0.207% | 0.347% |
| Stocks | -0.534% | 0.103% | 0.660% | 1.19% |
| Gold | -0.445% | 0.014% | 0.496% | 0.941% |
| Crude | -1.21% | 0.058% | 1.30% | 2.51% |
Source: quantile estimates for the four-asset study, daily sampling.
The ordering agrees with the one the standard deviations give: crude oil is the most dispersed, stocks and gold follow, bonds are the tightest. Quantiles scale with the sampling interval much as the mean does, so the monthly medians reach 0.640% for bonds and 1.56% for stocks, and the monthly interquartile ranges widen to 1.64% for bonds, 6.59% for stocks, 5.14% for gold and 11.9% for crude oil.
Everything so far concerns a single series. Extending to two is straightforward for the mean, since the sample mean of a pair is the two univariate means collected together. The second moment takes more work, because a pair of series has two variances and a covariance. Covariance measures linear dependence, and its estimator comes from the usual substitution of an average for the expectation.
Like the sample variance, this estimator is biased toward zero, for the same reason: two sample means are fitted inside it. Dividing by n minus 1 removes the bias.
Covariance inherits the units of both series multiplied together, which makes its magnitude hard to interpret and impossible to compare across pairs. Correlation fixes that: the estimated covariance is scaled by the two estimated standard deviations multiplied together.
The sample correlation is biased in finite samples even when unbiased estimators are used for both variances and the covariance, for the same reason that spoils the standard deviation: a ratio involving square roots is nonlinear.
A two-series calculation
Take the nine monthly equity returns from Example 1 alongside nine bond returns for the same months.
| Month | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
|---|---|---|---|---|---|---|---|---|---|
| Equity | 4 | -1 | 3 | 2 | -2 | 1 | 3 | -1 | 0 |
| Bond | 0.9 | 0.1 | 0.7 | 1.2 | -0.2 | 0.7 | 0.4 | 0.4 | 0.3 |
Source: constructed series used here to illustrate the covariance and correlation estimators.
The equity mean is 1 and the bond mean is 0.5. The equity deviations are 3, -2, 2, 1, -3, 0, 2, -2 and -1, and the bond deviations are 0.4, -0.4, 0.2, 0.7, -0.7, 0.2, -0.1, -0.1 and -0.2. The nine cross products sum to 5.4, so dividing by 9 gives a sample covariance of 0.6 and dividing by 8 gives the unbiased figure of 0.675. The squared bond deviations sum to 1.44, so the bond variance estimate is 0.16 and its standard deviation estimate is 0.4. With the equity standard deviation already known to be 2.0, the sample correlation is 0.6 divided by 0.8, or 0.75.
What estimated correlations look like in practice
| Pair | Daily | Weekly | Monthly |
|---|---|---|---|
| Bonds and stocks | -16.4% | -2.4% | 15.1% |
| Bonds and gold | 3.0% | 2.3% | 19.6% |
| Bonds and crude | -9.9% | -7.7% | -0.2% |
| Stocks and gold | -2.1% | 4.0% | -0.2% |
| Stocks and crude | 12.5% | 13.2% | 11.6% |
| Gold and crude | 13.0% | 15.2% | 18.9% |
Source: sample correlations for the four-asset study at three sampling frequencies.
The strongest pairing is gold with crude oil, running between 13% and 19% across the three frequencies. Bonds and stocks flip sign, negative on daily data and positive on monthly data, so a correlation measured at one horizon need not match one measured at another: short-horizon returns respond to issues specific to a single asset, longer-horizon returns to shared macroeconomic conditions. All these cross-asset numbers are small. Within an asset class the picture differs entirely: the Russell 1000 large-cap index correlates with the Russell 2000, a small-cap index tracking 2,000 firms, at 87%, 88% and 85% on daily, weekly and monthly data.
Estimating two means separately is no different from estimating one. What separate calculations miss is how the two estimates move together, and the Central Limit Theorem handles that once the estimators are stacked into a vector.
The vector is asymptotically normally distributed whenever the paired data are independent and identically distributed and each component has a finite variance. A covariance matrix replaces the scalar variance, gathering the two variances on the leading diagonal and the covariance in the off-diagonal positions.
In practice the population quantities are not available. Each series contributes its own sample variance, the pair contributes a sample covariance, and the two estimators are then treated as multivariate normal with that estimated matrix divided by n.
Applying it to two indices
| Series | Annualised mean | Annualised variance |
|---|---|---|
| Small-cap equity | 10.4 | 335.4 |
| Corporate bonds | 6.71 | 25.6 |
Source: estimated moments for the BoAML Total Return Index alongside the Russell 2000, monthly data from 1987 to 2018, with means, variances and the covariance annualised.
The annualised covariance between the two is 14.0, which puts their correlation at 0.151, positive but modest, and equity returns are far more volatile at 335.4 against 25.6. The sample runs to 359 monthly observations, so the theorem divides every entry of the covariance matrix by 359: the equity variance becomes 0.934, the bond variance 0.071, and the covariance 0.039.
Dependence in the data transmits directly into dependence between the estimates. If two series are correlated, their sample means are correlated too, and the correlation between the sample means equals the correlation between the underlying series. Treating the two estimation errors as independent therefore understates the joint uncertainty.
Variance extends to a pair of variables through covariance, and skewness and kurtosis extend the same way. Counting the extensions means counting how a total power splits between two variables: a cross moment of order p has p minus 1 genuine cross measures, giving no cross mean, one cross variance, two coskewness measures and three cokurtosis measures.
The two coskewness measures
Both capture one idea from opposite sides: how likely one variable is to take a large value in a particular direction when the other is large in magnitude. In this notation the univariate skewness estimators are the cases where one variable fills all three slots. Where direction is insensitive to magnitude, both measures are zero, and a bivariate normal has zero coskewness however strongly its components are correlated.
Estimated on monthly data for the same four series, the univariate skewness values are -0.365 for stocks, -0.179 for bonds, -0.101 for gold and 0.104 for crude oil. The two coskewness values for each pair are below.
| Pair, X then Y | Squaring X | Squaring Y |
|---|---|---|
| Bonds with stocks | -0.082 | 0.012 |
| Crude with stocks | -0.064 | -0.127 |
| Gold with stocks | -0.005 | 0.014 |
| Crude with gold | -0.010 | -0.098 |
| Bonds with gold | -0.018 | -0.032 |
| Bonds with crude | -0.008 | 0.040 |
Source: estimated coskewness for the four-asset study, monthly sampling.
Most of these estimates are negative. The bond and stock pair and the crude oil and stock pair carry the largest values, and neither relationship is symmetric. For bonds and stocks the measure squaring the bond return reads -0.082, so stock returns lean negative when bond volatility is high, while the measure squaring the stock return is 0.012, close enough to zero that the sign of bond returns looks unconnected to the volatility of stocks.
The three cokurtosis measures
The symmetric version is the easiest to read: it measures whether the two series tend to be large in magnitude at the same time, so a high value means turbulence in one market coincides with turbulence in the other. The asymmetric versions measure agreement of signs when the cubed variable is in its tail, and the extremes of the variable appearing three times dominate the result.
Kurtosis is normally judged against the value 3 for a normal distribution, but that benchmark does not carry over cleanly, because for a bivariate normal the cokurtosis is itself a function of the correlation. For normal data the symmetric measure runs from 1 up to 3, taking the value 1 when the variables are uncorrelated and rising symmetrically as the correlation moves away from zero. The asymmetric measures are linear in the correlation and run from -3 to 3.
Across the four asset classes, gold and stocks show the strongest link between their volatilities, and every symmetric estimate is positive, so high volatility in one market generally accompanies high volatility in another. For bonds and stocks the measure that cubes the bond return is -1.5, meaning stock returns tend to carry the opposite sign to bond returns when bonds are in their tail, while the measure that cubes the stock return is positive.