QTA 10: Stationary Time Series
Interest rates, credit spreads and growth rates carry pieces of their own past, and that memory is what makes forecasting worth attempting.
Any observed series separates into three parts. The trend describes how its level drifts over long stretches of time. The seasonal component describes swings that repeat on a calendar. The cyclical component is everything left over: waves produced by shocks and by the persistence with which the series holds on to them. Trend and seasonality are deterministic and can be written down and subtracted. The cyclical part cannot, and it is the subject of this lesson.
A stochastic process is an ordered collection of random variables, and the ordering matters, since prediction depends on knowing which observation came first. The plainest useful process draws each observation independently from a normal distribution, which offers no structure to exploit. Structure enters once one observation depends on the last, and the first-order autoregression, written AR(1), does exactly that:
Four series make the range of behaviour concrete. Monthly returns on the S&P 500 index look close to pure noise. The VIX index wanders from its average and returns slowly, so departures are long-lived without being permanent. The slope of the US Treasury yield curve, the ten-year yield minus the one-year yield, and its curvature, the difference between two slopes, both track the economy.
Almost every model here belongs to a family called linear processes, in which the observation is a deterministic piece plus a weighted sum of current and past shocks with constant weights. Many nonlinear processes admit a linear representation too.
Linear processes lead to linear models, which forecast the conditional mean given the information set: everything known at the current date.
Fitting a model to past data and using it on the future only makes sense if the relationships inside the series hold still. Covariance stationarity is the formal statement of that idea, and it constrains only the first two moments.
Autocovariance
Covariance normally compares two different random variables. Autocovariance compares one process with itself at two dates, which is what makes a series a time series rather than a scatter of unrelated draws. It is written with the Greek letter gamma, carrying a subscript for the date and one for the distance between the observations.
At lag zero the autocovariance compares the process with itself at the same date, so it collapses to the variance of the series.
The three requirements
A series is covariance stationary when three conditions hold together. The expected value is the same number at every date, so the series has one mean rather than a moving one. The variance is finite and identical at every date. And the autocovariance at any lag is finite, is the same at every date, and depends on the gap between the two observations and on nothing else.
The third condition carries most of the weight. It says the strength of the link between an observation and the one four periods earlier is a fixed property of the process, the same in the first year of the sample as in the last. Nothing is required of the third or fourth moments, of the shape of the distribution, or of independence.
Why the property matters
Two consequences follow. Stable relationships are the licence needed to estimate a model on history and apply it to observations the model has never seen. Parameters estimated from a non-stationary series are much harder to interpret, because the estimators are not asymptotically normally distributed. Worse, a pair of unrelated non-stationary series can show strong and apparently significant correlation, a spurious result that survives standard tests.
Autocovariance is expressed in squared units of the series, which makes raw numbers awkward to compare. Dividing by the variance removes the units and gives the autocorrelation at lag h.
Autocorrelations behave like ordinary correlations and are confined to the interval running from -1 to 1 inclusive.
The autocovariance function
Collecting the autocovariances across lags gives the autocovariance function, which takes a lag and returns the covariance between the series and its own past at that distance.
The absolute value makes the function symmetric: looking back four periods and looking forward four periods give the same answer. That symmetry comes from the third stationarity requirement, since shifting both observations forward by h periods changes nothing.
The autocorrelation function and the partial autocorrelation function
The autocorrelation function, universally shortened to ACF, is the same object scaled to the correlation range.
The ACF has a limitation. If today is correlated with yesterday, and yesterday with the day before, then today and the day before show correlation even with no direct link between them. The partial autocorrelation function, written PACF and usually denoted by the Greek letter alpha, strips that indirect transmission out by controlling for every observation lying between the two dates.
At the first lag nothing sits in between, so the two measures agree exactly. Beyond that they part company, and the PACF is a nonlinear transformation of the ACF rather than a rescaling of it. Both matter for one practical reason: each class of linear time-series model produces its own characteristic pair of shapes, and matching those against the shapes estimated from data is how a model gets chosen.
Every model in this lesson is assembled out of one raw material. A white noise process is a sequence of shocks with a mean of zero and a variance that is the same at every date.
Three properties define it. The mean is zero, which costs nothing in generality because a process whose error has some other mean can be rewritten by moving that mean into the deterministic part. The variance is constant and finite. And the shocks are uncorrelated at every non-zero lag, so autocovariance and autocorrelation both vanish beyond lag zero.
Those conditions are the requirements for covariance stationarity, so white noise is stationary by construction. The absence of correlation does the real work: it forces every trace of dependence in a modelled series to come from the model parameters rather than from links between the shocks, which is why residual checks are the standard test of a specification.
No particular distribution is required, since anything with finite variance becomes white noise once its mean is subtracted. Independent and identically distributed sequences with zero mean and finite variance are a special case, called independent white noise, and narrowing to the normal distribution gives Gaussian white noise. Normality is a stronger assumption than time-series analysis needs, and financial asset data frequently contradicts it.
Dependent white noise
Uncorrelated is not the same as independent. Asset returns are close to unforecastable in their mean while their volatility is visibly persistent, which makes such a series linearly independent and nonlinearly dependent at once. The leading example is an Autoregressive Conditional Heteroskedasticity process, known as ARCH, in which the variance of the current shock is driven by the size of the previous shock. Big moves follow big moves and quiet stretches follow quiet stretches, a pattern called volatility clustering. An ARCH process has a predictable variance, an unpredictable mean, and shocks that stay uncorrelated.
Wold’s theorem
Wold’s theorem explains why white noise holds this central position. Any mean-zero covariance-stationary process can be written as a weighted sum of current and past white noise shocks, with a leading weight of one and squared weights summing to a finite number.
Autoregressive models are the most heavily used time-series specifications in finance and economics. They connect the current value of a process to its own history, and the first-order version uses one lag.
The AR parameter controls persistence: the process is covariance stationary when its absolute value is below one and not stationary at all when it equals one.
The long-run mean
The intercept is not the mean. Because the process is stationary the expected value is the same at every date, so taking expectations through the model equation and writing that value as mu gives an equation in mu alone.
The variance
The variance operator works the same way. The covariance between the lagged value and the current shock is zero, because the lagged value was built from shocks dated one period earlier and before.
Ten shocks arrive in the order shown in the table below. Build the series for an AR(1) with an intercept of 0.5 and an AR parameter of 0.75, starting from zero, rounding each value to two decimal places before carrying it forward.
| Date | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
|---|---|---|---|---|---|---|---|---|---|---|
| Shock | -0.58 | 1.62 | -1.34 | 0.88 | -1.11 | 0.52 | 0.81 | -0.65 | 0.85 | -0.88 |
| AR(1) value | -0.08 | 2.06 | 0.71 | 1.91 | 0.82 | 1.64 | 2.54 | 1.76 | 2.67 | 1.62 |
Source: shock sequence and parameter values from the chapter practice questions.
Working out the autocovariances of an AR(1) takes one substitution and one recursion. Replacing the current value by its model equation inside the covariance, and using the fact that the current shock is uncorrelated with anything earlier, gives the first autocovariance.
The same substitution at an arbitrary lag links each autocovariance to the one before it, and applying it repeatedly turns the chain into a power.
Dividing through by the variance leaves an autocorrelation function that depends on the AR parameter alone.
Provided the absolute value of the AR parameter stays below one, the ACF decays geometrically as the lag lengthens. A negative parameter between -1 and 0 adds an oscillation, flipping the sign at every step. Negative parameters are uncommon in economic and financial series.
The PACF behaves in the opposite way. At the first lag it equals the AR parameter, and from the second lag onward it is exactly zero, because once the preceding observation is controlled for there is no further direct link.
Slow decay in the ACF paired with an abrupt cutoff in the PACF is the fingerprint of an autoregression. At a parameter of 0.7 the ACF slides towards zero while the PACF stands at 0.7 at lag one and flat at zero afterwards. Setting the parameter to -0.9 keeps that structure and adds the alternating sign.
Once models carry several lags, writing them out term by term becomes unwieldy. The lag operator, written as a capital L, shifts the time index backwards and obeys six rules.
Applying it once moves the index back one period, and applying it p times moves the index back p periods, so a superscript counts the shifts. Applied to a constant it returns the constant. A polynomial in the operator is written like any other, and applying it to a series gives a weighted sum of that series and its lags. Two such polynomials can be multiplied, and multiplication is commutative, so the order of application does not matter. Finally, subject to conditions on the coefficients, a lag polynomial can be inverted.
Inversion is the rule that carries the most weight. A first-order polynomial can be inverted whenever the absolute value of its coefficient is below one, and the inverse is an infinite geometric series in the operator.
Manual inversion is rarely necessary, but the concept earns its place twice. An autoregression is covariance stationary only when its lag polynomial inverts, and invertibility also pins down a unique specification under the Box-Jenkins route. Collecting the AR(1) terms on the left gives its lag polynomial form.
Inverting and expanding that polynomial recovers the Wold representation directly: the constant becomes the long-run mean and each past shock enters with a weight equal to a power of the AR parameter.
Two conversions show the mechanics. A model with an intercept of 15/32, a coefficient of 5/4 on the first lag and -3/8 on the second rearranges into the lag polynomial 1 – (5/4)L + (3/8)L squared, the signs flipping because the lagged terms cross to the left-hand side. Running the operation the other way, a moving average with weights 0.8 and 0.16 and an intercept of 0.3 becomes that intercept plus the polynomial 1 + 0.8L + 0.16L squared applied to the shock.
Judging invertibility by eye works for a single lag and fails beyond that. The characteristic equation supplies a systematic test: take the lag polynomial, replace the operator by an ordinary algebraic variable, and set the result equal to zero.
The characteristic equation has the same order as the lag polynomial with the powers reversed. Solving it produces p roots, and the polynomial is invertible, so the autoregression is covariance stationary, when every root is greater than one in absolute value. That is usually stated as the roots lying outside the unit circle. Roots can be complex valued, in which case absolute value means the complex modulus.
For a first-order polynomial the single root is the reciprocal of the AR parameter, so requiring it to exceed one in absolute value is identical to requiring the parameter itself to be smaller than one in absolute value.
An order-two example carries the method. Take an autoregression with a first AR parameter of 1.4 and a second of -0.45. Its lag polynomial is 1 – 1.4L + 0.45L squared, so the characteristic equation is 1 – 1.4z + 0.45z squared = 0, and that quadratic factors into (0.9z – 1)(0.5z – 1). Multiplying out confirms it: 0.45z squared, then -0.9z – 0.5z = -1.4z, then 1.
Setting each factor to zero gives roots of 1/0.9 and 1/0.5, roughly 1.111 and 2. Both exceed one in absolute value, so both lie outside the unit circle, the polynomial inverts, and the process is covariance stationary. The result is easily read the wrong way: the numbers 0.9 and 0.5 inside the factored form are below one, but they are not the roots. The roots are their reciprocals.
A quick screen is available first. Because the long-run mean divides the intercept by one minus the sum of the AR parameters, that sum must be below one, and parameters adding to one or more rule out covariance stationarity immediately. The screen is necessary rather than sufficient, so a sum below one still leaves the roots to be checked.
Extending the first-order model to p lags gives the general autoregression.
Taking expectations, with every lagged value sharing the same mean, gives the mean-reverting level.
That level acts as an attractor, the systematic part of each observation pulling the series back while shocks push it away. An AR(1) with a parameter of 0.97 drifts far from its mean and takes many periods to return, while the AR(2) written as 1.6 times the first lag minus 0.7 times the second crosses its mean far more often. The usual persistence measure is the largest absolute root of the characteristic polynomial; for that second process the roots are complex valued and the largest modulus is given as 1.119.
Take the covariance-stationary AR(2) with an intercept of 0.3, a first AR parameter of 1.4 and a second AR parameter of -0.6, driven by white noise.
The Yule-Walker system uses p + 1 equations to pin down the long-run variance and the first p autocorrelations, after which longer lags follow by recursion. The shapes generalise without surprises: the ACF decays towards zero and may oscillate, and the PACF drops sharply to zero after p lags. Three autoregressions fitted to the annualized growth rate of real GDP show a series with modest persistence.
| Model | Intercept | First lag | Second lag | Third lag | R squared | Absolute roots |
|---|---|---|---|---|---|---|
| AR(1) | 1.996 | 0.360 | 0.130 | 0.360 | ||
| AR(2) | 1.765 | 0.319 | 0.114 | 0.141 | 0.533, 0.213 | |
| AR(3) | 1.963 | 0.333 | 0.150 | -0.113 | 0.152 | 0.486, 0.486, 0.477 |
Source: chapter estimates on quarterly real GDP growth. The last column holds one divided by each characteristic root, in absolute value.
The intercept in the AR(1) row is not the average growth rate. Dividing 1.996 by 1 – 0.360 gives a mean-reverting level of about 3.12 percent.
A moving average builds the current observation out of shocks rather than past values of the series. The first-order version, written MA(1), uses the current shock and the one before it.
Because every shock has an expected value of zero, the constant in an MA is the mean of the process itself. A positive theta makes consecutive observations positively correlated, while a negative theta reverses the previous shock in the current period and produces aggressive mean reversion. Every moving average is covariance stationary whatever its parameter values, being a finite weighted sum of white noise terms.
Memory is strictly limited. One shock is shared between neighbouring observations and nothing is shared two periods away, so exactly one autocorrelation is non-zero.
The PACF does the opposite, staying non-zero at every lag while it oscillates and decays, inverting the pattern an autoregression produces. Adding lags of the shock generalises the model to an MA(q).
Use the same ten shocks as in Example 1: -0.58, 1.62, -1.34, 0.88, -1.11, 0.52, 0.81, -0.65, 0.85 and -0.88.
Invertibility mirrors stationarity. A moving average is invertible when all the roots of its characteristic equation, defined exactly as for an autoregression, lie outside the unit circle, and an invertible MA(q) can be rewritten as an autoregression of infinite order. A stationary AR(p) can be rewritten as a moving average of infinite order.
Nothing forces a choice between the two families. An ARMA process carries both, and the simplest case pairs one lag of the series with one lag of the shock.
Its mean comes from the autoregressive side alone, since the moving average terms add nothing to the expectation, so the ratio of intercept to one minus the AR parameter still applies. The autocovariances are messier.
Stationarity is decided by the autoregressive part alone. An ARMA(1,1) is covariance stationary when the absolute value of the AR parameter is below one, and the moving average parameter has no say, because a moving average is always stationary and touches one lag of the shock only. The autoregressive component reaches every lagged shock, so an AR parameter that is too large destroys stationarity whatever the MA side does.
Adding orders to both sides gives the general model.
What makes an ARMA identifiable is the joint behaviour of its correlation functions. Both decay slowly towards zero, possibly oscillating, and neither cuts off. An ARMA(1,1) with an AR parameter of 0.8 and a moving average parameter of -0.4 gives an ACF resembling that of an AR(1) but starting lower, because the moving average term cancels part of the correlation with the most recent shock.
Everything so far describes what a model implies. Estimating those quantities needs sample counterparts, and the sample autocovariance averages products of deviations from the full-sample mean.
Dividing by the sample variance gives the sample autocorrelation, which carries a slight bias towards zero that fades as T grows.
These statistics get used twice. Applied to the raw data they reveal the dependence structure and shortlist candidate specifications. Applied to the estimated residuals they test whether the fitted model has extracted all the predictable structure.
Graphical inspection comes first. Plots of the sample ACF and PACF normally carry the boundaries of a 95% confidence interval built on the assumption that the process is white noise, and estimates outside those lines are flagged as significant. Many violations point to a model that has missed the dynamics. A handful is expected even from a correct specification, because each lag carries its own 5% chance of a false rejection, so joint tests are the standard remedy.
How the parameters are estimated
A pure autoregression is fitted by ordinary least squares, since its explanatory variables are observed lagged values. The AR coefficient estimators are biased in finite samples, although consistent, because a given shock appears on the left of one equation and inside a lagged regressor in later ones. Once a moving average term is present least squares fails, since the lagged errors are not observable, so MA and ARMA models are estimated by maximum likelihood under an assumption of jointly normal innovations.
Both statistics test one hypothesis: that the autocorrelations up to lag h are all zero at once, against the alternative that at least one is not. Both have asymptotic chi-squared distributions, and a value above the critical value says the autocorrelations are not jointly zero. Box-Pierce scales the sum of squared sample autocorrelations by the sample size.
The logic is short. Under the null the square root of the sample size times an estimated autocorrelation is asymptotically standard normal, so its square is chi-squared with one degree of freedom, and the estimates are independent, so summing h of them gives a chi-squared with h degrees of freedom. Ljung-Box reweights the same ingredients so that the finite-sample distribution sits closer to the asymptotic one.
The correction matters when the sample is modest, fewer than about 100 observations, which makes Ljung-Box the preferred test of multiple autocorrelations. Choosing h affects the conclusion. After fitting an ARMA(p,q), h should exceed the larger of p and q, because residual autocorrelations inside the model orders have effectively been fitted away. Between 5 and 20 lags is standard for a small model with p and q below 2.
A series of 100 observations gives sample autocorrelations of 0.24, -0.04 and 0.08 at the first three lags.
Neither statistic is robust to heteroskedasticity, so a series with time-varying volatility can produce misleading rejections.
Choosing the lag lengths for the two components is the central problem in building an ARMA. Correlation plots narrow the field, and deciding between the survivors needs a measure of fit. The natural one is the mean squared error, the sample variance of whatever the fitted model leaves behind.
Minimising it directly does not work. Adding lags to either component always drives the residual variance down, exactly as adding regressors always raises R squared in a linear regression, so the criterion picks the largest model on offer and overfits. The fix is a penalty that grows with the number of parameters, turning the measure into an information criterion. The two in general use are the Akaike Information Criterion and the Bayesian Information Criterion, also known as the Schwarz criterion.
The AIC charges a flat two per parameter whatever the sample size. The BIC charges the natural logarithm of the sample size, a price that climbs slowly as data accumulates. Two consequences follow. Once the sample exceeds eight observations, because the natural logarithm of 8 is 2.07, the BIC penalty is heavier, so the BIC never selects a larger model than the AIC. The BIC is also consistent, identifying the true specification as the sample grows without limit.
Seen through hypothesis testing, the AIC admits any variable significant at a fixed test size, so it keeps admitting irrelevant variables and drifts towards models that are too large, while the effective test size of the BIC shrinks towards zero as the sample grows.
Twelve ARMA specifications are fitted to 250 observations, and the number of parameters is the sum of the two orders.
| Order | Residual variance | AIC | BIC |
|---|---|---|---|
| (1, 0) | 1.212 | 50.068 | 53.589 |
| (1, 1) | 1.147 | 38.287 | 45.330 |
| (1, 2) | 1.112 | 32.540 | 43.104 |
| (2, 0) | 1.175 | 44.317 | 51.360 |
| (2, 1) | 1.097 | 29.145 | 39.709 |
| (2, 2) | 1.097 | 31.145 | 45.231 |
| (3, 0) | 1.150 | 40.940 | 51.505 |
| (3, 1) | 1.097 | 31.145 | 45.231 |
| (3, 2) | 1.068 | 26.447 | 44.054 |
| (4, 0) | 1.133 | 39.217 | 53.303 |
| (4, 1) | 1.096 | 32.917 | 50.524 |
| (4, 2) | 1.065 | 27.744 | 48.872 |
Source: residual variances from the chapter. The criterion columns are computed from them.
The Box-Jenkins method
Box and Jenkins (1976) set out a three-step routine: identify a specification from the ACF and PACF plots, estimate its parameters, and check the residuals for remaining structure. Two principles then resolve the fact that different parameter sets can produce identical means and correlation functions. Parsimony takes the model with fewer parameters whenever the fit is the same. Invertibility selects the moving average parameters giving an invertible representation, which matters because an MA(q) admits two raised to the power q equivalent versions. Together they guarantee a unique specification.
A forecast is a conditional expectation. The one-step forecast is the expected value of the next observation given the information set at the forecast origin, which holds the whole history of the series, the whole history of the shocks even though those are inferred rather than observed, and anything else dated at or before the origin.
Three rules make the calculations mechanical. Anything dated at the origin or earlier is known, so its expectation is the realised value, and that covers residuals as well as observations. The expectation of any future shock is zero at every horizon. Forecasts are generated recursively from the first step outward, and wherever a future value appears it is replaced by the forecast already computed for that date.
For an AR(1) the first step keeps the intercept and the AR parameter times the last observed value, and the shock drops out. The second step replaces the unobserved next value with the first forecast, so it depends on the last observation through the square of the AR parameter. Repeating the substitution gives the general expression.
As the horizon lengthens the power of the AR parameter shrinks towards zero, because stationarity keeps its absolute value below one, so the term carrying the last observation fades and the forecast settles on the long-run mean.
This is the mean-reverting level again, and the duality belongs to covariance stationarity rather than to autoregressions, since today has a negligible influence on the distant future in any stationary series. Moving averages reach the mean abruptly: an MA(q) depends on at most q lagged shocks, so every forecast beyond horizon q equals the mean exactly.
Applying this to real GDP growth with the fitted AR(1), each forecast is 1.996 plus 0.360 times the previous one. From a first forecast of 2.919, the next is 1.996 + 0.360(2.919) = 3.047 and the one after that is 1.996 + 0.360(3.047) = 3.093. The mean-reverting level is 1.996 divided by 0.640, which is 3.119, and the forecasts reach it within two years. The default premium behaves differently: its ARMA(1,1) forecasts edge down by roughly four thousandths a month and are still falling after eight steps, which is what an AR parameter close to one does to convergence.
| Horizon | Default premium | Quarter | Real GDP growth |
|---|---|---|---|
| 1 | 1.139 | 201901 | 2.919 |
| 2 | 1.135 | 201902 | 3.047 |
| 3 | 1.131 | 201903 | 3.093 |
| 4 | 1.127 | 201904 | 3.110 |
| 5 | 1.123 | 202001 | 3.116 |
| 6 | 1.119 | 202002 | 3.118 |
| 7 | 1.116 | 202003 | 3.119 |
| 8 | 1.113 | 202004 | 3.119 |
Source: chapter forecasts, from an ARMA(1,1) and an AR(1) respectively.
The forecast error is the realised value minus the forecast made for it. At one step it is simply the next shock, in any ARMA model. Errors at longer horizons accumulate several shocks weighted by the model parameters.
Many macrofinancial series repeat themselves on the calendar, with home buying and construction running heavier in summer than in winter. Seasonality comes in two forms. Deterministic seasonality, a fixed level shift attached to particular months, makes a series non-stationary. Stochastic seasonality can be stationary, so an ARMA can handle it.
The seasonal frequency is annual in practice, so the gap is four periods in quarterly data and twelve in monthly data. A pure seasonal model uses only lags at that frequency: a quarterly pure seasonal AR(1) links each observation to the one a year earlier and to nothing in between, which amounts to four independent AR(1) processes interleaved and is rarely plausible for an economic series.
The workable structure keeps both horizons. A short-run lag polynomial handles dependence at the observation frequency and a seasonal lag polynomial handles it at the annual frequency, and the two are multiplied rather than added.
Multiplying out reveals what the specification really is: an autoregression of order 13 in which almost every coefficient is zero and the thirteenth is not free but equals the product of the other two. Two estimated parameters buy dependence across thirteen lags, which is the point of the multiplicative form.
Seasonality can be attached to the autoregressive side, the moving average side, or both, and the notation records all four orders with the seasonal horizon. The example above is an ARMA(1,0) multiplied by a seasonal (1,0) at a frequency of 12. Seasonal components are usually held to a single lag, because their precision depends on how many complete annual cycles the sample contains, and dividing 100 observations by a frequency of 12 leaves very few.
The mechanics are easy to check. Take a quarterly model written as the polynomials 1 – 0.4L and 1 – 0.9L raised to the fourth power, multiplied together and set equal to a shock. Expanding gives 1 – 0.4L – 0.9L to the fourth + 0.36L to the fifth, so the standard form places 0.4 on the first lag, 0.9 on the fourth and -0.36 on the fifth, the last again fixed as the product of the other two.
Model selection works as before, with the correlation functions inspected at both frequencies and information criteria choosing between specifications carrying observation-frequency lags, seasonal lags, or a mixture.