QTA 4: Multivariate Random Variables

When each component takes values from a short finite list, the whole joint PMF fits inside a table. Index the rows by the values of one component and the columns by the other, put the probability of each pair in the matching cell, and the result is a probability matrix: the mass function written out in full, relating realizations to probabilities, and the format most exam questions arrive in.

Take a company whose stock return can only be 0, 5% or -5%, and an analyst who rates it negative, neutral or positive, coded -1, 0 and 1. The two are plainly connected, since a well informed analyst should be more positive ahead of good outcomes, and a probability matrix records how strong that connection is.

Source: the chapter example relating a company stock return to an analyst rating.

Every cell answers one question about a combination of outcomes. A neutral rating sitting alongside a 5% fall in the share price carries 10% probability, and the same fall under a negative rating carries 20%. The zero in the top right corner is the most informative entry, because it rules out a 5% gain whenever the rating is negative.

Two conditions have to hold, both checkable by eye: no cell may be negative, and the nine cells must add to 1. Nothing forces the matrix to be square, since one component may have four values and the other two, and nothing requires ordered labels, although ordering makes the cumulative sums readable.

Pick any cell, then add every cell at or below its row label and its column label: that total is the joint CDF at the pair. The matrix therefore carries both the joint PMF and the joint CDF, which is why it is the working representation for discrete problems.

A joint distribution describes uncertainty across both components together, but often the question concerns only one. The marginal distribution is the distribution of a single component on its own. It is an ordinary univariate random variable, and its PMF uses the same notation as any univariate PMF.

Constructing one means summing the joint probability over every value the other component can take.

In a probability matrix this is arithmetic you can do by eye: summing down a column gives the marginal PMF of the column variable and summing across a row gives that of the row variable. For a 5% stock return the column holds 0%, 15% and 20%, which add to 35%. The other columns give 35% and 30%, and the rows give rating marginals of 30%, 40% and 30%.

Interpreting these edge totals is straightforward: ignoring returns, the probability of a negative rating is 30%, and ignoring ratings, the probability of a negative return is 35%. The marginal CDF accumulates the marginal PMF up to a chosen value. With k components the same summing works for any subset of 1 to k minus 1 components.

Two events are independent when the probability of both happening equals the product of their separate probabilities, and that definition carries across to random variables unchanged. Two components of a bivariate variable count as independent when the joint PMF equals the product of the marginal PMFs at every point in the support.

That gives a mechanical test. Compute both marginals, multiply them into a table of the same shape, then compare cell by cell against the joint distribution. One disagreement settles it, since the definition demands equality at every point rather than on average.

Running the test on the analyst rating example, the marginal probability of a -5% return is the first column sum, 35%, and the marginal probability of a negative rating is the first row sum, 30%. Their product is 10.5%, the upper left entry below. Repeating that across all nine positions gives the products of the marginals.

Source: the chapter example, with the right hand panel formed by multiplying the two marginal distributions.

Not one of the nine cells matches its partner, so the factorization fails and the two variables are dependent. A faster argument needs no arithmetic: independence means neither variable holds information about the other, and yet learning that the return was 5% rules out a negative rating completely.

The test also runs in reverse: given two marginals and an assumption of independence, multiplication determines the joint distribution outright. Suppose the expected return on the S&P 500 is -10%, 0% or +10% with probabilities 25%, 50% and 25%, while the Nikkei returns -5%, 0% or 8% with probabilities 20%, 60% and 20%. If the two were unrelated, the chance of -10% on the S&P 500 alongside -5% on the Nikkei would be 25% times 20%, or 5%.

Filling the grid the same way gives 10% and 5% across the top row, 15%, 30% and 15% in the middle, and 5%, 10% and 5% along the bottom. Any matrix built this way reproduces the marginals it came from.

A marginal distribution throws information away on purpose. A conditional distribution does the opposite, fixing one component at a known value and asking what remains uncertain about the other, by dividing the joint probability by the probability of the conditioning event.

In matrix form the recipe is to pick out the row or column being conditioned on and rescale it so it sums to 1. Suppose the analyst has issued a positive rating, so X₂ = 1. That row holds 5%, 5% and 20% against a marginal probability of 30%. Dividing each entry by 30% gives 16.6%, 16.6% and 66.6%, which are non-negative and add to one, as any PMF must.

Conditioning need not fix a single value. It can restrict a component to a set of outcomes, the form used in risk work when the conditioning event is a loss beyond a threshold. The numerator then sums the joint probabilities over the set and the denominator is the marginal probability of the set.

Take the stock return in the case where no positive rating was issued. The conditioning set is the ratings -1 and 0, whose marginal probability is 30% plus 40%, or 70%, and the numerators are the column sums over those rows: 30%, then 25%, then 15%. Dividing each by 70% gives 42.8%, 35.7% and 21.4%.

Rearranged, the definition expresses the joint PMF as a conditional PMF multiplied by a marginal PMF, in either order. That decomposition is how simulation models are built.

Independence now has a second and more intuitive statement. If the joint PMF equals the marginals multiplied together, dividing by one leaves the other standing, so the conditional and marginal PMFs must be identical. Here the unconditional return distribution is 35%, 30% and 35% while the conditional one is 16.6%, 16.6% and 66.6%.

An expectation is a probability weighted average of outcomes, and that stays true with two components. Only the bookkeeping changes: each outcome is now a pair, the function g takes both coordinates, and the weighting runs over a grid rather than a list.

Two features catch people out. The double sum is required whatever g depends on: a function may involve only x₁ and never mention x₂, and the sum still runs across the support of both, because the probability attached to any value of x₁ is spread across the values x₂ can take alongside it. And expectation does not commute with nonlinearity, so for a nonlinear g the expectation of the function differs from the function of the expectations.

Keeping the two roles separate is the practical skill: g gives the outcomes and f gives their probabilities. Write the products out, weight each by its cell probability, then add.

Moments of a bivariate random variable are built from expectations exactly as univariate moments are. The first moment is the mean, itself a vector holding the mean of each component.

The second moment is where something new appears, because it is a two by two matrix rather than a single number. Along its leading diagonal, the one that runs from the top left corner down to the bottom right, sit the two component variances. The other positions hold the covariance.

Covariance measures dispersion, like the variance, but dispersion of the two variables taken jointly. When both components sit above their own means together, or both below, the product of the deviations is positive; when one is high while the other is low, it is negative. The sign reports the direction the pair moves in and the magnitude how forcefully.

Covariance generalizes variance rather than competing with it, since the covariance of a variable with itself reduces to its variance. That is why a bivariate variable has three distinct second moments and not four: two variances and one covariance, the matrix being symmetric about its leading diagonal because swapping the order of the factors changes nothing.

Two notation schemes circulate and both turn up in exam questions, writing the variances either as sigma one squared and sigma two squared or as sigma eleven and sigma twenty two. The covariance is sigma twelve in both, as are the standard deviations sigma one and sigma two.

The trouble with a covariance is that it carries the units of both variables multiplied together, so its size says as much about the measuring scale as about the relationship. Dividing by the two standard deviations strips the units out and leaves the correlation, a pure number.

Correlation grades the strength of a linear relationship between two variables and always lies between -1 and 1, positive when the two rise together and negative when one falls as the other rises. Rearranged, the definition recovers the covariance from the correlation and the standard deviations, which is how a covariance matrix gets assembled from reported correlations and volatilities.

The extreme values arrive when the relationship is exactly linear. Let X₂ equal a plus b times X₁. The covariance is b times the variance of X₁, because a contributes nothing to a quantity defined on deviations from the mean, and the variance of X₂ is b squared times that variance. Everything cancels except the sign of b.

Independence is stronger than zero correlation

If two random variables are independent then their correlation must be zero, since independence says each contains no information about the other. The implication does not run the other way. Zero correlation rules out a linear relationship only, and variables can depend on each other in ways no straight line describes.

The standard counterexample is short. Let X₁ follow a standard normal distribution and let X₂ be its square. Their covariance is zero, because the covariance of X₁ with its own square is another way of writing the skewness of X₁, and the skewness of a normal is 0. Yet a realization of the first fixes the second exactly, and a realization of the second pins down the first up to a sign.

One consequence survives from zero correlation alone. If the correlation is 0 then so is the covariance, and the shortcut form leaves the expectation of the product equal to the product of the expectations, with no independence assumption needed.

Higher cross moments come from powers of the two variables summing to three or four. Two coskewness measures and three cokurtosis measures are distinct, all of them standardized, and they ask whether one variable takes a clear direction when the other is large in magnitude.

Variables get rescaled and shifted constantly, as returns move between percentage points and basis points, profits move between currencies, and positions get sized up or down. Knowing how the second moments respond saves recomputing anything from the raw distribution.

Covariance is defined on deviations from the mean, and adding a constant moves the variable and its mean by the same amount, so location shifts have no effect at all. Scale is different, because each component contributes multiplicatively.

The variance behaves the same way with a single scale factor: a shift by a leaves it alone while rescaling by b multiplies it by b squared. That is why the correlation is scale free. The numerator picks up the product of the two scale factors and the denominator picks up the absolute value of each, so only the signs survive.

Three consequences follow. Changing units alters the covariance but never the correlation, which is why correlation is the number quoted in risk reports. Reversing the direction of one position, so its scale factor is negative, flips the sign of the correlation and leaves the magnitude alone. Reversing both restores the original correlation, since the sign changes cancel.

Portfolio weights are the most common rescaling of all. Weighting the first variable by w and the second by 1 minus w gives a covariance of w times 1 minus w times the original covariance, which is one piece of the portfolio variance calculation in the next section.

The Big Firm and Small Firm figures make this visible. Their covariance in millions of US dollars was 23.30 and their correlation 0.383. Restate both profit streams in thousands and the covariance jumps by a factor of a million while the correlation stays at 0.383, which is the property that makes it comparable across markets.

Adding two random variables adds their means, whatever the relationship between them. Variance is not so accommodating: the variance of a sum adds the two variances and then twice the covariance, and that third term is where dependence enters.

For independent components the covariance is zero and the variances simply add. When the components move together the extra term is positive and the sum is more volatile than the pieces suggest; when they lean against each other the sum is calmer. Weighting carries the constants through as the previous section established.

Correlation and portfolio diversification

A portfolio return depends on the distribution of the asset returns and on the weights placed on them, a weight being the share of funds invested. Weights sum to 1 by construction, so a two asset portfolio has weights w and 1 minus w, and its return variance is the weighted sum formula with those constants.

Differentiating that variance with respect to w, setting the derivative to zero and rearranging gives the weight that makes the portfolio variance as small as it can be.

The chapter plots the resulting portfolio standard deviation for a first asset with equity market like volatility of 20% and a second with corporate bond like volatility of 10%, as the correlation varies between -1 and 1. It reaches a theoretical minimum of zero when the correlation is -1 or 1. Negative correlation makes the optimal weight positive, calling for a long position in both assets, while positive correlation makes it negative, calling for a short position in one of them.

The picture is not symmetric. Large positive correlations produce larger portfolio standard deviations than small negative ones, and the reason is the weighting itself: once the optimal weight turns negative, the second asset carries a weight above 1. That extra gross exposure is a source of variance in its own right, whereas a negative correlation puts both weights between 0 and 1 and the effect never appears.

Many applications ask what to expect from one variable once something is known about another. The expected loss on a portfolio, conditional on a sharply negative market move, is the archetype, and any stress scenario is the same question. The tool is the conditional expectation, an ordinary expectation computed with a conditional PMF in place of the marginal one.

Return to the stock return and analyst rating example. Conditional on a positive rating the return probabilities were 16.6%, 16.6% and 66.6%, so the conditional expected return is -5% times 16.6% plus 5% times 66.6%, which comes to 2.5%. The unconditional expected return is -5% times 35% plus 0 times 30% plus 5% times 35%, or exactly 0. So a positive rating shifts the expectation by two and a half percentage points, and that shift is the information the rating carries.

Conditioning on a set works the same way. Given that the rating was not positive, the conditional probabilities were 42.8%, 35.7% and 21.4%, so the conditional expected return is -5% times 42.8% plus 5% times 21.4%, which is -1.07%. The absence of a positive rating is bad news of a milder kind than a positive rating is good news.

Conditional variance and conditional standard deviation

Every moment has a conditional counterpart, obtained by replacing each expectation operator with a conditional one. The variance is the usual case in practice.

The unconditional standard deviation comes from the marginal distribution: squared returns weighted by 35%, 30% and 35% give a second moment of 0.00175, the mean is 0, and the square root of the difference is 4.18%. The same arithmetic with the conditional probabilities, where the conditional mean is 2.5%, gives 3.81%.

Conditioning did two things. It moved the centre of the distribution from 0 up to 2.5%, and it narrowed the spread from 4.18% to 3.81%. A conditioning variable that achieved neither would be telling you nothing worth knowing.

As a rule, random variables in finance and risk management are dependent, and that dependence comes from several directions at once: shifts in investor risk aversion, spillovers across assets or borders, crowded positioning. Conditioning can remove dependence rather than merely measure it.

Suppose an investor wants to isolate firms run in a conservative style, labelled Z. Say 12.5% of companies fall in that category, and the joint behaviour of returns and ratings differs between that group and the rest while the overall probabilities still reconcile to the original matrix, with the chance of a negative rating alongside a -5% return still totalling 20%.

Now restrict attention to the conservative group. Each probability in that panel is rescaled by one constant so the conditional probabilities sum to 1, and that constant is 12.5%, the total of the panel before rescaling.

Source: the chapter example on management style, rescaled by the 12.5% probability of the conservative category.

Apply the independence test to this conditional matrix and it passes everywhere. The upper left cell is 8%, and the product of its marginals is 40% times 20%, also 8%. Every other cell agrees. So within the conservative group the rating and the return are independent, even though the pair is not independent in the population as a whole.

The reading is that management style was driving both variables at once. Once the style is pinned down nothing links the rating and the return. Credit models are built on this idea, treating defaults as independent once a common factor is fixed.

Moving from discrete to continuous bivariate variables changes surprisingly little. The mass function becomes a probability density function, sums become integrals, and the definitions, the moments and the way linear transformations act on the covariance are identical in substance.

The joint PDF returns a value at a single point, and that value is not a probability. As with one continuous variable, positive probability attaches to intervals rather than to individual outcomes. Probability over a rectangular region comes from integrating the density across it, and integrating across the whole support gives one.

The joint CDF is the same integral with both lower bounds at minus infinity and the upper bounds set to its arguments. Where the density is undefined below some point, the lower limits move up to the smallest values with support.

Most continuous multivariate distributions have complicated densities. The simplest is the bivariate uniform with independent components, flat at 1 over the unit square, a square with sides of length 1, whose CDF is the product of the two coordinates. Uniform variables need not be independent, and dependent uniforms carry the dependence structure of any multivariate distribution.

Marginal and conditional densities

A marginal PDF removes one component from the joint PDF by integration, the analogue of summing a component out of a joint PMF, and the result is a univariate density. A conditional PDF is the ratio of the joint PDF to the marginal PDF of the conditioning variable.

Conditioning on an interval matters more in risk work than conditioning on a point, because the question is usually how one asset behaves given that another has fallen heavily. Consider the return on a hedge fund given that the S&P 500 monthly return sits in the bottom 5% of its distribution. Historical data show the S&P 500 return is above -6.19% in 95% of months, so the conditioning event is a return below -6.19%. The conditional density integrates the joint density across that region and divides by the probability of the region. This structure reappears when expected shortfall is defined.

An independent and identically distributed sequence, shortened to iid, is a collection of random variables generated by one univariate distribution and independent of each other. Identical distribution gives every member the same mean, variance and shape. Independence means no member carries information about any other, so every covariance between distinct members is zero.

Sequences of this kind are the standard building block for the shocks that drive models, particularly in time series work.

The pay off shows up as soon as the members are added. Expectations are linear, so the expectation of a sum is the sum of the expectations whether or not anything is independent, and identical distribution makes every term the same.

The variance needs both parts. In general it adds the variances together with twice every distinct pairwise covariance, and there are n times n minus 1, all divided by 2, such pairs. Independence sends every one of those covariances to zero, and identical distribution makes the remaining n variances equal.

Drop either assumption and the simplification weakens. Without independence the covariances stay in the expression. Without identical distribution the variance of the sum still adds the individual variances together, but each may differ, so no factor of n comes out.

The sum of n variables is not n times one variable

These look similar written down and behave completely differently. The variance of the sum of two iid variables is twice the common variance, because two independent draws are combined. The variance of twice a single variable is four times it, because one draw is scaled up.

That contrast matters for estimation. Since the variance of the sum grows linearly, dividing by n to form an average divides the variance by n squared, so the variance of the average falls as the sample grows. Every additional observation buys precision.

Sums of iid normal variables are themselves normally distributed, because iid normal variables are jointly normal with zero correlation. The direction of that argument matters. Two variables that each have a univariate normal distribution do not automatically form a bivariate normal pair, since their dependence may be of a kind no normal can produce, and a bivariate normal always looks elliptical when plotted. Add independence and the joint distribution is bivariate normal with a correlation of 0.