Probability Theory

Preface

This treatise presents probability theory in the axiomatic style expected of a student preparing for the ISI M.Stat entrance examination or IIT JAM Mathematical Statistics — and, given the substantial syllabus overlap, the CMI entrance test as well. It assumes fluency with real analysis (limits, continuity, Riemann integration, basic measure-theoretic vocabulary) and advanced linear algebra (eigenvalues, quadratic forms, positive-definiteness, orthogonal diagonalization). The emphasis throughout is on structural completeness: precise statements of theorems, complete proofs — with any unavoidable gap flagged explicitly rather than papered over — canonical tricks, and challenge problems calibrated to genuine entrance-examination difficulty. Every Challenge Problem carries a complete worked solution; readers are strongly encouraged to attempt each one under time pressure before reading further.

Table of Contents

1. Foundations, Combinatorial Probability, and Conditioning
2. Random Variables, Expectations, and Moments
3. Univariate Distributions and Transformations
4. Joint Distributions and Order Statistics
5. Multivariate Distributions (Multinomial & Normal)
6. Appendix: Master Formula Sheet
7. Closing Remark: The Unifying Theme

Conventions and Notation

• $(\Omega, \mathcal{F}, P)$ denotes a probability space; $\Omega$ the sample space, $\mathcal{F}$ a $\sigma$-field of events, $P$ a probability measure.
• Random variables are denoted $X, Y, Z, \dots$; realizations by lowercase $x, y, z$.
• $F_X$ denotes the CDF of $X$; $f_X$ the PDF/PMF.
• $E[X]$, $\mathrm{Var}(X)$, $\mathrm{Cov}(X,Y)$ have their usual meanings.
• $\mathbf{1}_A$ denotes the indicator random variable of event $A$.
• $\stackrel{d}{=}$ denotes equality in distribution; $X_n \stackrel{d}{\to} X$ denotes convergence in distribution.
• $\phi(t)$ denotes a characteristic function; $M_X(t)$ an MGF.
• $\mathrm{tr}(M)$ denotes the trace of a square matrix $M$; $\mathrm{Leb}$ denotes Lebesgue measure.
• $\mathbb{S}^{p-1} = \{\mathbf{x}\in\mathbb{R}^p : \|\mathbf{x}\|=1\}$ denotes the unit sphere in $\mathbb{R}^p$.

---

Part 1 — Foundations, Combinatorial Probability, and Conditioning

1.1 Sample Spaces and $\sigma$-Fields

Definition 1.1 (Sample space). The sample space $\Omega$ is the set of all possible outcomes of a random experiment.

Definition 1.2 ($\sigma$-field). A collection $\mathcal{F}$ of subsets of $\Omega$ is a $\sigma$-field if:
(i) $\Omega \in \mathcal{F}$; (ii) $A \in \mathcal{F} \implies A^c \in \mathcal{F}$; (iii) $A_1, A_2, \dots \in \mathcal{F} \implies \bigcup_{i=1}^\infty A_i \in \mathcal{F}$.

Remark (Why restrict to a $\sigma$-field at all?). When $\Omega$ is uncountable — e.g. $\Omega=[0,1]$ with $P$ meant to be "length" — it is a genuine theorem, not a technicality, that no countably additive, translation-invariant $P$ can be defined on every subset of $\Omega$: Vitali-type constructions (using the Axiom of Choice) produce sets to which no consistent length can be assigned. A $\sigma$-field is the largest class of sets on which $P$ can be forced to behave consistently, and the Borel $\sigma$-field — generated by the open intervals — is rich enough to contain every event of practical interest. This is also why a single outcome can satisfy $P(\{\omega\})=0$ while $P(\Omega)=1$: for uncountable $\Omega$ this is not a paradox but the generic situation (every continuous distribution assigns probability zero to every singleton), and correctly interpreting conditional expectation given a continuous random variable (Part 4) requires exactly this vocabulary.

Definition 1.3 (Kolmogorov's Axioms). $P: \mathcal{F} \to [0,1]$ is a probability measure if:

1. $P(\Omega) = 1$.
2. $P(A) \geq 0$ for all $A \in \mathcal{F}$.
3. (Countable additivity) For pairwise disjoint $A_1, A_2, \dots \in \mathcal{F}$,
   $P\left(\bigcup_{i=1}^\infty A_i\right) = \sum_{i=1}^\infty P(A_i).$

Theorem 1.1 (Basic consequences). From the axioms: (a) $P(\emptyset) = 0$; (b) $P(A^c) = 1 - P(A)$; (c) $A \subseteq B \implies P(A) \le P(B)$; (d) (continuity from below) if $A_1\subseteq A_2\subseteq\cdots$ then $P\!\left(\bigcup_n A_n\right)=\lim_n P(A_n)$; (e) (continuity from above) if $A_1\supseteq A_2\supseteq\cdots$ then $P\!\left(\bigcap_n A_n\right)=\lim_n P(A_n)$.

Proof. (a) Apply countable additivity to the pairwise disjoint sequence $\emptyset,\emptyset,\emptyset,\dots$ (disjointness holds trivially): since $\bigcup_i\emptyset=\emptyset$, we get $P(\emptyset)=\sum_{i=1}^\infty P(\emptyset)$. The right side is an infinite sum of copies of the single fixed number $P(\emptyset)\ge0$; this can equal the finite number $P(\emptyset)$ only if $P(\emptyset)=0$ — any positive value would make the sum diverge to $+\infty$.

(b) $\Omega = A\cup A^c$ is a disjoint union, so $1=P(\Omega)=P(A)+P(A^c)$ by additivity.

(c) $B = A \cup (B\setminus A)$ is a disjoint union, so $P(B) = P(A) + P(B\setminus A) \geq P(A)$, using $P(B\setminus A)\ge0$.

(d) Set $B_1=A_1$ and $B_n = A_n\setminus A_{n-1}$ for $n\ge2$; these are pairwise disjoint, and by induction $\bigcup_{i=1}^n B_i = A_n$ for every $n$ (the case $n=1$ is trivial, and $\bigcup_{i=1}^n B_i = A_{n-1}\cup(A_n\setminus A_{n-1})=A_{n-1}\cup A_n=A_n$ using $A_{n-1}\subseteq A_n$). Hence $\bigcup_{i=1}^\infty B_i=\bigcup_n A_n$, and countable additivity gives
$P\left(\bigcup_n A_n\right)=\sum_{i=1}^\infty P(B_i)=\lim_{n\to\infty}\sum_{i=1}^n P(B_i)=\lim_{n\to\infty}P\left(\bigcup_{i=1}^n B_i\right)=\lim_{n\to\infty}P(A_n).$

(e) Apply (d) to the increasing sequence $A_1^c\subseteq A_2^c\subseteq\cdots$: $P\!\left(\bigcup_n A_n^c\right)=\lim_n P(A_n^c)=\lim_n[1-P(A_n)]=1-\lim_n P(A_n)$ using (b). By De Morgan, $\bigcup_n A_n^c = \left(\bigcap_n A_n\right)^c$, so the left side equals $1-P\!\left(\bigcap_n A_n\right)$ by (b) again. Equate the two expressions and cancel the $1$'s. $\blacksquare$

1.2 Combinatorial Probability

Classical model. If $\Omega$ is finite and outcomes equally likely, $P(A) = |A|/|\Omega|$.

Trick (Stars and Bars). The number of ways to distribute $n$ identical objects into $k$ distinct boxes is $\binom{n+k-1}{k-1}$. This underlies occupancy problems, and its continuous limit is revisited in Part 4 (Theorem 4.5).

Theorem 1.2 (Inclusion-Exclusion). For events $A_1, \dots, A_n$,
$P\left(\bigcup_{i=1}^n A_i\right) = \sum_i P(A_i) - \sum_{i<j} P(A_i \cap A_j) + \cdots + (-1)^{n+1} P(A_1 \cap \cdots \cap A_n).$

Proof. Induct on $n$, with base case $n=2$: $P(A\cup B)=P(A)+P(B)-P(A\cap B)$, itself immediate from the disjoint decomposition $A\cup B = A \cup (B\setminus A)$ together with $P(B\setminus A)=P(B)-P(A\cap B)$ (Theorem 1.1(c)'s argument applied to $A\cap B\subseteq B$). For the inductive step, write $C=A_1\cup\cdots\cup A_{n-1}$ and apply the base case to $C\cup A_n$:
$P\left(\bigcup_{i=1}^n A_i\right) = P(C) + P(A_n) - P(C\cap A_n).$
Now $P(C)$ expands by the inductive hypothesis applied to $A_1,\dots,A_{n-1}$, and $C\cap A_n = \bigcup_{i=1}^{n-1}(A_i\cap A_n)$, so $P(C\cap A_n)$ also expands by the inductive hypothesis, now applied to the $n-1$ events $A_1\cap A_n,\dots,A_{n-1}\cap A_n$. Collecting all resulting terms by the size of the index set reproduces exactly the alternating sum claimed for $n$ events; the bookkeeping is routine, but the two invocations of the inductive hypothesis are the entire content of the step. $\blacksquare$

Worked Example 1.1 (Derangements). The probability that a random permutation of $n$ objects has no fixed point (a derangement) is
$P(D_n) = \sum_{k=0}^n \frac{(-1)^k}{k!} \to e^{-1} \text{ as } n \to \infty.$

Solution. Let $A_i=\{\pi(i)=i\}$. For any $k$ fixed indices $i_1,\dots,i_k$, freezing those $k$ positions and permuting the rest freely gives $P(A_{i_1}\cap\cdots\cap A_{i_k}) = (n-k)!/n!$, and there are $\binom{n}{k}$ such terms of size $k$, contributing
$\binom{n}{k}\cdot\frac{(n-k)!}{n!} = \frac{n!}{k!(n-k)!}\cdot\frac{(n-k)!}{n!}=\frac{1}{k!}$
to the $k$-th layer of inclusion–exclusion — the combinatorial factor cancels completely against the probability, which is the entire reason the final answer is so clean. Summing, $P\!\left(\bigcup A_i\right)=\sum_{k=1}^n(-1)^{k+1}/k!$, and $P(D_n)=1-P\!\left(\bigcup A_i\right)=\sum_{k=0}^n(-1)^k/k!$, the truncated Taylor series for $e^{-1}$.

1.3 Conditional Probability and Independence

Definition 1.4. $P(A\mid B) = P(A \cap B)/P(B)$ for $P(B) > 0$.

Definition 1.5 (Independence). $A, B$ are independent if $P(A \cap B) = P(A)P(B)$. Events $A_1, \dots, A_n$ are mutually independent if for every subset $I \subseteq \{1,\dots,n\}$ with $|I|\ge2$, $P\left(\bigcap_{i\in I} A_i\right) = \prod_{i \in I} P(A_i)$ — a strictly stronger condition than pairwise independence.

Remark (Pairwise $\neq$ Mutual, made explicit). Toss two fair coins independently, so $\Omega=\{HH,HT,TH,TT\}$ with each outcome having probability $1/4$. Let $A=\{HH,HT\}$ (first coin H), $B=\{HH,TH\}$ (second coin H), $C=\{HH,TT\}$ (both coins agree). Then $P(A)=P(B)=P(C)=1/2$, and in fact $A\cap B = A\cap C = B\cap C = \{HH\}$, so
$P(A\cap B)=P(A\cap C)=P(B\cap C)=\frac{1}{4}=P(A)P(B)=P(A)P(C)=P(B)P(C):\quad\textbf{pairwise independent.}$
Yet $A\cap B\cap C=\{HH\}$ too, so $P(A\cap B\cap C)=\frac{1}{4}\ne\frac{1}{8}=P(A)P(B)P(C)$: not mutually independent. The failure is driven entirely by the fact that any two of $A,B,C$ logically determine the third.

1.4 Bayes' Theorem and Total Probability

Theorem 1.3 (Law of Total Probability). If $\{B_i\}$ partitions $\Omega$ with $P(B_i) > 0$, then $P(A) = \sum_i P(A \mid B_i)P(B_i)$. The continuous analogue: if $\Theta$ has density $f_\Theta$, $P(A) = \int P(A \mid \Theta = \theta) f_\Theta(\theta)\, d\theta$.

Theorem 1.4 (Bayes). $P(B_i \mid A) = \dfrac{P(A\mid B_i) P(B_i)}{\sum_j P(A \mid B_j) P(B_j)}$.

1.5 Problem-Solving Tricks

Trick 1 (Pólya Urn). An urn starts with $b$ black and $r$ red balls. Draw a ball, replace it along with $c$ balls of the same color, repeat. A key invariant: the probability that the $n$-th draw is black equals $b/(b+r)$ for every $n$. This follows from exchangeability: for any specific sequence of colors with $k$ black and $n-k$ red draws, in any order, the chain rule gives a product of $n$ factors of the form $\frac{b+jc}{b+r+jc}$ or $\frac{r+jc}{b+r+jc}$ — and multiplying the same $n$ numbers in a different order does not change the product. So every ordering with $k$ black draws is equally likely, meaning the sequence is exchangeable; exchangeability in particular forces every individual draw to share the marginal law of the first draw, which is trivially $\mathrm{Bernoulli}(b/(b+r))$.

Trick 2 (Sequential Conditioning / Chain Rule). $P(A_1 \cap \cdots \cap A_n) = P(A_1)P(A_2\mid A_1) \cdots P(A_n \mid A_1 \cap \cdots \cap A_{n-1})$ — indispensable for problems involving sequential draws without replacement.

Trick 3 (Absorbing Markov Chains for Exact Probabilities). Gambler's ruin-type problems: let $p_i$ be the probability of eventual absorption at state $N$ starting from $i$, with $p_i = p\, p_{i+1} + q\, p_{i-1}$ ($q=1-p$), $p_0=0$, $p_N=1$. The characteristic equation of this linear recurrence is $px^2-x+q=0$; since $1-4pq=(p-q)^2$, the roots are $x=1$ and $x=q/p$. Solving with the boundary conditions gives closed forms in each case:
$p_i = \frac{1-(q/p)^i}{1-(q/p)^N}\quad(p\ne q), \qquad\qquad p_i=\frac{i}{N}\quad(p=q=\frac{1}{2},\text{ a double root at }x=1).$

Challenge Problem 1.1. A fair die is rolled repeatedly. Find the probability that the sequence of rolls contains the pattern $(1,2,3)$ (in that order, consecutively) before it contains the pattern $(1,1)$.

Solution. Track the state of progress: the longest suffix of the sequence rolled so far that is a proper prefix of $(1,2,3)$ or of $(1,1)$. The reachable non-absorbing states are start (no progress), $\texttt{"1"}$ (progress shared by both patterns), and $\texttt{"12"}$ (progress exclusive to $123$); the absorbing states are A wins ($123$ completes) and B wins ($11$ completes). Checking every state against every face of the die:

(The "12" $\to$ "1" transition on rolling a $1$ holds because the new suffix "121" matches neither pattern, "21" matches neither pattern, but "1" matches the shared prefix of both — so "1" is the longest valid match.)

Let $a,b,c$ denote $P(\text{A wins})$ starting from states start, "1", "12" respectively. First-step analysis (Trick 3's method, now with three transient states instead of one) gives
$a=\frac{5}{6}a+\frac{1}{6}b,\qquad b=\frac{1}{6}(0)+\frac{1}{6}c+\frac{4}{6}a,\qquad c=\frac{1}{6}(1)+\frac{1}{6}b+\frac{4}{6}a.$
The first equation simplifies to $a=b$. Substituting into the second: $a=\frac{1}{6}c+\frac{4}{6}a\implies c=2a$. Substituting both into the third:
$2a=\frac{1}{6}+\frac{1}{6}a+\frac{4}{6}a=\frac{1}{6}+\frac{5}{6}a\ \implies\ \frac{7}{6}a=\frac{1}{6}\ \implies\ a=\frac{1}{7}.$
$P(123\text{ before }11)=\frac{1}{7},\qquad\qquad P(11\text{ before }123)=\frac{6}{7}.$
This lopsided result is intuitive in hindsight: every time progress toward $123$ is "almost lost" by rolling a $1$ from state "12", the chain lands back at state "1" — which is exactly one more $1$ away from an immediate win for pattern $11$. The shared prefix systematically favors the shorter, self-overlapping pattern. $\blacksquare$

---

Part 2 — Random Variables, Expectations, and Moments

2.1 Random Variables and CDFs

Definition 2.1. A random variable $X: \Omega \to \mathbb{R}$ is $\mathcal{F}$-measurable: $\{\omega : X(\omega) \le x\} \in \mathcal{F}$ for all $x$. (This apparently weak requirement — checking only half-lines — is equivalent to full Borel measurability, since the sets $(-\infty,x]$, $x\in\mathbb{R}$, generate the Borel $\sigma$-field $\mathcal{B}(\mathbb{R})$.)

Definition 2.2 (CDF). $F_X(x) = P(X \le x)$.

Theorem 2.1 (Properties of a CDF). $F_X$ is non-decreasing, right-continuous, $\lim_{x\to-\infty}F_X = 0$, $\lim_{x\to\infty}F_X = 1$. Conversely, any function with these properties is the CDF of some random variable (existence via the probability integral transform / quantile function $F_X^{-1}(U)$ for $U \sim \mathrm{Unif}(0,1)$).

Random variables may be discrete (countable support, PMF $p_X$), continuous ($F_X$ absolutely continuous, density $f_X = F_X'$), or mixed (a convex combination, e.g., insurance claim sizes with an atom at zero).

Remark (Radon–Nikodym and the existence of densities). More precisely: $X$ is continuous means its law $P_X(A):=P(X\in A)$ — the pushforward of $P$ under $X$ — satisfies $P_X\ll\mathrm{Leb}$ (absolute continuity: $\mathrm{Leb}(A)=0\implies P_X(A)=0$). The Radon–Nikodym theorem then guarantees a density $f_X=dP_X/d\,\mathrm{Leb}$ exists, unique up to a Lebesgue-null set, with $P(X\in A)=\int_A f_X\,d\,\mathrm{Leb}$ for every Borel $A$. The familiar identity $f_X=F_X'$ recovers this density in the special case where $F_X$ is classically differentiable, which absolute continuity guarantees at Lebesgue-almost every point (Lebesgue's differentiation theorem). The "mixed" distributions of elementary probability — an atom superposed on a density, as in insurance claim sizes — are precisely the case where the singular part of the general Lebesgue decomposition $P_X = P_X^{\mathrm{ac}} + P_X^{\mathrm{sing}}$ is purely discrete; the fully general decomposition also permits pathological singular continuous components (e.g. measures supported on Cantor-type sets), which never arise from the standard constructions of this treatise and may safely be set aside.

2.2 Expectation

Definition 2.3. $E[X] = \sum_x x\, p_X(x)$ (discrete) or $\int x f_X(x)\,dx$ (continuous), when absolutely convergent.

Theorem 2.2 (LOTUS — Law of the Unconscious Statistician). $E[g(X)] = \sum_x g(x) p_X(x)$ or $\int g(x) f_X(x)\,dx$ — one need not derive the distribution of $g(X)$ to compute its expectation.

Theorem 2.3 (Linearity of Expectation). $E[aX + bY] = aE[X] + bE[Y]$, regardless of dependence between $X$ and $Y$. This is the single most powerful tool in combinatorial probability.

Trick (Indicator Variable Method). To compute $E[N]$ where $N$ counts occurrences of some event pattern, write $N = \sum_i \mathbf{1}_{A_i}$ and use $E[N] = \sum_i P(A_i)$, entirely sidestepping the (often intractable) distribution of $N$ itself.

Worked Example 2.1 (Matching Problem via Indicators). For a random permutation of $\{1,\dots,n\}$, let $N$ be the number of fixed points. With $A_i = \{\pi(i) = i\}$, $P(A_i) = 1/n$, so $E[N] = \sum_{i=1}^n \frac{1}{n} = 1$ — independent of $n$, and obtained without ever finding the distribution of $N$.

2.3 Moments, Variance, and Inequalities

Definition 2.4. $\mathrm{Var}(X) = E[(X-E[X])^2] = E[X^2] - (E[X])^2$. $\mathrm{Cov}(X,Y) = E[XY] - E[X]E[Y]$.

Worked Example 2.2 (Variance via indicators, and an MGF cross-check). For $S_n=\sum_{i=1}^n X_i$ with $X_i\stackrel{iid}{\sim}\mathrm{Bernoulli}(p)$: since $X_i^2=X_i$ (it is an indicator), $\mathrm{Var}(X_i)=E[X_i]-E[X_i]^2=p-p^2=p(1-p)$, and independence gives $\mathrm{Cov}(X_i,X_j)=0$ for $i\ne j$, so
$\mathrm{Var}(S_n)=\sum_{i=1}^n\mathrm{Var}(X_i) + \sum_{i \ne j}\mathrm{Cov}(X_i,X_j) = np(1-p).$
As a cross-check via $M_{S_n}(t)=(1-p+pe^t)^n$: differentiating twice and setting $t=0$ gives $M_{S_n}'(0)=np$ and $M_{S_n}''(0)=n(n-1)p^2+np$, so $\mathrm{Var}(S_n)=M_{S_n}''(0)-[M_{S_n}'(0)]^2 = n(n-1)p^2+np-n^2p^2=np(1-p)$ — agreeing exactly, as it must.

Theorem 2.4 (Markov's Inequality). For $X \geq 0$ and $a > 0$: $P(X \geq a) \leq E[X]/a$.

Proof. $E[X] = E[X\mathbf{1}_{X<a}]+E[X \mathbf{1}_{X \geq a}] \geq E[X \mathbf{1}_{X \geq a}] \geq a\, P(X \geq a)$, using $E[X\mathbf{1}_{X<a}]\ge0$ for the first inequality and $X\ge a$ on $\{X\ge a\}$ for the second. $\blacksquare$

Theorem 2.5 (Chebyshev's Inequality). $P(|X - \mu| \geq k\sigma) \leq 1/k^2$. Proof: apply Markov's inequality to the non-negative random variable $(X-\mu)^2$ with $a=k^2\sigma^2$. $\blacksquare$

Theorem 2.6 (Cauchy–Schwarz). $|E[XY]| \leq \sqrt{E[X^2]E[Y^2]}$, with equality iff $Y = cX$ a.s. This underlies $|\mathrm{Corr}(X,Y)| \leq 1$.

2.4 Moment Generating Functions

Definition 2.5. $M_X(t) = E[e^{tX}]$, defined for $t$ in some neighborhood of $0$.

Theorem 2.7 (Moment recovery). $M_X^{(k)}(0) = E[X^k]$, by differentiating under the expectation (Leibniz rule, justified when $M_X$ is finite in a neighborhood of $0$).

Theorem 2.8 (Uniqueness). If $M_X(t) = M_Y(t)$ for $t$ in a neighborhood of $0$, then $F_X = F_Y$. This makes MGFs a powerful tool for identifying distributions of sums of independent variables: $M_{X+Y}(t) = M_X(t)M_Y(t)$ when $X \perp Y$.

Remark (PGFs and Characteristic Functions). For non-negative integer-valued $X$, the probability generating function $G_X(s) = E[s^X]$ is convenient (e.g., for branching processes). For heavy-tailed distributions where the MGF fails to exist (e.g., Cauchy, where $E[e^{tX}] = \infty$ for $t \neq 0$), the characteristic function $\phi_X(t) = E[e^{itX}]$ always exists since $|e^{itX}| = 1$, and by Lévy's inversion theorem uniquely determines the distribution.

Remark (Lévy's Continuity Theorem). Characteristic functions earn their keep in asymptotic theory through: $X_n\stackrel{d}{\to}X$ if and only if $\phi_{X_n}(t)\to\phi_X(t)$ for every $t\in\mathbb{R}$. More usefully, if $\phi_{X_n}(t)\to g(t)$ pointwise for some function $g$ continuous at $t=0$, then $g$ is automatically the characteristic function of some random variable $X$, and $X_n\stackrel{d}{\to}X$. This is the standard mechanism behind a proof of the Central Limit Theorem (identify the pointwise limit of $\phi_{\bar X_n}$ as the $N(0,1)$ characteristic function $e^{-t^2/2}$, then invoke this theorem) — and it resurfaces in Challenge Problem 3.1 to explain a case where the naive CLT machinery cannot even get started.

Challenge Problem 2.1 (Occupancy: empty boxes). $n$ distinguishable balls are thrown independently and uniformly into $m$ distinguishable boxes. Let $W$ be the number of empty boxes. Find $E[W]$ and $\mathrm{Var}(W)$.

Solution. Let $I_j=\mathbf{1}\{\text{box }j\text{ is empty}\}$ for $j=1,\dots,m$, so $W=\sum_{j=1}^m I_j$. Each of the $n$ balls independently avoids box $j$ with probability $\frac{m-1}{m}$, so
$E[I_j]=P(\text{box }j\text{ empty})=\left(\frac{m-1}{m}\right)^n=:p_1,\qquad\qquad E[W]=mp_1=m\left(1-\frac{1}{m}\right)^n.$
For the variance we need the dependence between different indicators — this is what makes the problem genuinely harder than Worked Example 2.2's independent case. For $j\ne k$, both boxes $j,k$ are empty iff every ball avoids both, which happens independently for each ball with probability $\frac{m-2}{m}$ (for $m\ge2$), so
$E[I_jI_k]=P(\text{boxes }j,k\text{ both empty})=\left(\frac{m-2}{m}\right)^n=:p_3,\qquad\qquad \mathrm{Cov}(I_j,I_k)=p_3-p_1^2.$
Also $\mathrm{Var}(I_j)=p_1-p_1^2$, since each $I_j$ is Bernoulli. There are $m$ diagonal terms and $m(m-1)$ ordered off-diagonal pairs, so
$\mathrm{Var}(W)=m(p_1-p_1^2)+m(m-1)(p_3-p_1^2)=mp_1+m(m-1)p_3-m^2p_1^2.$
Explicitly,
$\mathrm{Var}(W)=m\left(\frac{m-1}{m}\right)^n+m(m-1)\left(\frac{m-2}{m}\right)^n-m^2\left(\frac{m-1}{m}\right)^{2n}.$
(As a check: $n=3,m=2$ gives $E[W]=\frac{1}{4}$, $\mathrm{Var}(W)=\frac{3}{16}$; direct enumeration of all $2^3=8$ equally likely outcomes confirms both values exactly.) $\blacksquare$

Remark. This "balls into boxes" computation is the discrete sibling of the spacing problem of Part 4 (Challenge Problem 4.2): both ask "how many of $k$ regions receive no mass?", with the continuum limit trading the present Binomial-flavored inclusion–exclusion for the Exponential/Gamma machinery of Theorem 4.5.

---

Part 3 — Univariate Distributions and Transformations

3.1 Standard Discrete Distributions

Theorem 3.1 (Poisson as limit of Binomial). If $X_n \sim \mathrm{Binomial}(n, \lambda/n)$, then $P(X_n = k) \to e^{-\lambda}\lambda^k/k!$ as $n \to \infty$.

Proof sketch. Expand $\binom{n}{k}(\lambda/n)^k(1-\lambda/n)^{n-k}$; the binomial coefficient contributes $n^k/k!$ asymptotically, $(1-\lambda/n)^n \to e^{-\lambda}$, and the remaining factors $\to 1$. $\blacksquare$

Trick (Memorylessness characterizes Geometric among discrete distributions). If $X$ is a positive-integer-valued random variable satisfying $P(X > s+t\mid X>s) = P(X>t)$ for all integers $s,t\ge0$, then $X$ is Geometric — a useful identification tool in exam problems.

3.2 Standard Continuous Distributions

Remark (Cauchy pathology). $\int_{-\infty}^\infty |x| f(x)\,dx = \infty$, so $E[X]$ fails to exist — not merely infinite, but genuinely undefined since the integral is not absolutely convergent. Yet the Cauchy distribution arises naturally as the ratio of two independent standard Normals (Challenge Problem 3.1), and again as the $t$-distribution on $1$ degree of freedom; its characteristic function $\phi(t) = e^{-|t|}$ exists despite the MGF's non-existence.

3.3 Transformations of Random Variables

Theorem 3.2 (CDF method). For $Y = g(X)$, $F_Y(y) = P(g(X) \le y)$, computed by identifying $\{x : g(x) \le y\}$ and integrating $f_X$ over that set.

Theorem 3.3 (Jacobian / change-of-variables, monotonic case). If $g$ is strictly monotonic and differentiable with inverse $g^{-1}$,
$f_Y(y) = f_X(g^{-1}(y)) \left|\frac{d}{dy}g^{-1}(y)\right|.$

Theorem 3.4 (Non-monotonic case). If $g$ is piecewise monotonic with branches $g_1, \dots, g_k$ and corresponding inverses $g_1^{-1}, \dots, g_k^{-1}$,
$f_Y(y) = \sum_{i=1}^k f_X(g_i^{-1}(y)) \left|\frac{d}{dy}g_i^{-1}(y)\right|.$

Worked Example 3.1. If $Z \sim N(0,1)$ and $Y = Z^2$, the two branches $z = \pm\sqrt{y}$ give
$f_Y(y) = \frac{1}{\sqrt{2\pi y}} e^{-y/2}, \quad y > 0,$
which is $\chi^2_1 = \mathrm{Gamma}(1/2, 1/2)$.

Challenge Problem 3.1 (Cauchy ratio and failure of the CLT). Let $X, Y$ be iid standard Normal. Show that $T = X/Y$ follows a standard Cauchy distribution, and explain — via Lévy's Continuity Theorem (§2.4) — why the Central Limit Theorem does not apply to averages of iid Cauchy random variables.

Solution. Transform $(X,Y)\to(T,Y)$ via $X=TY$, keeping $Y$ unchanged. The Jacobian of $(t,y)\mapsto(x,y)=(ty,y)$ is
$\left|\det\begin{pmatrix}\partial x/\partial t & \partial x/\partial y\\ \partial y/\partial t&\partial y/\partial y\end{pmatrix}\right|=\left|\det\begin{pmatrix}y&t\\0&1\end{pmatrix}\right|=|y|.$
Since $X\perp Y$, $f_{T,Y}(t,y)=f_X(ty)f_Y(y)|y|=\dfrac{1}{2\pi}|y|\,e^{-y^2(1+t^2)/2}$. Integrating out $y$ (the integrand is even in $y$):
$f_T(t)=\frac{1}{\pi}\int_0^\infty y\,e^{-y^2(1+t^2)/2}\,dy=\frac{1}{\pi}\cdot\frac{1}{1+t^2}\int_0^\infty e^{-u}\,du=\frac{1}{\pi(1+t^2)},$
using the substitution $u=y^2(1+t^2)/2$, $du = y(1+t^2)\,dy$. This is exactly the standard Cauchy density.

For the CLT failure: the characteristic function of standard Cauchy is $\phi(t)=e^{-|t|}$ (§3.2). For $\bar X_n=\frac{1}{n}\sum_{i=1}^n X_i$ with $X_i\stackrel{iid}{\sim}\mathrm{Cauchy}(0,1)$, independence gives
$\phi_{\bar X_n}(t)=\left[\phi_X\!\left(\frac{t}{n}\right)\right]^n=\left[e^{-|t|/n}\right]^n=e^{-|t|}=\phi_X(t)\quad\text{for every }n.$
So $\bar X_n\stackrel{d}{=}X_1$ for every sample size — the sample mean never concentrates, in flagrant violation of the usual CLT conclusion. This is not a counterexample to the CLT (whose hypotheses require $E[X^2]<\infty$, in particular $E|X|<\infty$); it is a demonstration of why the hypothesis is needed: without a finite mean there is nothing for $\bar X_n$ to concentrate around, and Lévy's Continuity Theorem shows the characteristic functions witness this directly, rather than merely failing to prove convergence. $\blacksquare$

Challenge Problem 3.2 (The Gamma–Beta algebra). Let $X\sim\mathrm{Gamma}(\alpha,\lambda)$ and $Y\sim\mathrm{Gamma}(\beta,\lambda)$ be independent. Show that $U=X+Y$ and $V=X/(X+Y)$ are independent, with $U\sim\mathrm{Gamma}(\alpha+\beta,\lambda)$ and $V\sim\mathrm{Beta}(\alpha,\beta)$.

Solution. This is the two-dimensional Jacobian transformation par excellence: independence emerges from the change of variables rather than being visible beforehand. Write $x=uv,\ y=u(1-v)$ (inverting $u=x+y,\ v=x/(x+y)$), for $u>0,\ v\in(0,1)$. The Jacobian is
$\left|\det\begin{pmatrix}\partial x/\partial u&\partial x/\partial v\\ \partial y/\partial u&\partial y/\partial v\end{pmatrix}\right|=\left|\det\begin{pmatrix}v&u\\1-v&-u\end{pmatrix}\right|=|-uv-u(1-v)|=u.$
By independence of $X,Y$,
$f_{U,V}(u,v)=f_X(uv)\,f_Y(u(1-v))\cdot u = \frac{\lambda^\alpha}{\Gamma(\alpha)}(uv)^{\alpha-1}e^{-\lambda uv}\cdot\frac{\lambda^\beta}{\Gamma(\beta)}(u(1-v))^{\beta-1}e^{-\lambda u(1-v)}\cdot u.$
Collecting powers of $u$ and using $v+(1-v)=1$ in the exponent,
$f_{U,V}(u,v)=\underbrace{\left[\frac{\lambda^{\alpha+\beta}}{\Gamma(\alpha+\beta)}u^{\alpha+\beta-1}e^{-\lambda u}\right]}_{\mathrm{Gamma}(\alpha+\beta,\lambda)\text{ density in }u}\times\underbrace{\left[\frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)}v^{\alpha-1}(1-v)^{\beta-1}\right]}_{\mathrm{Beta}(\alpha,\beta)\text{ density in }v},$
which factors as a function of $u$ alone times a function of $v$ alone — precisely independence, with both marginals read off directly. $\blacksquare$

Remark. This computation is the engine behind Theorem 4.5's Rényi representation: normalizing $k$ independent $\mathrm{Gamma}(1,\lambda)=\mathrm{Exponential}(\lambda)$ variables by their sum, iterated successively via exactly this Jacobian argument, produces the uniform Dirichlet law on the simplex. The discrete stars-and-bars principle (§1.2) and the continuous Dirichlet spacing law are, quite literally, the same computation carried out with sums replaced by integrals.

---

Part 4 — Joint Distributions and Order Statistics

4.1 Joint, Marginal, and Conditional Distributions

Definition 4.1. Joint CDF $F_{X,Y}(x,y) = P(X\le x, Y \le y)$; joint density $f_{X,Y} = \partial^2 F_{X,Y}/\partial x\,\partial y$ (continuous case). Marginals: $f_X(x) = \int f_{X,Y}(x,y)\,dy$.

Definition 4.2 (Independence). $X \perp Y \iff f_{X,Y}(x,y) = f_X(x)f_Y(y)$ for all $x,y$.

Definition 4.3 (Conditional density and expectation). $f_{Y\mid X}(y\mid x) = f_{X,Y}(x,y)/f_X(x)$; $E[Y\mid X=x] = \int y\, f_{Y\mid X}(y\mid x)\,dy$.

4.2 Adam's and Eve's Laws

Theorem 4.1 (Adam's Law / Law of Total Expectation). $E[Y] = E[E[Y\mid X]]$.

Theorem 4.2 (Eve's Law / Law of Total Variance). $\mathrm{Var}(Y) = E[\mathrm{Var}(Y\mid X)] + \mathrm{Var}(E[Y\mid X])$.

Proof. Let $g(X) = E[Y\mid X]$. Then $\mathrm{Var}(Y) = E[Y^2] - (E[Y])^2$. Write $E[Y^2] = E[E[Y^2\mid X]] = E[\mathrm{Var}(Y\mid X) + g(X)^2]$ by Adam's Law applied within each conditional slice. Also $(E[Y])^2 = (E[g(X)])^2$ by Adam's Law applied to $Y$ itself. Combining:
$\mathrm{Var}(Y) = E[\mathrm{Var}(Y\mid X)] + \left(E[g(X)^2] - (E[g(X)])^2\right) = E[\mathrm{Var}(Y\mid X)] + \mathrm{Var}(g(X)).\ \blacksquare$

Remark. Eve's Law is the theoretical basis of the ANOVA decomposition of variance into within-group and between-group components; §7 revisits it as an instance of the Pythagorean theorem in $L^2(\Omega,\mathcal F,P)$.

4.3 Order Statistics

Definition 4.4. For iid $X_1,\dots,X_n \sim F$, the order statistics are the sorted values $X_{(1)} \le X_{(2)} \le \cdots \le X_{(n)}$.

Theorem 4.3 (Marginal density of $X_{(k)}$).
$f_{X_{(k)}}(x) = \frac{n!}{(k-1)!(n-k)!} F(x)^{k-1}[1-F(x)]^{n-k} f(x).$

Proof sketch. $X_{(k)} \in [x, x+dx)$ requires exactly $k-1$ of the $n$ points below $x$, one in $[x,x+dx)$, and $n-k$ above; the multinomial coefficient counts the assignments, giving the stated density in the limit $dx \to 0$. $\blacksquare$

Theorem 4.4 (Joint density of $X_{(1)}, \dots, X_{(n)}$).
$f_{X_{(1)},\dots,X_{(n)}}(x_1,\dots,x_n) = n!\, f(x_1)\cdots f(x_n), \quad x_1 < x_2 < \cdots < x_n.$

Corollary (Range and extremes). $F_{X_{(n)}}(x) = F(x)^n$; $F_{X_{(1)}}(x) = 1-[1-F(x)]^n$. The distribution of the sample range $R = X_{(n)} - X_{(1)}$ is obtained from the joint density of $(X_{(1)}, X_{(n)})$ via a change of variables.

4.4 Spacings of Uniform Order Statistics

Theorem 4.5 (Uniform spacings). For $U_{(1)} \le \cdots \le U_{(n)}$ from $\mathrm{Unif}(0,1)$, define spacings $D_i = U_{(i)} - U_{(i-1)}$ (with $U_{(0)}=0$, $U_{(n+1)}=1$). The vector $(D_1,\dots,D_{n+1})$ is exchangeable and distributed as a uniform Dirichlet on the simplex; consequently, each spacing satisfies $D_i \stackrel{d}{=} \frac{E_i}{\sum_{j=1}^{n+1}E_j}$ for iid $\mathrm{Exponential}(1)$ variables $E_j$ — the Rényi representation, directly linking Uniform spacings to sums of iid Exponentials (i.e., Gamma variables); Challenge Problem 3.2 supplies the underlying Jacobian mechanism.

Remark (Discrete $\to$ continuous: stars-and-bars grown up). This is exactly the continuum limit of the discrete stars-and-bars principle of §1.2. In the discrete world, splitting a total of $n$ indistinguishable units into $k$ non-negative parts is counted by $\binom{n+k-1}{k-1}$ equally likely compositions. In the continuum, the "total" becomes the unit interval, the "parts" become the $k$ spacings cut out by $k-1$ random dividers, and the uniform-measure analogue of "every composition equally likely" is precisely the uniform density on the simplex $\{\mathbf{d}\ge\mathbf{0}:\sum_i d_i=1\}$ — the $\mathrm{Dirichlet}(1,\dots,1)$ law above. Stars-and-bars is Theorem 4.5 with the integral replaced by a sum.

Challenge Problem 4.1 (Proving the exponential-gap representation). Let $X_1,\dots,X_n\stackrel{iid}{\sim}\mathrm{Exponential}(\lambda)$ with order statistics $X_{(1)}\le\cdots\le X_{(n)}$. Prove that the successive gaps $G_k=X_{(k)}-X_{(k-1)}$ (with $X_{(0)}:=0$) are independent, with $G_k\sim\mathrm{Exponential}\big((n-k+1)\lambda\big)$; deduce closed forms for $E[X_{(k)}]$ and $\mathrm{Var}(X_{(k)})$.

Solution. By Theorem 4.4, the joint density of $(X_{(1)},\dots,X_{(n)})$ is $n!\,\lambda^ne^{-\lambda\sum x_i}$ on $x_1<\cdots<x_n$. Change variables to the gaps $g_k=x_k-x_{k-1}$ (so $x_k=g_1+\cdots+g_k$); this is a linear transformation with unit lower-triangular Jacobian matrix, so $|\det|=1$ — the simplest possible non-trivial instance of a multivariate Jacobian. Each gap $g_j$ contributes to $x_j,x_{j+1},\dots,x_n$, i.e. to $(n-j+1)$ of the $x_k$'s, so
$\sum_{k=1}^n x_k=\sum_{j=1}^n(n-j+1)g_j.$
Hence the joint density of the gaps is
$f(g_1,\dots,g_n)=n!\,\lambda^n\exp\left(-\lambda\sum_{j=1}^n(n-j+1)g_j\right),\qquad g_j>0.$
Since $\prod_{j=1}^n(n-j+1)=n!$ (it is just $n(n-1)\cdots1$ relabelled), the constant $n!\lambda^n$ may be rewritten as $\prod_{j=1}^n\big[(n-j+1)\lambda\big]$, and the density factors:
$f(g_1,\dots,g_n)=\prod_{j=1}^n\Big[(n-j+1)\lambda\, e^{-(n-j+1)\lambda g_j}\Big].$
Each factor is exactly an $\mathrm{Exponential}\big((n-j+1)\lambda\big)$ density, and a joint density that factors into a product of one-variable densities is precisely the statement that the $G_j$ are independent with those marginals — proving the claim. (Intuition: after the first $k-1$ failures, the remaining $n-k+1$ lifetimes are, by memorylessness, still fresh $\mathrm{Exponential}(\lambda)$ variables, and the minimum of $n-k+1$ independent $\mathrm{Exponential}(\lambda)$'s is $\mathrm{Exponential}((n-k+1)\lambda)$ — the algebra above is that heuristic made rigorous.)

Since $G_j\stackrel{d}{=}E_j/[(n-j+1)\lambda]$ for iid standard Exponentials $E_j$, and $X_{(k)}=\sum_{j=1}^kG_j$, independence of the $G_j$ immediately gives
$E[X_{(k)}]=\sum_{j=1}^k\frac{1}{(n-j+1)\lambda}=\frac{1}{\lambda}\sum_{i=n-k+1}^n\frac{1}{i},\qquad\qquad \mathrm{Var}(X_{(k)})=\sum_{j=1}^k\frac{1}{[(n-j+1)\lambda]^2}=\frac{1}{\lambda^2}\sum_{i=n-k+1}^n\frac{1}{i^2}.$
$\blacksquare$

Challenge Problem 4.2 (An indicator count for spacings, and a Poisson-process limit). With $D_1,\dots,D_{n+1}$ the uniform spacings of Theorem 4.5, fix $c>0$ and let $N_n(c)=\#\{i: D_i> c/n\}$ count the spacings exceeding threshold $c/n$. Find $E[N_n(c)]$ exactly using indicator variables, and show $N_n(c)/(n+1)\to e^{-c}$ as $n\to\infty$.

Solution. By exchangeability (Theorem 4.5), every $D_i$ shares the marginal law of $D_1=U_{(1)}$, the minimum of the $n$ uniform points — and the Corollary to Theorem 4.3 already gives $F_{X_{(1)}}(x)=1-[1-F(x)]^n$, i.e. for $\mathrm{Unif}(0,1)$, $P(D_1\le t)=1-(1-t)^n$, so $P(D_i>t)=(1-t)^n$ for every $i$. Writing $N_n(c)=\sum_{i=1}^{n+1}\mathbf{1}\{D_i>c/n\}$ and applying linearity of expectation (§2.2) — note this sidesteps entirely the fact that the $D_i$ are dependent, exactly as the Indicator Variable Trick intends:
$E[N_n(c)]=(n+1)\,P\!\left(D_1>\frac{c}{n}\right)=(n+1)\left(1-\frac{c}{n}\right)^n\qquad(n>c).$
Dividing by $n+1$ and letting $n\to\infty$, the standard limit $\left(1-\frac{c}{n}\right)^n\to e^{-c}$ gives
$\frac{E[N_n(c)]}{n+1}\longrightarrow e^{-c}.$
$\blacksquare$

Remark (why this is the right answer). The Rényi representation says $D_i\approx E_i/n$ for large $n$ (since $\sum_{j=1}^{n+1}E_j\approx n$ by the Law of Large Numbers), so $P(D_i>c/n)\approx P(E_i>c)=e^{-c}$ — matching exactly. More is true: near any fixed location, the $n$ uniform points locally resemble a rate-$n$ Poisson process, whose gaps are exactly $\mathrm{Exponential}(n)$; $N_n(c)/(n+1)$ is then literally an empirical estimate of the tail probability $P(\mathrm{Exponential}(1)>c)$ of the rescaled gap $nD_i$. A second-moment computation — entirely analogous to the box-occupancy variance of Challenge Problem 2.1, replacing "boxes empty" by "spacings large" — would additionally show $\mathrm{Var}(N_n(c)/(n+1))\to0$, upgrading this convergence in expectation to convergence in probability.

---

Part 5 — Multivariate Distributions (Multinomial & Normal)

5.1 Multinomial Distribution

Definition 5.1. $(N_1,\dots,N_k) \sim \mathrm{Multinomial}(n; p_1,\dots,p_k)$ has joint PMF
$P(N_1=n_1,\dots,N_k=n_k) = \frac{n!}{n_1!\cdots n_k!}p_1^{n_1}\cdots p_k^{n_k}, \quad \sum n_i = n.$

Theorem 5.1 (Marginals). $N_i \sim \mathrm{Binomial}(n, p_i)$.

Theorem 5.2 (Conditional distribution). Given $N_1 = n_1$, the remaining counts $(N_2,\dots,N_k)$ are Multinomial$(n-n_1; \frac{p_2}{1-p_1},\dots,\frac{p_k}{1-p_1})$.

Theorem 5.3 (Covariance structure). $\mathrm{Cov}(N_i,N_j) = -n p_i p_j$ for $i \neq j$ — always negative, reflecting the constraint $\sum N_i = n$.

5.2 Bivariate Normal Distribution

Definition 5.2. $(X,Y)$ is bivariate Normal with parameters $\mu_X,\mu_Y,\sigma_X^2,\sigma_Y^2,\rho$ if its joint density is
$f(x,y) = \frac{1}{2\pi\sigma_X\sigma_Y\sqrt{1-\rho^2}} \exp\left\{-\frac{1}{2(1-\rho^2)}\left[\frac{(x-\mu_X)^2}{\sigma_X^2} - \frac{2\rho(x-\mu_X)(y-\mu_Y)}{\sigma_X\sigma_Y} + \frac{(y-\mu_Y)^2}{\sigma_Y^2}\right]\right\}.$

Theorem 5.4 (Conditional distribution). $Y \mid X=x \sim N\left(\mu_Y + \rho\frac{\sigma_Y}{\sigma_X}(x-\mu_X),\ \sigma_Y^2(1-\rho^2)\right)$ — linear regression function, constant conditional variance.

Theorem 5.5 (Zero correlation $\iff$ independence, bivariate Normal only). For jointly Normal $(X,Y)$, $\rho = 0 \implies$ the cross term vanishes and $f(x,y)$ factors into $f_X(x)f_Y(y)$. This equivalence is special to joint normality — in general, zero correlation does not imply independence (a standard counterexample: $X\sim N(0,1)$, $Y = X^2$; $\mathrm{Cov}(X,Y)=0$ but $Y$ is a deterministic function of $X$).

Remark (A famous corollary). Theorem 5.5 is the reason $\bar X$ and $S^2$ are independent for a normal random sample $X_1,\dots,X_n\stackrel{iid}\sim N(\mu,\sigma^2)$: one checks directly that $\mathrm{Cov}(\bar X, X_i-\bar X)=0$ for every $i$, and $(\bar X, X_1-\bar X,\dots,X_n-\bar X)$ is jointly Gaussian (an affine image of $(X_1,\dots,X_n)$, Theorem 5.6 below), so zero cross-covariance upgrades to independence: $\bar X\perp(X_1-\bar X,\dots,X_n-\bar X)$, hence $\bar X\perp S^2$. This independence is the single fact that makes the $t$-distribution — and the entire theory of the one-sample $t$-test — tractable.

5.3 Multivariate Normal Distribution

Remark (Why $\Sigma$ is positive semi-definite — true for any random vector). Before specializing to Gaussians: for any random vector $\mathbf{Y}=(Y_1,\dots,Y_p)^T$ with finite second moments, $\Sigma=\mathrm{Cov}(\mathbf{Y})$ is automatically positive semi-definite, because for any $\mathbf{a}\in\mathbb{R}^p$,
$\mathbf{a}^T\Sigma\mathbf{a}=\mathrm{Var}(\mathbf{a}^T\mathbf{Y})\ge0.$
$\Sigma$ fails to be strictly positive definite exactly when some linear combination $\mathbf{a}^T\mathbf{Y}$ is almost-surely constant, i.e. the law of $\mathbf{Y}$ is degenerate, concentrated on a proper affine subspace of $\mathbb{R}^p$ (which has Lebesgue measure zero in $\mathbb{R}^p$, so no genuine $p$-dimensional density can exist there). This is exactly why Definition 5.3 below imposes $\Sigma\succ0$: the degenerate case is instead handled through the affine-image construction of Theorem 5.6.

Definition 5.3. $\mathbf{X} \sim N_p(\boldsymbol{\mu}, \Sigma)$, $\Sigma$ positive definite, has density
$f(\mathbf{x}) = \frac{1}{(2\pi)^{p/2}|\Sigma|^{1/2}} \exp\left\{-\frac{1}{2}(\mathbf{x}-\boldsymbol{\mu})^T\Sigma^{-1}(\mathbf{x}-\boldsymbol{\mu})\right\}.$

Theorem 5.6 (Affine transformations). If $\mathbf{X}\sim N_p(\boldsymbol\mu,\Sigma)$ and $\mathbf{Y}=A\mathbf{X}+\mathbf{b}$ for a $q\times p$ matrix $A$, then $\mathbf{Y}\sim N_q(A\boldsymbol\mu+\mathbf{b},\, A\Sigma A^T)$ — the multivariate Normal family is closed under affine maps, with no rank restriction on $A$ (degenerate normals arise when $A\Sigma A^T$ is singular).

Theorem 5.7 (Marginals). Any sub-vector of a multivariate Normal is multivariate Normal, with the corresponding sub-block of $\boldsymbol\mu$ and $\Sigma$ — an immediate corollary of Theorem 5.6 by choosing $A$ as a selection matrix.

Theorem 5.8 (Quadratic forms and $\chi^2$). If $\mathbf{X} \sim N_p(\mathbf{0}, \Sigma)$ with $\Sigma$ invertible, then $\mathbf{X}^T\Sigma^{-1}\mathbf{X} \sim \chi^2_p$.

Proof sketch. Write $\Sigma^{-1} = \Sigma^{-1/2}\Sigma^{-1/2}$ (symmetric square root), and let $\mathbf{Z}=\Sigma^{-1/2}\mathbf{X}$; by Theorem 5.6, $\mathbf{Z}\sim N_p(\mathbf{0}, I_p)$, so $\mathbf{X}^T\Sigma^{-1}\mathbf{X}=\mathbf{Z}^T\mathbf{Z}=\sum Z_i^2 \sim \chi^2_p$ by definition of the chi-squared distribution as a sum of $p$ iid squared standard Normals. $\blacksquare$

Challenge Problem 5.1 (Independence of quadratic forms, and the unbiasedness of $S^2$). Let $\mathbf{X}\sim N_p(\mathbf{0}, \Sigma)$, $\Sigma\succ0$, and let $A, B$ be symmetric $p\times p$ matrices.

(a) Show that if $A\Sigma B=0$, then $\mathbf{X}^TA\mathbf{X}$ and $\mathbf{X}^TB\mathbf{X}$ are independent (Craig's theorem, sufficiency direction).

(b) Show $\mathrm{Cov}(\mathbf{X}^TA\mathbf{X},\mathbf{X}^TB\mathbf{X})=2\,\mathrm{tr}(A\Sigma B\Sigma)$, and explain why this shows $A\Sigma B=0$ is not recoverable from second moments alone.

(c) For $X_1,\dots,X_n\stackrel{iid}\sim N(\mu,\sigma^2)$ — i.e. $\mathbf{X}\sim N_n(\mu\mathbf{1}_n,\sigma^2I_n)$, where $\mathbf{1}_n\in\mathbb{R}^n$ is the all-ones vector, not to be confused with the event-indicator $\mathbf{1}_A$ — use the identity $E[\mathbf{X}^TA\mathbf{X}]=\mathrm{tr}(A\Sigma)+\boldsymbol\mu^TA\boldsymbol\mu$ (the mean-adjusted generalization of Theorem 5.8's zero-mean trace identity) to show $S^2=\frac{1}{n-1}\sum_{i=1}^n(X_i-\bar X)^2$ is unbiased for $\sigma^2$.

(d) Show that if $Q$ is orthogonal and $\mathbf{X}\sim N_p(\mathbf{0},I_p)$, then $Q\mathbf{X}\stackrel{d}{=}\mathbf{X}$.

Solution.

(a) Write $\Sigma=\Sigma^{1/2}\Sigma^{1/2}$ and set $\mathbf{Z}=\Sigma^{-1/2}\mathbf{X}\sim N_p(\mathbf{0},I_p)$ (Theorem 5.6), so $\mathbf{X}=\Sigma^{1/2}\mathbf{Z}$. Then $\mathbf{X}^TA\mathbf{X}=\mathbf{Z}^TA'\mathbf{Z}$ and $\mathbf{X}^TB\mathbf{X}=\mathbf{Z}^TB'\mathbf{Z}$ where $A'=\Sigma^{1/2}A\Sigma^{1/2}$, $B'=\Sigma^{1/2}B\Sigma^{1/2}$ are symmetric, and
$A'B'=\Sigma^{1/2}A\,\Sigma\, B\Sigma^{1/2}=\Sigma^{1/2}(A\Sigma B)\Sigma^{1/2}=0.$
So it suffices to treat the whitened case $\Sigma=I$: symmetric $A',B'$ with $A'B'=0$ (hence also $B'A'=(A'B')^T=0^T=0$). Commuting symmetric matrices are simultaneously orthogonally diagonalizable: there is an orthogonal $Q$ with $Q^TA'Q=D_A=\mathrm{diag}(\alpha_i)$ and $Q^TB'Q=D_B=\mathrm{diag}(\beta_i)$, and $D_AD_B=Q^TA'B'Q=0$ forces $\alpha_i\beta_i=0$ for every $i$ — at each coordinate, at most one of $\alpha_i,\beta_i$ is nonzero. Let $\mathbf{W}=Q^T\mathbf{Z}\sim N_p(\mathbf{0},I_p)$ (Theorem 5.6 again, $Q$ orthogonal). Then
$\mathbf{Z}^TA'\mathbf{Z}=\mathbf{W}^TD_A\mathbf{W}=\sum_i\alpha_iW_i^2,\qquad\qquad \mathbf{Z}^TB'\mathbf{Z}=\mathbf{W}^TD_B\mathbf{W}=\sum_i\beta_iW_i^2,$
and since $\{i:\alpha_i\ne0\}$ and $\{i:\beta_i\ne0\}$ are disjoint index sets, these two sums are functions of disjoint subcollections of the mutually independent $W_1,\dots,W_p$ — hence independent. $\blacksquare$

(b) For $\mathbf{X}\sim N_p(\mathbf{0},\Sigma)$, expanding $\mathbf{X}^TA\mathbf{X}$ and $\mathbf{X}^TB\mathbf{X}$ in the whitened coordinates $\mathbf{Z}=\Sigma^{-1/2}\mathbf{X}\sim N(\mathbf{0},I)$ as in part (a) and applying Isserlis'/Wick's formula for fourth moments of independent standard Normals yields the general formula
$\mathrm{Cov}(\mathbf{X}^TA\mathbf{X},\mathbf{X}^TB\mathbf{X})=2\,\mathrm{tr}(A\Sigma B\Sigma).$
(Sanity check: $A=B=I_p,\Sigma=I_p$ gives $\mathrm{Var}(\chi^2_p)=2p$, correct.) If $A\Sigma B=0$ then certainly $\mathrm{tr}(A\Sigma B\Sigma)=\mathrm{tr}(0\cdot\Sigma)=0$, consistent with part (a)'s stronger conclusion. But the converse is genuinely weaker, because a matrix trace can vanish without the matrix itself vanishing: with $\Sigma=I$, $A=\begin{pmatrix}1&0\\0&-1\end{pmatrix}$, $B=\begin{pmatrix}0&1\\1&0\end{pmatrix}$ (both symmetric), $AB=\begin{pmatrix}0&1\\-1&0\end{pmatrix}$ has $\mathrm{tr}(AB)=0$ yet $AB\ne0$. So vanishing covariance — the second-moment shadow of Craig's condition — cannot by itself certify $A\Sigma B=0$. The full converse (necessity) of Craig's theorem is true, but it requires matching the entire joint moment generating function $E\big[e^{s\mathbf{X}^TA\mathbf{X}+t\mathbf{X}^TB\mathbf{X}}\big]=|I-2sA-2tB|^{-1/2}$ against the product $|I-2sA|^{-1/2}|I-2tB|^{-1/2}$ to all orders in $(s,t)$ — not merely the order-$st$ term that produces the covariance formula above — a genuinely more delicate argument that we do not pursue here. $\blacksquare$

(c) Let $A=I_n-\frac{1}{n}J$ where $J=\mathbf{1}_n\mathbf{1}_n^T$ is the $n\times n$ all-ones matrix; $A$ is symmetric, and $A\mathbf{1}_n=\mathbf{1}_n-\frac{1}{n}(n\mathbf{1}_n)=\mathbf{0}$. With $\boldsymbol\mu=\mu\mathbf{1}_n$, $\Sigma=\sigma^2I_n$:
$\mathrm{tr}(A\Sigma)=\sigma^2\,\mathrm{tr}(A)=\sigma^2\Big[n-\frac{1}{n}\,\mathrm{tr}(J)\Big]=\sigma^2\Big[n-\frac{1}{n}\cdot n\Big]=\sigma^2(n-1),\qquad\qquad \boldsymbol\mu^TA\boldsymbol\mu=\mu^2\mathbf{1}_n^TA\mathbf{1}_n=\mu^2\mathbf{1}_n^T\mathbf{0}=0.$
So $E[\mathbf{X}^TA\mathbf{X}]=\sigma^2(n-1)$. But algebraically $\mathbf{X}^TA\mathbf{X}=\mathbf{X}^T\mathbf{X}-\frac{1}{n}(\mathbf{1}_n^T\mathbf{X})^2=\sum_iX_i^2-\frac{1}{n}\Big(\sum_iX_i\Big)^2=\sum_i(X_i-\bar X)^2$, so $E\big[\sum_i(X_i-\bar X)^2\big]=\sigma^2(n-1)$, i.e.
$E[S^2]=E\left[\frac{1}{n-1}\sum_{i=1}^n(X_i-\bar X)^2\right]=\sigma^2.$
$\blacksquare$

(d) By Theorem 5.6, $Q\mathbf{X}\sim N_p(Q\mathbf{0},\,QI_pQ^T)=N_p(\mathbf{0},\,QQ^T)=N_p(\mathbf{0},I_p)$ since $Q$ is orthogonal. A multivariate Normal law is determined entirely by its mean and covariance, so $Q\mathbf{X}\stackrel{d}{=}\mathbf{X}$. $\blacksquare$

Challenge Problem 5.2 (Orthogonal invariance: radius and direction are independent). Let $\mathbf{X}\sim N_p(\mathbf{0},I_p)$, $p\ge2$. Define $R=\|\mathbf{X}\|$ and $\boldsymbol\Theta=\mathbf{X}/\|\mathbf{X}\|\in\mathbb{S}^{p-1}$ (well-defined a.s., since $P(\mathbf{X}=\mathbf{0})=0$). Show that $R$ and $\boldsymbol\Theta$ are independent, and that $\boldsymbol\Theta$ is uniformly distributed on $\mathbb{S}^{p-1}$.

Solution. This is Challenge Problem 5.1(d)'s rotational invariance, pushed to its natural conclusion. Fix any orthogonal $Q$. Two deterministic facts about the maps $R(\cdot),\boldsymbol\Theta(\cdot)$: $R(Q\mathbf{x})=\|Q\mathbf{x}\|=\|\mathbf{x}\|=R(\mathbf{x})$ for every $\mathbf{x}$ (orthogonal maps preserve norm), and $\boldsymbol\Theta(Q\mathbf{x})=\dfrac{Q\mathbf{x}}{\|Q\mathbf{x}\|}=Q\dfrac{\mathbf{x}}{\|\mathbf{x}\|}=Q\boldsymbol\Theta(\mathbf{x})$. Combined with $Q\mathbf{X}\stackrel{d}{=}\mathbf{X}$ (part (d) above), we get, for every orthogonal $Q$:
$(R,\boldsymbol\Theta)\ \stackrel{d}{=}\ \big(R(Q\mathbf{X}),\boldsymbol\Theta(Q\mathbf{X})\big)=(R,\,Q\boldsymbol\Theta).$
Fix a Borel set $A\subseteq(0,\infty)$ and define the finite measure $\nu_A(B):=P(R\in A,\boldsymbol\Theta\in B)$ on $\mathbb{S}^{p-1}$. The displayed distributional identity gives $\nu_A(B)=P(R\in A,Q\boldsymbol\Theta\in B)=P(R\in A,\boldsymbol\Theta\in Q^{-1}B)=\nu_A(Q^{-1}B)$ for every orthogonal $Q$; replacing $Q$ by $Q^{-1}$ (which ranges over the same group) shows $\nu_A$ is invariant under every rotation of the sphere. The orthogonal group acts transitively on $\mathbb{S}^{p-1}$, and the invariant probability measure under a transitive compact group action is unique (uniqueness of Haar measure, specialized to $\mathbb{S}^{p-1}\cong O(p)/O(p-1)$); hence $\nu_A$ must be a constant multiple of the normalized uniform surface measure $\sigma$ on $\mathbb{S}^{p-1}$. Taking $B=\mathbb{S}^{p-1}$ pins down the constant: $\nu_A(\mathbb{S}^{p-1})=P(R\in A)$, so
$P(R\in A,\boldsymbol\Theta\in B)=P(R\in A)\cdot\sigma(B)\qquad\text{for all Borel }A,B.$
Setting $A=(0,\infty)$ shows $\boldsymbol\Theta$ has law $\sigma$ — uniform on the sphere — and then the display is the independence of $R$ and $\boldsymbol\Theta$ (their joint law factors as a product over the generating rectangles, which extends to the full product $\sigma$-field by a standard $\pi$–$\lambda$ argument). $\blacksquare$

Remark (the computational route, for comparison). One can reach the same conclusion by brute-force integration: the $p$-dimensional polar-coordinate volume element is $d\mathbf{x}=r^{p-1}\,dr\,d\sigma(\boldsymbol\theta)$ (the $(p-1)$-dimensional surface measure at radius $r$ scales as $r^{p-1}$ times the unit-sphere measure), so the standard Gaussian density $(2\pi)^{-p/2}e^{-\|\mathbf{x}\|^2/2}$ becomes, in $(r,\boldsymbol\theta)$ coordinates,
$f_{R,\boldsymbol\Theta}(r,\boldsymbol\theta)=(2\pi)^{-p/2}r^{p-1}e^{-r^2/2},$
which manifestly does not depend on $\boldsymbol\theta$, giving the same factorization and additionally identifying $R$'s density as proportional to $r^{p-1}e^{-r^2/2}$ — the $\chi_p$ distribution (consistent with $R^2=\mathbf{X}^T\mathbf{X}\sim\chi_p^2$ from Theorem 5.8 with $\Sigma=I$). The symmetry argument above is preferred here because it needs no unproven formula for the sphere's volume element and cleanly isolates why the result is true: it is nothing but Theorem 5.6's orthogonal invariance, upgraded from a statement about $\mathbf{X}$ to a statement about $\boldsymbol\Theta$ alone.

---

Appendix — Master Formula Sheet

---

Closing Remark — The Unifying Theme

Beneath the combinatorial identities and distributional families of this treatise lies a single structural fact: the space of random variables with finite second moment, $L^2(\Omega,\mathcal{F},P)$, is a Hilbert space, with inner product $\langle X, Y\rangle = E[XY]$. Under this lens:

• Expectation is an inner product with the constant function $1$: $E[X] = \langle X, 1\rangle$.
• Variance is squared distance from the mean: $\mathrm{Var}(X) = \|X - E[X]\|^2$.
• Covariance is an inner product between centered variables, and correlation $\rho$ is precisely the cosine of the angle between $X-\mu_X$ and $Y-\mu_Y$ — explaining geometrically why Cauchy–Schwarz forces $|\rho|\le 1$, with equality exactly when the vectors are collinear (i.e., $Y$ is an affine function of $X$).
• Conditional expectation $E[Y\mid X]$ is the orthogonal projection of $Y$ onto the closed subspace $M$ of $\sigma(X)$-measurable, finite-variance random variables — the random-variable analogue of projecting a vector onto a subspace in $\mathbb{R}^n$. Concretely: writing $\widehat Y=E[Y\mid X]$, the projection residual $Y-\widehat Y$ is orthogonal to every element of $M$, in particular to $\widehat Y - E[Y]\in M$ (constants are trivially $\sigma(X)$-measurable). The Pythagorean theorem applied to $(Y-\widehat Y)+(\widehat Y-E[Y])$ then gives $\|Y-E[Y]\|^2=\|Y-\widehat Y\|^2+\|\widehat Y-E[Y]\|^2$, i.e.
  $\mathrm{Var}(Y)=\underbrace{E\big[\mathrm{Var}(Y\mid X)\big]}_{\|Y-\widehat Y\|^2\text{, via the tower property}}+\underbrace{\mathrm{Var}\big(E[Y\mid X]\big)}_{\|\widehat Y-E[Y]\|^2}$
  — Eve's Law is nothing but the Pythagorean theorem, applied to this one projection.
• The multivariate Normal's covariance matrix $\Sigma$ plays the role of a Gram matrix / positive-definite quadratic form (§5.3's opening Remark makes this precise for any random vector, Gaussian or not), and orthogonal transformations preserving $\Sigma=I$ correspond exactly to the rotational symmetry of the standard multivariate Normal density — the content of Challenge Problems 5.1(d) and 5.2, where that symmetry is shown to separate a Gaussian vector cleanly into an independent radius and a uniformly-distributed direction.

Probability theory, viewed this way, is measure theory (for existence and integration — Kolmogorov's axioms, Radon–Nikodym densities, $\sigma$-fields rich enough to avoid Vitali-type pathologies) married to linear algebra (for the geometry of dependence — positive-definite covariance matrices, orthogonal diagonalization, quadratic-form independence) — a synthesis worth carrying forward into statistical inference, where estimators are themselves often best understood as projections in this same $L^2$ space.