Date: 20/07/2026
Linear Algebra
Preface
This treatise is written for a reader who already possesses fluency with abstract algebraic structures — groups, rings, fields — and with the grammar of proof-writing (direct proof, contradiction, induction, contrapositive). It does not re-derive what a field is, nor does it pause to motivate why linear algebra matters; the reader is assumed to already believe that. Instead, the emphasis throughout is on:
- Structural completeness — every definition is stated in full generality (over or , or an arbitrary field where the distinction is immaterial), and every theorem is stated with its precise hypotheses.
- Exam-grade rigor — the proofs given are the ones you would be expected to reproduce, or at least sketch cogently, in an ISI/JAM/GATE written examination.
- The statistical undercurrent — wherever a piece of linear algebra has a direct statistical avatar (idempotents as projections in OLS, positive-definiteness as the admissibility condition for a covariance matrix, the pseudoinverse as the tool of generalized least squares), this is flagged as a Remark.
Where a fully general proof is genuinely outside the scope of an entrance-exam treatise (e.g. the existence of Jordan Canonical Form over ), the theorem is stated precisely, its proof is outlined with the key lemmas named, and the reader is pointed to the standard references (Hoffman–Kunze, Horn & Johnson, Axler).
Work through the Challenge Problems seriously. They are calibrated to ISI M.Stat subjective-paper difficulty, not to routine JAM objective-paper difficulty — if a Challenge Problem takes you forty minutes and two false starts, that is exactly the intended experience.
Table of Contents
- Conventions and Notation
- Part 1 — Vector Spaces and Linear Transformations
- 1.1 Vector Spaces and Subspaces
- 1.2 Span, Independence, Basis, Dimension
- 1.3 Linear Transformations and Matrix Representation
- 1.4 The Four Fundamental Subspaces and Rank–Nullity
- 1.5 Challenge Problems
- Part 2 — Determinant, Trace, Rank, and Inverse
- 2.1 Rank Revisited: Sylvester's Inequality
- 2.2 Trace
- 2.3 Determinants
- 2.4 The Inverse, Adjugate, and Matrix Identities
- 2.5 Problem-Solving Tricks
- 2.6 Challenge Problems
- Part 3 — Systems of Linear Equations
- 3.1 Matrix Formulation and Consistency
- 3.2 The Rouché–Capelli Theorem
- 3.3 Structure of the Solution Set
- 3.4 The Moore–Penrose Pseudoinverse
- 3.5 Challenge Problems
- Part 4 — Eigenvalues, Eigenvectors, and Diagonalization
- 4.1 Eigenvalues, Eigenvectors, Characteristic Polynomial
- 4.2 Multiplicities and Diagonalizability
- 4.3 Cayley–Hamilton and the Minimal Polynomial
- 4.4 Similarity Invariants
- 4.5 Challenge Problems
- Part 5 — Spectral Theory and Special Matrices
- 5.1 Symmetric, Orthogonal, Idempotent, Nilpotent
- 5.2 The Spectral Theorem
- 5.3 Positive Definite and Semi-Definite Matrices
- 5.4 Singular Value Decomposition
- 5.5 Challenge Problems
- Appendix: Master Formula Sheet
- Closing Remark: The Unifying Theme
Conventions and Notation
- denotes a general field; where a statement requires or specifically, this is stated.
- denote vector spaces over ; denotes dimension.
- is the space of matrices over ; .
- denotes transpose; denotes conjugate transpose (Hermitian adjoint). Over , .
- (or ) is the identity matrix; is the zero matrix/vector, disambiguated by context.
- , , , denote rank, trace, determinant, nullity.
- : column space; : row space; : null space (kernel); : left null space.
- denotes an inner product; on this is unless otherwise stated.
- means is positive semi-definite; means positive definite. For symmetric , means (the Loewner order).
- denotes the spectrum (multiset of eigenvalues) of .
- We use , , and (direct sum) in the standard sense.
- All vector spaces considered are finite-dimensional unless explicitly stated otherwise.
- closes a proof; closes a worked example.
Part 1 — Vector Spaces and Linear Transformations
1.1 Vector Spaces and Subspaces
Definition 1.1.1 (Vector Space).
A vector space over a field is a set equipped with two operations, vector addition and scalar multiplication , satisfying the following axioms for all and :
- is an abelian group: associativity, commutativity, existence of additive identity , existence of additive inverses.
- (distributivity over vector addition).
- (distributivity over scalar addition).
- (compatibility of scalar multiplication with field multiplication).
- for the multiplicative identity .
Remark. For , is called a real vector space; for , a complex vector space. In statistics, real vector spaces dominate (data lives in ), but complex vector spaces are indispensable in spectral theory, since a real matrix may have complex eigenvalues.
Definition 1.1.2 (Subspace).
A subset is a subspace if is itself a vector space under the operations inherited from . Equivalently (and this is the operational criterion), is a subspace iff:
(i) ;
(ii) is closed under addition: ;
(iii) is closed under scalar multiplication: .
(Conditions (ii) and (iii) can be compressed into a single closure-under-linear-combination condition.)
Theorem 1.1.3. If are subspaces of , then is a subspace of . In general, is not a subspace unless or .
Proof. since and . Closure under addition and scalar multiplication in follows immediately from closure in each of and separately. For the second claim: suppose neither nor . Pick and . If , then either , forcing — contradiction — or , forcing by the symmetric argument — contradiction. Hence , so fails closure.
Definition 1.1.4 (Sum of Subspaces). For subspaces , the sum is
This is the smallest subspace of containing . If , the sum is called a direct sum, written , and every has a unique decomposition .
1.2 Span, Independence, Basis, Dimension
Definition 1.2.1 (Linear Span). For ,
This is a subspace of — indeed, the intersection of all subspaces containing .
Definition 1.2.2 (Linear Independence). is linearly independent if
Otherwise is linearly dependent: some is expressible as a linear combination of the others.
Definition 1.2.3 (Basis). is a basis if is linearly independent and .
Theorem 1.2.4 (Steinitz Exchange Lemma). Let be linearly independent in and let span . Then , and of the can be replaced by the to yield a spanning set.
Proof sketch. Induct on . Since spans , write ; some (as by independence), so we may swap out for , and the resulting set still spans . Repeat this exchange for ; independence of the guarantees at each stage that some -vector remains available to eject (if none did, the current would already lie in the span of the previously-inserted , contradicting independence), forcing .
Corollary 1.2.5 (Well-Definedness of Dimension). Any two bases of have the same cardinality. This common cardinality is called .
Theorem 1.2.6 (Dimension Formula for Sums).
(Full proof given in §1.5, Challenge Problem 1 — this is the flagship result of Part 1.)
Trick 1.2.7. To check linear independence of vectors in quickly, stack them as rows/columns of a matrix and row-reduce; independence rank equals . For abstract vector spaces (e.g. polynomial spaces), evaluate at well-chosen points or differentiate, turning the question into a Wronskian/determinant computation.
Example 1.2.8. In (polynomials of degree ), are linearly independent? Express in the monomial basis coordinates , stack as rows, row reduce:
rank , so the four vectors are independent (and hence a basis of the -dimensional space ).
1.3 Linear Transformations and Matrix Representation
Definition 1.3.1 (Linear Transformation). is linear if for all .
Definition 1.3.2 (Matrix Representation). Fix ordered bases of and of . The matrix representation has -th column equal to the -coordinate vector of . Then for any ,
Theorem 1.3.3 (Change of Basis). Let be two bases of and let be the change-of-basis matrix whose columns are the -coordinates of the -basis vectors (so for every ). If is linear with matrix , then
That is, matrices representing the same linear operator in different bases are similar.
Proof. Since holds for every , equivalently for every . Apply this to :
Since this holds for all , the matrix represents in basis .
Remark. Similarity is the correct notion of "sameness" for linear operators — it will recur throughout Part 4, since eigenvalues, trace, determinant, rank, and the characteristic polynomial are all similarity invariants. It is also the abstract statement behind principal component analysis: choosing a new orthonormal basis (the eigenbasis of a covariance matrix ) is nothing but a change of basis of exactly this kind, engineered so that 's representation in the new basis is diagonal (§5.2).
1.4 The Four Fundamental Subspaces and Rank–Nullity
For , viewed as a linear map :
Definition 1.4.1.
- Column space — the range of .
- Null space — the kernel of .
- Row space .
- Left null space .
Theorem 1.4.2 (Row Rank = Column Rank). For any , . This common value is .
Proof. Row-reduce to row echelon form ( invertible, a product of elementary matrices). Row operations preserve the row space (each row of is a linear combination of rows of and vice versa via ), so number of nonzero (pivot) rows of . Also, the pivot columns of correspond exactly to a maximal linearly independent subset of columns of (since preserves linear dependence relations among columns, being invertible), giving number of pivot columns number of pivot rows.
Theorem 1.4.3 (Rank–Nullity Theorem). For a linear map with ,
Equivalently, for : .
Proof. Let be a basis of (). Extend to a basis of . Claim: is a basis of .
Spanning: any since .
Independence: if , then , so for some , i.e. ; independence of the extended basis forces all .
Hence , giving .
Theorem 1.4.4 (Orthogonality of the Fundamental Subspaces, over ).
Proof. If and , then . So . By Theorem 1.4.3 applied to , ; by Theorem 1.4.2, , so . Orthogonality forces (any vector in the intersection is orthogonal to itself, hence zero), so this dimension count forces . The second pair follows by applying the first pair to (noting by definition).
Remark (Statistical Avatar). This theorem is the abstract source of a fact every regression student memorizes without always seeing why it is true. In ordinary least squares with design matrix , the residual vector satisfies the normal equations , i.e. — the left null space of — while the fitted values lie in by construction. Theorem 1.4.4, applied with in place of , says precisely : the residual vector is orthogonal to the fitted values, and to every regressor, automatically — not a special virtue of least squares to be separately verified, but a direct instance of an orthogonality that holds for any matrix . Least squares merely chooses so that is the point of closest to ; orthogonality of the resulting residual is then forced by this theorem.
Trick 1.4.5. Rank–Nullity is the single most exam-tested identity in this subject. Any question of the form "given constraints on or , find the other" collapses to bookkeeping via this theorem. Combined with , it immediately yields: if (more unknowns than equations), , i.e. has a nontrivial solution.
1.5 Challenge Problems
Challenge Problem 1. (Flagship result — full proof required.) Let be subspaces of a finite-dimensional vector space . Prove that
Proof. Let , , . Choose a basis of . Extend it to a basis of , and separately extend it to a basis of .
Claim: is a basis of .
Spanning: Immediate, since every element of is a combination of and every element of is a combination of , and is spanned by .
Independence: Suppose
Rearranging, (right side is a combination of -basis vectors). But too, so . Hence for some scalars , i.e.
Since is a basis of (hence independent), all and all . Substituting back into gives ; independence of the basis of forces all . Hence has only the trivial solution, proving independence.
Therefore .
Challenge Problem 2. Let be a linear operator on a finite-dimensional space with (idempotent, i.e. a projection — see also §5.1). Prove that .
Proof outline. For , write . Check trivially, and so ; this gives . For directness, suppose ; then for some , and , so , forcing . Hence the sum is direct. (This result reappears with statistical content in §5.1 and the Closing Remark.)
Challenge Problem 3 (ISI-style, abstract operator). Let be a finite-dimensional vector space and linear with . Prove that , and hence .
Proof outline. By Rank–Nullity applied to restricted to : consider , well-defined since . Its image is , which by hypothesis has the same dimension as (the domain of this restricted map), forcing the restricted map to be injective (Rank–Nullity on the restriction: ). But , giving the result. Directness of the sum then follows from dimension counting via Rank–Nullity, as in Challenge Problem 2.
Challenge Problem 4 (A trap for the unwary — conceptual, not computational). Students who have just proved for two subspaces frequently conjecture the analogous inclusion–exclusion formula for three subspaces :
Show by an explicit low-dimensional counterexample that is false in general, and identify precisely which step of the two-subspace proof fails to generalize.
Solution. Take and
Each is -dimensional, and (already ), so .
Every pairwise intersection is trivial: since are independent; since are independent; since are independent. Consequently as well. Substituting into the right side of :
So overcounts by exactly , and is false.
Where the two-subspace proof breaks down. The proof of Challenge Problem 1 rests on the modular law , which holds for any two subspaces. Iterating it naively for three subspaces implicitly assumes the subspace lattice is distributive — that distributes over , i.e. — because that is what licenses expanding a three-term inclusion–exclusion the way it expands for sets (where genuinely does distribute over ). The lattice of subspaces of a vector space of dimension is modular but not distributive, and the counterexample above exhibits the failure directly: (dimension ), while (dimension ) — the two sides disagree. This is the precise mechanism by which fails: pairwise (and triple) intersection dimensions alone do not determine , because they do not encode how the intersections sit relative to one another inside the sum. What does survive to any number of subspaces is the weaker subadditivity inequality
obtained by inducting on the two-subspace formula and discarding the (nonnegative) intersection term at each stage — this is the fact used, for instance, in Challenge Problem 2 of §2.6.
Part 2 — Determinant, Trace, Rank, and Inverse
2.1 Rank Revisited: Sylvester's Inequality
Theorem 2.1.1 (Sub-additivity/Sub-multiplicativity of Rank under Products). For , ,
The left inequality is Sylvester's Rank Inequality.
Proof.
Upper bound: since every column of is times something, hence a combination of columns of ; so . Also by the same column-space argument applied to .
Lower bound (Sylvester): Consider the linear map , the restriction of to the subspace . We have and , so by Rank–Nullity applied to this restricted map,
Now , so (Rank–Nullity for ). Substituting,
Remark. Sylvester's inequality is the standard tool for bounding and for showing that products of "sufficiently full rank" matrices retain rank.
2.2 Trace
Definition 2.2.1. For , .
Theorem 2.2.2 (Cyclic Property). For , : .
Proof. .
Corollary 2.2.3 (Invariance under Similarity). If for invertible , then .
Proof. Apply the cyclic property with , : .
Remark. Trace is therefore a similarity invariant, and (Part 4) equals the sum of eigenvalues (with algebraic multiplicity) — this is often the fastest route to eigenvalue sum without computing the characteristic polynomial explicitly. It also underlies a fact used implicitly in PCA: since a change to an orthonormal eigenbasis is a similarity transformation (indeed an orthogonal one), — the total variance — is exactly preserved by rotating into principal components; PCA redistributes variance across new coordinate axes without creating or destroying any of it.
Trick 2.2.4. , with equality iff . This is the standard trick for proving a matrix is zero: show ; equivalently, for a vector, .
2.3 Determinants
Definition 2.3.1 (Axiomatic/Multilinear Definition). is the unique function that is (i) multilinear in the rows, (ii) alternating (equal rows determinant ), and (iii) normalized so . Equivalently, via the Leibniz formula,
Theorem 2.3.2 (Core Properties).
(a) (multiplicativity).
(b) .
(c) is invertible .
(d) Row/column swap negates ; scaling a row by scales by ; adding a multiple of one row to another leaves unchanged.
(e) for .
(These follow directly from the multilinear-alternating axioms; multiplicativity (a) is most cleanly proved via the fact that both sides are, as functions of for fixed invertible , alternating multilinear functions of the rows of agreeing at , hence equal by uniqueness of the alternating multilinear form up to scalar — or via elementary matrix factorization of .)
Theorem 2.3.3 (Cauchy–Binet Formula). For , with ,
where the sum is over all -element subsets of , is the submatrix of formed by columns indexed by , and is the submatrix of formed by rows indexed by .
Remark. Cauchy–Binet generalizes multiplicativity of determinants to non-square factors, and specializes (taking , so ) to — an important positivity fact used repeatedly in Part 5.
Theorem 2.3.4 (Block Matrix Determinants). For a block matrix with square and invertible,
If instead is square and invertible,
In particular, if or (block-triangular), .
Proof. Use the block factorization
The two outer factors are unit lower/upper block-triangular, hence have determinant (triangular with 's on the diagonal blocks). Taking determinants of both sides and using multiplicativity and the block-triangular case gives the result. The second formula is symmetric (pivot on instead).
The matrix (or ) is called the Schur complement of (resp. ) in ; it reappears throughout §2.4–§2.6 and again in §5.3.
Corollary 2.3.5 (Matrix Determinant Lemma). For invertible and ,
In particular, .
Proof. Consider the block matrix . Applying Theorem 2.3.4 with the invertible top-left block :
Applying Theorem 2.3.4 instead with the invertible bottom-right block:
Equating the two evaluations of gives the claim; setting gives the stated special case.
Remark. This is the determinant-side counterpart of the Sherman–Morrison formula (Theorem 2.4.3 below): both arise from viewing a rank-one perturbation through the same bordered matrix , evaluated via its two Schur complements. It also explains where Sherman–Morrison's denominator comes from: is not an arbitrary normalizing constant, it is literally — so it vanishes exactly when is singular, consistent with Sherman–Morrison's invertibility hypothesis.
2.4 The Inverse, Adjugate, and Matrix Identities
Definition 2.4.1 (Adjugate). For , the adjugate (classical adjoint) is the transpose of the cofactor matrix: , where is the minor of .
Theorem 2.4.2. . Consequently, if ,
Proof. by cofactor expansion along row . For , , which is the cofactor expansion of the determinant of the matrix obtained from by replacing row with a copy of row — a matrix with a repeated row, hence determinant .
Theorem 2.4.3 (Sherman–Morrison Formula). For invertible and with ,
Proof. Direct verification: multiply the RHS by and simplify using the scalar :
Theorem 2.4.4 (Woodbury Identity — matrix generalization). For conformable with invertible:
Remark. Sherman–Morrison/Woodbury is the workhorse for rank-one/low-rank updates: recomputing a regression's inverse-covariance after adding one observation (an update rather than an re-inversion) is a direct application. Running this in reverse — removing rather than adding an observation — is exactly how closed-form leave-one-out diagnostics in regression (the "hat value" , the PRESS statistic, Cook's distance) are derived without literally refitting the model times: deleting row of is a rank-one downdate of , and Sherman–Morrison converts an refit into an correction.
2.5 Problem-Solving Tricks
Trick 2.5.1 (Rank Factorization). Any with can be written with , , both of rank (take = a basis of as columns, = the coefficients expressing each column of in that basis). This turns rank questions about into questions about smaller full-rank factors, and is the standard route to prove (over ), via (Trick 2.2.4).
Trick 2.5.2 (Trace/Determinant for Nilpotency). is nilpotent ( for some ) all eigenvalues of are for all (Newton's identities relate power sums of eigenvalues to elementary symmetric polynomials, i.e. to coefficients of the characteristic polynomial). A quick necessary condition often sufficient for exam purposes: if then , hence , i.e. .
Trick 2.5.3 (Schur Complement for Block Inversion). To invert with invertible, use the block-LDU factorization from Theorem 2.3.4 to get
This is the standard route for inverting a covariance matrix partitioned into blocks — central to conditional distributions of the multivariate normal.
2.6 Challenge Problems
Challenge Problem 1 (Block Determinant Identity). Let with (commuting). Prove that
Proof. Work in and set . Consider the matrix, in blocks,
Direct block multiplication — using only that is a scalar matrix, so it commutes with every block; commutativity of is not needed for this step — gives
and the two right-hand sides agree entry by entry, so
Now is invertible: by Theorem 2.3.4, pivoting on the invertible top-left block ,
So rearranges to : is similar over to . Similar matrices have equal determinants (Theorem 4.4.1, or directly from multiplicativity), so
This identity holds for any square real — commutativity has not yet been used. It enters only in the final step, where permits
so that by multiplicativity once more. Combining with :
Remark. The intermediate identity is worth isolating: it holds unconditionally and is the determinant of the real representation of the complex matrix acting on . When commute, (entrywise conjugate, as are real), so — recovering positivity of directly, without needing to know (which in fact requires more than mere commutativity to guarantee).
Challenge Problem 2 (Rank Bound for Sums). For , prove
Proof. Upper bound: , and by Theorem 1.2.6 (dropping the intersection term). Lower bound: apply the upper bound to : , i.e. ; symmetrically for .
Challenge Problem 3 (Schur Complement Positivity — bridges to Part 5). Let be symmetric with . Prove that iff the Schur complement .
Proof. By Theorem 2.3.4's factorization (pivoting on the invertible top-left block ),
Writing , the third factor is exactly (since , using symmetric). So
a congruence: and are congruent via the invertible matrix (unit lower-triangular, ).
For , set ; since is invertible, is a bijection. By ,
() If : since , always and by hypothesis, so for every , i.e. .
() If : given any , since is invertible, there is a unique with — namely , . For this , (since ). As was arbitrary, .
Remark (Statistical Payoff). This is precisely the tool for checking that a partitioned covariance matrix is admissible: if is the covariance matrix of jointly, with , then automatically (it is a genuine covariance matrix), and this theorem tells you the conditional covariance matrix of in the multivariate normal model — which is exactly the Schur complement — is itself a bona fide (PSD) covariance matrix. Conditioning can only shrink uncertainty (in the Loewner order — see Challenge Problem 4 of §5.5), never produce a nonsensical negative-variance direction, and the Schur complement is the precise algebraic reason why.
Challenge Problem 4 (Rank Additivity via the Schur Complement). Let be symmetric with invertible. Prove that
Proof. Mirror the factorization of Theorem 2.3.4, pivoting instead on the invertible bottom-right block :
(Verified by the identical block-multiplication check used to prove Theorem 2.3.4, with the two pivot blocks' roles exchanged; expanding the right side reproduces entry by entry, using to cancel the off-diagonal blocks.) Both outer factors are unit-triangular, hence invertible. Left- or right-multiplying by an invertible matrix does not change rank (it carries a maximal independent set of columns to another maximal independent set), so
For a block-diagonal matrix , columns supported in the "-rows" and those supported in the "-rows" cannot participate in a common linear dependency (any dependency forces the -part and -part to vanish separately), so a maximal independent set of columns is the concatenation of maximal independent sets from each block: . Hence
Remark. This is a rank-additivity identity for partitioned symmetric matrices, and it is the rank-theoretic sibling of the determinant and definiteness statements already proved for the same Schur complement (Theorem 2.3.4, Challenge Problem 3). It has a clean statistical use: for a design (intercept plus covariates), partition this way and pivot on the invertible top-left block. The rank — and hence the identifiability — of the full regression reduces to the rank of the Schur complement , which is exactly times the sample covariance matrix of the columns of . This is the algebraic seed of the Frisch–Waugh–Lovell theorem: the covariates' contribution to the model, after partialling out the intercept, is governed entirely by their own centered cross-product matrix.
Part 3 — Systems of Linear Equations
3.1 Matrix Formulation and Consistency
Definition 3.1.1. A linear system in unknowns and equations is written , , , . It is homogeneous if , else non-homogeneous. The system is consistent if a solution exists.
Remark. is consistent — a restatement of the definition of column space, but the operational move of the whole section is to turn this membership question into a rank comparison.
3.2 The Rouché–Capelli Theorem
Definition 3.2.1 (Augmented Matrix). is with appended as an extra column.
Theorem 3.2.2 (Rouché–Capelli / Kronecker–Capelli). The system is consistent iff
Moreover, if consistent:
- If (number of unknowns), the solution is unique.
- If , the solution set is an affine subspace of dimension (a -parameter family).
Proof. () If has a solution, then , so appending as a column adds no new dimension to the column space: .
() If , then since and both have dimension , we get , so , i.e. the system is consistent.
Uniqueness/parametrization: If is one solution, the general solution is (verify: ). By Rank–Nullity, . If , , giving uniqueness; else the solution set is a coset of the -dimensional subspace .
3.3 Structure of the Solution Set
Theorem 3.3.1 (Superposition/Affine Structure). The general solution of a consistent system is
Remark (Geometry). The solution set is a translate of the null space, i.e. an affine subspace parallel to . The "degrees of freedom" of the system is exactly — the number of free variables remaining after row-reduction. This is precisely the notion of degrees of freedom that resurfaces in ANOVA decompositions and the residual degrees of freedom in linear regression.
Trick 3.3.2. To parametrize practically: row-reduce to reduced row echelon form, identify free columns (non-pivot), and for each free variable set it to and the rest to , solving for the pivot variables — this generates a basis for directly.
3.4 The Moore–Penrose Pseudoinverse
Definition 3.4.1. For (possibly non-square, possibly rank-deficient), the Moore–Penrose pseudoinverse is the unique matrix satisfying the four Penrose conditions:
Theorem 3.4.2 (Existence and Uniqueness). exists and is unique for every .
(Proof outline: existence is constructive via the SVD, where inverts the nonzero singular values in place — see §5.4. Uniqueness follows by a direct algebraic argument from the four Penrose conditions.)
Theorem 3.4.3 (Closed Forms in Full-Rank Cases).
- If has full column rank (): (the "left inverse"; ).
- If has full row rank (): (the "right inverse"; ).
Theorem 3.4.4 (Pseudoinverse Solves Least Squares). For possibly-inconsistent , is the minimum-norm least-squares solution: it minimizes , and among all minimizers, it has the smallest .
Remark. This is the crucial statistical avatar of this section: when (the design matrix) has full column rank, is exactly the OLS estimator . When is singular (rank-deficient design, e.g. collinear regressors), the pseudoinverse still delivers a well-defined "solution" — the minimum-norm one — which is precisely what many software implementations of regression fall back to (see Challenge Problem 4 below for exactly what remains well-defined in that case).
3.5 Challenge Problems
Challenge Problem 1 (Parameterized Consistency). For which values of does the system
have (a) a unique solution, (b) infinitely many solutions, (c) no solution?
Solution. Write the system as with
Step 1: Compute . Every row of sums to , so adding columns and into column () produces a common factor:
In the remaining determinant, subtract row from rows and (, ):
(cofactor expansion along column , which has a single nonzero entry). Hence
This vanishes exactly at (a double root) and (a simple root); the leading minus sign does not affect the case analysis below (only the roots matter for that), but it is worth being explicit about — a bare , missing the sign, is a common arithmetic slip and gives the wrong value if the determinant itself is ever needed (e.g. via Cramer's rule).
Step 2: Case analysis (Rouché–Capelli, Theorem 3.2.2).
Case : , so number of unknowns. Unique solution.
Case : becomes the all-ones matrix , of rank . Every equation of the system reduces to , so also has rank ; the system is consistent with , giving a -parameter family. Infinitely many solutions.
Case : Each row of sums to , i.e. where ; since is symmetric, this simultaneously says lies in the left null space, . A direct check of the leading minor, , confirms exactly (not less). By the Fredholm alternative — is consistent iff is orthogonal to every vector in , which is precisely Theorem 1.4.4 (, so ) — consistency here requires . But . No solution.
Remark. The identification of as spanning at the critical value , followed by the one-line check , is the Rouché–Capelli theorem in its most efficient operational form: rather than row-reducing the full augmented matrix, one only needs a spanning set for the left null space of (often visible by inspection, as here from the zero row-sum) and a single inner product against .
Challenge Problem 2 (Pseudoinverse Identity). Prove that for any ,
and that this reduces to Theorem 3.4.3 when has full column rank.
Proof. Let be an SVD of (Theorem 5.4.1 — logically this problem sits most naturally after §5.4, but is placed here for thematic continuity with the pseudoinverse). Then
using . This exhibits in exactly the spectral form of Theorem 5.2.1: an orthogonal matrix times a diagonal PSD matrix times its transpose. The pseudoinverse of a symmetric PSD matrix given in this form is obtained by inverting the nonzero diagonal entries in place:
Hence
using . It remains to check . Both sides are and zero off the "diagonal," so compare the -th diagonal entry, : if , the left side gives , matching 's convention; if , both sides give . Hence exactly, and
This proves the identity for every , with no rank hypothesis.
When has full column rank (), all singular values are nonzero, so is invertible and , recovering , i.e. Theorem 3.4.3.
Remark. This identity is the theoretical justification for computing OLS coefficients as even when one thinks of the conceptually "right" object as : the two expressions provably coincide whenever has full column rank, and the pseudoinverse route degrades gracefully — delivering the minimum-norm solution — exactly when stops existing.
Challenge Problem 3 (Structural). Let . Prove that has a solution for every iff (i.e. has full row rank), and relate this to the surjectivity of the linear map .
Proof. Immediate from Rouché–Capelli: consistency for all means for every ; taking would strictly increase the rank on the right unless already, i.e. , i.e. . This is exactly the statement that is onto .
Challenge Problem 4 (Well-Posedness of Fitted Values Under Collinearity). Let be a design matrix with (so the regressors are collinear and the normal equations do not pin down a unique ). Prove that nevertheless every solution of the normal equations produces the same fitted-value vector , and that this common value equals for the hat matrix .
Proof. Suppose both satisfy the normal equations: . Then . Writing ,
(Trick 2.2.4, with in place of ), so . The fitted-value vector is identical for every solution of the normal equations, regardless of the rank deficiency of .
To identify the common value: the minimum-norm solution (Theorem 3.4.4) satisfies the normal equations, since transposing the Penrose identity and using symmetry of (Penrose condition 3) gives , hence for every . Since every solution gives the same fitted value as does, and , every solution satisfies
Remark. This is the precise sense in which "OLS is well-defined even with collinear regressors": the coefficients are not identifiable (any two solutions differ by a vector in , itself nontrivial exactly when ), but every quantity actually observable from the fit — fitted values, residuals, , the hat matrix, predictions at points within — is identical no matter which generalized-inverse solution one's software happens to return. This is why collinearity is diagnosed as a problem for interpreting individual coefficients, not as a problem for prediction.
Part 4 — Eigenvalues, Eigenvectors, and Diagonalization
4.1 Eigenvalues, Eigenvectors, Characteristic Polynomial
Definition 4.1.1. For , (or an algebraic closure thereof) is an eigenvalue if there exists with ; is the corresponding eigenvector. The eigenspace .
Definition 4.1.2. The characteristic polynomial is (or , differing by sign by convention — both conventions appear across texts; we use , monic of degree ). Eigenvalues are precisely the roots of .
Theorem 4.1.3. . In particular, the sum of the eigenvalues (with algebraic multiplicity, over ) equals , and their product equals .
(Follows from Vieta's formulas applied to the roots of , using the expansion of the determinant ; the coefficient of is , where is the -th elementary symmetric polynomial in the eigenvalues.)
4.2 Multiplicities and Diagonalizability
Definition 4.2.1. The algebraic multiplicity of eigenvalue is its multiplicity as a root of . The geometric multiplicity .
Theorem 4.2.2. For every eigenvalue , .
Proof. trivially (eigenspace is nontrivial by definition of eigenvalue). For : let be a basis of , extend to a basis of . In this basis, is similar to a block matrix (since for ). Then (block-triangular determinant), so divides , i.e. .
Theorem 4.2.3 (Diagonalizability Criterion). is diagonalizable (similar to a diagonal matrix, over the splitting field of ) iff for every eigenvalue , iff (or its algebraic closure) admits a basis of eigenvectors of .
Theorem 4.2.4 (Sufficient Condition). If has distinct eigenvalues, is diagonalizable. (Eigenvectors for distinct eigenvalues are automatically linearly independent — a standard induction argument: suppose a minimal dependent combination (all ); apply , and separately multiply the original relation by , then subtract, to get , a strictly shorter nontrivial dependency (since , for ), contradicting minimality unless no such dependency existed at all.)
4.3 Cayley–Hamilton and the Minimal Polynomial
Theorem 4.3.1 (Cayley–Hamilton). Every satisfies its own characteristic polynomial: .
Proof sketch (adjugate method). Let , a matrix whose entries are polynomials in of degree ; write for constant matrices . From Theorem 2.4.2, . Expanding both sides as polynomials in with matrix coefficients and equating coefficients of like powers of gives a system relating each to and to the coefficients of ; multiplying the -coefficient equation on the left by and summing over causes every term to appear once with a sign and once with a sign, telescoping the entire left side to while the right side sums to exactly , yielding . (A fully rigorous write-up requires care in tracking the boundary terms of the telescoping sum — standard in Hoffman–Kunze §6.4 or Horn & Johnson §2.4; the outline above captures the mechanism expected on an exam.)
Definition 4.3.2. The minimal polynomial is the monic polynomial of least degree with .
Theorem 4.3.3. divides every polynomial with (in particular , by Cayley–Hamilton), and and have exactly the same roots (possibly with different multiplicities).
Proof. Divisibility: if , polynomial division gives with (or ). Then . If , this contradicts minimality of among nonzero annihilating polynomials; hence and .
Eigenvalue root of : if , , then for every , so for every polynomial . In particular ; since , .
Root of eigenvalue: let be a root of , and write , . Minimality forces ; pick with . Then , so is an eigenvector with eigenvalue .
The roots of are thus exactly the eigenvalues of , which are exactly the roots of .
Theorem 4.3.4. is diagonalizable iff has no repeated roots (i.e. splits into distinct linear factors over the algebraic closure).
Proof. () Suppose is diagonalizable with distinct eigenvalues . The polynomial annihilates : in an eigenbasis, is diagonal, and every diagonal entry is for the relevant eigenvalue ; hence . By Theorem 4.3.3, , and a divisor of a polynomial with only simple roots has only simple roots; so has no repeated roots (and, since every is also a root of by Theorem 4.3.3, in fact exactly).
() Suppose with distinct. Define the Lagrange basis polynomials . The polynomial has degree but distinct roots , forcing identically. Setting gives . Using together with the partial-fraction structure of the , one further checks , for , and , so each is a projection onto (a subspace of) the -eigenspace; then exhibits (or its closure) as a direct sum of eigenspaces of , i.e. has a full eigenbasis. (The verification that each is idempotent and mutually annihilating is a short computation from the partial-fraction identity for ; see Hoffman–Kunze §7.1 or Axler Ch. 8 for the complete construction.)
Remark. This gives a slick alternative diagonalizability test that avoids computing individual eigenspace dimensions: compute (often via minimal-degree relations found by inspection, e.g. for idempotents/involutions/nilpotents where is forced to divide , , respectively) and check for repeated roots.
4.4 Similarity Invariants
Theorem 4.4.1. If (similar, i.e. ), then share: characteristic polynomial, eigenvalues (with algebraic and geometric multiplicities), minimal polynomial, trace, determinant, rank.
(Each follows from direct substitution: e.g. , using .)
Remark (Jordan Canonical Form — stated without full proof). Over an algebraically closed field (e.g. ), every is similar to a Jordan canonical form — a block-diagonal matrix of Jordan blocks ( the nilpotent shift matrix with 's on the superdiagonal). This is the finest similarity invariant: two matrices are similar iff they have the same Jordan form up to block reordering. The full existence/uniqueness proof (via the structure theorem for finitely generated modules over a PID, applied to acting on via ) is beyond entrance-exam scope; what is exam-relevant is: (i) the number of Jordan blocks for eigenvalue equals ; (ii) the sizes of the blocks are determined by the ranks of powers of ; (iii) is diagonalizable iff every Jordan block has size , consistent with Theorem 4.3.4.
4.5 Challenge Problems
Challenge Problem 1 (Minimal Polynomial Trick). Let satisfy . Prove is diagonalizable (over , in the sense of being similar to a diagonal matrix with real entries).
Proof. divides , a product of distinct linear factors over . Hence itself is a product of a subset of these distinct linear factors, so has no repeated roots; by Theorem 4.3.4, is diagonalizable, and since all roots of lie in , the diagonalization is over .
Challenge Problem 2 (Patterned Matrix Diagonalizability). Let be the all-ones matrix. Show that (i.e. on the diagonal, off-diagonal) is diagonalizable for all , and find its eigenvalues explicitly.
Proof. itself is symmetric, hence diagonalizable (Spectral Theorem, §5.2) with eigenvalues (multiplicity 1, eigenvector ) and (multiplicity , eigenvectors spanning ). Since is an affine function of (i.e. ), shares the same eigenvectors as , with eigenvalues (multiplicity 1) and (multiplicity ). Since has an eigenbasis, is diagonalizable. (This matrix is exactly the equicorrelated covariance structure up to scaling — ubiquitous in exchangeable-random-variable models; see Challenge Problem 2 of §5.5.)
Challenge Problem 3 (Matrix Polynomial via Minimal Polynomial). Let with minimal polynomial . Compute in terms of and .
Proof. Since has distinct roots , is diagonalizable with eigenvalues only . Perform polynomial division: for some remainder (degree ). Evaluate at the roots: (from ), (from ). Solve: , . Since , . (This "polynomial remainder via minimal polynomial" trick generalizes to any matrix function once 's roots are known, and is the fast route to computing high matrix powers without diagonalizing explicitly.)
Challenge Problem 4 (Simultaneous Diagonalization). Let satisfy , and suppose both and are diagonalizable. Prove that and are simultaneously diagonalizable: there is a single basis of consisting of eigenvectors of both and .
Proof. Step 1: every eigenspace of is -invariant. Let be an eigenvalue of and , i.e. . Then
so too. Hence .
Step 2: restricted to each eigenspace of is diagonalizable. Since is diagonalizable, its minimal polynomial has distinct roots (Theorem 4.3.4). Let , a well-defined operator on by Step 1. Since on all of , restricting to gives ; so the minimal polynomial of divides (Theorem 4.3.3), and a divisor of a squarefree polynomial is squarefree. By Theorem 4.3.4 again (applied to on ), is diagonalizable.
Step 3: assemble the common eigenbasis. Since is diagonalizable, (sum over the distinct eigenvalues of ). By Step 2, choose within each a basis of eigenvectors of ; every such vector is automatically an eigenvector of itself (since agrees with on ), and it is already an eigenvector of with eigenvalue (by membership in ). The union of these bases over all eigenvalues of is a basis of — the concatenation of bases of the summands in a direct-sum decomposition — consisting entirely of common eigenvectors of and .
Remark (Statistical Payoff). This is the linear-algebraic core of several multivariate methods that use two covariance-type matrices at once — canonical correlation analysis, MANOVA, and Fisher's linear discriminant analysis all hinge on a generalized eigenvalue problem for a pair of matrices that need not literally commute. The standard resolution "whitens" by (when ) to convert the pencil into an ordinary eigenvalue problem for the single symmetric matrix ; the present theorem is the special case that already applies without whitening whenever the two matrices happen to commute outright — for instance, whenever both are diagonal in the same natural basis by construction, as with covariance matrices possessing a shared block or circulant structure.
Part 5 — Spectral Theory and Special Matrices
5.1 Symmetric, Orthogonal, Idempotent, Nilpotent
Definition 5.1.1. is:
- Symmetric if .
- Orthogonal if (equivalently, columns/rows form an orthonormal basis).
- Idempotent (a projection matrix) if .
- Nilpotent if for some .
Theorem 5.1.2 (Eigenvalue Constraints from Structure).
- If is orthogonal, every eigenvalue (possibly complex) satisfies . (Proof: , using since .)
- If is idempotent, every eigenvalue is or . (Proof: for .)
- If is nilpotent, every eigenvalue is . (Proof: .)
Theorem 5.1.3 (Trace and Rank of an Idempotent). If is idempotent, .
Proof. By Challenge Problem 2 of §1.5, , and acts as the identity on (since for , ) and as on . Choosing a basis adapted to this direct sum (basis of followed by basis of ), is similar to where . Trace is a similarity invariant (Corollary 2.2.3), so .
Remark. This is the identity behind "trace of the hat matrix = number of estimated parameters" in OLS — see the Closing Remark.
5.2 The Spectral Theorem
Theorem 5.2.1 (Spectral Theorem for Real Symmetric Matrices). If is symmetric, then:
(a) All eigenvalues of are real.
(b) is orthogonally diagonalizable: there exists an orthogonal matrix and diagonal (real entries) with .
(c) Eigenvectors corresponding to distinct eigenvalues are orthogonal.
Proof.
(a) Let , (possibly complex). Then . Taking conjugate transpose of the scalar : (since real symmetric, ), so is real. Since is real and positive, is real.
(c) Let , , . Then (using symmetry ), so , forcing .
(b) By induction on . Take an eigenvalue (exists, real, by (a) and the Fundamental Theorem of Algebra applied to ) with unit eigenvector . Extend to an orthonormal basis of ; in this basis is similar (via an orthogonal change of basis matrix, since orthonormal-to-orthonormal change of basis is orthogonal) to where is symmetric — symmetry is preserved since the conjugating matrix is orthogonal ( is symmetric whenever is and is orthogonal, as ). Apply the inductive hypothesis to and combine the orthogonal matrices block-diagonally.
Remark. This theorem is the load-bearing wall of Part 5: every subsequent statistical fact — PCA as eigendecomposition of a covariance matrix, positive-definiteness as admissibility for a covariance, SVD via — descends directly from Theorem 5.2.1.
5.3 Positive Definite and Semi-Definite Matrices
Definition 5.3.1. Symmetric is:
- Positive semi-definite () if .
- Positive definite () if .
Theorem 5.3.2 (Characterizations). For symmetric , TFAE:
(i) (resp. ).
(ii) All eigenvalues of are (resp. ).
(iii) All leading principal minors of are (Sylvester's Criterion — for strict positive-definiteness only; the semi-definite analogue requires all principal minors, not just leading ones, to be ).
(iv) for some invertible (resp. for some , possibly singular/rectangular).
Proof sketch. (i)(ii): By Theorem 5.2.1, ; set (a bijective change of variables since orthogonal), so , which is positive for all (equivalently all ) iff every .
(ii)(iv): with , (entrywise square root, valid since eigenvalues ), so with .
(iv)(ii)/(i): always; only if , which (for invertible) forces , giving strict positivity.
(iii) is the classical Sylvester determinantal criterion; proof proceeds by induction using the Schur complement (Theorem 2.3.4) applied to leading blocks — the sign of each successive Schur complement is controlled by the ratio of consecutive leading principal minors.
Remark (A Classic Trap). The caveat in (iii) is not pedantry: the tempting guess that "all leading principal minors " characterizes PSD-ness is false. Take
Both leading principal minors are and , yet has eigenvalue , so . The correct semi-definite criterion genuinely requires checking all principal minors (not just the leading ones) — considerably more expensive, and the reason Sylvester's criterion is usually quoted only for the strict () case in practice.
Corollary 5.3.3 (Positive Definite Square Root). If (resp. ), there is a symmetric (resp. ) with ; write . It satisfies when .
Proof. From (Theorem 5.2.1) with , (resp. ), set , . Then , (resp. ) by Theorem 5.3.2(ii) since its eigenvalues inherit the sign of , and (using ). The identity follows by the same computation applied to : both sides equal . (Uniqueness of the PSD square root is true but not needed below, so we omit it.)
Remark (Statistical Payoff). A covariance matrix of a random vector must be (since for all ), and is required for a non-degenerate multivariate normal density (so that exists and the density's quadratic form in the exponent is a genuine "bowl"). This connection is elaborated in the Closing Remark.
5.4 Singular Value Decomposition
Theorem 5.4.1 (SVD). Every can be written
where , are orthogonal, and is diagonal (in the rectangular sense: , off-diagonal entries ) with — the singular values of .
Proof outline. is symmetric and positive semi-definite (it is of the form with , Theorem 5.3.2(iv)), so by the Spectral Theorem, with orthogonal, . Define and, for , ; one checks are orthonormal, since (using orthonormality of the , and when ). For indices with : , so automatically — no separate is forced by , and the construction extends to a full orthonormal basis of arbitrarily on the remaining coordinates. In either case holds for every (trivially when ), i.e. , giving .
Remark. So: the singular values of are the square roots of the eigenvalues of (equivalently of , which shares the same nonzero eigenvalues); the right singular vectors are eigenvectors of ; the left singular vectors are eigenvectors of . SVD is thus "the Spectral Theorem applied to , transported back to " — it exists for every matrix, not just symmetric/square ones, which is precisely why it underlies PCA (eigendecomposition of the empirical covariance is literally an SVD of the centered data matrix ).
5.5 Challenge Problems
Challenge Problem 1 (Idempotent Diagonalizable, trace = rank, full proof). Let be idempotent (not assumed symmetric). Prove is diagonalizable and .
Proof. Diagonalizability: , so the minimal polynomial divides , a product of distinct linear factors; by Theorem 4.3.4, is diagonalizable. Trace = rank: this was proved directly in Theorem 5.1.3 via the direct-sum decomposition ; alternatively, having established diagonalizability, is similar to where number of eigenvalue-'s (using for idempotents: trivially, and conversely ), and sum of the diagonal entries of the similar diagonal matrix .
Challenge Problem 2 (Structured Covariance Eigenvalues). Let be an equicorrelated covariance matrix ( = all-ones matrix), with to ensure . Find the eigenvalues of and verify positive-definiteness reduces to the stated constraint on .
Proof. Write in the form of Challenge Problem 2 of §4.5, , by matching and (so ). That problem's conclusion transfers directly: eigenvalues of are
By Theorem 5.3.2(ii), iff both are strictly positive: , and . This recovers exactly the stated admissible range — the classical bound on intraclass correlation.
Challenge Problem 3 (Advanced, SVD/rank interplay). Let with SVD and . Prove that the best rank- approximation to in Frobenius norm () is , and that the approximation error is .
Proof outline (Eckart–Young–Mirsky Theorem). Orthogonal invariance of the Frobenius norm — for orthogonal , since is invariant under by cyclicity of trace (Theorem 2.2.2) — reduces the problem to approximating the diagonal matrix by a rank- matrix . Among all rank- matrices , one shows (via a Ky Fan norm / interlacing argument, beyond entrance-exam scope to derive from scratch, but standard — see Horn & Johnson §3.4 or Golub–Van Loan) that is minimized by zeroing out the smallest diagonal entries of , i.e. by , giving and, transporting back via , the stated result. (The reduction to the diagonal case above is fully rigorous and exam-fair; the optimality of truncation among diagonal competitors is the part cited rather than re-derived.)
Challenge Problem 4 (Order-Reversal of Matrix Inversion). Let be symmetric matrices with and (that is, , the Loewner order). Prove that and
Proof. : for any , (using and ). So , and both exist (Corollary 5.3.3).
Reduction to a normalized case. Let , symmetric since . Congruence preserves the Loewner order: for symmetric and any matrix , for all , i.e. . So from , congruence by gives
and from , the same congruence (strict version) gives . So is symmetric with : every eigenvalue of lies in .
Inverting. Diagonalize (, ). Then has eigenvalues , so for any , writing : . Hence .
Undoing the reduction. (using with ). So . Congruence by once more:
and the left side collapses to while the right side is , giving
Remark (Statistical Payoff). This is the linear-algebraic engine behind every "more information can only help" statement in estimation theory. If are two unbiased estimators with (the Gauss–Markov/Loewner sense in which is more efficient), this theorem says the corresponding precision matrices (inverse covariances) satisfy the reverse inequality — more efficient estimation corresponds to more precision, exactly as intuition demands, but the direction-reversal on inverting is precisely this theorem, and it is easy to get backwards under exam pressure.
Appendix: Master Formula Sheet
Trace
| Identity | Condition |
|---|---|
| always | |
| conformable | |
| cyclic, NOT in general | |
| sum of eigenvalues (with algebraic multiplicity) | |
| similarity invariance | |
| idempotent | |
| equality iff |
Determinant
| Identity | Condition |
|---|---|
| square, conformable | |
| always | |
| invertible | |
| product of eigenvalues | |
| block-triangular | |
| invertible; Schur complement | |
| Cauchy–Binet, | |
| matrix determinant lemma (Cor. 2.3.5); when |
Rank
| Identity | Condition |
|---|---|
| Rank–Nullity | |
| row rank = column rank | |
| always | |
| Sylvester's inequality | |
| $ | \text{rank}(A)-\text{rank}(B) |
| over | |
| invertible; Guttman-type rank additivity (§2.6 CP4) |
Inverse / Block Inversion
| Identity | Condition |
|---|---|
| Sherman–Morrison | |
| Woodbury | |
| , | block inversion via Schur complement |
Matrix Calculus (brief)
| Derivative | Result |
|---|---|
| ; if symmetric | |
| (Hessian) | ; if symmetric |
| — setting this to is the normal equations | |
| ; if symmetric | |
Closing Remark: The Unifying Theme
Strip away the exam-paper packaging and this treatise has told one story from five angles: linear algebra is the geometry that makes multivariate statistics legible.
A random vector has a covariance matrix , and the very first thing you learn to check about — that it is symmetric and — is not an incidental algebraic nicety but a theorem (§5.3, Theorem 5.3.2) in disguise: for every direction is exactly the defining inequality of positive semi-definiteness. When , the Spectral Theorem (§5.2) hands you an orthogonal basis of principal axes — eigenvectors of — along which the components of decorrelate, and the multivariate normal density is literally the equation of an ellipsoid whose principal axes are 's eigenvectors and whose semi-axis lengths are proportional to the square roots of 's eigenvalues — equivalently, inversely proportional to the square roots of 's eigenvalues, so that directions of large variance become the long axes of the density's contours, exactly as intuition demands. Positive-definiteness is not a technical side-condition on this picture; it is the condition that the ellipsoid is a genuine, non-degenerate bowl rather than a flattened, singular one — and Challenge Problem 4 of §5.5 is the statement that shrinking this ellipsoid (in the Loewner order) is the same thing as growing its inverse, the precise sense in which more precise estimation means a tighter concentration ellipsoid.
And the idempotent matrices of §5.1 are the other half of the same coin. In ordinary least squares, (an instance of the pseudoinverse of §3.4, since when has full column rank) produces fitted values . The hat matrix is symmetric and idempotent — — which is precisely the algebraic signature of an orthogonal projection (Challenge Problem 2 of §1.5: , and here that decomposition is the decomposition of into the column space of and its orthogonal complement, the residual space — the very orthogonality proved abstractly in Theorem 1.4.4). Theorem 5.1.3 — — combined with the fact that (so when has full column rank ) tells you the trace of the hat matrix equals , the number of estimated regression coefficients — exactly the "degrees of freedom used up" that appears in every -test and every unbiasedness calculation for in a linear model.
So when you next diagonalize a symmetric matrix, or check that a Schur complement is positive, or compute for a hat matrix — you are not doing algebra and then applying it to statistics. You are doing the same piece of mathematics twice, once in the language of vector spaces and once in the language of random variables, and linear algebra is simply the more honest of the two languages about what is actually going on.