← Study Notes

Mathematics

Linear Algebra

English · 225 pp. PDF

Open PDF ↗ Download PDF ↓

Contents

1. 1 Systems of Linear Equations
   1. 1.1 Basic Concepts
   2. 1.2 Elementary Row Operations
   3. 1.3 Gaussian Elimination
   4. 1.4 Homogeneous Systems
   5. 1.5 Applications
2. 2 Matrix Algebra
   1. 2.1 Matrix Basics
   2. 2.2 Matrix-Vector Multiplication
   3. 2.3 Matrix Multiplication
   4. 2.4 Matrix Inverses
   5. 2.5 Elementary Matrices
   6. 2.6 Linear Transformations
   7. 2.7 LU Factorization
   8. 2.8 Applications
3. 3 Determinants and Diagonalization
   1. 3.1 Cofactor Expansion
   2. 3.2 Determinants and Inverses
   3. 3.3 Eigenvalues and Eigenvectors
   4. 3.4 Diagonalization
   5. 3.5 Applications
4. 4 Vector Geometry
   1. 4.1 Vectors and Lines
   2. 4.2 Projections and Planes
   3. 4.3 Cross Product
   4. 4.4 Operators in ℝ³
   5. 4.5 Application to Computer Graphics
5. 5 The Vector Space ℝⁿ
   1. 5.1 Subspaces and Spanning Sets
   2. 5.2 Independence and Dimension
   3. 5.3 Orthogonality in ℝⁿ
   4. 5.4 Rank of a Matrix
   5. 5.5 Similarity and Diagonalization
   6. 5.6 Best Approximation and Least Squares
   7. 5.7 Application to Correlation and Variance
6. 6 Abstract Vector Spaces
   1. 6.1 Examples and Properties
   2. 6.2 Subspaces and Spanning Sets
   3. 6.3 Linear Independence and Basis
   4. 6.4 Finite-Dimensional Spaces
   5. 6.5 Applications
7. 7 Linear Transformations
   1. 7.1 Examples and Properties
   2. 7.2 Kernel and Image
   3. 7.3 Isomorphisms and Composition
   4. 7.4 Applications
8. 8 Orthogonality and Decomposition
   1. 8.1 Orthogonal Complements and Projections
   2. 8.2 Orthogonal Diagonalization
   3. 8.3 Positive Definite Matrices
   4. 8.4 QR-Factorization
   5. 8.5 Computing Eigenvalues
   6. 8.6 Singular Value Decomposition (SVD)
   7. 8.7 Complex Matrices
   8. 8.8 Applications of Matrix Decompositions
9. 9 Change of Basis
   1. 9.1 Coordinate Vectors and Matrix Representations
   2. 9.2 Change of Basis Matrix
   3. 9.3 Change of Basis for Linear Transformations
   4. 9.4 Applications of Change of Basis
10. 10 Inner Product Spaces
    1. 10.1 Definition and Examples
    2. 10.2 Orthogonality in Inner Product Spaces
    3. 10.3 Orthogonal Projections and Gram-Schmidt Process
    4. 10.4 Orthogonal Complements and Direct Sums
    5. 10.5 Applications to Approximation Theory
11. 11 Canonical Forms
    1. 11.1 Generalized Eigenvectors and Eigenspaces
    2. 11.2 Jordan Canonical Form
    3. 11.3 Minimal Polynomial
    4. 11.4 Rational Canonical Form
    5. 11.5 Applications and Computational Aspects
    6. 11.6 Advanced Topics and Connections

About this note. These are English notes for a first/second course in Linear Algebra. They follow a Topic–Section–Subsection hierarchy and mix definitions, proofs, worked examples, geometric intuition, and short exercises. ¹

1 Systems of Linear Equations

Linear algebra begins with the study of systems of linear equations, which are among the most fundamental objects in mathematics. Such systems arise naturally in countless applications across science, engineering, economics, and beyond. This chapter develops the theoretical foundation and computational tools necessary to solve and understand these systems completely.

1.1 Basic Concepts

1.1.1 Definition of a Linear Equation

Before diving into systems, we must first understand what constitutes a linear equation.

Definition 1.1 (Linear Equation). A linear equation in the variables \(x_1, x_2, \ldots, x_n\) is an equation that can be written in the form \[\begin{equation} a_1 x_1 + a_2 x_2 + \cdots + a_n x_n = b \end{equation}\] where \(a_1, a_2, \ldots, a_n\) and \(b\) are real constants, and \(a_1, a_2, \ldots, a_n\) are called the coefficients of the variables.

Remark 2. The term "linear" reflects the fact that each variable appears to the first power only, and no products of variables appear. This geometric interpretation will become clearer as we progress.

Example 1.3. The following are examples of linear equations: \[\begin{align} 3x_1 - 2x_2 + 7x_3 &= 14 \quad \text{(linear in three variables)}\\ -x + 5y &= 0 \quad \text{(linear in two variables)}\\ 2z &= -8 \quad \text{(linear in one variable)} \end{align}\] The following are not linear equations: \[\begin{align} x_1^2 + x_2 &= 5 \quad \text{(contains } x_1^2\text{)}\\ 3x_1 x_2 - x_3 &= 1 \quad \text{(contains product } x_1 x_2\text{)}\\ \sin(x) + 2y &= 0 \quad \text{(contains transcendental function)} \end{align}\]

Now we can define what we mean by a system of linear equations:

Definition 1.4 (System of Linear Equations). A system of linear equations (or linear system) is a collection of one or more linear equations involving the same variables. A general system of \(m\) linear equations in \(n\) variables can be written as: \[\begin{align} a_{11}x_1 + a_{12}x_2 + \cdots + a_{1n}x_n &= b_1\\ a_{21}x_1 + a_{22}x_2 + \cdots + a_{2n}x_n &= b_2\\ &\vdots\\ a_{m1}x_1 + a_{m2}x_2 + \cdots + a_{mn}x_n &= b_m \end{align}\] where \(a_{ij}\) and \(b_i\) are real constants.

Notation 5. We use the double subscript notation \(a_{ij}\) where the first index \(i\) indicates the equation number (row) and the second index \(j\) indicates the variable number (column). This systematic notation will prove invaluable when we introduce matrix representations.

Definition 1.6 (Solution). A solution to a system of linear equations is an ordered \(n\)-tuple \((s_1, s_2, \ldots, s_n)\) such that when we substitute \(x_1 = s_1\), \(x_2 = s_2\), …, \(x_n = s_n\) into each equation of the system, all equations are satisfied simultaneously.

Definition 1.7 (Solution Set). The solution set of a linear system is the set of all solutions to the system.

1.1.2 Solution Sets (unique, none, infinite)

One of the most fundamental questions in linear algebra is: "Given a system of linear equations, what can we say about its solutions?" The remarkable answer is that there are exactly three possibilities, and no others.

Theorem 1.8 (Fundamental Theorem about Linear Systems). Every system of linear equations has exactly one of the following:

1. No solution (the system is inconsistent)

2. Exactly one solution (the system is consistent and determinate)

3. Infinitely many solutions (the system is consistent and indeterminate)

Remark 9. This theorem is profound: a linear system can never have exactly two solutions, or exactly 17 solutions, or any finite number greater than one. The proof of this fact will emerge naturally from our study of the solution process.

Let us examine each case with detailed examples:

Example 1.10 (System with No Solution). Consider the system: \[\begin{align} x + y &= 3\\ x + y &= 5 \end{align}\] These equations are contradictory: no pair \((x,y)\) can simultaneously satisfy both \(x + y = 3\) and \(x + y = 5\). Geometrically, these represent two parallel lines that never intersect.

Example 1.11 (System with Exactly One Solution). Consider the system: \[\begin{align} x + y &= 3\\ x - y &= 1 \end{align}\] Adding the equations: \(2x = 4\), so \(x = 2\). Substituting back: \(2 + y = 3\), so \(y = 1\). The unique solution is \((2, 1)\). Geometrically, this represents two lines intersecting at exactly one point.

Verification: We should always verify our solutions: \[\begin{align} 2 + 1 &= 3 \quad \checkmark\\ 2 - 1 &= 1 \quad \checkmark \end{align}\]

Example 1.12 (System with Infinitely Many Solutions). Consider the system: \[\begin{align} x + y &= 3\\ 2x + 2y &= 6 \end{align}\] The second equation is exactly twice the first, so they represent the same line. Every point on the line \(x + y = 3\) is a solution. We can express the solutions parametrically: if we let \(x = t\) (where \(t\) is any real number), then \(y = 3 - t\). The solution set is \(\{(t, 3-t) : t \in \mathbb{R}\}\).

Key Idea

The geometric interpretation provides invaluable intuition: in two dimensions, each linear equation represents a line. The solution set corresponds to the intersection of these lines:

• No intersection \(\Rightarrow\) no solution

• One intersection point \(\Rightarrow\) unique solution

• Lines coincide \(\Rightarrow\) infinitely many solutions

This geometric perspective extends to higher dimensions, where equations represent hyperplanes.

1.1.3 Parametric Representation of Solutions

When a system has infinitely many solutions, we need a systematic way to describe all solutions. This leads us to parametric representations.

Definition 1.13 (Parametric Solution). A parametric solution to a linear system expresses the solution set using one or more parameters (free variables). Each choice of parameter values corresponds to a specific solution.

Example 1.14 (Parametric Representation in Detail). Consider the system: \[\begin{align} x_1 + 2x_2 - x_3 &= 4\\ 2x_1 + 4x_2 - 2x_3 &= 8 \end{align}\]

First, we observe that the second equation is exactly twice the first, so we effectively have only one equation in three unknowns: \[x_1 + 2x_2 - x_3 = 4\]

We can solve for one variable in terms of the others. Let’s solve for \(x_1\): \[x_1 = 4 - 2x_2 + x_3\]

Since \(x_2\) and \(x_3\) can be chosen arbitrarily, we introduce parameters \(s\) and \(t\): \[\begin{align} x_2 &= s\\ x_3 &= t\\ x_1 &= 4 - 2s + t \end{align}\]

The complete solution set can be written as: \[\{(4 - 2s + t, s, t) : s, t \in \mathbb{R}\}\]

Vector Form: We can also express this in vector form: \[\boldsymbol{x} = \begin{pmatrix} 4 - 2s + t \\ s \\ t \end{pmatrix} = \begin{pmatrix} 4 \\ 0 \\ 0 \end{pmatrix} + s\begin{pmatrix} -2 \\ 1 \\ 0 \end{pmatrix} + t\begin{pmatrix} 1 \\ 0 \\ 1 \end{pmatrix}\]

This representation shows that the solution set is a plane in three-dimensional space passing through the point \((4, 0, 0)\).

Remark 15. The choice of which variables to use as parameters is not unique. However, the solution set itself is uniquely determined by the system. Different parametric representations describe the same geometric object.

1.2 Elementary Row Operations

To solve linear systems systematically, we need to develop algebraic tools that preserve solutions while simplifying the system’s form. This leads us to matrix representations and elementary row operations.

1.2.1 Augmented and Coefficient Matrices

Definition 1.16 (Coefficient Matrix). For the linear system \[\begin{align} a_{11}x_1 + a_{12}x_2 + \cdots + a_{1n}x_n &= b_1\\ a_{21}x_1 + a_{22}x_2 + \cdots + a_{2n}x_n &= b_2\\ &\vdots\\ a_{m1}x_1 + a_{m2}x_2 + \cdots + a_{mn}x_n &= b_m \end{align}\] the coefficient matrix is the \(m \times n\) matrix \[\mathbf{A} = \begin{pmatrix} a_{11} & a_{12} & \cdots & a_{1n}\\ a_{21} & a_{22} & \cdots & a_{2n}\\ \vdots & \vdots & \ddots & \vdots\\ a_{m1} & a_{m2} & \cdots & a_{mn} \end{pmatrix}\]

Definition 1.17 (Augmented Matrix). The augmented matrix is the \(m \times (n+1)\) matrix formed by appending the constant vector as an additional column: \[[\mathbf{A}\mid\boldsymbol{b}] = \bgroup % \left[\begin{array}{cccc|c} a_{11} & a_{12} & \cdots & a_{1n} & b_1\\ a_{21} & a_{22} & \cdots & a_{2n} & b_2\\ \vdots & \vdots & \ddots & \vdots & \vdots\\ a_{m1} & a_{m2} & \cdots & a_{mn} & b_m \end{array}\right]\egroup\]

Example 1.18 (Matrix Representation). The system \[\begin{align} 2x_1 - 3x_2 + x_3 &= 7\\ x_1 + x_2 - 2x_3 &= -4\\ 3x_1 - 2x_2 + 5x_3 &= 0 \end{align}\] has coefficient matrix \[\mathbf{A} = \begin{pmatrix} 2 & -3 & 1\\ 1 & 1 & -2\\ 3 & -2 & 5 \end{pmatrix}\] and augmented matrix \[[\mathbf{A}\mid\boldsymbol{b}] = \bgroup % \left[\begin{array}{rrr|r} 2 & -3 & 1 & 7\\ 1 & 1 & -2 & -4\\ 3 & -2 & 5 & 0 \end{array}\right]\egroup\]

Remark 19. The augmented matrix contains all the information of the linear system. The vertical bar is a notational convenience to separate the coefficients from the constants, but mathematically it’s just a matrix.

1.2.2 Equivalence of Systems

Definition 1.20 (Equivalent Systems). Two linear systems are equivalent if they have the same solution set.

Our goal is to transform a given system into an equivalent system that is easier to solve. This leads us to operations that preserve the solution set.

Theorem 1.21 (Solution-Preserving Operations). The following operations on a linear system produce an equivalent system:

1. Interchange two equations

2. Multiply an equation by a nonzero constant

3. Add a multiple of one equation to another equation

Proof. We prove that each operation preserves the solution set.

Operation 1 (Interchange): If \((s_1, s_2, \ldots, s_n)\) satisfies both equations before interchange, it clearly satisfies both after interchange, since the same equations are present in different order.

Operation 2 (Scaling): Suppose equation \(i\) is \(a_{i1}x_1 + \cdots + a_{in}x_n = b_i\), and we multiply by nonzero constant \(k\). If \((s_1, \ldots, s_n)\) is a solution, then \(a_{i1}s_1 + \cdots + a_{in}s_n = b_i\). Multiplying both sides by \(k\): \(k(a_{i1}s_1 + \cdots + a_{in}s_n) = kb_i\), which shows the solution satisfies the new equation. Conversely, if the new equation is satisfied, dividing by \(k\) shows the original equation is satisfied.

Operation 3 (Replacement): Suppose we replace equation \(j\) with equation \(j\) plus \(k\) times equation \(i\). If \((s_1, \ldots, s_n)\) satisfies the original system, then both equations \(i\) and \(j\) are satisfied. The new equation \(j\) becomes: \[(a_{j1} + ka_{i1})s_1 + \cdots + (a_{jn} + ka_{in})s_n = b_j + kb_i\] This equals \((a_{j1}s_1 + \cdots + a_{jn}s_n) + k(a_{i1}s_1 + \cdots + a_{in}s_n) = b_j + kb_i\), which is satisfied. The converse follows by subtracting \(k\) times equation \(i\) from the new equation \(j\). ◻

1.2.3 Elementary Row Operations and Their Inverses

When we perform solution-preserving operations on a system, we correspondingly perform operations on its augmented matrix.

Definition 1.22 (Elementary Row Operations). The three elementary row operations on a matrix are:

1. Row Interchange: \(R_i \leftrightarrow R_j\) (swap rows \(i\) and \(j\))

2. Row Scaling: \(kR_i\) (multiply row \(i\) by nonzero scalar \(k\))

3. Row Replacement: \(R_j + kR_i\) (add \(k\) times row \(i\) to row \(j\))

Theorem 1.23 (Invertibility of Elementary Row Operations). Each elementary row operation is reversible:

1. The inverse of \(R_i \leftrightarrow R_j\) is \(R_i \leftrightarrow R_j\) (same operation)

2. The inverse of \(kR_i\) is \(\frac{1}{k}R_i\) (provided \(k \neq 0\))

3. The inverse of \(R_j + kR_i\) is \(R_j - kR_i\)

Example 1.24 (Elementary Row Operations in Action). Starting with the augmented matrix: \[\bgroup % \left[\begin{array}{rr|r} 1 & 2 & 5\\ 3 & 1 & 4 \end{array}\right]\egroup\]

Apply \(R_2 - 3R_1\): \[\bgroup % \left[\begin{array}{rr|r} 1 & 2 & 5\\ 0 & -5 & -11 \end{array}\right]\egroup\]

Apply \(-\frac{1}{5}R_2\): \[\bgroup % \left[\begin{array}{rr|r} 1 & 2 & 5\\ 0 & 1 & \frac{11}{5} \end{array}\right]\egroup\]

Apply \(R_1 - 2R_2\): \[\bgroup % \left[\begin{array}{rr|r} 1 & 0 & \frac{3}{5}\\ 0 & 1 & \frac{11}{5} \end{array}\right]\egroup\]

This corresponds to the system \(x_1 = \frac{3}{5}\), \(x_2 = \frac{11}{5}\), giving us the unique solution \(\left(\frac{3}{5}, \frac{11}{5}\right)\).

1.3 Gaussian Elimination

Gaussian elimination is the systematic process of using elementary row operations to solve linear systems. It transforms the augmented matrix into a form where the solution can be read directly.

1.3.1 Row-Echelon and Reduced Row-Echelon Forms

Definition 1.25 (Leading Entry). In a nonzero row of a matrix, the leading entry (or pivot) is the leftmost nonzero entry.

Definition 1.26 (Row-Echelon Form). A matrix is in row-echelon form (REF) if:

1. All nonzero rows are above any rows of all zeros

2. Each leading entry of a row is in a column to the right of the leading entry of the row above it

3. All entries in a column below a leading entry are zeros

Definition 1.27 (Reduced Row-Echelon Form). A matrix is in reduced row-echelon form (RREF) if:

1. It is in row-echelon form

2. Each leading entry is 1

3. Each leading entry is the only nonzero entry in its column

Example 1.28 (Row-Echelon Forms). The following matrix is in row-echelon form: \[\begin{pmatrix} 2 & -1 & 0 & 3\\ 0 & 3 & 1 & -2\\ 0 & 0 & 0 & 4\\ 0 & 0 & 0 & 0 \end{pmatrix}\]

The following matrix is in reduced row-echelon form: \[\begin{pmatrix} 1 & 0 & -2 & 0\\ 0 & 1 & 3 & 0\\ 0 & 0 & 0 & 1\\ 0 & 0 & 0 & 0 \end{pmatrix}\]

Theorem 1.29 (Uniqueness of Reduced Row-Echelon Form). Each matrix is row equivalent to one and only one reduced row-echelon form matrix.

Remark 30. This uniqueness theorem is crucial: it means that regardless of the sequence of elementary row operations we choose, we will always arrive at the same RREF. This gives us confidence that our solution method is well-defined.

1.3.2 Gaussian Algorithm (step-by-step)

The Gaussian elimination algorithm systematically transforms any matrix to row-echelon form, and can be extended to reduced row-echelon form.

Example 1.31 (Complete Gaussian Elimination). Solve the system: \[\begin{align} x_1 + 3x_2 - 2x_3 &= 5\\ 3x_1 + 5x_2 + 6x_3 &= 7\\ 2x_1 + 4x_2 + 3x_3 &= 8 \end{align}\]

Step 1: Write the augmented matrix: \[\bgroup % \left[\begin{array}{rrr|r} 1 & 3 & -2 & 5\\ 3 & 5 & 6 & 7\\ 2 & 4 & 3 & 8 \end{array}\right]\egroup\]

Step 2: Eliminate below the first pivot: \(R_2 - 3R_1\): \[\bgroup % \left[\begin{array}{rrr|r} 1 & 3 & -2 & 5\\ 0 & -4 & 12 & -8\\ 2 & 4 & 3 & 8 \end{array}\right]\egroup\]

\(R_3 - 2R_1\): \[\bgroup % \left[\begin{array}{rrr|r} 1 & 3 & -2 & 5\\ 0 & -4 & 12 & -8\\ 0 & -2 & 7 & -2 \end{array}\right]\egroup\]

Step 3: Work on the second column. Scale the second row: \(-\frac{1}{4}R_2\): \[\bgroup % \left[\begin{array}{rrr|r} 1 & 3 & -2 & 5\\ 0 & 1 & -3 & 2\\ 0 & -2 & 7 & -2 \end{array}\right]\egroup\]

Step 4: Eliminate below the second pivot: \(R_3 + 2R_2\): \[\bgroup % \left[\begin{array}{rrr|r} 1 & 3 & -2 & 5\\ 0 & 1 & -3 & 2\\ 0 & 0 & 1 & 2 \end{array}\right]\egroup\]

This is now in row-echelon form. For RREF, we continue:

Step 5: Eliminate above the third pivot: \(R_2 + 3R_3\): \[\bgroup % \left[\begin{array}{rrr|r} 1 & 3 & -2 & 5\\ 0 & 1 & 0 & 8\\ 0 & 0 & 1 & 2 \end{array}\right]\egroup\]

\(R_1 + 2R_3\): \[\bgroup % \left[\begin{array}{rrr|r} 1 & 3 & 0 & 9\\ 0 & 1 & 0 & 8\\ 0 & 0 & 1 & 2 \end{array}\right]\egroup\]

Step 6: Eliminate above the second pivot: \(R_1 - 3R_2\): \[\bgroup % \left[\begin{array}{rrr|r} 1 & 0 & 0 & -15\\ 0 & 1 & 0 & 8\\ 0 & 0 & 1 & 2 \end{array}\right]\egroup\]

Solution: The unique solution is \((x_1, x_2, x_3) = (-15, 8, 2)\).

Verification: \[\begin{align} -15 + 3(8) - 2(2) &= -15 + 24 - 4 = 5 \quad \checkmark\\ 3(-15) + 5(8) + 6(2) &= -45 + 40 + 12 = 7 \quad \checkmark\\ 2(-15) + 4(8) + 3(2) &= -30 + 32 + 6 = 8 \quad \checkmark \end{align}\]

1.3.3 Consistency and Rank Conditions

Definition 1.32 (Consistent System). A linear system is consistent if it has at least one solution, and inconsistent if it has no solution.

Theorem 1.33 (Test for Consistency). A linear system is inconsistent if and only if the reduced row-echelon form of the augmented matrix has a row of the form: \[[0 \quad 0 \quad \cdots \quad 0 \quad | \quad 1]\] Such a row corresponds to the equation \(0 = 1\), which is impossible.

Definition 1.34 (Rank of a Matrix). The rank of a matrix \(\mathbf{A}\), denoted \(\operatorname{rank}(\mathbf{A})\), is the number of leading entries (pivots) in any row-echelon form of \(\mathbf{A}\).

Theorem 1.35 (Rank and Consistency). Consider the linear system \(\mathbf{A}\boldsymbol{x} = \boldsymbol{b}\) with augmented matrix \([\mathbf{A}|\boldsymbol{b}]\). The system is consistent if and only if: \[\operatorname{rank}(\mathbf{A}) = \operatorname{rank}([\mathbf{A}|\boldsymbol{b}])\]

Theorem 1.36 (Number of Solutions). For a consistent system \(\mathbf{A}\boldsymbol{x} = \boldsymbol{b}\) where \(\mathbf{A}\) is \(m \times n\):

• If \(\operatorname{rank}(\mathbf{A}) = n\), the system has a unique solution

• If \(\operatorname{rank}(\mathbf{A}) < n\), the system has infinitely many solutions

The number of free variables is \(n - \operatorname{rank}(\mathbf{A})\).

Example 1.37 (Inconsistent System). Consider: \[\bgroup % \left[\begin{array}{rr|r} 1 & 2 & 3\\ 2 & 4 & 5 \end{array}\right]\egroup\]

Apply \(R_2 - 2R_1\): \[\bgroup % \left[\begin{array}{rr|r} 1 & 2 & 3\\ 0 & 0 & -1 \end{array}\right]\egroup\]

The second row represents \(0 = -1\), so the system is inconsistent. Here \(\operatorname{rank}(\mathbf{A}) = 1\) but \(\operatorname{rank}([\mathbf{A}|\boldsymbol{b}]) = 2\).

1.4 Homogeneous Systems

Definition 1.38 (Homogeneous System). A system of linear equations is homogeneous if all the constant terms are zero: \[\begin{align} a_{11}x_1 + a_{12}x_2 + \cdots + a_{1n}x_n &= 0\\ a_{21}x_1 + a_{22}x_2 + \cdots + a_{2n}x_n &= 0\\ &\vdots\\ a_{m1}x_1 + a_{m2}x_2 + \cdots + a_{mn}x_n &= 0 \end{align}\] This can be written in matrix form as \(\mathbf{A}\boldsymbol{x} = \boldsymbol{0}\).

Remark 39. The word "homogeneous" comes from the Greek meaning "of the same kind" - all terms on the right-hand side are the same (zero).

Theorem 1.40 (Properties of Homogeneous Systems). Every homogeneous system \(\mathbf{A}\boldsymbol{x} = \boldsymbol{0}\) has the following properties:

1. The system is always consistent (since \(\boldsymbol{x} = \boldsymbol{0}\) is always a solution)

2. If \(\boldsymbol{u}\) and \(\boldsymbol{v}\) are solutions, then \(\boldsymbol{u} + \boldsymbol{v}\) is also a solution

3. If \(\boldsymbol{u}\) is a solution and \(c\) is any scalar, then \(c\boldsymbol{u}\) is also a solution

4. The solution set forms a subspace of \(\mathbb{R}^n\)

Proof.

1. \(\mathbf{A}\boldsymbol{0} = \boldsymbol{0}\) by properties of matrix multiplication

2. If \(\mathbf{A}\boldsymbol{u} = \boldsymbol{0}\) and \(\mathbf{A}\boldsymbol{v} = \boldsymbol{0}\), then \(\mathbf{A}(\boldsymbol{u} + \boldsymbol{v}) = \mathbf{A}\boldsymbol{u} + \mathbf{A}\boldsymbol{v} = \boldsymbol{0} + \boldsymbol{0} = \boldsymbol{0}\)

3. If \(\mathbf{A}\boldsymbol{u} = \boldsymbol{0}\), then \(\mathbf{A}(c\boldsymbol{u}) = c(\mathbf{A}\boldsymbol{u}) = c\boldsymbol{0} = \boldsymbol{0}\)

4. Properties 1-3 establish that the solution set is closed under addition and scalar multiplication, and contains the zero vector

 ◻

Theorem 1.41 (Nontrivial Solutions). A homogeneous system \(\mathbf{A}\boldsymbol{x} = \boldsymbol{0}\) has nontrivial solutions (solutions other than \(\boldsymbol{x} = \boldsymbol{0}\)) if and only if the system has at least one free variable, which occurs when \(\operatorname{rank}(\mathbf{A}) < n\) (where \(n\) is the number of variables).

1.4.1 Null Space of a Matrix

Definition 1.42 (Null Space). The null space (or kernel) of an \(m \times n\) matrix \(\mathbf{A}\) is the set of all solutions to the homogeneous system \(\mathbf{A}\boldsymbol{x} = \boldsymbol{0}\): \[\text{Nul}(\mathbf{A}) = \{\boldsymbol{x} \in \mathbb{R}^n : \mathbf{A}\boldsymbol{x} = \boldsymbol{0}\}\]

Theorem 1.43 (Null Space is a Subspace). For any \(m \times n\) matrix \(\mathbf{A}\), the null space \(\text{Nul}(\mathbf{A})\) is a subspace of \(\mathbb{R}^n\).

Example 1.44 (Finding the Null Space). Find the null space of \(\mathbf{A} = \begin{pmatrix} 1 & 3 & -4 \\ 2 & -1 & 3 \end{pmatrix}\).

We solve \(\mathbf{A}\boldsymbol{x} = \boldsymbol{0}\): \[\bgroup % \left[\begin{array}{rrr|r} 1 & 3 & -4 & 0\\ 2 & -1 & 3 & 0 \end{array}\right]\egroup\]

Apply \(R_2 - 2R_1\): \[\bgroup % \left[\begin{array}{rrr|r} 1 & 3 & -4 & 0\\ 0 & -7 & 11 & 0 \end{array}\right]\egroup\]

Apply \(-\frac{1}{7}R_2\): \[\bgroup % \left[\begin{array}{rrr|r} 1 & 3 & -4 & 0\\ 0 & 1 & -\frac{11}{7} & 0 \end{array}\right]\egroup\]

Apply \(R_1 - 3R_2\): \[\bgroup % \left[\begin{array}{rrr|r} 1 & 0 & \frac{5}{7} & 0\\ 0 & 1 & -\frac{11}{7} & 0 \end{array}\right]\egroup\]

This gives us: \[\begin{align} x_1 + \frac{5}{7}x_3 &= 0 \quad \Rightarrow \quad x_1 = -\frac{5}{7}x_3\\ x_2 - \frac{11}{7}x_3 &= 0 \quad \Rightarrow \quad x_2 = \frac{11}{7}x_3 \end{align}\]

Setting \(x_3 = t\) (free variable): \[\text{Nul}(\mathbf{A}) = \left\{t\begin{pmatrix} -5/7 \\ 11/7 \\ 1 \end{pmatrix} : t \in \mathbb{R}\right\} = \operatorname{span}\left\{\begin{pmatrix} -5 \\ 11 \\ 7 \end{pmatrix}\right\}\]

1.4.2 Basis of Solution Space

Definition 1.45 (Basis of Null Space). A basis for the null space of a matrix \(\mathbf{A}\) is a linearly independent set of vectors that spans \(\text{Nul}(\mathbf{A})\).

Theorem 1.46 (Constructing a Basis for the Null Space). To find a basis for \(\text{Nul}(\mathbf{A})\):

1. Row reduce \(\mathbf{A}\) to reduced row-echelon form

2. Identify the free variables

3. For each free variable, set it to 1 and all other free variables to 0, then solve for the basic variables

4. The resulting vectors form a basis for \(\text{Nul}(\mathbf{A})\)

Example 1.47 (Basis with Multiple Free Variables). Find a basis for the null space of: \[\mathbf{A} = \begin{pmatrix} 1 & 2 & -1 & 3 & 0\\ 2 & 4 & 1 & 0 & 5\\ 1 & 2 & 2 & -3 & 5 \end{pmatrix}\]

Row reducing to RREF: \[\bgroup % \left[\begin{array}{rrrrr|r} 1 & 2 & 0 & 1 & \frac{5}{3} & 0\\ 0 & 0 & 1 & -2 & \frac{5}{3} & 0\\ 0 & 0 & 0 & 0 & 0 & 0 \end{array}\right]\egroup\]

The basic variables are \(x_1\) and \(x_3\); the free variables are \(x_2\), \(x_4\), and \(x_5\).

From the RREF: \[\begin{align} x_1 + 2x_2 + x_4 + \frac{5}{3}x_5 &= 0\\ x_3 - 2x_4 + \frac{5}{3}x_5 &= 0 \end{align}\]

For the basis vectors:

Vector 1 (\(x_2 = 1\), \(x_4 = x_5 = 0\)): \[x_1 = -2, \quad x_3 = 0 \quad \Rightarrow \quad \boldsymbol{v}_1 = \begin{pmatrix} -2 \\ 1 \\ 0 \\ 0 \\ 0 \end{pmatrix}\]

Vector 2 (\(x_4 = 1\), \(x_2 = x_5 = 0\)): \[x_1 = -1, \quad x_3 = 2 \quad \Rightarrow \quad \boldsymbol{v}_2 = \begin{pmatrix} -1 \\ 0 \\ 2 \\ 1 \\ 0 \end{pmatrix}\]

Vector 3 (\(x_5 = 1\), \(x_2 = x_4 = 0\)): \[x_1 = -\frac{5}{3}, \quad x_3 = -\frac{5}{3} \quad \Rightarrow \quad \boldsymbol{v}_3 = \begin{pmatrix} -5/3 \\ 0 \\ -5/3 \\ 0 \\ 1 \end{pmatrix}\]

Therefore: \(\text{Nul}(\mathbf{A}) = \operatorname{span}\{\boldsymbol{v}_1, \boldsymbol{v}_2, \boldsymbol{v}_3\}\).

1.5 Applications

Linear systems arise naturally in numerous practical contexts. We examine three important application areas that demonstrate the power and ubiquity of linear algebra.

1.5.1 Network Flow

Network flow problems model the movement of commodities through a network of nodes and edges. The fundamental principle is conservation of flow: at each node, the total inflow must equal the total outflow.

Example 1.48 (Traffic Flow Analysis). Consider a network of one-way streets in a city center. Let \(x_i\) denote the number of cars per hour flowing along street \(i\).

Applying conservation of flow at each node:

Node A: \(300 = x_1 + x_2\) Node B: \(x_1 = 200 + x_3\) Node C: \(150 + x_2 = x_4\) Node D: \(x_3 + x_4 = 250\)

This gives us the system: \[\begin{align} x_1 + x_2 + 0x_3 + 0x_4 &= 300\\ x_1 + 0x_2 - x_3 + 0x_4 &= 200\\ 0x_1 + x_2 + 0x_3 - x_4 &= -150\\ 0x_1 + 0x_2 + x_3 + x_4 &= 250 \end{align}\]

Row reducing the augmented matrix: \[\bgroup % \left[\begin{array}{rrrr|r} 1 & 1 & 0 & 0 & 300\\ 1 & 0 & -1 & 0 & 200\\ 0 & 1 & 0 & -1 & -150\\ 0 & 0 & 1 & 1 & 250 \end{array}\right]\egroup \sim \bgroup % \left[\begin{array}{rrrr|r} 1 & 0 & 0 & 1 & 50\\ 0 & 1 & 0 & -1 & -150\\ 0 & 0 & 1 & 1 & 250\\ 0 & 0 & 0 & 0 & 0 \end{array}\right]\egroup\]

Setting \(x_4 = t\) (free variable): \[\begin{align} x_1 &= 50 - t\\ x_2 &= t - 150\\ x_3 &= 250 - t\\ x_4 &= t \end{align}\]

For physical feasibility, all flows must be non-negative: \(x_1 \geq 0 \Rightarrow t \leq 50\), \(x_2 \geq 0 \Rightarrow t \geq 150\), \(x_3 \geq 0 \Rightarrow t \leq 250\)

Therefore \(150 \leq t \leq 50\), which is impossible! This indicates the given external flows are incompatible with the network structure - the traffic demands cannot be satisfied.

1.5.2 Electrical Networks

Electrical circuit analysis relies on two fundamental laws that lead to systems of linear equations.

Definition 1.49 (Kirchhoff’s Laws).

1. Kirchhoff’s Current Law (KCL): The sum of currents entering any node equals the sum of currents leaving that node

2. Kirchhoff’s Voltage Law (KVL): The sum of voltage drops around any closed loop is zero

Example 1.50 (DC Circuit Analysis). Consider the circuit with three loops and resistors \(R_1 = 2\Omega\), \(R_2 = 4\Omega\), \(R_3 = 1\Omega\), and voltage sources \(V_1 = 12V\), \(V_2 = 8V\). Let \(i_1\), \(i_2\), \(i_3\) be the loop currents (clockwise positive). Loop 1: \(2i_1 + 4(i_1 - i_2) = 12\) Loop 2: \(4(i_2 - i_1) + 1(i_2 - i_3) = 0\) Loop 3: \(1(i_3 - i_2) = -8\) Simplifying: \[\begin{align} 6i_1 - 4i_2 + 0i_3 &= 12\\ -4i_1 + 5i_2 - i_3 &= 0\\ 0i_1 - i_2 + i_3 &= -8 \end{align}\] Solving this system: \[\bgroup % \left[\begin{array}{rrr|r} 6 & -4 & 0 & 12\\ -4 & 5 & -1 & 0\\ 0 & -1 & 1 & -8 \end{array}\right]\egroup \sim \bgroup % \left[\begin{array}{rrr|r} 1 & 0 & 0 & 2\\ 0 & 1 & 0 & 0\\ 0 & 0 & 1 & -8 \end{array}\right]\egroup\] Therefore: \(i_1 = 2A\), \(i_2 = 0A\), \(i_3 = -8A\). The actual branch currents can be found from the loop currents: - Current through \(R_1\): \(i_1 = 2A\) - Current through \(R_2\): \(i_1 - i_2 = 2 - 0 = 2A\) - Current through \(R_3\): \(i_2 - i_3 = 0 - (-8) = 8A\)

1.5.3 Chemical Reactions

Chemical equation balancing is a classic application of homogeneous linear systems, based on the principle of conservation of mass.

Example 1.51 (Balancing Chemical Equations). Balance the combustion reaction: \[x_1 \text{C}_2\text{H}_6 + x_2 \text{O}_2 \rightarrow x_3 \text{CO}_2 + x_4 \text{H}_2\text{O}\]

Conservation of each element gives us:

Carbon (C): \(2x_1 = x_3\) Hydrogen (H): \(6x_1 = 2x_4\) Oxygen (O): \(2x_2 = 2x_3 + x_4\)

This homogeneous system can be written as: \[\begin{pmatrix} 2 & 0 & -1 & 0\\ 6 & 0 & 0 & -2\\ 0 & 2 & -2 & -1 \end{pmatrix}\begin{pmatrix} x_1 \\ x_2 \\ x_3 \\ x_4 \end{pmatrix} = \begin{pmatrix} 0 \\ 0 \\ 0 \end{pmatrix}\]

Row reducing: \[\bgroup % \left[\begin{array}{rrrr|r} 2 & 0 & -1 & 0 & 0\\ 6 & 0 & 0 & -2 & 0\\ 0 & 2 & -2 & -1 & 0 \end{array}\right]\egroup \sim \bgroup % \left[\begin{array}{rrrr|r} 1 & 0 & 0 & -\frac{1}{3} & 0\\ 0 & 1 & 0 & -\frac{7}{6} & 0\\ 0 & 0 & 1 & -\frac{2}{3} & 0 \end{array}\right]\egroup\]

Setting \(x_4 = t\): \[x_1 = \frac{t}{3}, \quad x_2 = \frac{7t}{6}, \quad x_3 = \frac{2t}{3}, \quad x_4 = t\]

For integer coefficients, we choose \(t = 6\): \[x_1 = 2, \quad x_2 = 7, \quad x_3 = 4, \quad x_4 = 6\]

The balanced equation is: \[2\text{C}_2\text{H}_6 + 7\text{O}_2 \rightarrow 4\text{CO}_2 + 6\text{H}_2\text{O}\]

Verification: - Carbon: \(2 \times 2 = 4\) on left, \(4 \times 1 = 4\) on right - Hydrogen: \(2 \times 6 = 12\) on left, \(6 \times 2 = 12\) on right - Oxygen: \(7 \times 2 = 14\) on left, \(4 \times 2 + 6 \times 1 = 14\) on right

Remark 52. These applications demonstrate that linear systems are not merely abstract mathematical objects, but fundamental tools for modeling real-world phenomena. The systematic methods we’ve developed - matrix representations, elementary row operations, and Gaussian elimination - provide powerful computational tools for solving practical problems across diverse fields.

This completes our comprehensive study of systems of linear equations. We have established the theoretical foundations, developed systematic solution methods, and explored practical applications. These concepts will serve as the cornerstone for our continued journey through linear algebra.

2 Matrix Algebra

Matrices are rectangular arrays of numbers that serve as one of the most powerful tools in mathematics and its applications. They provide a compact way to represent and manipulate linear transformations, systems of equations, and countless other mathematical objects. This chapter develops the algebraic structure of matrices, establishing the rules and properties that govern matrix operations.

2.1 Matrix Basics

2.1.1 Matrix Addition and Scalar Multiplication

Definition 2.1 (Matrix). An \(m \times n\) matrix is a rectangular array of real numbers arranged in \(m\) rows and \(n\) columns: \[\mathbf{A} = \begin{pmatrix} a_{11} & a_{12} & \cdots & a_{1n}\\ a_{21} & a_{22} & \cdots & a_{2n}\\ \vdots & \vdots & \ddots & \vdots\\ a_{m1} & a_{m2} & \cdots & a_{mn} \end{pmatrix}\] The number \(a_{ij}\) is called the \((i,j)\)-entry of \(\mathbf{A}\), located in the \(i\)-th row and \(j\)-th column.

Notation 54. We denote the set of all \(m \times n\) matrices with real entries as \(\mathbb{R}^{m \times n}\). When the dimensions are clear from context, we may write \(\mathbf{A} = (a_{ij})\) to indicate that \(\mathbf{A}\) has entries \(a_{ij}\).

Remark 55. The double subscript notation is crucial for matrix algebra. The first index always refers to the row, the second to the column. This convention, while arbitrary, is universally adopted and must be followed consistently.

Definition 2.4 (Matrix Equality). Two matrices \(\mathbf{A} = (a_{ij})\) and \(\mathbf{B} = (b_{ij})\) are equal if:

1. They have the same dimensions (\(m \times n\))

2. \(a_{ij} = b_{ij}\) for all \(i = 1, 2, \ldots, m\) and \(j = 1, 2, \ldots, n\)

Now we define the fundamental operations on matrices:

Definition 2.5 (Matrix Addition). If \(\mathbf{A} = (a_{ij})\) and \(\mathbf{B} = (b_{ij})\) are both \(m \times n\) matrices, then their sum \(\mathbf{A} + \mathbf{B}\) is the \(m \times n\) matrix whose \((i,j)\)-entry is \(a_{ij} + b_{ij}\): \[(\mathbf{A} + \mathbf{B})_{ij} = a_{ij} + b_{ij}\]

Remark 58. Matrix addition is only defined for matrices of the same size. We cannot add a \(2 \times 3\) matrix to a \(3 \times 2\) matrix, even though both have six entries. The structural arrangement matters fundamentally.

Definition 2.7 (Scalar Multiplication). If \(\mathbf{A} = (a_{ij})\) is an \(m \times n\) matrix and \(c\) is a real number (scalar), then the scalar multiple \(c\mathbf{A}\) is the \(m \times n\) matrix whose \((i,j)\)-entry is \(ca_{ij}\): \[(c\mathbf{A})_{ij} = ca_{ij}\]

Example 2.8 (Basic Matrix Operations). Let \(\mathbf{A} = \begin{pmatrix} 2 & -1 & 3 \\ 0 & 4 & -2 \end{pmatrix}\) and \(\mathbf{B} = \begin{pmatrix} 1 & 5 & -1 \\ 3 & -2 & 7 \end{pmatrix}\).

Matrix Addition: \[\mathbf{A} + \mathbf{B} = \begin{pmatrix} 2+1 & -1+5 & 3+(-1) \\ 0+3 & 4+(-2) & -2+7 \end{pmatrix} = \begin{pmatrix} 3 & 4 & 2 \\ 3 & 2 & 5 \end{pmatrix}\]

Scalar Multiplication: \[3\mathbf{A} = \begin{pmatrix} 3 \cdot 2 & 3 \cdot (-1) & 3 \cdot 3 \\ 3 \cdot 0 & 3 \cdot 4 & 3 \cdot (-2) \end{pmatrix} = \begin{pmatrix} 6 & -3 & 9 \\ 0 & 12 & -6 \end{pmatrix}\]

Combined Operations: \[2\mathbf{A} - 3\mathbf{B} = \begin{pmatrix} 4 & -2 & 6 \\ 0 & 8 & -4 \end{pmatrix} - \begin{pmatrix} 3 & 15 & -3 \\ 9 & -6 & 21 \end{pmatrix} = \begin{pmatrix} 1 & -17 & 9 \\ -9 & 14 & -25 \end{pmatrix}\]

The algebraic properties of matrix addition and scalar multiplication mirror those of real number arithmetic:

Theorem 2.9 (Properties of Matrix Addition and Scalar Multiplication). Let \(\mathbf{A}\), \(\mathbf{B}\), and \(\mathbf{C}\) be \(m \times n\) matrices, and let \(r\) and \(s\) be scalars. Then:

Addition Properties:

1. \(\mathbf{A} + \mathbf{B} = \mathbf{B} + \mathbf{A}\) (commutativity)

2. \((\mathbf{A} + \mathbf{B}) + \mathbf{C} = \mathbf{A} + (\mathbf{B} + \mathbf{C})\) (associativity)

3. \(\mathbf{A} + \mathbf{0} = \mathbf{A}\) (additive identity, where \(\mathbf{0}\) is the zero matrix)

4. \(\mathbf{A} + (-\mathbf{A}) = \mathbf{0}\) (additive inverse)

Scalar Multiplication Properties:

1. \(r(\mathbf{A} + \mathbf{B}) = r\mathbf{A} + r\mathbf{B}\) (distributivity over matrix addition)

2. \((r + s)\mathbf{A} = r\mathbf{A} + s\mathbf{A}\) (distributivity over scalar addition)

3. \(r(s\mathbf{A}) = (rs)\mathbf{A}\) (associativity of scalar multiplication)

4. \(1 \cdot \mathbf{A} = \mathbf{A}\) (scalar multiplicative identity)

Proof. We prove property (1) and (5); the others follow similarly.

Property (1): For any \(i,j\): \[(\mathbf{A} + \mathbf{B})_{ij} = a_{ij} + b_{ij} = b_{ij} + a_{ij} = (\mathbf{B} + \mathbf{A})_{ij}\] Since this holds for all entries, \(\mathbf{A} + \mathbf{B} = \mathbf{B} + \mathbf{A}\).

Property (5): For any \(i,j\): \[(r(\mathbf{A} + \mathbf{B}))_{ij} = r(\mathbf{A} + \mathbf{B})_{ij} = r(a_{ij} + b_{ij}) = ra_{ij} + rb_{ij} = (r\mathbf{A})_{ij} + (r\mathbf{B})_{ij} = (r\mathbf{A} + r\mathbf{B})_{ij}\] ◻

Definition 2.10 (Zero Matrix). The \(m \times n\) zero matrix, denoted \(\mathbf{0}\) or \(\mathbf{0}_{m \times n}\), is the matrix whose entries are all zero.

Definition 2.11 (Negative of a Matrix). The negative of a matrix \(\mathbf{A}\), denoted \(-\mathbf{A}\), is the matrix \((-1)\mathbf{A}\).

Key Idea

These properties establish that the set of all \(m \times n\) matrices, together with matrix addition and scalar multiplication, forms a vector space. This is our first encounter with this fundamental algebraic structure, which will be explored in detail in later chapters.

2.1.2 Transpose and Symmetry

Definition 2.12 (Transpose). The transpose of an \(m \times n\) matrix \(\mathbf{A} = (a_{ij})\) is the \(n \times m\) matrix \(\mathbf{A}^{\mathsf{T}}\) whose \((i,j)\)-entry is \(a_{ji}\): \[(\mathbf{A}^{\mathsf{T}})_{ij} = a_{ji}\] In other words, the rows of \(\mathbf{A}\) become the columns of \(\mathbf{A}^{\mathsf{T}}\), and vice versa.

Example 2.13 (Computing Transposes). \[\begin{align} \text{If } \mathbf{A} &= \begin{pmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{pmatrix}, \text{ then } \mathbf{A}^{\mathsf{T}} = \begin{pmatrix} 1 & 4 \\ 2 & 5 \\ 3 & 6 \end{pmatrix}\\[0.5em] \text{If } \mathbf{B} &= \begin{pmatrix} 2 \\ -1 \\ 3 \end{pmatrix}, \text{ then } \mathbf{B}^{\mathsf{T}} = \begin{pmatrix} 2 & -1 & 3 \end{pmatrix}\\[0.5em] \text{If } \mathbf{C} &= \begin{pmatrix} 5 \end{pmatrix}, \text{ then } \mathbf{C}^{\mathsf{T}} = \begin{pmatrix} 5 \end{pmatrix} \end{align}\]

Theorem 2.14 (Properties of Transpose). Let \(\mathbf{A}\) and \(\mathbf{B}\) be matrices of appropriate sizes, and let \(r\) be a scalar. Then:

1. \((\mathbf{A}^{\mathsf{T}})^{\mathsf{T}} = \mathbf{A}\)

2. \((\mathbf{A} + \mathbf{B})^{\mathsf{T}} = \mathbf{A}^{\mathsf{T}} + \mathbf{B}^{\mathsf{T}}\)

3. \((r\mathbf{A})^{\mathsf{T}} = r\mathbf{A}^{\mathsf{T}}\)

4. \((\mathbf{A}\mathbf{B})^{\mathsf{T}} = \mathbf{B}^{\mathsf{T}}\mathbf{A}^{\mathsf{T}}\) (when the product \(\mathbf{A}\mathbf{B}\) is defined)

Proof. Property (1): For any \(i,j\): \[((\mathbf{A}^{\mathsf{T}})^{\mathsf{T}})_{ij} = (\mathbf{A}^{\mathsf{T}})_{ji} = a_{ij}\] Thus \((\mathbf{A}^{\mathsf{T}})^{\mathsf{T}} = \mathbf{A}\).

Property (2): For any \(i,j\): \[((\mathbf{A} + \mathbf{B})^{\mathsf{T}})_{ij} = (\mathbf{A} + \mathbf{B})_{ji} = a_{ji} + b_{ji} = (\mathbf{A}^{\mathsf{T}})_{ij} + (\mathbf{B}^{\mathsf{T}})_{ij}\]

Property (3): For any \(i,j\): \[((r\mathbf{A})^{\mathsf{T}})_{ij} = (r\mathbf{A})_{ji} = ra_{ji} = r(\mathbf{A}^{\mathsf{T}})_{ij}\]

Property (4) will be proved after we define matrix multiplication. ◻

Definition 2.15 (Symmetric Matrix). A square matrix \(\mathbf{A}\) is symmetric if \(\mathbf{A} = \mathbf{A}^{\mathsf{T}}\), i.e., \(a_{ij} = a_{ji}\) for all \(i,j\).

Definition 2.16 (Skew-Symmetric Matrix). A square matrix \(\mathbf{A}\) is skew-symmetric (or antisymmetric) if \(\mathbf{A} = -\mathbf{A}^{\mathsf{T}}\), i.e., \(a_{ij} = -a_{ji}\) for all \(i,j\).

Example 2.17 (Symmetric and Skew-Symmetric Matrices). Symmetric matrix: \[\mathbf{A} = \begin{pmatrix} 1 & 2 & 3 \\ 2 & 5 & -1 \\ 3 & -1 & 0 \end{pmatrix}\] Note that \(a_{12} = a_{21} = 2\), \(a_{13} = a_{31} = 3\), and \(a_{23} = a_{32} = -1\).

Skew-symmetric matrix: \[\mathbf{B} = \begin{pmatrix} 0 & 2 & -3 \\ -2 & 0 & 1 \\ 3 & -1 & 0 \end{pmatrix}\] Note that the diagonal entries are all zero (since \(a_{ii} = -a_{ii}\) implies \(a_{ii} = 0\)), and off-diagonal entries satisfy \(a_{ij} = -a_{ji}\).

Theorem 2.18 (Decomposition into Symmetric and Skew-Symmetric Parts). Every square matrix \(\mathbf{A}\) can be uniquely written as the sum of a symmetric matrix and a skew-symmetric matrix: \[\mathbf{A} = \frac{1}{2}(\mathbf{A} + \mathbf{A}^{\mathsf{T}}) + \frac{1}{2}(\mathbf{A} - \mathbf{A}^{\mathsf{T}})\] where \(\frac{1}{2}(\mathbf{A} + \mathbf{A}^{\mathsf{T}})\) is symmetric and \(\frac{1}{2}(\mathbf{A} - \mathbf{A}^{\mathsf{T}})\) is skew-symmetric.

Proof. Let \(\mathbf{S} = \frac{1}{2}(\mathbf{A} + \mathbf{A}^{\mathsf{T}})\) and \(\mathbf{K} = \frac{1}{2}(\mathbf{A} - \mathbf{A}^{\mathsf{T}})\).

Verification that \(\mathbf{S}\) is symmetric: \[\mathbf{S}^{\mathsf{T}} = \left(\frac{1}{2}(\mathbf{A} + \mathbf{A}^{\mathsf{T}})\right)^{\mathsf{T}} = \frac{1}{2}(\mathbf{A}^{\mathsf{T}} + (\mathbf{A}^{\mathsf{T}})^{\mathsf{T}}) = \frac{1}{2}(\mathbf{A}^{\mathsf{T}} + \mathbf{A}) = \mathbf{S}\]

Verification that \(\mathbf{K}\) is skew-symmetric: \[\mathbf{K}^{\mathsf{T}} = \left(\frac{1}{2}(\mathbf{A} - \mathbf{A}^{\mathsf{T}})\right)^{\mathsf{T}} = \frac{1}{2}(\mathbf{A}^{\mathsf{T}} - (\mathbf{A}^{\mathsf{T}})^{\mathsf{T}}) = \frac{1}{2}(\mathbf{A}^{\mathsf{T}} - \mathbf{A}) = -\mathbf{K}\]

Verification of the decomposition: \[\mathbf{S} + \mathbf{K} = \frac{1}{2}(\mathbf{A} + \mathbf{A}^{\mathsf{T}}) + \frac{1}{2}(\mathbf{A} - \mathbf{A}^{\mathsf{T}}) = \frac{1}{2}(2\mathbf{A}) = \mathbf{A}\]

Uniqueness: If \(\mathbf{A} = \mathbf{S}_1 + \mathbf{K}_1 = \mathbf{S}_2 + \mathbf{K}_2\) where \(\mathbf{S}_1, \mathbf{S}_2\) are symmetric and \(\mathbf{K}_1, \mathbf{K}_2\) are skew-symmetric, then: \[\mathbf{S}_1 - \mathbf{S}_2 = \mathbf{K}_2 - \mathbf{K}_1\] The left side is symmetric, the right side is skew-symmetric. The only matrix that is both symmetric and skew-symmetric is the zero matrix, so \(\mathbf{S}_1 = \mathbf{S}_2\) and \(\mathbf{K}_1 = \mathbf{K}_2\). ◻

2.2 Matrix-Vector Multiplication

Before tackling general matrix multiplication, we first study the multiplication of a matrix by a vector, which provides essential intuition and forms the foundation for more complex operations.

2.2.1 Dot Product Interpretation

Definition 2.19 (Dot Product). The dot product (or inner product) of two vectors \(\boldsymbol{u} = \begin{pmatrix} u_1 \\ u_2 \\ \vdots \\ u_n \end{pmatrix}\) and \(\boldsymbol{v} = \begin{pmatrix} v_1 \\ v_2 \\ \vdots \\ v_n \end{pmatrix}\) in \(\mathbb{R}^n\) is: \[\boldsymbol{u} \cdot \boldsymbol{v} = u_1v_1 + u_2v_2 + \cdots + u_nv_n = \sum_{i=1}^n u_iv_i\]

Definition 2.20 (Matrix-Vector Product (Row Perspective)). If \(\mathbf{A}\) is an \(m \times n\) matrix and \(\boldsymbol{x}\) is a vector in \(\mathbb{R}^n\), then the product \(\mathbf{A}\boldsymbol{x}\) is the vector in \(\mathbb{R}^m\) whose \(i\)-th entry is the dot product of the \(i\)-th row of \(\mathbf{A}\) with \(\boldsymbol{x}\): \[(\mathbf{A}\boldsymbol{x})_i = \text{row}_i(\mathbf{A}) \cdot \boldsymbol{x} = \sum_{j=1}^n a_{ij}x_j\]

Example 2.21 (Matrix-Vector Multiplication). \[\begin{pmatrix} 2 & -1 & 3 \\ 0 & 4 & -2 \\ 1 & 1 & 1 \end{pmatrix} \begin{pmatrix} 1 \\ 2 \\ -1 \end{pmatrix} = \begin{pmatrix} 2(1) + (-1)(2) + 3(-1) \\ 0(1) + 4(2) + (-2)(-1) \\ 1(1) + 1(2) + 1(-1) \end{pmatrix} = \begin{pmatrix} -3 \\ 10 \\ 2 \end{pmatrix}\]

Detailed computation: \[\begin{align} \text{First entry:} \quad &(2, -1, 3) \cdot (1, 2, -1) = 2 - 2 - 3 = -3\\ \text{Second entry:} \quad &(0, 4, -2) \cdot (1, 2, -1) = 0 + 8 + 2 = 10\\ \text{Third entry:} \quad &(1, 1, 1) \cdot (1, 2, -1) = 1 + 2 - 1 = 2 \end{align}\]

Remark 74. The matrix-vector product \(\mathbf{A}\boldsymbol{x}\) is only defined when the number of columns of \(\mathbf{A}\) equals the number of entries in \(\boldsymbol{x}\). The result is a vector with the same number of entries as the number of rows of \(\mathbf{A}\).

There is an alternative, equally important way to view matrix-vector multiplication:

Theorem 2.23 (Column Interpretation of Matrix-Vector Product). If \(\mathbf{A} = [\boldsymbol{a}_1 \mid \boldsymbol{a}_2 \mid \cdots \mid \boldsymbol{a}_n]\) where \(\boldsymbol{a}_j\) denotes the \(j\)-th column of \(\mathbf{A}\), and \(\boldsymbol{x} = \begin{pmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{pmatrix}\), then: \[\mathbf{A}\boldsymbol{x} = x_1\boldsymbol{a}_1 + x_2\boldsymbol{a}_2 + \cdots + x_n\boldsymbol{a}_n\]

Proof. Let’s compute the \(i\)-th entry of both sides:

Left side: \((\mathbf{A}\boldsymbol{x})_i = \sum_{j=1}^n a_{ij}x_j\)

Right side: \((x_1\boldsymbol{a}_1 + x_2\boldsymbol{a}_2 + \cdots + x_n\boldsymbol{a}_n)_i = x_1(\boldsymbol{a}_1)_i + x_2(\boldsymbol{a}_2)_i + \cdots + x_n(\boldsymbol{a}_n)_i = x_1a_{i1} + x_2a_{i2} + \cdots + x_na_{in} = \sum_{j=1}^n a_{ij}x_j\)

Since the \(i\)-th entries are equal for all \(i\), the vectors are equal. ◻

Example 2.24 (Column Interpretation). Using the same example as before: \[\begin{pmatrix} 2 & -1 & 3 \\ 0 & 4 & -2 \\ 1 & 1 & 1 \end{pmatrix} \begin{pmatrix} 1 \\ 2 \\ -1 \end{pmatrix} = 1\begin{pmatrix} 2 \\ 0 \\ 1 \end{pmatrix} + 2\begin{pmatrix} -1 \\ 4 \\ 1 \end{pmatrix} + (-1)\begin{pmatrix} 3 \\ -2 \\ 1 \end{pmatrix}\]

\[= \begin{pmatrix} 2 \\ 0 \\ 1 \end{pmatrix} + \begin{pmatrix} -2 \\ 8 \\ 2 \end{pmatrix} + \begin{pmatrix} -3 \\ 2 \\ -1 \end{pmatrix} = \begin{pmatrix} -3 \\ 10 \\ 2 \end{pmatrix}\]

Key Idea

The column interpretation reveals that \(\mathbf{A}\boldsymbol{x}\) is a linear combination of the columns of \(\mathbf{A}\), with coefficients given by the entries of \(\boldsymbol{x}\). This perspective is fundamental to understanding the geometric meaning of matrix transformations.

2.2.2 Matrices as Linear Transformations

Matrix-vector multiplication can be viewed as a function that maps vectors to vectors. This viewpoint opens up rich geometric interpretations.

Definition 2.25 (Matrix Transformation). For an \(m \times n\) matrix \(\mathbf{A}\), the matrix transformation \(T_{\mathbf{A}}: \mathbb{R}^n \to \mathbb{R}^m\) is defined by: \[T_{\mathbf{A}}(\boldsymbol{x}) = \mathbf{A}\boldsymbol{x}\]

Theorem 2.26 (Linearity of Matrix Transformations). Every matrix transformation is linear, meaning:

1. \(T_{\mathbf{A}}(\boldsymbol{u} + \boldsymbol{v}) = T_{\mathbf{A}}(\boldsymbol{u}) + T_{\mathbf{A}}(\boldsymbol{v})\) for all \(\boldsymbol{u}, \boldsymbol{v} \in \mathbb{R}^n\)

2. \(T_{\mathbf{A}}(c\boldsymbol{u}) = cT_{\mathbf{A}}(\boldsymbol{u})\) for all \(\boldsymbol{u} \in \mathbb{R}^n\) and scalars \(c\)

Equivalently, \(T_{\mathbf{A}}(c\boldsymbol{u} + d\boldsymbol{v}) = cT_{\mathbf{A}}(\boldsymbol{u}) + dT_{\mathbf{A}}(\boldsymbol{v})\) for all vectors \(\boldsymbol{u}, \boldsymbol{v}\) and scalars \(c, d\).

Proof. Property 1: Using the \(i\)-th entry: \[\begin{align} (T_{\mathbf{A}}(\boldsymbol{u} + \boldsymbol{v}))_i &= (\mathbf{A}(\boldsymbol{u} + \boldsymbol{v}))_i = \sum_{j=1}^n a_{ij}(u_j + v_j)\\ &= \sum_{j=1}^n a_{ij}u_j + \sum_{j=1}^n a_{ij}v_j = (\mathbf{A}\boldsymbol{u})_i + (\mathbf{A}\boldsymbol{v})_i\\ &= (T_{\mathbf{A}}(\boldsymbol{u}) + T_{\mathbf{A}}(\boldsymbol{v}))_i \end{align}\]

Property 2: Similarly: \[(T_{\mathbf{A}}(c\boldsymbol{u}))_i = (\mathbf{A}(c\boldsymbol{u}))_i = \sum_{j=1}^n a_{ij}(cu_j) = c\sum_{j=1}^n a_{ij}u_j = c(\mathbf{A}\boldsymbol{u})_i = (cT_{\mathbf{A}}(\boldsymbol{u}))_i\] ◻

Let’s explore some important geometric transformations in \(\mathbb{R}^2\):

Example 2.27 (Rotation Transformation). The matrix \(\mathbf{R}_{\theta} = \begin{pmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{pmatrix}\) represents counterclockwise rotation by angle \(\theta\) about the origin.

Verification: Consider the standard basis vectors: \[\mathbf{R}_{\theta}\begin{pmatrix} 1 \\ 0 \end{pmatrix} = \begin{pmatrix} \cos\theta \\ \sin\theta \end{pmatrix}, \quad \mathbf{R}_{\theta}\begin{pmatrix} 0 \\ 1 \end{pmatrix} = \begin{pmatrix} -\sin\theta \\ \cos\theta \end{pmatrix}\]

These are indeed the results of rotating \((1,0)\) and \((0,1)\) by angle \(\theta\).

Specific example: Rotation by \(90°\) (\(\theta = \pi/2\)): \[\mathbf{R}_{\pi/2} = \begin{pmatrix} 0 & -1 \\ 1 & 0 \end{pmatrix}\]

Check: \(\begin{pmatrix} 0 & -1 \\ 1 & 0 \end{pmatrix}\begin{pmatrix} 3 \\ 2 \end{pmatrix} = \begin{pmatrix} -2 \\ 3 \end{pmatrix}\)

The point \((3,2)\) is rotated to \((-2,3)\), which is indeed a \(90°\) counterclockwise rotation.

Example 2.28 (Reflection Transformation). The matrix \(\mathbf{F} = \begin{pmatrix} 1 & 0 \\ 0 & -1 \end{pmatrix}\) represents reflection across the \(x\)-axis.

\[\mathbf{F}\begin{pmatrix} x \\ y \end{pmatrix} = \begin{pmatrix} x \\ -y \end{pmatrix}\]

The \(x\)-coordinate remains unchanged, while the \(y\)-coordinate is negated, which is exactly what reflection across the \(x\)-axis should do.

Example 2.29 (Scaling Transformation). The matrix \(\mathbf{S} = \begin{pmatrix} a & 0 \\ 0 & b \end{pmatrix}\) represents scaling by factor \(a\) in the \(x\)-direction and factor \(b\) in the \(y\)-direction.

\[\mathbf{S}\begin{pmatrix} x \\ y \end{pmatrix} = \begin{pmatrix} ax \\ by \end{pmatrix}\]

When \(a = b\), this is uniform scaling. When \(a \neq b\), the transformation stretches or compresses differently in each direction.

Example 2.30 (Shear Transformation). The matrix \(\mathbf{H} = \begin{pmatrix} 1 & k \\ 0 & 1 \end{pmatrix}\) represents horizontal shear with parameter \(k\).

\[\mathbf{H}\begin{pmatrix} x \\ y \end{pmatrix} = \begin{pmatrix} x + ky \\ y \end{pmatrix}\]

Points are shifted horizontally by an amount proportional to their \(y\)-coordinate, while the \(y\)-coordinate remains unchanged.

2.3 Matrix Multiplication

Building on matrix-vector multiplication, we now define the multiplication of two matrices. This operation, while initially seeming complex, has a natural motivation and profound applications.

Definition 2.31 (Matrix Multiplication). If \(\mathbf{A}\) is an \(m \times p\) matrix and \(\mathbf{B}\) is a \(p \times n\) matrix, then the product \(\mathbf{A}\mathbf{B}\) is the \(m \times n\) matrix whose \((i,j)\)-entry is: \[(\mathbf{A}\mathbf{B})_{ij} = \sum_{k=1}^p a_{ik}b_{kj}\] This is the dot product of the \(i\)-th row of \(\mathbf{A}\) with the \(j\)-th column of \(\mathbf{B}\).

Remark 84. The product \(\mathbf{A}\mathbf{B}\) is only defined when the number of columns of \(\mathbf{A}\) equals the number of rows of \(\mathbf{B}\). The resulting matrix has dimensions determined by the outer dimensions: if \(\mathbf{A}\) is \(m \times p\) and \(\mathbf{B}\) is \(p \times n\), then \(\mathbf{A}\mathbf{B}\) is \(m \times n\).

Example 2.33 (Matrix Multiplication Computation). \[\begin{pmatrix} 1 & 2 \\ 3 & 4 \\ 5 & 6 \end{pmatrix} \begin{pmatrix} 7 & 8 & 9 \\ 10 & 11 & 12 \end{pmatrix}\]

Step-by-step computation: \[\begin{align} (1,1)\text{-entry:} \quad &(1, 2) \cdot (7, 10) = 1 \cdot 7 + 2 \cdot 10 = 27\\ (1,2)\text{-entry:} \quad &(1, 2) \cdot (8, 11) = 1 \cdot 8 + 2 \cdot 11 = 30\\ (1,3)\text{-entry:} \quad &(1, 2) \cdot (9, 12) = 1 \cdot 9 + 2 \cdot 12 = 33\\ (2,1)\text{-entry:} \quad &(3, 4) \cdot (7, 10) = 3 \cdot 7 + 4 \cdot 10 = 61\\ (2,2)\text{-entry:} \quad &(3, 4) \cdot (8, 11) = 3 \cdot 8 + 4 \cdot 11 = 68\\ (2,3)\text{-entry:} \quad &(3, 4) \cdot (9, 12) = 3 \cdot 9 + 4 \cdot 12 = 75\\ (3,1)\text{-entry:} \quad &(5, 6) \cdot (7, 10) = 5 \cdot 7 + 6 \cdot 10 = 95\\ (3,2)\text{-entry:} \quad &(5, 6) \cdot (8, 11) = 5 \cdot 8 + 6 \cdot 11 = 106\\ (3,3)\text{-entry:} \quad &(5, 6) \cdot (9, 12) = 5 \cdot 9 + 6 \cdot 12 = 117 \end{align}\]

Result: \[\begin{pmatrix} 1 & 2 \\ 3 & 4 \\ 5 & 6 \end{pmatrix} \begin{pmatrix} 7 & 8 & 9 \\ 10 & 11 & 12 \end{pmatrix} = \begin{pmatrix} 27 & 30 & 33 \\ 61 & 68 & 75 \\ 95 & 106 & 117 \end{pmatrix}\]

2.3.1 Column Interpretation

Just as with matrix-vector multiplication, there’s an insightful column interpretation:

Theorem 2.34 (Column Interpretation of Matrix Multiplication). If \(\mathbf{A}\) is \(m \times p\) and \(\mathbf{B} = [\boldsymbol{b}_1 \mid \boldsymbol{b}_2 \mid \cdots \mid \boldsymbol{b}_n]\) where \(\boldsymbol{b}_j\) is the \(j\)-th column of \(\mathbf{B}\), then: \[\mathbf{A}\mathbf{B} = [\mathbf{A}\boldsymbol{b}_1 \mid \mathbf{A}\boldsymbol{b}_2 \mid \cdots \mid \mathbf{A}\boldsymbol{b}_n]\] In other words, the \(j\)-th column of \(\mathbf{A}\mathbf{B}\) is \(\mathbf{A}\) times the \(j\)-th column of \(\mathbf{B}\).

Proof. The \(j\)-th column of \(\mathbf{A}\mathbf{B}\) has entries \((\mathbf{A}\mathbf{B})_{ij}\) for \(i = 1, 2, \ldots, m\). By definition: \[(\mathbf{A}\mathbf{B})_{ij} = \sum_{k=1}^p a_{ik}b_{kj}\]

On the other hand, \((\mathbf{A}\boldsymbol{b}_j)_i = \sum_{k=1}^p a_{ik}(\boldsymbol{b}_j)_k = \sum_{k=1}^p a_{ik}b_{kj}\).

Since these expressions are equal, the \(j\)-th column of \(\mathbf{A}\mathbf{B}\) equals \(\mathbf{A}\boldsymbol{b}_j\). ◻

Example 2.35 (Using Column Interpretation). Let’s verify our previous example using the column interpretation: \[\mathbf{A} = \begin{pmatrix} 1 & 2 \\ 3 & 4 \\ 5 & 6 \end{pmatrix}, \quad \mathbf{B} = \begin{pmatrix} 7 & 8 & 9 \\ 10 & 11 & 12 \end{pmatrix}\]

First column of \(\mathbf{A}\mathbf{B}\): \[\mathbf{A}\begin{pmatrix} 7 \\ 10 \end{pmatrix} = \begin{pmatrix} 1 & 2 \\ 3 & 4 \\ 5 & 6 \end{pmatrix}\begin{pmatrix} 7 \\ 10 \end{pmatrix} = \begin{pmatrix} 27 \\ 61 \\ 95 \end{pmatrix}\]

Second column of \(\mathbf{A}\mathbf{B}\): \[\mathbf{A}\begin{pmatrix} 8 \\ 11 \end{pmatrix} = \begin{pmatrix} 1 & 2 \\ 3 & 4 \\ 5 & 6 \end{pmatrix}\begin{pmatrix} 8 \\ 11 \end{pmatrix} = \begin{pmatrix} 30 \\ 68 \\ 106 \end{pmatrix}\]

Third column of \(\mathbf{A}\mathbf{B}\): \[\mathbf{A}\begin{pmatrix} 9 \\ 12 \end{pmatrix} = \begin{pmatrix} 1 & 2 \\ 3 & 4 \\ 5 & 6 \end{pmatrix}\begin{pmatrix} 9 \\ 12 \end{pmatrix} = \begin{pmatrix} 33 \\ 75 \\ 117 \end{pmatrix}\]

This confirms our previous result.

2.3.2 Associativity Proof

One of the most important properties of matrix multiplication is associativity. Although the proof is computational, it’s essential for understanding why we can write expressions like \(\mathbf{A}\mathbf{B}\mathbf{C}\) without ambiguity.

Theorem 2.36 (Associativity of Matrix Multiplication). If \(\mathbf{A}\), \(\mathbf{B}\), and \(\mathbf{C}\) are matrices such that the products \(\mathbf{A}\mathbf{B}\) and \(\mathbf{B}\mathbf{C}\) are defined, then: \[(\mathbf{A}\mathbf{B})\mathbf{C} = \mathbf{A}(\mathbf{B}\mathbf{C})\]

Proof. Suppose \(\mathbf{A}\) is \(m \times p\), \(\mathbf{B}\) is \(p \times q\), and \(\mathbf{C}\) is \(q \times n\). We need to show that the \((i,j)\)-entries of both sides are equal.

Left side \(((\mathbf{A}\mathbf{B})\mathbf{C})_{ij}\): \[= \sum_{k=1}^q (\mathbf{A}\mathbf{B})_{ik} \cdot c_{kj} = \sum_{k=1}^q \left(\sum_{\ell=1}^p a_{i\ell}b_{\ell k}\right) c_{kj} = \sum_{k=1}^q \sum_{\ell=1}^p a_{i\ell}b_{\ell k}c_{kj}\]

Right side \((\mathbf{A}(\mathbf{B}\mathbf{C}))_{ij}\): \[= \sum_{\ell=1}^p a_{i\ell} \cdot (\mathbf{B}\mathbf{C})_{\ell j} = \sum_{\ell=1}^p a_{i\ell} \left(\sum_{k=1}^q b_{\ell k}c_{kj}\right) = \sum_{\ell=1}^p \sum_{k=1}^q a_{i\ell}b_{\ell k}c_{kj}\]

Since addition of real numbers is commutative and associative, we can rearrange the double sum: \[\sum_{k=1}^q \sum_{\ell=1}^p a_{i\ell}b_{\ell k}c_{kj} = \sum_{\ell=1}^p \sum_{k=1}^q a_{i\ell}b_{\ell k}c_{kj}\]

Therefore, the \((i,j)\)-entries are equal, proving associativity. ◻

Remark 89. This proof shows that associativity follows from the associativity and commutativity of real number addition and multiplication. The rearrangement of the double sum is the key insight.

Theorem 2.38 (Properties of Matrix Multiplication). Let \(\mathbf{A}\), \(\mathbf{B}\), and \(\mathbf{C}\) be matrices of appropriate sizes, and let \(r\) be a scalar. Then:

1. \(\mathbf{A}(\mathbf{B}\mathbf{C}) = (\mathbf{A}\mathbf{B})\mathbf{C}\) (associativity)

2. \(\mathbf{A}(\mathbf{B} + \mathbf{C}) = \mathbf{A}\mathbf{B} + \mathbf{A}\mathbf{C}\) (left distributivity)

3. \((\mathbf{B} + \mathbf{C})\mathbf{A} = \mathbf{B}\mathbf{A} + \mathbf{C}\mathbf{A}\) (right distributivity)

4. \((r\mathbf{A})\mathbf{B} = r(\mathbf{A}\mathbf{B}) = \mathbf{A}(r\mathbf{B})\) (scalar associativity)

5. \(\mathbf{A}\mathbf{0} = \mathbf{0}\) and \(\mathbf{0}\mathbf{A} = \mathbf{0}\) (multiplication by zero)

Example 2.39 (Matrix Multiplication is Not Commutative). In general, \(\mathbf{A}\mathbf{B} \neq \mathbf{B}\mathbf{A}\). Consider: \[\mathbf{A} = \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix}, \quad \mathbf{B} = \begin{pmatrix} 0 & 1 \\ 0 & 0 \end{pmatrix}\]

\[\mathbf{A}\mathbf{B} = \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix}\begin{pmatrix} 0 & 1 \\ 0 & 0 \end{pmatrix} = \begin{pmatrix} 0 & 1 \\ 0 & 3 \end{pmatrix}\]

\[\mathbf{B}\mathbf{A} = \begin{pmatrix} 0 & 1 \\ 0 & 0 \end{pmatrix}\begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix} = \begin{pmatrix} 3 & 4 \\ 0 & 0 \end{pmatrix}\]

Clearly \(\mathbf{A}\mathbf{B} \neq \mathbf{B}\mathbf{A}\).

2.3.3 Directed Graphs and Composition of Maps

Matrix multiplication has elegant interpretations in terms of directed graphs and function composition.

Definition 2.40 (Adjacency Matrix). For a directed graph with \(n\) vertices, the adjacency matrix \(\mathbf{A}\) is the \(n \times n\) matrix where \(a_{ij} = 1\) if there is an edge from vertex \(i\) to vertex \(j\), and \(a_{ij} = 0\) otherwise.

Theorem 2.41 (Powers of Adjacency Matrix). If \(\mathbf{A}\) is the adjacency matrix of a directed graph, then \((\mathbf{A}^k)_{ij}\) equals the number of paths of length \(k\) from vertex \(i\) to vertex \(j\).

Proof. We use induction on \(k\).

Base case (\(k = 1\)): \((\mathbf{A}^1)_{ij} = a_{ij}\) is 1 if there’s a direct edge from \(i\) to \(j\) (a path of length 1), and 0 otherwise.

Inductive step: Assume the statement holds for paths of length \(k\). For paths of length \(k+1\): \[(\mathbf{A}^{k+1})_{ij} = (\mathbf{A}^k \mathbf{A})_{ij} = \sum_{\ell=1}^n (\mathbf{A}^k)_{i\ell} \cdot a_{\ell j}\]

By the inductive hypothesis, \((\mathbf{A}^k)_{i\ell}\) counts paths of length \(k\) from \(i\) to \(\ell\). The term \(a_{\ell j}\) is 1 if there’s an edge from \(\ell\) to \(j\), and 0 otherwise. Therefore, \((\mathbf{A}^k)_{i\ell} \cdot a_{\ell j}\) counts paths of length \(k+1\) from \(i\) to \(j\) that pass through \(\ell\). Summing over all possible intermediate vertices \(\ell\) gives the total number of paths of length \(k+1\) from \(i\) to \(j\). ◻

Example 2.42 (Graph Paths). Consider the directed graph with adjacency matrix: \[\mathbf{A} = \begin{pmatrix} 0 & 1 & 1 \\ 1 & 0 & 1 \\ 0 & 1 & 0 \end{pmatrix}\]

This represents a graph where: - Vertex 1 connects to vertices 2 and 3 - Vertex 2 connects to vertices 1 and 3 - Vertex 3 connects to vertex 2

Let’s compute \(\mathbf{A}^2\): \[\mathbf{A}^2 = \begin{pmatrix} 0 & 1 & 1 \\ 1 & 0 & 1 \\ 0 & 1 & 0 \end{pmatrix}\begin{pmatrix} 0 & 1 & 1 \\ 1 & 0 & 1 \\ 0 & 1 & 0 \end{pmatrix} = \begin{pmatrix} 1 & 1 & 1 \\ 0 & 2 & 1 \\ 1 & 0 & 1 \end{pmatrix}\]

Interpretation: - \((\mathbf{A}^2)_{22} = 2\) means there are 2 paths of length 2 from vertex 2 to itself: \(2 \to 1 \to 2\) and \(2 \to 3 \to 2\) - \((\mathbf{A}^2)_{13} = 1\) means there is 1 path of length 2 from vertex 1 to vertex 3: \(1 \to 2 \to 3\)

Theorem 2.43 (Composition of Linear Transformations). If \(T: \mathbb{R}^n \to \mathbb{R}^m\) and \(S: \mathbb{R}^m \to \mathbb{R}^p\) are linear transformations with matrix representations \(\mathbf{A}\) and \(\mathbf{B}\) respectively, then the composition \(S \circ T: \mathbb{R}^n \to \mathbb{R}^p\) has matrix representation \(\mathbf{B}\mathbf{A}\).

Proof. For any \(\boldsymbol{x} \in \mathbb{R}^n\): \[(S \circ T)(\boldsymbol{x}) = S(T(\boldsymbol{x})) = S(\mathbf{A}\boldsymbol{x}) = \mathbf{B}(\mathbf{A}\boldsymbol{x}) = (\mathbf{B}\mathbf{A})\boldsymbol{x}\]

Since this holds for all \(\boldsymbol{x}\), the matrix representation of \(S \circ T\) is \(\mathbf{B}\mathbf{A}\). ◻

Remark 96. Note the order: the composition \(S \circ T\) (apply \(T\) first, then \(S\)) corresponds to the matrix product \(\mathbf{B}\mathbf{A}\) (not \(\mathbf{A}\mathbf{B}\)). This is because in function notation, we read from right to left: \((S \circ T)(\boldsymbol{x}) = S(T(\boldsymbol{x}))\).

2.4 Matrix Inverses

The concept of matrix inverse generalizes the notion of reciprocal from real numbers to matrices. However, not all matrices have inverses, making the study of invertibility conditions crucial.

Definition 2.45 (Identity Matrix). The \(n \times n\) identity matrix \(\mathbf{I}_n\) (or simply \(\mathbf{I}\) when the size is clear) is the matrix with 1’s on the main diagonal and 0’s elsewhere: \[\mathbf{I}_n = \begin{pmatrix} 1 & 0 & 0 & \cdots & 0\\ 0 & 1 & 0 & \cdots & 0\\ 0 & 0 & 1 & \cdots & 0\\ \vdots & \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & 0 & \cdots & 1 \end{pmatrix}\]

Theorem 2.46 (Properties of Identity Matrix). For any \(m \times n\) matrix \(\mathbf{A}\):

1. \(\mathbf{A}\mathbf{I}_n = \mathbf{A}\) (right identity)

2. \(\mathbf{I}_m\mathbf{A} = \mathbf{A}\) (left identity)

Definition 2.47 (Matrix Inverse). An \(n \times n\) matrix \(\mathbf{A}\) is invertible (or nonsingular) if there exists an \(n \times n\) matrix \(\mathbf{B}\) such that: \[\mathbf{A}\mathbf{B} = \mathbf{I}_n \quad \text{and} \quad \mathbf{B}\mathbf{A} = \mathbf{I}_n\] In this case, \(\mathbf{B}\) is called the inverse of \(\mathbf{A}\) and is denoted \(\mathbf{A}^{-1}\).

Theorem 2.48 (Uniqueness of Inverse). If a matrix \(\mathbf{A}\) is invertible, then its inverse is unique.

Proof. Suppose \(\mathbf{B}\) and \(\mathbf{C}\) are both inverses of \(\mathbf{A}\). Then: \[\mathbf{B} = \mathbf{B}\mathbf{I} = \mathbf{B}(\mathbf{A}\mathbf{C}) = (\mathbf{B}\mathbf{A})\mathbf{C} = \mathbf{I}\mathbf{C} = \mathbf{C}\] ◻

Theorem 2.49 (Properties of Matrix Inverse). Let \(\mathbf{A}\) and \(\mathbf{B}\) be invertible \(n \times n\) matrices. Then:

1. \((\mathbf{A}^{-1})^{-1} = \mathbf{A}\)

2. \((\mathbf{A}\mathbf{B})^{-1} = \mathbf{B}^{-1}\mathbf{A}^{-1}\) (reverse order!)

3. \((\mathbf{A}^{\mathsf{T}})^{-1} = (\mathbf{A}^{-1})^{\mathsf{T}}\)

4. If \(\mathbf{A}\) is invertible, then \(\mathbf{A}^{-1}\) is also invertible

Proof. Property 1: Since \(\mathbf{A}\mathbf{A}^{-1} = \mathbf{I}\) and \(\mathbf{A}^{-1}\mathbf{A} = \mathbf{I}\), the matrix \(\mathbf{A}\) satisfies the definition of being the inverse of \(\mathbf{A}^{-1}\).

Property 2: We need to verify that \(\mathbf{B}^{-1}\mathbf{A}^{-1}\) is the inverse of \(\mathbf{A}\mathbf{B}\): \[(\mathbf{A}\mathbf{B})(\mathbf{B}^{-1}\mathbf{A}^{-1}) = \mathbf{A}(\mathbf{B}\mathbf{B}^{-1})\mathbf{A}^{-1} = \mathbf{A}\mathbf{I}\mathbf{A}^{-1} = \mathbf{A}\mathbf{A}^{-1} = \mathbf{I}\] \[(\mathbf{B}^{-1}\mathbf{A}^{-1})(\mathbf{A}\mathbf{B}) = \mathbf{B}^{-1}(\mathbf{A}^{-1}\mathbf{A})\mathbf{B} = \mathbf{B}^{-1}\mathbf{I}\mathbf{B} = \mathbf{B}^{-1}\mathbf{B} = \mathbf{I}\]

Property 3: We show that \((\mathbf{A}^{-1})^{\mathsf{T}}\) is the inverse of \(\mathbf{A}^{\mathsf{T}}\): \[\mathbf{A}^{\mathsf{T}}(\mathbf{A}^{-1})^{\mathsf{T}} = (\mathbf{A}^{-1}\mathbf{A})^{\mathsf{T}} = \mathbf{I}^{\mathsf{T}} = \mathbf{I}\] \[(\mathbf{A}^{-1})^{\mathsf{T}}\mathbf{A}^{\mathsf{T}} = (\mathbf{A}\mathbf{A}^{-1})^{\mathsf{T}} = \mathbf{I}^{\mathsf{T}} = \mathbf{I}\] ◻

2.4.1 Invertibility Conditions

Theorem 2.50 (Characterizations of Invertibility). For an \(n \times n\) matrix \(\mathbf{A}\), the following statements are equivalent:

1. \(\mathbf{A}\) is invertible

2. The system \(\mathbf{A}\boldsymbol{x} = \boldsymbol{b}\) has a unique solution for every \(\boldsymbol{b} \in \mathbb{R}^n\)

3. The system \(\mathbf{A}\boldsymbol{x} = \boldsymbol{0}\) has only the trivial solution

4. \(\mathbf{A}\) is row equivalent to \(\mathbf{I}_n\)

5. \(\operatorname{rank}(\mathbf{A}) = n\)

6. The columns of \(\mathbf{A}\) are linearly independent

7. The columns of \(\mathbf{A}\) span \(\mathbb{R}^n\)

8. \(\det(\mathbf{A}) \neq 0\) (determinant, to be defined later)

Remark 103. This theorem, sometimes called the "Invertible Matrix Theorem," is one of the most important results in linear algebra. It shows that invertibility is connected to fundamental concepts like linear independence, spanning, rank, and determinants.

Example 2.52 (Computing Matrix Inverse). Find the inverse of \(\mathbf{A} = \begin{pmatrix} 2 & 1 \\ 3 & 2 \end{pmatrix}\).

Step 1: Form the augmented matrix: \[\bgroup % \left[\begin{array}{cc|cc} 2 & 1 & 1 & 0\\ 3 & 2 & 0 & 1 \end{array}\right]\egroup\]

Step 2: Row reduce to get identity on the left: \[\frac{1}{2}R_1: \bgroup % \left[\begin{array}{cc|cc} 1 & 1/2 & 1/2 & 0\\ 3 & 2 & 0 & 1 \end{array}\right]\egroup\]

\[R_2 - 3R_1: \bgroup % \left[\begin{array}{cc|cc} 1 & 1/2 & 1/2 & 0\\ 0 & 1/2 & -3/2 & 1 \end{array}\right]\egroup\]

\[2R_2: \bgroup % \left[\begin{array}{cc|cc} 1 & 1/2 & 1/2 & 0\\ 0 & 1 & -3 & 2 \end{array}\right]\egroup\]

\[R_1 - \frac{1}{2}R_2: \bgroup % \left[\begin{array}{cc|cc} 1 & 0 & 2 & -1\\ 0 & 1 & -3 & 2 \end{array}\right]\egroup\]

Result: \(\mathbf{A}^{-1} = \begin{pmatrix} 2 & -1 \\ -3 & 2 \end{pmatrix}\)

Verification: \[\begin{pmatrix} 2 & 1 \\ 3 & 2 \end{pmatrix}\begin{pmatrix} 2 & -1 \\ -3 & 2 \end{pmatrix} = \begin{pmatrix} 4-3 & -2+2 \\ 6-6 & -3+4 \end{pmatrix} = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix} \quad \checkmark\]

2.4.2 Adjugate and Cramer’s Rule

For \(2 \times 2\) matrices, there’s a direct formula for the inverse:

Theorem 2.53 (Inverse of \(2 \times 2\) Matrix). For a \(2 \times 2\) matrix \(\mathbf{A} = \begin{pmatrix} a & b \\ c & d \end{pmatrix}\), if \(ad - bc \neq 0\), then: \[\mathbf{A}^{-1} = \frac{1}{ad - bc}\begin{pmatrix} d & -b \\ -c & a \end{pmatrix}\] The quantity \(ad - bc\) is called the determinant of \(\mathbf{A}\).

Proof. We verify that the given matrix is indeed the inverse: \[\begin{pmatrix} a & b \\ c & d \end{pmatrix} \cdot \frac{1}{ad-bc}\begin{pmatrix} d & -b \\ -c & a \end{pmatrix} = \frac{1}{ad-bc}\begin{pmatrix} ad-bc & -ab+ba \\ cd-dc & -cb+da \end{pmatrix} = \frac{1}{ad-bc}\begin{pmatrix} ad-bc & 0 \\ 0 & ad-bc \end{pmatrix} = \mathbf{I}\] ◻

Example 2.54 (Using the \(2 \times 2\) Formula). Find the inverse of \(\mathbf{A} = \begin{pmatrix} 3 & 1 \\ 5 & 2 \end{pmatrix}\).

The determinant is \(ad - bc = 3 \cdot 2 - 1 \cdot 5 = 6 - 5 = 1\).

Therefore: \[\mathbf{A}^{-1} = \frac{1}{1}\begin{pmatrix} 2 & -1 \\ -5 & 3 \end{pmatrix} = \begin{pmatrix} 2 & -1 \\ -5 & 3 \end{pmatrix}\]

For larger matrices, the adjugate matrix (or classical adjoint) provides a theoretical formula for the inverse, though it’s computationally inefficient for large matrices.

Definition 2.55 (Cofactor and Adjugate). For an \(n \times n\) matrix \(\mathbf{A}\), the \((i,j)\)-cofactor is \(C_{ij} = (-1)^{i+j}M_{ij}\), where \(M_{ij}\) is the determinant of the \((n-1) \times (n-1)\) matrix obtained by deleting row \(i\) and column \(j\) from \(\mathbf{A}\). The adjugate matrix is: \[\operatorname{adj}(\mathbf{A}) = \begin{pmatrix} C_{11} & C_{21} & \cdots & C_{n1} \\ C_{12} & C_{22} & \cdots & C_{n2} \\ \vdots & \vdots & \ddots & \vdots \\ C_{1n} & C_{2n} & \cdots & C_{nn} \end{pmatrix}\] Note that the cofactors are transposed: the \((i,j)\)-entry of \(\operatorname{adj}(\mathbf{A})\) is \(C_{ji}\).

Theorem 2.56 (Inverse via Adjugate). If \(\mathbf{A}\) is an invertible \(n \times n\) matrix, then: \[\mathbf{A}^{-1} = \frac{1}{\det(\mathbf{A})}\operatorname{adj}(\mathbf{A})\]

Theorem 2.57 (Cramer’s Rule). Consider the system \(\mathbf{A}\boldsymbol{x} = \boldsymbol{b}\) where \(\mathbf{A}\) is an invertible \(n \times n\) matrix. Then: \[x_i = \frac{\det(\mathbf{A}_i)}{\det(\mathbf{A})}\] where \(\mathbf{A}_i\) is the matrix obtained by replacing the \(i\)-th column of \(\mathbf{A}\) with \(\boldsymbol{b}\).

Example 2.58 (Cramer’s Rule Application). Solve the system: \[\begin{align} 2x_1 + x_2 &= 5\\ 3x_1 + 2x_2 &= 8 \end{align}\]

We have \(\mathbf{A} = \begin{pmatrix} 2 & 1 \\ 3 & 2 \end{pmatrix}\) and \(\boldsymbol{b} = \begin{pmatrix} 5 \\ 8 \end{pmatrix}\).

\(\det(\mathbf{A}) = 2 \cdot 2 - 1 \cdot 3 = 4 - 3 = 1\)

For \(x_1\): \(\mathbf{A}_1 = \begin{pmatrix} 5 & 1 \\ 8 & 2 \end{pmatrix}\), so \(\det(\mathbf{A}_1) = 5 \cdot 2 - 1 \cdot 8 = 10 - 8 = 2\)

For \(x_2\): \(\mathbf{A}_2 = \begin{pmatrix} 2 & 5 \\ 3 & 8 \end{pmatrix}\), so \(\det(\mathbf{A}_2) = 2 \cdot 8 - 5 \cdot 3 = 16 - 15 = 1\)

Therefore: \(x_1 = \frac{2}{1} = 2\) and \(x_2 = \frac{1}{1} = 1\).

Verification: \(2(2) + 1 = 5\) and \(3(2) + 2(1) = 8\)

2.5 Elementary Matrices

Elementary matrices provide a powerful connection between row operations and matrix multiplication, revealing the underlying algebraic structure of Gaussian elimination.

2.5.1 Row Operations as Matrices

Definition 2.59 (Elementary Matrix). An elementary matrix is a matrix obtained by performing exactly one elementary row operation on an identity matrix.

There are three types of elementary matrices, corresponding to the three types of elementary row operations:

Definition 2.60 (Types of Elementary Matrices).

1. Type I (Row Interchange): \(\mathbf{E}_{ij}\) swaps rows \(i\) and \(j\) of the identity matrix

2. Type II (Row Scaling): \(\mathbf{E}_i(k)\) multiplies row \(i\) of the identity matrix by nonzero scalar \(k\)

3. Type III (Row Addition): \(\mathbf{E}_{ij}(k)\) adds \(k\) times row \(j\) to row \(i\) of the identity matrix

Example 2.61 (Elementary Matrices in \(3 \times 3\)). Starting with \(\mathbf{I}_3 = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{pmatrix}\):

Type I: \(\mathbf{E}_{12} = \begin{pmatrix} 0 & 1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 1 \end{pmatrix}\) (swap rows 1 and 2)

Type II: \(\mathbf{E}_2(3) = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 3 & 0 \\ 0 & 0 & 1 \end{pmatrix}\) (multiply row 2 by 3)

Type III: \(\mathbf{E}_{13}(-2) = \begin{pmatrix} 1 & 0 & -2 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{pmatrix}\) (add -2 times row 3 to row 1)

The fundamental property of elementary matrices is:

Theorem 2.62 (Elementary Matrix Action). If \(\mathbf{E}\) is an elementary matrix and \(\mathbf{A}\) is any matrix with the same number of rows as \(\mathbf{E}\), then \(\mathbf{E}\mathbf{A}\) is the matrix obtained by performing the same elementary row operation on \(\mathbf{A}\) that was used to create \(\mathbf{E}\) from the identity matrix.

Proof. We verify this for Type III operations; the others are similar.

Let \(\mathbf{E} = \mathbf{E}_{ij}(k)\) be the elementary matrix that adds \(k\) times row \(j\) to row \(i\). The \(\ell\)-th row of \(\mathbf{E}\mathbf{A}\) is: \[(\mathbf{E}\mathbf{A})_{\ell \cdot} = \sum_{m=1}^n e_{\ell m} \cdot \text{row}_m(\mathbf{A})\]

For \(\ell \neq i\): \(e_{\ell m} = \delta_{\ell m}\) (Kronecker delta), so \((\mathbf{E}\mathbf{A})_{\ell \cdot} = \text{row}_\ell(\mathbf{A})\).

For \(\ell = i\): \(e_{im} = \delta_{im} + k\delta_{jm}\), so: \[(\mathbf{E}\mathbf{A})_{i \cdot} = \text{row}_i(\mathbf{A}) + k \cdot \text{row}_j(\mathbf{A})\]

This is exactly the result of adding \(k\) times row \(j\) to row \(i\) in \(\mathbf{A}\). ◻

Example 2.63 (Elementary Matrix Multiplication). Let \(\mathbf{A} = \begin{pmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \\ 7 & 8 & 9 \end{pmatrix}\) and \(\mathbf{E} = \mathbf{E}_{21}(-4) = \begin{pmatrix} 1 & 0 & 0 \\ -4 & 1 & 0 \\ 0 & 0 & 1 \end{pmatrix}\).

\[\mathbf{E}\mathbf{A} = \begin{pmatrix} 1 & 0 & 0 \\ -4 & 1 & 0 \\ 0 & 0 & 1 \end{pmatrix}\begin{pmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \\ 7 & 8 & 9 \end{pmatrix} = \begin{pmatrix} 1 & 2 & 3 \\ 0 & -3 & -6 \\ 7 & 8 & 9 \end{pmatrix}\]

This is indeed the result of the row operation \(R_2 - 4R_1\) applied to \(\mathbf{A}\).

Theorem 2.64 (Invertibility of Elementary Matrices). Every elementary matrix is invertible, and its inverse is also an elementary matrix of the same type:

1. \((\mathbf{E}_{ij})^{-1} = \mathbf{E}_{ij}\) (row interchange is self-inverse)

2. \((\mathbf{E}_i(k))^{-1} = \mathbf{E}_i(1/k)\) for \(k \neq 0\)

3. \((\mathbf{E}_{ij}(k))^{-1} = \mathbf{E}_{ij}(-k)\)

Corollary 2.65 (Matrix Invertibility via Elementary Matrices). A matrix \(\mathbf{A}\) is invertible if and only if it is a product of elementary matrices.

Theorem 2.66 (Fundamental Theorem of Invertible Matrices). For an \(n \times n\) matrix \(\mathbf{A}\), the following are equivalent:

1. \(\mathbf{A}\) is invertible

2. \(\mathbf{A}\) is row equivalent to \(\mathbf{I}_n\)

3. \(\mathbf{A}\) is a product of elementary matrices

Moreover, if \(\mathbf{A}\) is invertible and \(\mathbf{A} = \mathbf{E}_1\mathbf{E}_2 \cdots \mathbf{E}_k\) where the \(\mathbf{E}_i\) are elementary matrices, then: \[\mathbf{A}^{-1} = \mathbf{E}_k^{-1} \cdots \mathbf{E}_2^{-1}\mathbf{E}_1^{-1}\]

2.5.2 Smith Normal Form

The Smith Normal Form is a canonical form for matrices over the integers, generalizing the reduced row-echelon form.

Definition 2.67 (Smith Normal Form). A matrix is in Smith Normal Form if:

1. It is diagonal (all off-diagonal entries are zero)

2. The diagonal entries \(d_1, d_2, \ldots, d_r\) are positive integers

3. \(d_i\) divides \(d_{i+1}\) for \(i = 1, 2, \ldots, r-1\)

4. All other entries are zero

Theorem 2.68 (Existence of Smith Normal Form). Every integer matrix can be transformed to Smith Normal Form using elementary row and column operations over the integers (operations with determinant \(\pm 1\)).

Example 2.69 (Computing Smith Normal Form). Transform \(\mathbf{A} = \begin{pmatrix} 2 & 4 & 6 \\ 4 & 8 & 12 \\ 6 & 12 & 18 \end{pmatrix}\) to Smith Normal Form.

Step 1: Use the first entry as pivot: \(R_2 - 2R_1\): \(\begin{pmatrix} 2 & 4 & 6 \\ 0 & 0 & 0 \\ 6 & 12 & 18 \end{pmatrix}\)

\(R_3 - 3R_1\): \(\begin{pmatrix} 2 & 4 & 6 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{pmatrix}\)

Step 2: Use column operations: \(C_2 - 2C_1\): \(\begin{pmatrix} 2 & 0 & 6 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{pmatrix}\)

\(C_3 - 3C_1\): \(\begin{pmatrix} 2 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{pmatrix}\)

The Smith Normal Form is \(\begin{pmatrix} 2 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{pmatrix}\).

2.6 Linear Transformations

Linear transformations provide the geometric interpretation of matrix operations, bridging the gap between algebraic computations and geometric intuition.

2.6.1 Standard Matrices

Definition 2.70 (Linear Transformation). A function \(T: \mathbb{R}^n \to \mathbb{R}^m\) is a linear transformation if:

1. \(T(\boldsymbol{u} + \boldsymbol{v}) = T(\boldsymbol{u}) + T(\boldsymbol{v})\) for all \(\boldsymbol{u}, \boldsymbol{v} \in \mathbb{R}^n\)

2. \(T(c\boldsymbol{u}) = cT(\boldsymbol{u})\) for all \(\boldsymbol{u} \in \mathbb{R}^n\) and scalars \(c\)

Equivalently: \(T(c\boldsymbol{u} + d\boldsymbol{v}) = cT(\boldsymbol{u}) + dT(\boldsymbol{v})\) for all vectors and scalars.

Theorem 2.71 (Standard Matrix Representation). Every linear transformation \(T: \mathbb{R}^n \to \mathbb{R}^m\) can be represented as matrix multiplication. Specifically, there exists a unique \(m \times n\) matrix \(\mathbf{A}\) such that \(T(\boldsymbol{x}) = \mathbf{A}\boldsymbol{x}\) for all \(\boldsymbol{x} \in \mathbb{R}^n\).

The matrix \(\mathbf{A}\) is called the standard matrix of \(T\) and is given by: \[\mathbf{A} = [T(\boldsymbol{e}_1) \mid T(\boldsymbol{e}_2) \mid \cdots \mid T(\boldsymbol{e}_n)]\] where \(\boldsymbol{e}_1, \boldsymbol{e}_2, \ldots, \boldsymbol{e}_n\) are the standard basis vectors in \(\mathbb{R}^n\).

Proof. Existence: For any \(\boldsymbol{x} = \begin{pmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{pmatrix} = x_1\boldsymbol{e}_1 + x_2\boldsymbol{e}_2 + \cdots + x_n\boldsymbol{e}_n\), linearity gives: \[T(\boldsymbol{x}) = T(x_1\boldsymbol{e}_1 + x_2\boldsymbol{e}_2 + \cdots + x_n\boldsymbol{e}_n) = x_1T(\boldsymbol{e}_1) + x_2T(\boldsymbol{e}_2) + \cdots + x_nT(\boldsymbol{e}_n)\]

If we define \(\mathbf{A} = [T(\boldsymbol{e}_1) \mid T(\boldsymbol{e}_2) \mid \cdots \mid T(\boldsymbol{e}_n)]\), then by the column interpretation of matrix-vector multiplication: \[\mathbf{A}\boldsymbol{x} = x_1T(\boldsymbol{e}_1) + x_2T(\boldsymbol{e}_2) + \cdots + x_nT(\boldsymbol{e}_n) = T(\boldsymbol{x})\]

Uniqueness: If \(T(\boldsymbol{x}) = \mathbf{A}\boldsymbol{x} = \mathbf{B}\boldsymbol{x}\) for all \(\boldsymbol{x}\), then \((\mathbf{A} - \mathbf{B})\boldsymbol{x} = \boldsymbol{0}\) for all \(\boldsymbol{x}\). In particular, for each standard basis vector \(\boldsymbol{e}_j\): \((\mathbf{A} - \mathbf{B})\boldsymbol{e}_j = \boldsymbol{0}\). But \((\mathbf{A} - \mathbf{B})\boldsymbol{e}_j\) is just the \(j\)-th column of \(\mathbf{A} - \mathbf{B}\), so all columns of \(\mathbf{A} - \mathbf{B}\) are zero, meaning \(\mathbf{A} = \mathbf{B}\). ◻

Example 2.72 (Finding Standard Matrix). Find the standard matrix of the linear transformation \(T: \mathbb{R}^3 \to \mathbb{R}^2\) defined by: \[T\begin{pmatrix} x_1 \\ x_2 \\ x_3 \end{pmatrix} = \begin{pmatrix} 2x_1 - x_2 + 3x_3 \\ x_1 + 4x_2 - x_3 \end{pmatrix}\]

We compute \(T\) applied to each standard basis vector: \[T(\boldsymbol{e}_1) = T\begin{pmatrix} 1 \\ 0 \\ 0 \end{pmatrix} = \begin{pmatrix} 2(1) - 0 + 3(0) \\ 1 + 4(0) - 0 \end{pmatrix} = \begin{pmatrix} 2 \\ 1 \end{pmatrix}\]

\[T(\boldsymbol{e}_2) = T\begin{pmatrix} 0 \\ 1 \\ 0 \end{pmatrix} = \begin{pmatrix} 2(0) - 1 + 3(0) \\ 0 + 4(1) - 0 \end{pmatrix} = \begin{pmatrix} -1 \\ 4 \end{pmatrix}\]

\[T(\boldsymbol{e}_3) = T\begin{pmatrix} 0 \\ 0 \\ 1 \end{pmatrix} = \begin{pmatrix} 2(0) - 0 + 3(1) \\ 0 + 4(0) - 1 \end{pmatrix} = \begin{pmatrix} 3 \\ -1 \end{pmatrix}\]

Therefore, the standard matrix is: \[\mathbf{A} = \begin{pmatrix} 2 & -1 & 3 \\ 1 & 4 & -1 \end{pmatrix}\]

2.6.2 Reflections, Rotations, Projections in \(\mathbb{R}^2\)

Linear transformations in \(\mathbb{R}^2\) have particularly nice geometric interpretations. Let’s explore the most important types:

Example 2.73 (Rotation by Angle \(\theta\)). The transformation that rotates every point counterclockwise by angle \(\theta\) about the origin has standard matrix: \[\mathbf{R}_\theta = \begin{pmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{pmatrix}\]

Derivation: The standard basis vectors transform as: - \(\boldsymbol{e}_1 = (1,0)\) rotates to \((\cos\theta, \sin\theta)\) - \(\boldsymbol{e}_2 = (0,1)\) rotates to \((-\sin\theta, \cos\theta)\)

Therefore: \(\mathbf{R}_\theta = [\text{rotated } \boldsymbol{e}_1 \mid \text{rotated } \boldsymbol{e}_2] = \begin{pmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{pmatrix}\)

Special cases: - \(\theta = 90°\): \(\mathbf{R}_{90°} = \begin{pmatrix} 0 & -1 \\ 1 & 0 \end{pmatrix}\) - \(\theta = 180°\): \(\mathbf{R}_{180°} = \begin{pmatrix} -1 & 0 \\ 0 & -1 \end{pmatrix}\) - \(\theta = 270°\): \(\mathbf{R}_{270°} = \begin{pmatrix} 0 & 1 \\ -1 & 0 \end{pmatrix}\)

Example 2.74 (Reflection across a Line through Origin). Reflection across the \(x\)-axis: \[\mathbf{F}_x = \begin{pmatrix} 1 & 0 \\ 0 & -1 \end{pmatrix}\] This keeps \(x\)-coordinates unchanged and negates \(y\)-coordinates.

Reflection across the \(y\)-axis: \[\mathbf{F}_y = \begin{pmatrix} -1 & 0 \\ 0 & 1 \end{pmatrix}\]

Reflection across the line \(y = x\): \[\mathbf{F}_{y=x} = \begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix}\] This swaps \(x\) and \(y\) coordinates.

Reflection across line making angle \(\alpha\) with positive \(x\)-axis: \[\mathbf{F}_\alpha = \begin{pmatrix} \cos(2\alpha) & \sin(2\alpha) \\ \sin(2\alpha) & -\cos(2\alpha) \end{pmatrix}\]

Example 2.75 (Orthogonal Projection onto a Line). The orthogonal projection onto the line through the origin making angle \(\theta\) with the positive \(x\)-axis has matrix: \[\mathbf{P}_\theta = \begin{pmatrix} \cos^2\theta & \cos\theta\sin\theta \\ \cos\theta\sin\theta & \sin^2\theta \end{pmatrix}\]

Special cases: - Projection onto \(x\)-axis (\(\theta = 0\)): \(\mathbf{P}_x = \begin{pmatrix} 1 & 0 \\ 0 & 0 \end{pmatrix}\) - Projection onto \(y\)-axis (\(\theta = 90°\)): \(\mathbf{P}_y = \begin{pmatrix} 0 & 0 \\ 0 & 1 \end{pmatrix}\) - Projection onto line \(y = x\) (\(\theta = 45°\)): \(\mathbf{P}_{y=x} = \begin{pmatrix} 1/2 & 1/2 \\ 1/2 & 1/2 \end{pmatrix}\)

Key property: Projection matrices satisfy \(\mathbf{P}^2 = \mathbf{P}\) (applying the projection twice gives the same result as applying it once).

Example 2.76 (Scaling Transformation). Uniform scaling by factor \(k\): \[\mathbf{S}_k = \begin{pmatrix} k & 0 \\ 0 & k \end{pmatrix} = k\mathbf{I}\]

Non-uniform scaling: Scale by factor \(a\) in \(x\)-direction, factor \(b\) in \(y\)-direction: \[\mathbf{S}_{a,b} = \begin{pmatrix} a & 0 \\ 0 & b \end{pmatrix}\]

When \(|a| > 1\) or \(|b| > 1\), we have expansion; when \(0 < |a| < 1\) or \(0 < |b| < 1\), we have contraction.

Example 2.77 (Shear Transformations). Horizontal shear: Fix the \(x\)-axis and shear parallel to it: \[\mathbf{H}_k = \begin{pmatrix} 1 & k \\ 0 & 1 \end{pmatrix}\] This maps \((x,y) \mapsto (x + ky, y)\).

Vertical shear: Fix the \(y\)-axis and shear parallel to it: \[\mathbf{V}_k = \begin{pmatrix} 1 & 0 \\ k & 1 \end{pmatrix}\] This maps \((x,y) \mapsto (x, kx + y)\).

Key Idea

These basic transformations can be composed (multiplied) to create more complex transformations. For example: - A rotation followed by a scaling: \(\mathbf{S}_k \mathbf{R}_\theta\) - A reflection followed by a rotation: \(\mathbf{R}_\theta \mathbf{F}_x\)

The order matters because matrix multiplication is not commutative!

2.7 LU Factorization

The LU factorization decomposes a matrix into the product of a lower triangular matrix and an upper triangular matrix, providing an efficient method for solving multiple systems with the same coefficient matrix.

Definition 2.78 (Triangular Matrices). - A matrix \(\mathbf{L}\) is lower triangular if \(\ell_{ij} = 0\) whenever \(i < j\) (all entries above the main diagonal are zero) - A matrix \(\mathbf{U}\) is upper triangular if \(u_{ij} = 0\) whenever \(i > j\) (all entries below the main diagonal are zero) - A triangular matrix is unit triangular if all diagonal entries equal 1

Definition 2.79 (LU Factorization). An LU factorization of an \(n \times n\) matrix \(\mathbf{A}\) is a representation \(\mathbf{A} = \mathbf{L}\mathbf{U}\) where \(\mathbf{L}\) is lower triangular and \(\mathbf{U}\) is upper triangular.

Theorem 2.80 (Existence of LU Factorization). If an \(n \times n\) matrix \(\mathbf{A}\) can be reduced to row-echelon form using only row replacement operations (no row interchanges), then \(\mathbf{A}\) has an LU factorization where \(\mathbf{L}\) is unit lower triangular.

Example 2.81 (LU Factorization Computation). Find the LU factorization of \(\mathbf{A} = \begin{pmatrix} 2 & 4 & -2 \\ 1 & 5 & 2 \\ 1 & 3 & 1 \end{pmatrix}\).

Step 1: Reduce to row-echelon form using row replacement:

\(R_2 - \frac{1}{2}R_1\): \[\begin{pmatrix} 2 & 4 & -2 \\ 0 & 3 & 3 \\ 1 & 3 & 1 \end{pmatrix}\]

\(R_3 - \frac{1}{2}R_1\): \[\begin{pmatrix} 2 & 4 & -2 \\ 0 & 3 & 3 \\ 0 & 1 & 2 \end{pmatrix}\]

\(R_3 - \frac{1}{3}R_2\): \[\mathbf{U} = \begin{pmatrix} 2 & 4 & -2 \\ 0 & 3 & 3 \\ 0 & 0 & 1 \end{pmatrix}\]

Step 2: The elementary matrices used were: - \(\mathbf{E}_1 = \mathbf{E}_{21}(-1/2)\): subtract \(\frac{1}{2}\) times row 1 from row 2 - \(\mathbf{E}_2 = \mathbf{E}_{31}(-1/2)\): subtract \(\frac{1}{2}\) times row 1 from row 3 - \(\mathbf{E}_3 = \mathbf{E}_{32}(-1/3)\): subtract \(\frac{1}{3}\) times row 2 from row 3

Step 3: For LU factorization with unit lower triangular \(\mathbf{L}\), we can construct \(\mathbf{L}\) directly by placing the multipliers in the appropriate positions:

\[\mathbf{L} = \begin{pmatrix} 1 & 0 & 0 \\ 1/2 & 1 & 0 \\ 1/2 & 1/3 & 1 \end{pmatrix}\]

Verification: \[\mathbf{L}\mathbf{U} = \begin{pmatrix} 1 & 0 & 0 \\ 1/2 & 1 & 0 \\ 1/2 & 1/3 & 1 \end{pmatrix}\begin{pmatrix} 2 & 4 & -2 \\ 0 & 3 & 3 \\ 0 & 0 & 1 \end{pmatrix}\]

\[= \begin{pmatrix} 2 & 4 & -2 \\ 1 + 0 & 2 + 3 & -1 + 3 \\ 1 + 0 + 0 & 2 + 1 + 0 & -1 + 1 + 1 \end{pmatrix} = \begin{pmatrix} 2 & 4 & -2 \\ 1 & 5 & 2 \\ 1 & 3 & 1 \end{pmatrix} = \mathbf{A} \quad \checkmark\]

Theorem 2.82 (Solving Systems with LU Factorization). If \(\mathbf{A} = \mathbf{L}\mathbf{U}\), then the system \(\mathbf{A}\boldsymbol{x} = \boldsymbol{b}\) can be solved efficiently by:

1. Forward substitution: Solve \(\mathbf{L}\boldsymbol{y} = \boldsymbol{b}\) for \(\boldsymbol{y}\)

2. Back substitution: Solve \(\mathbf{U}\boldsymbol{x} = \boldsymbol{y}\) for \(\boldsymbol{x}\)

Example 2.83 (Using LU Factorization to Solve Systems). Solve \(\mathbf{A}\boldsymbol{x} = \boldsymbol{b}\) where \(\mathbf{A}\) is from the previous example and \(\boldsymbol{b} = \begin{pmatrix} 2 \\ 11 \\ 7 \end{pmatrix}\).

Step 1: Forward substitution with \(\mathbf{L}\boldsymbol{y} = \boldsymbol{b}\): \[\begin{pmatrix} 1 & 0 & 0 \\ 1/2 & 1 & 0 \\ 1/2 & 1/3 & 1 \end{pmatrix}\begin{pmatrix} y_1 \\ y_2 \\ y_3 \end{pmatrix} = \begin{pmatrix} 2 \\ 11 \\ 7 \end{pmatrix}\]

From the first equation: \(y_1 = 2\) From the second equation: \(\frac{1}{2}y_1 + y_2 = 11 \Rightarrow 1 + y_2 = 11 \Rightarrow y_2 = 10\) From the third equation: \(\frac{1}{2}y_1 + \frac{1}{3}y_2 + y_3 = 7 \Rightarrow 1 + \frac{10}{3} + y_3 = 7 \Rightarrow y_3 = 7 - 1 - \frac{10}{3} = \frac{8}{3}\)

So \(\boldsymbol{y} = \begin{pmatrix} 2 \\ 10 \\ 8/3 \end{pmatrix}\).

Step 2: Back substitution with \(\mathbf{U}\boldsymbol{x} = \boldsymbol{y}\): \[\begin{pmatrix} 2 & 4 & -2 \\ 0 & 3 & 3 \\ 0 & 0 & 1 \end{pmatrix}\begin{pmatrix} x_1 \\ x_2 \\ x_3 \end{pmatrix} = \begin{pmatrix} 2 \\ 10 \\ 8/3 \end{pmatrix}\]

From the third equation: \(x_3 = \frac{8}{3}\) From the second equation: \(3x_2 + 3x_3 = 10 \Rightarrow 3x_2 + 8 = 10 \Rightarrow x_2 = \frac{2}{3}\) From the first equation: \(2x_1 + 4x_2 - 2x_3 = 2 \Rightarrow 2x_1 + \frac{8}{3} - \frac{16}{3} = 2 \Rightarrow 2x_1 = 2 + \frac{8}{3} = \frac{14}{3} \Rightarrow x_1 = \frac{7}{3}\)

Therefore: \(\boldsymbol{x} = \begin{pmatrix} 7/3 \\ 2/3 \\ 8/3 \end{pmatrix}\).

Theorem 2.84 (PLU Factorization). Every invertible matrix \(\mathbf{A}\) has a PLU factorization \(\mathbf{A} = \mathbf{P}\mathbf{L}\mathbf{U}\) where:

• \(\mathbf{P}\) is a permutation matrix (product of row interchange matrices)

• \(\mathbf{L}\) is unit lower triangular

• \(\mathbf{U}\) is upper triangular

Remark 137. The PLU factorization always exists and accounts for the row interchanges that may be necessary during Gaussian elimination. The permutation matrix \(\mathbf{P}\) records the row swaps performed.

2.8 Applications

Matrix algebra finds extensive applications across numerous fields. We examine two important areas: economic modeling and stochastic processes.

2.8.1 Input-Output Economic Models

Input-output analysis, developed by economist Wassily Leontief (Nobel Prize 1973), uses matrices to model the interdependencies in an economy.

Definition 2.86 (Leontief Input-Output Model). Consider an economy with \(n\) sectors. Let:

• \(x_i\) = total output of sector \(i\)

• \(d_i\) = external demand for sector \(i\)’s products

• \(a_{ij}\) = amount of sector \(i\)’s output needed to produce one unit of sector \(j\)’s output

The consumption matrix \(\mathbf{C} = (a_{ij})\) and the fundamental equation is: \[\boldsymbol{x} = \mathbf{C}\boldsymbol{x} + \boldsymbol{d}\] where \(\boldsymbol{x} = (x_1, x_2, \ldots, x_n)^{\mathsf{T}}\) and \(\boldsymbol{d} = (d_1, d_2, \ldots, d_n)^{\mathsf{T}}\).

Theorem 2.87 (Leontief Inverse). The input-output equation \(\boldsymbol{x} = \mathbf{C}\boldsymbol{x} + \boldsymbol{d}\) can be rewritten as: \[(\mathbf{I} - \mathbf{C})\boldsymbol{x} = \boldsymbol{d}\]

If \(\mathbf{I} - \mathbf{C}\) is invertible, then: \[\boldsymbol{x} = (\mathbf{I} - \mathbf{C})^{-1}\boldsymbol{d}\]

The matrix \((\mathbf{I} - \mathbf{C})^{-1}\) is called the Leontief inverse.

Example 2.88 (Two-Sector Economy). Consider a simplified economy with two sectors: Agriculture (A) and Manufacturing (M). The consumption matrix is:

\[\mathbf{C} = \begin{pmatrix} 0.2 & 0.4 \\ 0.3 & 0.1 \end{pmatrix}\]

This means:

• To produce $1 of agricultural output requires $0.20 of agricultural input and $0.30 of manufacturing input

• To produce $1 of manufacturing output requires $0.40 of agricultural input and $0.10 of manufacturing input

If external demand is \(\boldsymbol{d} = \begin{pmatrix} 50 \\ 80 \end{pmatrix}\) (in millions of dollars), find the required total outputs.

Solution: \[\mathbf{I} - \mathbf{C} = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix} - \begin{pmatrix} 0.2 & 0.4 \\ 0.3 & 0.1 \end{pmatrix} = \begin{pmatrix} 0.8 & -0.4 \\ -0.3 & 0.9 \end{pmatrix}\]

To find the inverse: \[\det(\mathbf{I} - \mathbf{C}) = 0.8 \times 0.9 - (-0.4) \times (-0.3) = 0.72 - 0.12 = 0.6\]

\[(\mathbf{I} - \mathbf{C})^{-1} = \frac{1}{0.6}\begin{pmatrix} 0.9 & 0.4 \\ 0.3 & 0.8 \end{pmatrix} = \begin{pmatrix} 1.5 & 2/3 \\ 0.5 & 4/3 \end{pmatrix}\]

Therefore: \[\boldsymbol{x} = \begin{pmatrix} 1.5 & 2/3 \\ 0.5 & 4/3 \end{pmatrix}\begin{pmatrix} 50 \\ 80 \end{pmatrix} = \begin{pmatrix} 75 + 160/3 \\ 25 + 320/3 \end{pmatrix} = \begin{pmatrix} 128.33 \\ 131.67 \end{pmatrix}\]

Interpretation: To meet external demands of $50M agricultural and $80M manufacturing products, the economy must produce $128.33M agricultural output and $131.67M manufacturing output.

Economic meaning of Leontief inverse entries: \((\mathbf{I} - \mathbf{C})^{-1}_{ij}\) represents the total output required from sector \(i\) to deliver one additional unit of final demand for sector \(j\)’s product.

2.8.2 Markov Chains

Markov chains model systems that transition between different states over time, where the probability of the next state depends only on the current state.

Definition 2.89 (Markov Chain). A Markov chain is a sequence of random variables \(X_0, X_1, X_2, \ldots\) where each \(X_n\) takes values in a finite state space \(\{1, 2, \ldots, k\}\), and: \[P(X_{n+1} = j \mid X_n = i, X_{n-1} = i_{n-1}, \ldots, X_0 = i_0) = P(X_{n+1} = j \mid X_n = i)\]

The transition matrix \(\mathbf{P} = (p_{ij})\) where \(p_{ij} = P(X_{n+1} = j \mid X_n = i)\).

Definition 2.90 (Stochastic Matrix). A matrix \(\mathbf{P}\) is stochastic (or row stochastic) if:

1. All entries are non-negative: \(p_{ij} \geq 0\)

2. Each row sums to 1: \(\sum_{j=1}^k p_{ij} = 1\)

Theorem 2.91 (Evolution of State Distribution). If \(\boldsymbol{\pi}^{(n)}\) represents the probability distribution over states at time \(n\), then: \[\boldsymbol{\pi}^{(n+1)} = \boldsymbol{\pi}^{(n)}\mathbf{P}\]

More generally: \[\boldsymbol{\pi}^{(n)} = \boldsymbol{\pi}^{(0)}\mathbf{P}^n\]

Example 2.92 (Weather Model). Consider a simplified weather model with three states: Sunny (S), Cloudy (C), and Rainy (R). The transition matrix is: \[\mathbf{P} = \begin{pmatrix} 0.7 & 0.2 & 0.1 \\ 0.3 & 0.4 & 0.3 \\ 0.2 & 0.3 & 0.5 \end{pmatrix}\] Interpretation:

• If today is sunny, tomorrow will be sunny with probability 0.7, cloudy with probability 0.2, rainy with probability 0.1

• If today is cloudy, tomorrow will be sunny with probability 0.3, cloudy with probability 0.4, rainy with probability 0.3

• If today is rainy, tomorrow will be sunny with probability 0.2, cloudy with probability 0.3, rainy with probability 0.5

Question: If today is sunny, what’s the probability distribution after 2 days? Solution: Starting with \(\boldsymbol{\pi}^{(0)} = (1, 0, 0)\) (certainly sunny), we need \(\mathbf{P}^2\): \[\mathbf{P}^2 = \begin{pmatrix} 0.7 & 0.2 & 0.1 \\ 0.3 & 0.4 & 0.3 \\ 0.2 & 0.3 & 0.5 \end{pmatrix}^2 = \begin{pmatrix} 0.57 & 0.25 & 0.18 \\ 0.39 & 0.31 & 0.30 \\ 0.33 & 0.31 & 0.36 \end{pmatrix}\] Verification of first row: \[\begin{align} 0.7 \times 0.7 + 0.2 \times 0.3 + 0.1 \times 0.2 &= 0.49 + 0.06 + 0.02 = 0.57\\ 0.7 \times 0.2 + 0.2 \times 0.4 + 0.1 \times 0.3 &= 0.14 + 0.08 + 0.03 = 0.25\\ 0.7 \times 0.1 + 0.2 \times 0.3 + 0.1 \times 0.5 &= 0.07 + 0.06 + 0.05 = 0.18 \end{align}\] Therefore: \(\boldsymbol{\pi}^{(2)} = (1, 0, 0)\begin{pmatrix} 0.57 & 0.25 & 0.18 \\ 0.39 & 0.31 & 0.30 \\ 0.33 & 0.31 & 0.36 \end{pmatrix} = (0.57, 0.25, 0.18)\) After 2 days starting from sunny, there’s a 57% chance of sunny, 25% chance of cloudy, and 18% chance of rainy weather.

Definition 2.93 (Steady-State Distribution). A probability vector \(\boldsymbol{\pi}\) is a steady-state distribution (or stationary distribution) if: \[\boldsymbol{\pi}\mathbf{P} = \boldsymbol{\pi}\]

In other words, \(\boldsymbol{\pi}\) is a left eigenvector of \(\mathbf{P}\) with eigenvalue 1.

Theorem 2.94 (Existence of Steady-State). Every finite Markov chain has at least one steady-state distribution. If the chain is irreducible (can reach any state from any other state) and aperiodic (not cyclical), then:

1. The steady-state distribution is unique

2. \(\lim_{n \to \infty} \mathbf{P}^n = \begin{pmatrix} \boldsymbol{\pi} \\ \boldsymbol{\pi} \\ \vdots \\ \boldsymbol{\pi} \end{pmatrix}\) (each row is \(\boldsymbol{\pi}\))

3. The long-run proportion of time spent in each state equals the steady-state probabilities

Example 2.95 (Finding Steady-State Distribution). Find the steady-state distribution for the weather model. We need to solve \(\boldsymbol{\pi}\mathbf{P} = \boldsymbol{\pi}\) with \(\pi_1 + \pi_2 + \pi_3 = 1\). This gives us the system: \[\begin{align} 0.7\pi_1 + 0.3\pi_2 + 0.2\pi_3 &= \pi_1\\ 0.2\pi_1 + 0.4\pi_2 + 0.3\pi_3 &= \pi_2\\ 0.1\pi_1 + 0.3\pi_2 + 0.5\pi_3 &= \pi_3\\ \pi_1 + \pi_2 + \pi_3 &= 1 \end{align}\] Rearranging the first three equations: \[\begin{align} -0.3\pi_1 + 0.3\pi_2 + 0.2\pi_3 &= 0\\ 0.2\pi_1 - 0.6\pi_2 + 0.3\pi_3 &= 0\\ 0.1\pi_1 + 0.3\pi_2 - 0.5\pi_3 &= 0 \end{align}\] From equation (1): \(\pi_1 = \pi_2 + \frac{2}{3}\pi_3\)
From equation (2): \(\pi_2 = \frac{13}{12}\pi_3\)
Substituting: \(\pi_1 = \frac{13}{12}\pi_3 + \frac{2}{3}\pi_3 = \frac{21}{12}\pi_3\)
Using the constraint \(\pi_1 + \pi_2 + \pi_3 = 1\): \[\frac{21}{12}\pi_3 + \frac{13}{12}\pi_3 + \pi_3 = 1\] \[\frac{46}{12}\pi_3 = 1 \implies \pi_3 = \frac{12}{46}\] Therefore: \[\boldsymbol{\pi} = \left(\frac{21}{46}, \frac{13}{46}, \frac{12}{46}\right) \approx (0.457, 0.283, 0.261)\] Verification: Check that \(\boldsymbol{\pi}\mathbf{P} = \boldsymbol{\pi}\): \[\left(\frac{21}{46}, \frac{13}{46}, \frac{12}{46}\right) \begin{pmatrix} 0.7 & 0.2 & 0.1 \\ 0.3 & 0.4 & 0.3 \\ 0.2 & 0.3 & 0.5 \end{pmatrix} = \left(\frac{21}{46}, \frac{13}{46}, \frac{12}{46}\right) \checkmark\] Interpretation: In the long run, about 45.7% of days are sunny, 28.3% are cloudy, and 26.1% are rainy, regardless of the starting weather condition.

Key Idea

Matrix algebra provides the computational foundation for these applications:

• Economic modeling: Matrix inversion solves for equilibrium production levels

• Markov chains: Matrix powers predict future state distributions, and eigenvalue problems find long-term behavior

• General principle: Real-world systems often involve linear relationships that can be modeled and analyzed using matrix techniques

This completes our comprehensive study of matrix algebra. We have developed the fundamental operations, explored the geometric interpretations through linear transformations, established computational algorithms, and demonstrated practical applications. These tools and concepts form the foundation for advanced topics in linear algebra, including vector spaces, eigenvalue problems, and orthogonality.

3 Determinants and Diagonalization

The determinant is one of the most important scalar-valued functions defined on square matrices. It encodes fundamental information about linear transformations, including whether they are invertible, how they scale areas and volumes, and their eigenvalue structure. Combined with the theory of eigenvalues and eigenvectors, determinants unlock the powerful technique of diagonalization, which simplifies many computational problems and reveals the underlying geometric structure of linear transformations.

3.1 Cofactor Expansion

3.1.1 Definition and Properties

We begin by defining determinants recursively, starting with the simplest cases and building to the general definition.

Definition 3.1 (Determinant for Small Matrices).

1. For a \(1 \times 1\) matrix \(\mathbf{A} = (a)\): \(\det(\mathbf{A}) = a\)

2. For a \(2 \times 2\) matrix \(\mathbf{A} = \begin{pmatrix} a & b \\ c & d \end{pmatrix}\): \(\det(\mathbf{A}) = ad - bc\)

Remark 149. The \(2 \times 2\) determinant formula \(ad - bc\) has a clear geometric interpretation: it represents the signed area of the parallelogram formed by the column vectors \((a,c)\) and \((b,d)\). The sign indicates orientation—positive for counterclockwise, negative for clockwise.

Example 3.3 (Computing \(2 \times 2\) Determinants). \[\det\begin{pmatrix} 3 & 1 \\ 2 & 4 \end{pmatrix} = 3 \cdot 4 - 1 \cdot 2 = 12 - 2 = 10\]

\[\det\begin{pmatrix} 5 & -2 \\ 1 & 3 \end{pmatrix} = 5 \cdot 3 - (-2) \cdot 1 = 15 + 2 = 17\]

\[\det\begin{pmatrix} 2 & 6 \\ 1 & 3 \end{pmatrix} = 2 \cdot 3 - 6 \cdot 1 = 6 - 6 = 0\]

The last example shows a singular (non-invertible) matrix, as its determinant is zero.

For larger matrices, we need the concept of minors and cofactors:

Definition 3.4 (Minor and Cofactor). Let \(\mathbf{A}\) be an \(n \times n\) matrix.

1. The \((i,j)\)-minor \(M_{ij}\) is the determinant of the \((n-1) \times (n-1)\) matrix obtained by deleting row \(i\) and column \(j\) from \(\mathbf{A}\)

2. The \((i,j)\)-cofactor is \(C_{ij} = (-1)^{i+j}M_{ij}\)

Remark 152. The factor \((-1)^{i+j}\) creates a "checkerboard" pattern of signs: \[\begin{pmatrix} + & - & + & - & \cdots\\ - & + & - & + & \cdots\\ + & - & + & - & \cdots\\ - & + & - & + & \cdots\\ \vdots & \vdots & \vdots & \vdots & \ddots \end{pmatrix}\] This alternating sign pattern is crucial for the determinant’s properties.

Definition 3.6 (Determinant by Cofactor Expansion). For an \(n \times n\) matrix \(\mathbf{A}\), the determinant can be computed by expansion along any row or column:

Expansion along row \(i\): \[\det(\mathbf{A}) = \sum_{j=1}^n a_{ij}C_{ij} = \sum_{j=1}^n a_{ij}(-1)^{i+j}M_{ij}\]

Expansion along column \(j\): \[\det(\mathbf{A}) = \sum_{i=1}^n a_{ij}C_{ij} = \sum_{i=1}^n a_{ij}(-1)^{i+j}M_{ij}\]

Theorem 3.7 (Well-Definedness of Determinant). The determinant is well-defined: expansion along any row or column yields the same value.

Remark 155. This theorem is non-trivial and requires careful proof by induction. The fact that we get the same answer regardless of which row or column we choose is a remarkable property that reflects the deep algebraic structure underlying determinants.

Example 3.9 (Computing a \(3 \times 3\) Determinant). Compute \(\det(\mathbf{A})\) where \(\mathbf{A} = \begin{pmatrix} 2 & 1 & 3 \\ 0 & 4 & 1 \\ 5 & 2 & 0 \end{pmatrix}\).

Method 1: Expand along row 2 (chosen because it has a zero): \[\det(\mathbf{A}) = 0 \cdot C_{21} + 4 \cdot C_{22} + 1 \cdot C_{23}\]

\[C_{22} = (-1)^{2+2}M_{22} = M_{22} = \det\begin{pmatrix} 2 & 3 \\ 5 & 0 \end{pmatrix} = 2 \cdot 0 - 3 \cdot 5 = -15\]

\[C_{23} = (-1)^{2+3}M_{23} = -M_{23} = -\det\begin{pmatrix} 2 & 1 \\ 5 & 2 \end{pmatrix} = -(2 \cdot 2 - 1 \cdot 5) = -(4 - 5) = 1\]

Therefore: \(\det(\mathbf{A}) = 0 + 4(-15) + 1(1) = -60 + 1 = -59\)

Method 2: Expand along column 3 (to verify): \[\det(\mathbf{A}) = 3 \cdot C_{13} + 1 \cdot C_{23} + 0 \cdot C_{33}\]

\[C_{13} = (-1)^{1+3}M_{13} = M_{13} = \det\begin{pmatrix} 0 & 4 \\ 5 & 2 \end{pmatrix} = 0 \cdot 2 - 4 \cdot 5 = -20\]

We already computed \(C_{23} = 1\).

Therefore: \(\det(\mathbf{A}) = 3(-20) + 1(1) + 0 = -60 + 1 = -59\)

The fundamental properties of determinants emerge from their definition:

Theorem 3.10 (Basic Properties of Determinants). Let \(\mathbf{A}\) and \(\mathbf{B}\) be \(n \times n\) matrices, and let \(k\) be a scalar. Then:

1. \(\det(\mathbf{A}^{\mathsf{T}}) = \det(\mathbf{A})\) (determinant of transpose)

2. \(\det(k\mathbf{A}) = k^n\det(\mathbf{A})\) (scalar multiplication)

3. \(\det(\mathbf{A}\mathbf{B}) = \det(\mathbf{A})\det(\mathbf{B})\) (multiplicativity)

4. If \(\mathbf{A}\) has a row (or column) of zeros, then \(\det(\mathbf{A}) = 0\)

5. If two rows (or columns) of \(\mathbf{A}\) are identical, then \(\det(\mathbf{A}) = 0\)

6. If two rows (or columns) are swapped, the determinant changes sign

7. The determinant is linear in each row (and each column) when others are held fixed

Theorem 3.11 (Effects of Elementary Row Operations). Let \(\mathbf{A}\) be an \(n \times n\) matrix and \(\mathbf{A}'\) be the result of applying an elementary row operation:

1. Row swap: \(\det(\mathbf{A}') = -\det(\mathbf{A})\)

2. Row scaling: If row \(i\) is multiplied by \(k\), then \(\det(\mathbf{A}') = k\det(\mathbf{A})\)

3. Row addition: If a multiple of one row is added to another, then \(\det(\mathbf{A}') = \det(\mathbf{A})\)

Example 3.12 (Using Row Operations to Compute Determinants). Compute \(\det(\mathbf{A})\) where \(\mathbf{A} = \begin{pmatrix} 1 & 2 & 3 \\ 2 & 5 & 7 \\ 1 & 3 & 4 \end{pmatrix}\) using row operations.

\[\begin{vmatrix} 1 & 2 & 3 \\ 2 & 5 & 7 \\ 1 & 3 & 4 \end{vmatrix} \xrightarrow{R_2 - 2R_1} \begin{vmatrix} 1 & 2 & 3 \\ 0 & 1 & 1 \\ 1 & 3 & 4 \end{vmatrix} \xrightarrow{R_3 - R_1} \begin{vmatrix} 1 & 2 & 3 \\ 0 & 1 & 1 \\ 0 & 1 & 1 \end{vmatrix}\]

Since rows 2 and 3 are now identical, \(\det(\mathbf{A}) = 0\).

Alternative approach: Continue to upper triangular form: \[\begin{vmatrix} 1 & 2 & 3 \\ 0 & 1 & 1 \\ 0 & 1 & 1 \end{vmatrix} \xrightarrow{R_3 - R_2} \begin{vmatrix} 1 & 2 & 3 \\ 0 & 1 & 1 \\ 0 & 0 & 0 \end{vmatrix}\]

For an upper triangular matrix, the determinant equals the product of diagonal entries: \(1 \cdot 1 \cdot 0 = 0\).

3.1.2 Inductive Proofs

The recursive definition of determinants naturally leads to proofs by induction. We present key results with full proofs.

Theorem 3.13 (Determinant of Upper Triangular Matrix). If \(\mathbf{U}\) is an upper triangular \(n \times n\) matrix, then \(\det(\mathbf{U})\) equals the product of its diagonal entries: \[\det(\mathbf{U}) = u_{11}u_{22} \cdots u_{nn}\]

Proof. We proceed by induction on \(n\).

Base case (\(n = 1\)): For \(\mathbf{U} = (u_{11})\), clearly \(\det(\mathbf{U}) = u_{11}\).

Base case (\(n = 2\)): For \(\mathbf{U} = \begin{pmatrix} u_{11} & u_{12} \\ 0 & u_{22} \end{pmatrix}\): \[\det(\mathbf{U}) = u_{11} \cdot u_{22} - u_{12} \cdot 0 = u_{11}u_{22}\]

Inductive step: Assume the theorem holds for \((n-1) \times (n-1)\) upper triangular matrices. For an \(n \times n\) upper triangular matrix \(\mathbf{U}\), expand along the first column:

\[\det(\mathbf{U}) = u_{11}C_{11} + 0 \cdot C_{21} + \cdots + 0 \cdot C_{n1} = u_{11}C_{11}\]

Since \(C_{11} = (-1)^{1+1}M_{11} = M_{11}\), and \(M_{11}\) is the determinant of an \((n-1) \times (n-1)\) upper triangular matrix (obtained by deleting the first row and column), the inductive hypothesis gives:

\[M_{11} = u_{22}u_{33} \cdots u_{nn}\]

Therefore: \(\det(\mathbf{U}) = u_{11} \cdot u_{22}u_{33} \cdots u_{nn} = u_{11}u_{22} \cdots u_{nn}\) ◻

Corollary 3.14 (Determinant and Invertibility). An \(n \times n\) matrix \(\mathbf{A}\) is invertible if and only if \(\det(\mathbf{A}) \neq 0\).

Proof. \(\mathbf{A}\) is invertible if and only if it is row equivalent to the identity matrix \(\mathbf{I}_n\). Since row operations either preserve the determinant or multiply it by a non-zero constant, \(\mathbf{A}\) is row equivalent to \(\mathbf{I}_n\) if and only if \(\det(\mathbf{A})\) and \(\det(\mathbf{I}_n) = 1\) have the same sign property (both zero or both non-zero). Thus \(\mathbf{A}\) is invertible if and only if \(\det(\mathbf{A}) \neq 0\). ◻

Theorem 3.15 (Multilinearity of Determinant). The determinant is multilinear: it is linear in each row when all other rows are held fixed.

Proof. We prove linearity in the first row; the argument for other rows is identical.

Let \(\mathbf{A}\) have rows \(\boldsymbol{r}_1, \boldsymbol{r}_2, \ldots, \boldsymbol{r}_n\). We need to show: \[\det(\mathbf{A}) = a\det(\mathbf{A}') + b\det(\mathbf{A}'')\] where \(\mathbf{A}'\) has first row \(\boldsymbol{r}'_1\) and \(\mathbf{A}''\) has first row \(\boldsymbol{r}''_1\), while \(\mathbf{A}\) has first row \(a\boldsymbol{r}'_1 + b\boldsymbol{r}''_1\).

Expanding along the first row: \[\det(\mathbf{A}) = \sum_{j=1}^n (a r'_{1j} + b r''_{1j})C_{1j} = a\sum_{j=1}^n r'_{1j}C_{1j} + b\sum_{j=1}^n r''_{1j}C_{1j} = a\det(\mathbf{A}') + b\det(\mathbf{A}'')\]

The cofactors \(C_{1j}\) are the same in all three determinants since they depend only on rows 2 through \(n\). ◻

3.2 Determinants and Inverses

The determinant provides a direct formula for matrix inverses and a computational method for solving linear systems.

Theorem 3.16 (Adjugate Matrix Formula for Inverse). If \(\mathbf{A}\) is an invertible \(n \times n\) matrix, then: \[\mathbf{A}^{-1} = \frac{1}{\det(\mathbf{A})}\operatorname{adj}(\mathbf{A})\] where \(\operatorname{adj}(\mathbf{A})\) is the adjugate matrix (or classical adjoint) with entries \((\operatorname{adj}(\mathbf{A}))_{ij} = C_{ji}\) (note the transposed indices).

Proof. We need to verify that \(\mathbf{A} \cdot \operatorname{adj}(\mathbf{A}) = \det(\mathbf{A})\mathbf{I}\).

Consider the \((i,j)\)-entry of \(\mathbf{A} \cdot \operatorname{adj}(\mathbf{A})\): \[(\mathbf{A} \cdot \operatorname{adj}(\mathbf{A}))_{ij} = \sum_{k=1}^n a_{ik}(\operatorname{adj}(\mathbf{A}))_{kj} = \sum_{k=1}^n a_{ik}C_{jk}\]

Case 1 (\(i = j\)): This sum is the cofactor expansion of \(\det(\mathbf{A})\) along row \(i\): \[\sum_{k=1}^n a_{ik}C_{ik} = \det(\mathbf{A})\]

Case 2 (\(i \neq j\)): This sum is the cofactor expansion of a matrix \(\mathbf{B}\) identical to \(\mathbf{A}\) except that row \(j\) is replaced by row \(i\). Since \(\mathbf{B}\) has two identical rows, \(\det(\mathbf{B}) = 0\): \[\sum_{k=1}^n a_{ik}C_{jk} = 0\]

Therefore: \(\mathbf{A} \cdot \operatorname{adj}(\mathbf{A}) = \det(\mathbf{A})\mathbf{I}\), which gives the desired formula. ◻

Example 3.17 (Computing Inverse via Adjugate). Find the inverse of \(\mathbf{A} = \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix}\).

Step 1: Compute the determinant: \[\det(\mathbf{A}) = 1 \cdot 4 - 2 \cdot 3 = 4 - 6 = -2\]

Step 2: Compute the cofactors: \[\begin{align} C_{11} &= (-1)^{1+1}M_{11} = \det(4) = 4\\ C_{12} &= (-1)^{1+2}M_{12} = -\det(3) = -3\\ C_{21} &= (-1)^{2+1}M_{21} = -\det(2) = -2\\ C_{22} &= (-1)^{2+2}M_{22} = \det(1) = 1 \end{align}\]

Step 3: Form the adjugate matrix: \[\operatorname{adj}(\mathbf{A}) = \begin{pmatrix} C_{11} & C_{21} \\ C_{12} & C_{22} \end{pmatrix} = \begin{pmatrix} 4 & -2 \\ -3 & 1 \end{pmatrix}\]

Step 4: Apply the formula: \[\mathbf{A}^{-1} = \frac{1}{-2}\begin{pmatrix} 4 & -2 \\ -3 & 1 \end{pmatrix} = \begin{pmatrix} -2 & 1 \\ 3/2 & -1/2 \end{pmatrix}\]

Verification: \[\begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix}\begin{pmatrix} -2 & 1 \\ 3/2 & -1/2 \end{pmatrix} = \begin{pmatrix} -2 + 3 & 1 - 1 \\ -6 + 6 & 3 - 2 \end{pmatrix} = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix} \quad \checkmark\]

Theorem 3.18 (Cramer’s Rule). Consider the linear system \(\mathbf{A}\boldsymbol{x} = \boldsymbol{b}\) where \(\mathbf{A}\) is an invertible \(n \times n\) matrix. Then the solution is: \[x_i = \frac{\det(\mathbf{A}_i)}{\det(\mathbf{A})}\] where \(\mathbf{A}_i\) is the matrix obtained by replacing the \(i\)-th column of \(\mathbf{A}\) with the vector \(\boldsymbol{b}\).

Proof. Since \(\mathbf{A}\) is invertible, the unique solution is \(\boldsymbol{x} = \mathbf{A}^{-1}\boldsymbol{b}\). Using the adjugate formula: \[x_i = (\mathbf{A}^{-1}\boldsymbol{b})_i = \frac{1}{\det(\mathbf{A})}(\operatorname{adj}(\mathbf{A})\boldsymbol{b})_i = \frac{1}{\det(\mathbf{A})}\sum_{j=1}^n C_{ij}b_j\]

The sum \(\sum_{j=1}^n C_{ij}b_j\) is the cofactor expansion of \(\det(\mathbf{A}_i)\) along the \(i\)-th column, since \(\mathbf{A}_i\) has the same cofactors as \(\mathbf{A}\) for the \(i\)-th column, but with entries \(b_j\) instead of \(a_{ji}\). ◻

Example 3.19 (Solving with Cramer’s Rule). Solve the system: \[\begin{align} 2x_1 + 3x_2 &= 7\\ x_1 - x_2 &= 1 \end{align}\]

Step 1: Set up matrices: \[\mathbf{A} = \begin{pmatrix} 2 & 3 \\ 1 & -1 \end{pmatrix}, \quad \boldsymbol{b} = \begin{pmatrix} 7 \\ 1 \end{pmatrix}\]

Step 2: Compute \(\det(\mathbf{A})\): \[\det(\mathbf{A}) = 2(-1) - 3(1) = -2 - 3 = -5\]

Step 3: Form \(\mathbf{A}_1\) and \(\mathbf{A}_2\): \[\mathbf{A}_1 = \begin{pmatrix} 7 & 3 \\ 1 & -1 \end{pmatrix}, \quad \mathbf{A}_2 = \begin{pmatrix} 2 & 7 \\ 1 & 1 \end{pmatrix}\]

Step 4: Compute determinants: \[\det(\mathbf{A}_1) = 7(-1) - 3(1) = -7 - 3 = -10\] \[\det(\mathbf{A}_2) = 2(1) - 7(1) = 2 - 7 = -5\]

Step 5: Apply Cramer’s rule: \[x_1 = \frac{\det(\mathbf{A}_1)}{\det(\mathbf{A})} = \frac{-10}{-5} = 2\] \[x_2 = \frac{\det(\mathbf{A}_2)}{\det(\mathbf{A})} = \frac{-5}{-5} = 1\]

Verification: \(2(2) + 3(1) = 4 + 3 = 7\) and \(2 - 1 = 1\)

Remark 167. While Cramer’s rule provides an elegant theoretical solution, it’s computationally expensive for large systems since it requires computing \(n+1\) determinants. Gaussian elimination is much more efficient for practical computation.

3.3 Eigenvalues and Eigenvectors

Eigenvalues and eigenvectors reveal the fundamental directional properties of linear transformations, identifying special directions that are preserved (up to scaling) under the transformation.

Definition 3.21 (Eigenvalue and Eigenvector). Let \(\mathbf{A}\) be an \(n \times n\) matrix. A non-zero vector \(\boldsymbol{v} \in \mathbb{R}^n\) is called an eigenvector of \(\mathbf{A}\) if there exists a scalar \(\lambda\) such that: \[\mathbf{A}\boldsymbol{v} = \lambda\boldsymbol{v}\] The scalar \(\lambda\) is called the eigenvalue corresponding to the eigenvector \(\boldsymbol{v}\).

Remark 169. The equation \(\mathbf{A}\boldsymbol{v} = \lambda\boldsymbol{v}\) means that the linear transformation represented by \(\mathbf{A}\) acts on \(\boldsymbol{v}\) by simply scaling it by the factor \(\lambda\). The direction of \(\boldsymbol{v}\) is preserved (if \(\lambda > 0\)) or reversed (if \(\lambda < 0\)).

Example 3.23 (Geometric Interpretation of Eigenvectors). Consider the matrix \(\mathbf{A} = \begin{pmatrix} 3 & 1 \\ 0 & 2 \end{pmatrix}\).

Testing potential eigenvectors: \[\mathbf{A}\begin{pmatrix} 1 \\ 0 \end{pmatrix} = \begin{pmatrix} 3 \\ 0 \end{pmatrix} = 3\begin{pmatrix} 1 \\ 0 \end{pmatrix}\]

So \(\boldsymbol{v}_1 = \begin{pmatrix} 1 \\ 0 \end{pmatrix}\) is an eigenvector with eigenvalue \(\lambda_1 = 3\).

\[\mathbf{A}\begin{pmatrix} 1 \\ -1 \end{pmatrix} = \begin{pmatrix} 2 \\ -2 \end{pmatrix} = 2\begin{pmatrix} 1 \\ -1 \end{pmatrix}\]

So \(\boldsymbol{v}_2 = \begin{pmatrix} 1 \\ -1 \end{pmatrix}\) is an eigenvector with eigenvalue \(\lambda_2 = 2\).

Geometric meaning: The transformation \(\mathbf{A}\) stretches vectors along the \(x\)-axis (direction of \(\boldsymbol{v}_1\)) by a factor of 3, and stretches vectors along the direction of \(\boldsymbol{v}_2\) by a factor of 2.

3.3.1 Characteristic Polynomial

To find eigenvalues systematically, we rearrange the eigenvalue equation:

Theorem 3.24 (Characteristic Equation). \(\lambda\) is an eigenvalue of matrix \(\mathbf{A}\) if and only if: \[\det(\mathbf{A} - \lambda\mathbf{I}) = 0\]

Proof. \(\lambda\) is an eigenvalue if and only if there exists a non-zero vector \(\boldsymbol{v}\) such that \(\mathbf{A}\boldsymbol{v} = \lambda\boldsymbol{v}\). This is equivalent to: \[(\mathbf{A} - \lambda\mathbf{I})\boldsymbol{v} = \boldsymbol{0}\]

This homogeneous system has a non-trivial solution if and only if the coefficient matrix \(\mathbf{A} - \lambda\mathbf{I}\) is singular, which occurs if and only if \(\det(\mathbf{A} - \lambda\mathbf{I}) = 0\). ◻

Definition 3.25 (Characteristic Polynomial). The characteristic polynomial of an \(n \times n\) matrix \(\mathbf{A}\) is: \[p(\lambda) = \det(\mathbf{A} - \lambda\mathbf{I})\] This is a polynomial of degree \(n\) in \(\lambda\). The eigenvalues of \(\mathbf{A}\) are precisely the roots of \(p(\lambda) = 0\).

Example 3.26 (Finding Eigenvalues via Characteristic Polynomial). Find the eigenvalues of \(\mathbf{A} = \begin{pmatrix} 4 & -2 \\ 1 & 1 \end{pmatrix}\).

Step 1: Form \(\mathbf{A} - \lambda\mathbf{I}\): \[\mathbf{A} - \lambda\mathbf{I} = \begin{pmatrix} 4 & -2 \\ 1 & 1 \end{pmatrix} - \lambda\begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix} = \begin{pmatrix} 4-\lambda & -2 \\ 1 & 1-\lambda \end{pmatrix}\]

Step 2: Compute the characteristic polynomial: \[p(\lambda) = \det\begin{pmatrix} 4-\lambda & -2 \\ 1 & 1-\lambda \end{pmatrix} = (4-\lambda)(1-\lambda) - (-2)(1)\] \[= (4-\lambda)(1-\lambda) + 2 = 4 - 4\lambda - \lambda + \lambda^2 + 2 = \lambda^2 - 5\lambda + 6\]

Step 3: Solve \(p(\lambda) = 0\): \[\lambda^2 - 5\lambda + 6 = 0\] \[(\lambda - 2)(\lambda - 3) = 0\]

Therefore, the eigenvalues are \(\lambda_1 = 2\) and \(\lambda_2 = 3\).

Definition 3.27 (Eigenspace). For each eigenvalue \(\lambda\) of matrix \(\mathbf{A}\), the eigenspace \(E_\lambda\) is the set of all eigenvectors corresponding to \(\lambda\), together with the zero vector: \[E_\lambda = \{\boldsymbol{v} \in \mathbb{R}^n : \mathbf{A}\boldsymbol{v} = \lambda\boldsymbol{v}\} = \text{Nul}(\mathbf{A} - \lambda\mathbf{I})\]

Theorem 3.28 (Eigenspaces are Subspaces). For each eigenvalue \(\lambda\), the eigenspace \(E_\lambda\) is a subspace of \(\mathbb{R}^n\).

Proof. \(E_\lambda = \text{Nul}(\mathbf{A} - \lambda\mathbf{I})\) is the null space of a matrix, which is always a subspace. ◻

Definition 3.29 (Algebraic and Geometric Multiplicity).

• The algebraic multiplicity of eigenvalue \(\lambda\) is its multiplicity as a root of the characteristic polynomial

• The geometric multiplicity of eigenvalue \(\lambda\) is \(\dim(E_\lambda)\), the dimension of the eigenspace

Example 3.30 (Computing Eigenvectors and Eigenspaces). Continue the previous example by finding eigenvectors for \(\mathbf{A} = \begin{pmatrix} 4 & -2 \\ 1 & 1 \end{pmatrix}\).

For \(\lambda_1 = 2\): Solve \((\mathbf{A} - 2\mathbf{I})\boldsymbol{v} = \boldsymbol{0}\): \[\begin{pmatrix} 2 & -2 \\ 1 & -1 \end{pmatrix}\begin{pmatrix} v_1 \\ v_2 \end{pmatrix} = \begin{pmatrix} 0 \\ 0 \end{pmatrix}\]

This gives us \(2v_1 - 2v_2 = 0\), or \(v_1 = v_2\). The eigenspace is: \[E_2 = \operatorname{span}\left\{\begin{pmatrix} 1 \\ 1 \end{pmatrix}\right\}\]

For \(\lambda_2 = 3\): Solve \((\mathbf{A} - 3\mathbf{I})\boldsymbol{v} = \boldsymbol{0}\): \[\begin{pmatrix} 1 & -2 \\ 1 & -2 \end{pmatrix}\begin{pmatrix} v_1 \\ v_2 \end{pmatrix} = \begin{pmatrix} 0 \\ 0 \end{pmatrix}\]

This gives us \(v_1 - 2v_2 = 0\), or \(v_1 = 2v_2\). The eigenspace is: \[E_3 = \operatorname{span}\left\{\begin{pmatrix} 2 \\ 1 \end{pmatrix}\right\}\]

Both eigenvalues have algebraic multiplicity 1 and geometric multiplicity 1.

3.3.2 Geometric Meaning

Eigenvalues and eigenvectors have profound geometric interpretations that illuminate the behavior of linear transformations.

Key Idea

Geometric Interpretation of Eigenvalues and Eigenvectors:

• Eigenvectors represent the "principal directions" of a linear transformation—directions that are preserved under the transformation

• Eigenvalues represent the scaling factors along these principal directions

• Positive eigenvalues preserve orientation; negative eigenvalues reverse it

• Eigenvalues with absolute value greater than 1 cause expansion; less than 1 cause contraction

• Zero eigenvalues correspond to directions that are collapsed to the origin

Example 3.31 (Eigenvalues of Geometric Transformations). Rotation by \(90°\): \(\mathbf{R} = \begin{pmatrix} 0 & -1 \\ 1 & 0 \end{pmatrix}\)

Characteristic polynomial: \(\det\begin{pmatrix} -\lambda & -1 \\ 1 & -\lambda \end{pmatrix} = \lambda^2 + 1\)

Eigenvalues: \(\lambda^2 = -1 \Rightarrow \lambda = \pm i\) (complex!)

This makes geometric sense: rotation by \(90°\) has no real eigenvectors because no direction is preserved.

Reflection across \(x\)-axis: \(\mathbf{F} = \begin{pmatrix} 1 & 0 \\ 0 & -1 \end{pmatrix}\)

Eigenvalues: \(\lambda_1 = 1\) (eigenspace: \(x\)-axis), \(\lambda_2 = -1\) (eigenspace: \(y\)-axis)

This reflects the fact that the \(x\)-axis is preserved (\(\lambda = 1\)) while the \(y\)-axis is reversed (\(\lambda = -1\)).

Projection onto \(x\)-axis: \(\mathbf{P} = \begin{pmatrix} 1 & 0 \\ 0 & 0 \end{pmatrix}\)

Eigenvalues: \(\lambda_1 = 1\) (eigenspace: \(x\)-axis), \(\lambda_2 = 0\) (eigenspace: \(y\)-axis)

The \(x\)-axis is preserved, while the \(y\)-direction is collapsed to zero.

Theorem 3.32 (Properties of Eigenvalues). Let \(\mathbf{A}\) be an \(n \times n\) matrix with eigenvalues \(\lambda_1, \lambda_2, \ldots, \lambda_n\) (counting multiplicities). Then:

1. \(\det(\mathbf{A}) = \lambda_1 \lambda_2 \cdots \lambda_n\) (product of eigenvalues)

2. \(\operatorname{tr}(\mathbf{A}) = \lambda_1 + \lambda_2 + \cdots + \lambda_n\) (sum of eigenvalues)

3. \(\mathbf{A}\) is invertible if and only if all eigenvalues are non-zero

4. If \(\mathbf{A}\) is invertible, then the eigenvalues of \(\mathbf{A}^{-1}\) are \(1/\lambda_1, 1/\lambda_2, \ldots, 1/\lambda_n\)

Proof. Property 1: The characteristic polynomial is \(p(\lambda) = \det(\mathbf{A} - \lambda\mathbf{I})\). When we expand this determinant, the constant term (coefficient of \(\lambda^0\)) equals \(\det(\mathbf{A})\). By Vieta’s formulas for polynomials, this constant term also equals \((-1)^n\) times the product of roots. For the characteristic polynomial, this gives \(\det(\mathbf{A}) = \lambda_1 \lambda_2 \cdots \lambda_n\).

Property 2: Similarly, the coefficient of \(\lambda^{n-1}\) in the characteristic polynomial equals \((-1)^{n-1}(\lambda_1 + \cdots + \lambda_n)\). This coefficient also equals \((-1)^{n-1}\operatorname{tr}(\mathbf{A})\) by direct expansion of the determinant.

Properties 3 and 4: Follow immediately from Property 1 and the definition of eigenvalues. ◻

3.4 Diagonalization

Diagonalization is the process of finding a basis with respect to which a linear transformation has a diagonal matrix representation. This dramatically simplifies many computations and reveals the underlying structure of the transformation.

Definition 3.33 (Diagonalizable Matrix). An \(n \times n\) matrix \(\mathbf{A}\) is diagonalizable if there exists an invertible matrix \(\mathbf{P}\) and a diagonal matrix \(\mathbf{D}\) such that: \[\mathbf{A} = \mathbf{P}\mathbf{D}\mathbf{P}^{-1}\] Equivalently, \(\mathbf{P}^{-1}\mathbf{A}\mathbf{P} = \mathbf{D}\).

Remark 181. The equation \(\mathbf{P}^{-1}\mathbf{A}\mathbf{P} = \mathbf{D}\) represents a change of basis: if \(\mathbf{P}\) has columns \(\boldsymbol{p}_1, \boldsymbol{p}_2, \ldots, \boldsymbol{p}_n\), then these vectors form a basis in which the linear transformation represented by \(\mathbf{A}\) has the simple diagonal form \(\mathbf{D}\).

3.4.1 Conditions for Diagonalization

Theorem 3.35 (Diagonalization Theorem). An \(n \times n\) matrix \(\mathbf{A}\) is diagonalizable if and only if \(\mathbf{A}\) has \(n\) linearly independent eigenvectors.

Moreover, if \(\mathbf{A}\) is diagonalizable, then:

• \(\mathbf{P}\) can be chosen to have eigenvectors of \(\mathbf{A}\) as its columns

• \(\mathbf{D}\) has the corresponding eigenvalues on its diagonal

• \(\mathbf{A}\mathbf{P} = \mathbf{P}\mathbf{D}\)

Proof. (\(\Rightarrow\)) Suppose \(\mathbf{A} = \mathbf{P}\mathbf{D}\mathbf{P}^{-1}\) where \(\mathbf{D} = \operatorname{diag}(\lambda_1, \lambda_2, \ldots, \lambda_n)\) and \(\mathbf{P} = [\boldsymbol{p}_1 \mid \boldsymbol{p}_2 \mid \cdots \mid \boldsymbol{p}_n]\).

Then \(\mathbf{A}\mathbf{P} = \mathbf{P}\mathbf{D}\), which gives: \[\mathbf{A}[\boldsymbol{p}_1 \mid \boldsymbol{p}_2 \mid \cdots \mid \boldsymbol{p}_n] = [\boldsymbol{p}_1 \mid \boldsymbol{p}_2 \mid \cdots \mid \boldsymbol{p}_n]\begin{pmatrix} \lambda_1 & & \\ & \ddots & \\ & & \lambda_n \end{pmatrix}\]

Comparing columns: \(\mathbf{A}\boldsymbol{p}_i = \lambda_i\boldsymbol{p}_i\) for \(i = 1, 2, \ldots, n\).

Since \(\mathbf{P}\) is invertible, its columns \(\boldsymbol{p}_1, \boldsymbol{p}_2, \ldots, \boldsymbol{p}_n\) are linearly independent eigenvectors of \(\mathbf{A}\).

(\(\Leftarrow\)) Suppose \(\mathbf{A}\) has \(n\) linearly independent eigenvectors \(\boldsymbol{p}_1, \boldsymbol{p}_2, \ldots, \boldsymbol{p}_n\) with corresponding eigenvalues \(\lambda_1, \lambda_2, \ldots, \lambda_n\).

Let \(\mathbf{P} = [\boldsymbol{p}_1 \mid \boldsymbol{p}_2 \mid \cdots \mid \boldsymbol{p}_n]\) and \(\mathbf{D} = \operatorname{diag}(\lambda_1, \lambda_2, \ldots, \lambda_n)\).

Since the eigenvectors are linearly independent, \(\mathbf{P}\) is invertible. The eigenvalue equations \(\mathbf{A}\boldsymbol{p}_i = \lambda_i\boldsymbol{p}_i\) combine to give \(\mathbf{A}\mathbf{P} = \mathbf{P}\mathbf{D}\), which yields \(\mathbf{A} = \mathbf{P}\mathbf{D}\mathbf{P}^{-1}\). ◻

Theorem 3.36 (Sufficient Condition for Diagonalizability). If an \(n \times n\) matrix has \(n\) distinct eigenvalues, then it is diagonalizable.

Proof. Eigenvectors corresponding to distinct eigenvalues are linearly independent (this is a standard result in linear algebra). Since there are \(n\) distinct eigenvalues, there are \(n\) linearly independent eigenvectors, so the matrix is diagonalizable by the previous theorem. ◻

Theorem 3.37 (Geometric vs Algebraic Multiplicity Condition). An \(n \times n\) matrix \(\mathbf{A}\) is diagonalizable if and only if, for each eigenvalue \(\lambda\): \[\text{geometric multiplicity of } \lambda = \text{algebraic multiplicity of } \lambda\]

Remark 185. This theorem provides a practical test: compute the characteristic polynomial to find algebraic multiplicities, then compute eigenspaces to find geometric multiplicities. If they match for all eigenvalues, the matrix is diagonalizable.

3.4.2 Diagonalization Algorithm

Example 3.39 (Complete Diagonalization). Diagonalize \(\mathbf{A} = \begin{pmatrix} 3 & 1 & 0 \\ 0 & 3 & 0 \\ 0 & 0 & 2 \end{pmatrix}\).

Step 1: Find the characteristic polynomial: \[\det(\mathbf{A} - \lambda\mathbf{I}) = \det\begin{pmatrix} 3-\lambda & 1 & 0 \\ 0 & 3-\lambda & 0 \\ 0 & 0 & 2-\lambda \end{pmatrix}\]

Since this is upper triangular, the determinant is the product of diagonal entries: \[p(\lambda) = (3-\lambda)^2(2-\lambda)\]

Step 2: Eigenvalues are \(\lambda_1 = 3\) (algebraic multiplicity 2) and \(\lambda_2 = 2\) (algebraic multiplicity 1).

Step 3: Find eigenspaces:

For \(\lambda_1 = 3\): Solve \((\mathbf{A} - 3\mathbf{I})\boldsymbol{v} = \boldsymbol{0}\): \[\begin{pmatrix} 0 & 1 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & -1 \end{pmatrix}\begin{pmatrix} v_1 \\ v_2 \\ v_3 \end{pmatrix} = \begin{pmatrix} 0 \\ 0 \\ 0 \end{pmatrix}\]

This gives \(v_2 = 0\) and \(v_3 = 0\), so \(v_1\) is free. Therefore: \[E_3 = \operatorname{span}\left\{\begin{pmatrix} 1 \\ 0 \\ 0 \end{pmatrix}\right\}\]

The geometric multiplicity of \(\lambda_1 = 3\) is 1, but the algebraic multiplicity is 2. Since these don’t match, \(\mathbf{A}\) is not diagonalizable.

Verification: We can still find the eigenspace for \(\lambda_2 = 2\): \[(\mathbf{A} - 2\mathbf{I})\boldsymbol{v} = \begin{pmatrix} 1 & 1 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0 \end{pmatrix}\boldsymbol{v} = \boldsymbol{0}\]

This gives \(v_1 + v_2 = 0\) and \(v_2 = 0\), so \(v_1 = v_2 = 0\) and \(v_3\) is free: \[E_2 = \operatorname{span}\left\{\begin{pmatrix} 0 \\ 0 \\ 1 \end{pmatrix}\right\}\]

Total number of linearly independent eigenvectors: \(1 + 1 = 2 < 3\), confirming that \(\mathbf{A}\) is not diagonalizable.

Example 3.40 (Successful Diagonalization). Diagonalize \(\mathbf{B} = \begin{pmatrix} 4 & -2 \\ 1 & 1 \end{pmatrix}\) (from our earlier example).

We already found eigenvalues \(\lambda_1 = 2\) and \(\lambda_2 = 3\) with eigenvectors \(\boldsymbol{v}_1 = \begin{pmatrix} 1 \\ 1 \end{pmatrix}\) and \(\boldsymbol{v}_2 = \begin{pmatrix} 2 \\ 1 \end{pmatrix}\).

Form the matrices: \[\mathbf{P} = \begin{pmatrix} 1 & 2 \\ 1 & 1 \end{pmatrix}, \quad \mathbf{D} = \begin{pmatrix} 2 & 0 \\ 0 & 3 \end{pmatrix}\]

Verification: \[\mathbf{B}\mathbf{P} = \begin{pmatrix} 4 & -2 \\ 1 & 1 \end{pmatrix}\begin{pmatrix} 1 & 2 \\ 1 & 1 \end{pmatrix} = \begin{pmatrix} 2 & 6 \\ 2 & 3 \end{pmatrix}\]

\[\mathbf{P}\mathbf{D} = \begin{pmatrix} 1 & 2 \\ 1 & 1 \end{pmatrix}\begin{pmatrix} 2 & 0 \\ 0 & 3 \end{pmatrix} = \begin{pmatrix} 2 & 6 \\ 2 & 3 \end{pmatrix}\]

Since \(\mathbf{B}\mathbf{P} = \mathbf{P}\mathbf{D}\), we have successfully diagonalized \(\mathbf{B}\).

Complete factorization: We can compute \(\mathbf{P}^{-1}\): \[\det(\mathbf{P}) = 1 \cdot 1 - 2 \cdot 1 = -1\] \[\mathbf{P}^{-1} = \frac{1}{-1}\begin{pmatrix} 1 & -2 \\ -1 & 1 \end{pmatrix} = \begin{pmatrix} -1 & 2 \\ 1 & -1 \end{pmatrix}\]

Therefore: \(\mathbf{B} = \mathbf{P}\mathbf{D}\mathbf{P}^{-1} = \begin{pmatrix} 1 & 2 \\ 1 & 1 \end{pmatrix}\begin{pmatrix} 2 & 0 \\ 0 & 3 \end{pmatrix}\begin{pmatrix} -1 & 2 \\ 1 & -1 \end{pmatrix}\)

3.5 Applications

The power of eigenvalues and diagonalization becomes apparent in their wide-ranging applications across mathematics, science, and engineering.

3.5.1 Linear Dynamical Systems

Dynamical systems model how quantities change over time. When the evolution is linear, eigenvalues determine the long-term behavior.

Definition 3.41 (Discrete Linear Dynamical System). A discrete linear dynamical system is a sequence of states \(\boldsymbol{x}_0, \boldsymbol{x}_1, \boldsymbol{x}_2, \ldots\) where: \[\boldsymbol{x}_{k+1} = \mathbf{A}\boldsymbol{x}_k\] for some matrix \(\mathbf{A}\). The solution is \(\boldsymbol{x}_k = \mathbf{A}^k\boldsymbol{x}_0\).

Theorem 3.42 (Behavior of Linear Dynamical Systems). If \(\mathbf{A}\) is diagonalizable with \(\mathbf{A} = \mathbf{P}\mathbf{D}\mathbf{P}^{-1}\), then: \[\boldsymbol{x}_k = \mathbf{A}^k\boldsymbol{x}_0 = \mathbf{P}\mathbf{D}^k\mathbf{P}^{-1}\boldsymbol{x}_0\]

Since \(\mathbf{D}^k = \operatorname{diag}(\lambda_1^k, \lambda_2^k, \ldots, \lambda_n^k)\), the long-term behavior depends on the eigenvalues:

• If all \(|\lambda_i| < 1\), then \(\boldsymbol{x}_k \to \boldsymbol{0}\) (stable equilibrium)

• If any \(|\lambda_i| > 1\), then \(\|\boldsymbol{x}_k\| \to \infty\) (unstable)

• If \(|\lambda_i| = 1\) for all \(i\), the system may be periodic or neutrally stable

Example 3.43 (Population Dynamics). Consider a predator-prey model where \(x_k\) represents the prey population and \(y_k\) represents the predator population at time \(k\). The evolution might be modeled by: \[\begin{pmatrix} x_{k+1} \\ y_{k+1} \end{pmatrix} = \begin{pmatrix} 1.1 & -0.2 \\ 0.1 & 0.9 \end{pmatrix}\begin{pmatrix} x_k \\ y_k \end{pmatrix}\]

Analysis: Find eigenvalues of \(\mathbf{A} = \begin{pmatrix} 1.1 & -0.2 \\ 0.1 & 0.9 \end{pmatrix}\):

\[\det(\mathbf{A} - \lambda\mathbf{I}) = (1.1-\lambda)(0.9-\lambda) - (-0.2)(0.1) = \lambda^2 - 2\lambda + 1.01\]

Using the quadratic formula: \[\lambda = \frac{2 \pm \sqrt{4 - 4.04}}{2} = \frac{2 \pm \sqrt{-0.04}}{2} = 1 \pm 0.1i\]

Since \(|\lambda| = \sqrt{1^2 + 0.1^2} = \sqrt{1.01} > 1\), the system is unstable—populations will oscillate with increasing amplitude.

Biological interpretation: The complex eigenvalues indicate oscillatory behavior (predator-prey cycles), while the magnitude greater than 1 suggests the oscillations grow over time, possibly indicating an unrealistic model or parameter values.

3.5.2 Linear Recurrences

Many sequences satisfy linear recurrence relations that can be solved using eigenvalue methods.

Example 3.44 (Fibonacci Sequence). The Fibonacci sequence \(F_0 = 0, F_1 = 1, F_{n+1} = F_n + F_{n-1}\) can be written as: \[\begin{pmatrix} F_{n+1} \\ F_n \end{pmatrix} = \begin{pmatrix} 1 & 1 \\ 1 & 0 \end{pmatrix}\begin{pmatrix} F_n \\ F_{n-1} \end{pmatrix}\]

Let \(\mathbf{A} = \begin{pmatrix} 1 & 1 \\ 1 & 0 \end{pmatrix}\). The characteristic polynomial is: \[\det(\mathbf{A} - \lambda\mathbf{I}) = \lambda^2 - \lambda - 1\]

The eigenvalues are: \[\lambda = \frac{1 \pm \sqrt{5}}{2}\]

These are the golden ratio \(\phi = \frac{1+\sqrt{5}}{2} \approx 1.618\) and \(\hat{\phi} = \frac{1-\sqrt{5}}{2} \approx -0.618\).

Binet’s Formula: Using diagonalization, we can derive the explicit formula: \[F_n = \frac{1}{\sqrt{5}}\left[\left(\frac{1+\sqrt{5}}{2}\right)^n - \left(\frac{1-\sqrt{5}}{2}\right)^n\right]\]

Since \(|\hat{\phi}| < 1\), the second term vanishes for large \(n\), giving the approximation: \[F_n \approx \frac{\phi^n}{\sqrt{5}}\]

3.5.3 Systems of Differential Equations

Eigenvalues also determine the behavior of systems of linear differential equations.

Definition 3.45 (Linear System of ODEs). A homogeneous linear system of ordinary differential equations has the form: \[\frac{d\boldsymbol{x}}{dt} = \mathbf{A}\boldsymbol{x}\] where \(\boldsymbol{x}(t) \in \mathbb{R}^n\) and \(\mathbf{A}\) is a constant \(n \times n\) matrix.

Theorem 3.46 (General Solution via Eigenvalues). If \(\mathbf{A}\) has real eigenvalues \(\lambda_1, \lambda_2, \ldots, \lambda_n\) with corresponding eigenvectors \(\boldsymbol{v}_1, \boldsymbol{v}_2, \ldots, \boldsymbol{v}_n\), then the general solution is: \[\boldsymbol{x}(t) = c_1 e^{\lambda_1 t}\boldsymbol{v}_1 + c_2 e^{\lambda_2 t}\boldsymbol{v}_2 + \cdots + c_n e^{\lambda_n t}\boldsymbol{v}_n\] where \(c_1, c_2, \ldots, c_n\) are constants determined by initial conditions.

Example 3.47 (Coupled Oscillator System). Consider two masses connected by springs, leading to the system: \[\frac{d}{dt}\begin{pmatrix} x_1 \\ x_2 \end{pmatrix} = \begin{pmatrix} 0 & 1 \\ -2 & -1 \end{pmatrix}\begin{pmatrix} x_1 \\ x_2 \end{pmatrix}\]

Find eigenvalues: \[\det\begin{pmatrix} -\lambda & 1 \\ -2 & -1-\lambda \end{pmatrix} = \lambda(\lambda+1) + 2 = \lambda^2 + \lambda + 2\]

\[\lambda = \frac{-1 \pm \sqrt{1-8}}{2} = \frac{-1 \pm i\sqrt{7}}{2}\]

Interpretation: The complex eigenvalues with negative real parts indicate damped oscillations—the system oscillates while gradually returning to equilibrium.

General solution: For complex eigenvalues \(\alpha \pm \beta i\), the solution involves terms like \(e^{\alpha t}(\cos(\beta t) + i\sin(\beta t))\). The negative real part \(\alpha = -1/2\) ensures decay, while the imaginary part \(\beta = \sqrt{7}/2\) determines the oscillation frequency.

3.5.4 Google PageRank

One of the most famous applications of eigenvalue theory is Google’s PageRank algorithm, which ranks web pages based on their link structure.

Definition 3.48 (PageRank Model). Consider a web with \(n\) pages. The link matrix \(\mathbf{L}\) has entries: \[L_{ij} = \begin{cases} 1/k_j & \text{if page } j \text{ links to page } i\\ 0 & \text{otherwise} \end{cases}\] where \(k_j\) is the number of outgoing links from page \(j\).

The Google matrix is: \[\mathbf{G} = d\mathbf{L} + \frac{1-d}{n}\mathbf{J}\] where \(d \approx 0.85\) is the damping factor and \(\mathbf{J}\) is the matrix of all ones.

Theorem 3.49 (PageRank as Stationary Distribution). The PageRank vector \(\boldsymbol{\pi}\) is the stationary distribution of the Markov chain defined by \(\mathbf{G}\): \[\mathbf{G}\boldsymbol{\pi} = \boldsymbol{\pi}\] In other words, \(\boldsymbol{\pi}\) is an eigenvector of \(\mathbf{G}\) with eigenvalue 1.

Example 3.50 (Simple PageRank Calculation). Consider a mini-web with 3 pages:

• Page 1 links to pages 2 and 3

• Page 2 links to page 3

• Page 3 links to page 1

The link matrix is: \[\mathbf{L} = \begin{pmatrix} 0 & 0 & 1 \\ 1/2 & 0 & 0 \\ 1/2 & 1 & 0 \end{pmatrix}\]

With damping factor \(d = 0.8\): \[\mathbf{G} = 0.8\begin{pmatrix} 0 & 0 & 1 \\ 1/2 & 0 & 0 \\ 1/2 & 1 & 0 \end{pmatrix} + 0.2\begin{pmatrix} 1/3 & 1/3 & 1/3 \\ 1/3 & 1/3 & 1/3 \\ 1/3 & 1/3 & 1/3 \end{pmatrix}\]

\[= \begin{pmatrix} 1/15 & 1/15 & 13/15 \\ 7/15 & 1/15 & 1/15 \\ 7/15 & 13/15 & 1/15 \end{pmatrix}\]

To find the PageRank vector, solve \((\mathbf{G} - \mathbf{I})\boldsymbol{\pi} = \boldsymbol{0}\) with the constraint \(\pi_1 + \pi_2 + \pi_3 = 1\).

The solution is approximately \(\boldsymbol{\pi} \approx (0.387, 0.226, 0.387)\), indicating that pages 1 and 3 have higher PageRank than page 2.

Key Idea

The Power of Eigenvalue Methods:

• Diagonalization transforms complex matrix powers into simple scalar powers

• Eigenvalues determine stability and long-term behavior of dynamical systems

• Eigenvectors identify principal directions and steady-state distributions

• Applications span from pure mathematics (recurrence relations) to modern technology (web search algorithms)

The marriage of linear algebra theory with computational practice in eigenvalue problems exemplifies the power of mathematical abstraction to solve real-world problems.

This completes our comprehensive study of determinants and diagonalization. We have developed the theoretical foundations, explored computational methods, and demonstrated the wide-ranging applications of these fundamental concepts. The eigenvalue-eigenvector framework provides a powerful lens for understanding linear transformations and their applications across diverse fields.

4 Vector Geometry

Vector geometry bridges the gap between algebraic manipulation and geometric intuition, providing powerful tools for describing and analyzing geometric objects in two and three dimensions. This chapter develops the geometric interpretation of vectors, establishes methods for describing lines and planes, introduces the cross product, and explores applications to computer graphics and spatial reasoning.

4.1 Vectors and Lines

Vectors provide a natural language for describing geometric objects and their relationships. We begin by establishing the geometric interpretation of vector operations and then apply these concepts to describe lines in space.

Definition 4.1 (Geometric Vector). A geometric vector is a directed line segment characterized by:

1. Magnitude (or length): the distance between the initial and terminal points

2. Direction: the orientation from the initial point to the terminal point

Two vectors are considered equal if they have the same magnitude and direction, regardless of their position in space.

Remark 199. This definition captures the essential property that vectors represent displacements or movements through space. The position where a vector "starts" is irrelevant—only the magnitude and direction matter. This is why we can "slide" vectors around in space without changing their mathematical properties.

Definition 4.3 (Position Vector). Given a point \(P\) in space, the position vector \(\overrightarrow{OP}\) (or simply \(\boldsymbol{p}\)) is the vector from the origin \(O\) to the point \(P\). This establishes a one-to-one correspondence between points and position vectors.

Theorem 4.4 (Vector Between Two Points). If \(A\) and \(B\) are points with position vectors \(\boldsymbol{a}\) and \(\boldsymbol{b}\) respectively, then the vector from \(A\) to \(B\) is: \[\overrightarrow{AB} = \boldsymbol{b} - \boldsymbol{a}\]

Proof. Using the triangle rule for vector addition: \(\boldsymbol{a} + \overrightarrow{AB} = \boldsymbol{b}\), which gives \(\overrightarrow{AB} = \boldsymbol{b} - \boldsymbol{a}\). ◻

Example 4.5 (Vector Calculations with Points). Let \(A = (1, 2, 3)\), \(B = (4, -1, 2)\), and \(C = (0, 3, -1)\).

Vector from \(A\) to \(B\): \[\overrightarrow{AB} = \boldsymbol{b} - \boldsymbol{a} = \begin{pmatrix} 4 \\ -1 \\ 2 \end{pmatrix} - \begin{pmatrix} 1 \\ 2 \\ 3 \end{pmatrix} = \begin{pmatrix} 3 \\ -3 \\ -1 \end{pmatrix}\]

Magnitude of \(\overrightarrow{AB}\): \[\|\overrightarrow{AB}\| = \sqrt{3^2 + (-3)^2 + (-1)^2} = \sqrt{9 + 9 + 1} = \sqrt{19}\]

Midpoint of segment \(AB\): The midpoint \(M\) has position vector: \[\boldsymbol{m} = \frac{\boldsymbol{a} + \boldsymbol{b}}{2} = \frac{1}{2}\begin{pmatrix} 5 \\ 1 \\ 5 \end{pmatrix} = \begin{pmatrix} 2.5 \\ 0.5 \\ 2.5 \end{pmatrix}\]

Now we use vectors to describe lines in space:

Definition 4.6 (Parametric Equation of a Line). A line passing through point \(P_0\) with position vector \(\boldsymbol{p}_0\) and parallel to direction vector \(\boldsymbol{d}\) has the parametric equation: \[\boldsymbol{r}(t) = \boldsymbol{p}_0 + t\boldsymbol{d}\] where \(t \in \mathbb{R}\) is the parameter, and \(\boldsymbol{r}(t)\) gives the position vector of any point on the line.

Remark 204. The parameter \(t\) has geometric meaning:

• \(t = 0\) corresponds to the point \(P_0\)

• \(t = 1\) corresponds to the point \(P_0 + \boldsymbol{d}\)

• \(t < 0\) corresponds to points "behind" \(P_0\) in the direction opposite to \(\boldsymbol{d}\)

• The distance from \(P_0\) to the point \(\boldsymbol{r}(t)\) is \(|t|\|\boldsymbol{d}\|\)

Example 4.8 (Line Through Two Points). Find the parametric equation of the line through \(A = (1, 2, -1)\) and \(B = (3, 0, 2)\).

Method 1: Use \(A\) as the base point and \(\overrightarrow{AB}\) as the direction: \[\overrightarrow{AB} = \begin{pmatrix} 3 \\ 0 \\ 2 \end{pmatrix} - \begin{pmatrix} 1 \\ 2 \\ -1 \end{pmatrix} = \begin{pmatrix} 2 \\ -2 \\ 3 \end{pmatrix}\]

The line equation is: \[\boldsymbol{r}(t) = \begin{pmatrix} 1 \\ 2 \\ -1 \end{pmatrix} + t\begin{pmatrix} 2 \\ -2 \\ 3 \end{pmatrix} = \begin{pmatrix} 1 + 2t \\ 2 - 2t \\ -1 + 3t \end{pmatrix}\]

Verification: - At \(t = 0\): \(\boldsymbol{r}(0) = (1, 2, -1) = A\) - At \(t = 1\): \(\boldsymbol{r}(1) = (3, 0, 2) = B\)

Method 2: We could also use \(B\) as the base point: \[\boldsymbol{r}(s) = \begin{pmatrix} 3 \\ 0 \\ 2 \end{pmatrix} + s\begin{pmatrix} 2 \\ -2 \\ 3 \end{pmatrix}\]

This is the same line with a different parameterization. The relationship is \(s = t - 1\).

Definition 4.9 (Symmetric Equations of a Line). If the direction vector \(\boldsymbol{d} = (a, b, c)\) has no zero components, the line can also be written in symmetric form: \[\frac{x - x_0}{a} = \frac{y - y_0}{b} = \frac{z - z_0}{c}\] where \((x_0, y_0, z_0)\) is a known point on the line.

Example 4.10 (Converting Between Parametric and Symmetric Forms). Convert the line \(\boldsymbol{r}(t) = (1, 2, -1) + t(2, -2, 3)\) to symmetric form.

From the parametric equations: \[\begin{align} x &= 1 + 2t\\ y &= 2 - 2t\\ z &= -1 + 3t \end{align}\]

Solving for \(t\) in each equation: \[t = \frac{x-1}{2} = \frac{y-2}{-2} = \frac{z+1}{3}\]

Therefore, the symmetric form is: \[\frac{x-1}{2} = \frac{y-2}{-2} = \frac{z+1}{3}\]

Theorem 4.11 (Distance from Point to Line). The distance from a point \(Q\) with position vector \(\boldsymbol{q}\) to a line passing through point \(P\) with direction vector \(\boldsymbol{d}\) is: \[\text{distance} = \frac{\|\overrightarrow{PQ} \times \boldsymbol{d}\|}{\|\boldsymbol{d}\|}\] where \(\times\) denotes the cross product (to be defined later).

Remark 209. This formula has a beautiful geometric interpretation: \(\|\overrightarrow{PQ} \times \boldsymbol{d}\|\) gives the area of the parallelogram formed by vectors \(\overrightarrow{PQ}\) and \(\boldsymbol{d}\), while \(\|\boldsymbol{d}\|\) is the base length. The ratio gives the height, which is precisely the perpendicular distance from \(Q\) to the line.

Theorem 4.13 (Intersection of Two Lines). Two lines in space either:

1. Intersect at exactly one point

2. Are parallel (never intersect)

3. Are skew (do not intersect and are not parallel—this can only happen in 3D)

Example 4.14 (Finding Line Intersection). Determine if these lines intersect: \[L_1: \boldsymbol{r}_1(t) = (1, 2, 0) + t(1, -1, 2)\] \[L_2: \boldsymbol{r}_2(s) = (3, 0, 4) + s(2, 1, -1)\]

For intersection, we need \(\boldsymbol{r}_1(t) = \boldsymbol{r}_2(s)\) for some values of \(t\) and \(s\): \[(1, 2, 0) + t(1, -1, 2) = (3, 0, 4) + s(2, 1, -1)\]

This gives the system: \[\begin{align} 1 + t &= 3 + 2s \quad \Rightarrow \quad t - 2s = 2\\ 2 - t &= s \quad \Rightarrow \quad -t - s = -2\\ 2t &= 4 - s \quad \Rightarrow \quad 2t + s = 4 \end{align}\]

From equations (1) and (2): \(t = 2 + 2s\) and \(t = 2 - s\) Setting equal: \(2 + 2s = 2 - s \Rightarrow 3s = 0 \Rightarrow s = 0\) Thus \(t = 2\).

Check equation (3): \(2(2) + 0 = 4\)

The lines intersect at the point: \[\boldsymbol{r}_1(2) = (1, 2, 0) + 2(1, -1, 2) = (3, 0, 4)\] \[\boldsymbol{r}_2(0) = (3, 0, 4) + 0(2, 1, -1) = (3, 0, 4)\]

4.2 Projections and Planes

Projections decompose vectors into components, while planes extend the concept of lines to two-dimensional surfaces in three-dimensional space.

Definition 4.15 (Scalar Projection). The scalar projection of vector \(\boldsymbol{u}\) onto vector \(\boldsymbol{v}\) is: \[\text{comp}_{\boldsymbol{v}}\boldsymbol{u} = \frac{\boldsymbol{u} \cdot \boldsymbol{v}}{\|\boldsymbol{v}\|}\] This represents the signed length of the projection of \(\boldsymbol{u}\) onto the direction of \(\boldsymbol{v}\).

Definition 4.16 (Vector Projection). The vector projection (or orthogonal projection) of vector \(\boldsymbol{u}\) onto vector \(\boldsymbol{v}\) is: \[\text{proj}_{\boldsymbol{v}}\boldsymbol{u} = \frac{\boldsymbol{u} \cdot \boldsymbol{v}}{\boldsymbol{v} \cdot \boldsymbol{v}}\boldsymbol{v} = \frac{\boldsymbol{u} \cdot \boldsymbol{v}}{\|\boldsymbol{v}\|^2}\boldsymbol{v}\]

Remark 214. The vector projection gives the component of \(\boldsymbol{u}\) in the direction of \(\boldsymbol{v}\), while the scalar projection gives just the magnitude (with sign) of this component. The relationship is: \[\text{proj}_{\boldsymbol{v}}\boldsymbol{u} = \left(\text{comp}_{\boldsymbol{v}}\boldsymbol{u}\right) \cdot \frac{\boldsymbol{v}}{\|\boldsymbol{v}\|}\]

Theorem 4.18 (Properties of Projections). For any vectors \(\boldsymbol{u}\) and \(\boldsymbol{v}\) (with \(\boldsymbol{v} \neq \boldsymbol{0}\)):

1. \(\text{proj}_{\boldsymbol{v}}\boldsymbol{u}\) is parallel to \(\boldsymbol{v}\)

2. \(\boldsymbol{u} - \text{proj}_{\boldsymbol{v}}\boldsymbol{u}\) is orthogonal to \(\boldsymbol{v}\)

3. \(\|\text{proj}_{\boldsymbol{v}}\boldsymbol{u}\| = |\text{comp}_{\boldsymbol{v}}\boldsymbol{u}|\)

4. \(\boldsymbol{u} = \text{proj}_{\boldsymbol{v}}\boldsymbol{u} + (\boldsymbol{u} - \text{proj}_{\boldsymbol{v}}\boldsymbol{u})\) (orthogonal decomposition)

Proof. Property 1: By definition, \(\text{proj}_{\boldsymbol{v}}\boldsymbol{u} = k\boldsymbol{v}\) where \(k = \frac{\boldsymbol{u} \cdot \boldsymbol{v}}{\|\boldsymbol{v}\|^2}\).

Property 2: We compute: \[\begin{align} (\boldsymbol{u} - \text{proj}_{\boldsymbol{v}}\boldsymbol{u}) \cdot \boldsymbol{v} &= \boldsymbol{u} \cdot \boldsymbol{v} - \text{proj}_{\boldsymbol{v}}\boldsymbol{u} \cdot \boldsymbol{v}\\ &= \boldsymbol{u} \cdot \boldsymbol{v} - \frac{\boldsymbol{u} \cdot \boldsymbol{v}}{\|\boldsymbol{v}\|^2}(\boldsymbol{v} \cdot \boldsymbol{v})\\ &= \boldsymbol{u} \cdot \boldsymbol{v} - \boldsymbol{u} \cdot \boldsymbol{v} = 0 \end{align}\]

Properties 3 and 4: Follow directly from the definitions and Property 1. ◻

Example 4.19 (Computing Projections). Let \(\boldsymbol{u} = (3, 1, 2)\) and \(\boldsymbol{v} = (1, 2, -1)\).

Scalar projection: \[\boldsymbol{u} \cdot \boldsymbol{v} = 3(1) + 1(2) + 2(-1) = 3 + 2 - 2 = 3\] \[\|\boldsymbol{v}\| = \sqrt{1^2 + 2^2 + (-1)^2} = \sqrt{6}\] \[\text{comp}_{\boldsymbol{v}}\boldsymbol{u} = \frac{3}{\sqrt{6}} = \frac{3\sqrt{6}}{6} = \frac{\sqrt{6}}{2}\]

Vector projection: \[\text{proj}_{\boldsymbol{v}}\boldsymbol{u} = \frac{3}{6}(1, 2, -1) = \frac{1}{2}(1, 2, -1) = \left(\frac{1}{2}, 1, -\frac{1}{2}\right)\]

Orthogonal component: \[\boldsymbol{u} - \text{proj}_{\boldsymbol{v}}\boldsymbol{u} = (3, 1, 2) - \left(\frac{1}{2}, 1, -\frac{1}{2}\right) = \left(\frac{5}{2}, 0, \frac{5}{2}\right)\]

Verification of orthogonality: \[\left(\frac{5}{2}, 0, \frac{5}{2}\right) \cdot (1, 2, -1) = \frac{5}{2} + 0 - \frac{5}{2} = 0\]

Now we turn to planes in three-dimensional space:

Definition 4.20 (Plane Equation). A plane in \(\mathbb{R}^3\) can be described by:

1. Normal form: \(\boldsymbol{n} \cdot (\boldsymbol{r} - \boldsymbol{r}_0) = 0\), where \(\boldsymbol{n}\) is a normal vector and \(\boldsymbol{r}_0\) is the position vector of a known point on the plane

2. Standard form: \(ax + by + cz = d\), where \((a, b, c)\) is a normal vector

3. Parametric form: \(\boldsymbol{r}(s, t) = \boldsymbol{r}_0 + s\boldsymbol{u} + t\boldsymbol{v}\), where \(\boldsymbol{u}\) and \(\boldsymbol{v}\) are non-parallel vectors in the plane

Theorem 4.21 (Equivalence of Plane Representations). The normal form \(\boldsymbol{n} \cdot (\boldsymbol{r} - \boldsymbol{r}_0) = 0\) expands to the standard form \(ax + by + cz = d\) where:

• \((a, b, c) = \boldsymbol{n}\) is the normal vector

• \(d = \boldsymbol{n} \cdot \boldsymbol{r}_0 = ax_0 + by_0 + cz_0\)

Example 4.22 (Finding the Equation of a Plane). Find the equation of the plane passing through points \(A = (1, 2, 3)\), \(B = (2, 1, 4)\), and \(C = (0, 3, 2)\).

Step 1: Find two vectors in the plane: \[\overrightarrow{AB} = (2, 1, 4) - (1, 2, 3) = (1, -1, 1)\] \[\overrightarrow{AC} = (0, 3, 2) - (1, 2, 3) = (-1, 1, -1)\]

Step 2: Find a normal vector using the cross product: \[\boldsymbol{n} = \overrightarrow{AB} \times \overrightarrow{AC} = \begin{vmatrix} \boldsymbol{i} & \boldsymbol{j} & \boldsymbol{k} \\ 1 & -1 & 1 \\ -1 & 1 & -1 \end{vmatrix}\]

\[= \boldsymbol{i}((-1)(-1) - (1)(1)) - \boldsymbol{j}((1)(-1) - (1)(-1)) + \boldsymbol{k}((1)(1) - (-1)(-1))\] \[= \boldsymbol{i}(1 - 1) - \boldsymbol{j}(-1 + 1) + \boldsymbol{k}(1 - 1) = (0, 0, 0)\]

This indicates the three points are collinear! Let’s verify: \[\overrightarrow{AC} = (-1, 1, -1) = -1 \cdot (1, -1, 1) = -\overrightarrow{AB}\]

Indeed, the points are collinear, so they do not determine a unique plane.

Corrected example: Let’s use \(A = (1, 2, 3)\), \(B = (2, 1, 4)\), and \(C = (0, 3, 1)\).

\[\overrightarrow{AB} = (1, -1, 1)\] \[\overrightarrow{AC} = (-1, 1, -2)\]

\[\boldsymbol{n} = \overrightarrow{AB} \times \overrightarrow{AC} = \begin{vmatrix} \boldsymbol{i} & \boldsymbol{j} & \boldsymbol{k} \\ 1 & -1 & 1 \\ -1 & 1 & -2 \end{vmatrix}\]

\[= \boldsymbol{i}((-1)(-2) - (1)(1)) - \boldsymbol{j}((1)(-2) - (1)(-1)) + \boldsymbol{k}((1)(1) - (-1)(-1))\] \[= \boldsymbol{i}(2 - 1) - \boldsymbol{j}(-2 + 1) + \boldsymbol{k}(1 - 1) = (1, 1, 0)\]

Step 3: Use point \(A\) to find the plane equation: \[1(x - 1) + 1(y - 2) + 0(z - 3) = 0\] \[x - 1 + y - 2 = 0\] \[x + y = 3\]

Verification: - Point \(A\): \(1 + 2 = 3\) - Point \(B\): \(2 + 1 = 3\) - Point \(C\): \(0 + 3 = 3\)

Theorem 4.23 (Distance from Point to Plane). The distance from a point \(P_0 = (x_0, y_0, z_0)\) to the plane \(ax + by + cz + d = 0\) is: \[\text{distance} = \frac{|ax_0 + by_0 + cz_0 + d|}{\sqrt{a^2 + b^2 + c^2}}\]

Proof. Let \(Q\) be any point on the plane and \(\boldsymbol{n} = (a, b, c)\) be the normal vector. The distance is the absolute value of the scalar projection of \(\overrightarrow{QP_0}\) onto the unit normal vector: \[\text{distance} = \left|\text{comp}_{\boldsymbol{n}}\overrightarrow{QP_0}\right| = \frac{|\overrightarrow{QP_0} \cdot \boldsymbol{n}|}{\|\boldsymbol{n}\|}\]

The numerator simplifies to \(|ax_0 + by_0 + cz_0 + d|\) when we use the fact that \(Q\) satisfies the plane equation. ◻

Example 4.24 (Distance from Point to Plane). Find the distance from point \(P = (2, 1, -1)\) to the plane \(3x - 4y + z = 7\).

First, rewrite in standard form: \(3x - 4y + z - 7 = 0\).

Using the distance formula: \[\text{distance} = \frac{|3(2) - 4(1) + 1(-1) - 7|}{\sqrt{3^2 + (-4)^2 + 1^2}} = \frac{|6 - 4 - 1 - 7|}{\sqrt{9 + 16 + 1}} = \frac{|-6|}{\sqrt{26}} = \frac{6}{\sqrt{26}} = \frac{6\sqrt{26}}{26} = \frac{3\sqrt{26}}{13}\]

4.3 Cross Product

The cross product is a binary operation unique to three-dimensional space that produces a vector perpendicular to both input vectors. It has fundamental applications in physics, engineering, and computer graphics.

Definition 4.25 (Cross Product). For vectors \(\boldsymbol{u} = (u_1, u_2, u_3)\) and \(\boldsymbol{v} = (v_1, v_2, v_3)\) in \(\mathbb{R}^3\), their cross product is: \[\boldsymbol{u} \times \boldsymbol{v} = \begin{vmatrix} \boldsymbol{i} & \boldsymbol{j} & \boldsymbol{k} \\ u_1 & u_2 & u_3 \\ v_1 & v_2 & v_3 \end{vmatrix} = (u_2v_3 - u_3v_2, u_3v_1 - u_1v_3, u_1v_2 - u_2v_1)\] where \(\boldsymbol{i}\), \(\boldsymbol{j}\), \(\boldsymbol{k}\) are the standard basis vectors.

Remark 223. The determinant notation is a mnemonic device—we’re not actually computing the determinant of a matrix with vector entries. Instead, we expand along the first row, treating \(\boldsymbol{i}\), \(\boldsymbol{j}\), \(\boldsymbol{k}\) as placeholders that remind us which component we’re computing.

Example 4.27 (Computing Cross Products). Compute \(\boldsymbol{u} \times \boldsymbol{v}\) where \(\boldsymbol{u} = (2, 1, -1)\) and \(\boldsymbol{v} = (1, 3, 2)\).

\[\boldsymbol{u} \times \boldsymbol{v} = \begin{vmatrix} \boldsymbol{i} & \boldsymbol{j} & \boldsymbol{k} \\ 2 & 1 & -1 \\ 1 & 3 & 2 \end{vmatrix}\]

\[= \boldsymbol{i}\begin{vmatrix} 1 & -1 \\ 3 & 2 \end{vmatrix} - \boldsymbol{j}\begin{vmatrix} 2 & -1 \\ 1 & 2 \end{vmatrix} + \boldsymbol{k}\begin{vmatrix} 2 & 1 \\ 1 & 3 \end{vmatrix}\]

\[= \boldsymbol{i}(1 \cdot 2 - (-1) \cdot 3) - \boldsymbol{j}(2 \cdot 2 - (-1) \cdot 1) + \boldsymbol{k}(2 \cdot 3 - 1 \cdot 1)\]

\[= \boldsymbol{i}(2 + 3) - \boldsymbol{j}(4 + 1) + \boldsymbol{k}(6 - 1) = 5\boldsymbol{i} - 5\boldsymbol{j} + 5\boldsymbol{k} = (5, -5, 5)\]

Verification of orthogonality: \[\boldsymbol{u} \cdot (\boldsymbol{u} \times \boldsymbol{v}) = (2, 1, -1) \cdot (5, -5, 5) = 10 - 5 - 5 = 0\] \[\boldsymbol{v} \cdot (\boldsymbol{u} \times \boldsymbol{v}) = (1, 3, 2) \cdot (5, -5, 5) = 5 - 15 + 10 = 0\]

Theorem 4.28 (Properties of Cross Product). For vectors \(\boldsymbol{u}\), \(\boldsymbol{v}\), \(\boldsymbol{w}\) in \(\mathbb{R}^3\) and scalar \(c\):

1. \(\boldsymbol{u} \times \boldsymbol{v} = -(\boldsymbol{v} \times \boldsymbol{u})\) (anti-commutativity)

2. \(\boldsymbol{u} \times (\boldsymbol{v} + \boldsymbol{w}) = \boldsymbol{u} \times \boldsymbol{v} + \boldsymbol{u} \times \boldsymbol{w}\) (distributivity)

3. \((c\boldsymbol{u}) \times \boldsymbol{v} = c(\boldsymbol{u} \times \boldsymbol{v}) = \boldsymbol{u} \times (c\boldsymbol{v})\) (scalar multiplication)

4. \(\boldsymbol{u} \times \boldsymbol{0} = \boldsymbol{0}\) and \(\boldsymbol{u} \times \boldsymbol{u} = \boldsymbol{0}\)

5. \(\boldsymbol{u} \times \boldsymbol{v} = \boldsymbol{0}\) if and only if \(\boldsymbol{u}\) and \(\boldsymbol{v}\) are parallel

6. \((\boldsymbol{u} \times \boldsymbol{v}) \cdot \boldsymbol{u} = 0\) and \((\boldsymbol{u} \times \boldsymbol{v}) \cdot \boldsymbol{v} = 0\) (orthogonality)

Proof. We prove the key properties:

Property 1 (Anti-commutativity): Direct computation shows that swapping the rows in the determinant changes the sign.

Property 5: \(\boldsymbol{u} \times \boldsymbol{v} = \boldsymbol{0}\) if and only if the determinant has linearly dependent rows, which occurs if and only if \(\boldsymbol{u}\) and \(\boldsymbol{v}\) are parallel.

Property 6: This follows from the geometric interpretation that the cross product is perpendicular to both input vectors. ◻

Theorem 4.29 (Geometric Properties of Cross Product). For non-zero vectors \(\boldsymbol{u}\) and \(\boldsymbol{v}\):

1. \(\|\boldsymbol{u} \times \boldsymbol{v}\| = \|\boldsymbol{u}\|\|\boldsymbol{v}\|\sin\theta\), where \(\theta\) is the angle between \(\boldsymbol{u}\) and \(\boldsymbol{v}\)

2. \(\|\boldsymbol{u} \times \boldsymbol{v}\|\) equals the area of the parallelogram spanned by \(\boldsymbol{u}\) and \(\boldsymbol{v}\)

3. The direction of \(\boldsymbol{u} \times \boldsymbol{v}\) follows the right-hand rule

Proof. Property 1: This is a fundamental identity that can be proved using the relationship between cross product and dot product: \[\|\boldsymbol{u} \times \boldsymbol{v}\|^2 = \|\boldsymbol{u}\|^2\|\boldsymbol{v}\|^2 - (\boldsymbol{u} \cdot \boldsymbol{v})^2 = \|\boldsymbol{u}\|^2\|\boldsymbol{v}\|^2(1 - \cos^2\theta) = \|\boldsymbol{u}\|^2\|\boldsymbol{v}\|^2\sin^2\theta\]

Property 2: The area of a parallelogram with adjacent sides \(\boldsymbol{u}\) and \(\boldsymbol{v}\) is base × height = \(\|\boldsymbol{u}\| \cdot \|\boldsymbol{v}\|\sin\theta\), which equals \(\|\boldsymbol{u} \times \boldsymbol{v}\|\) by Property 1. ◻

Example 4.30 (Area Using Cross Product). Find the area of the triangle with vertices \(A = (1, 0, 2)\), \(B = (3, 1, 0)\), and \(C = (2, -1, 3)\).

Step 1: Find two sides of the triangle: \[\overrightarrow{AB} = (3, 1, 0) - (1, 0, 2) = (2, 1, -2)\] \[\overrightarrow{AC} = (2, -1, 3) - (1, 0, 2) = (1, -1, 1)\]

Step 2: Compute the cross product: \[\overrightarrow{AB} \times \overrightarrow{AC} = \begin{vmatrix} \boldsymbol{i} & \boldsymbol{j} & \boldsymbol{k} \\ 2 & 1 & -2 \\ 1 & -1 & 1 \end{vmatrix}\]

\[= \boldsymbol{i}(1 \cdot 1 - (-2)(-1)) - \boldsymbol{j}(2 \cdot 1 - (-2)(1)) + \boldsymbol{k}(2(-1) - 1 \cdot 1)\] \[= \boldsymbol{i}(1 - 2) - \boldsymbol{j}(2 + 2) + \boldsymbol{k}(-2 - 1) = (-1, -4, -3)\]

Step 3: Find the magnitude: \[\|\overrightarrow{AB} \times \overrightarrow{AC}\| = \sqrt{(-1)^2 + (-4)^2 + (-3)^2} = \sqrt{1 + 16 + 9} = \sqrt{26}\]

Step 4: The triangle area is half the parallelogram area: \[\text{Area} = \frac{1}{2}\|\overrightarrow{AB} \times \overrightarrow{AC}\| = \frac{\sqrt{26}}{2}\]

Definition 4.31 (Scalar Triple Product). The scalar triple product of vectors \(\boldsymbol{u}\), \(\boldsymbol{v}\), and \(\boldsymbol{w}\) is: \[\boldsymbol{u} \cdot (\boldsymbol{v} \times \boldsymbol{w}) = \begin{vmatrix} u_1 & u_2 & u_3 \\ v_1 & v_2 & v_3 \\ w_1 & w_2 & w_3 \end{vmatrix}\]

Theorem 4.32 (Geometric Meaning of Scalar Triple Product). The absolute value of the scalar triple product \(|\boldsymbol{u} \cdot (\boldsymbol{v} \times \boldsymbol{w})|\) equals the volume of the parallelepiped (3D parallelogram) spanned by vectors \(\boldsymbol{u}\), \(\boldsymbol{v}\), and \(\boldsymbol{w}\).

Proof. The volume of a parallelepiped is base area × height. The base area is \(\|\boldsymbol{v} \times \boldsymbol{w}\|\), and the height is the absolute value of the scalar projection of \(\boldsymbol{u}\) onto \(\boldsymbol{v} \times \boldsymbol{w}\): \[\text{height} = \left|\text{comp}_{\boldsymbol{v} \times \boldsymbol{w}}\boldsymbol{u}\right| = \frac{|\boldsymbol{u} \cdot (\boldsymbol{v} \times \boldsymbol{w})|}{\|\boldsymbol{v} \times \boldsymbol{w}\|}\]

Therefore: \[\text{Volume} = \|\boldsymbol{v} \times \boldsymbol{w}\| \cdot \frac{|\boldsymbol{u} \cdot (\boldsymbol{v} \times \boldsymbol{w})|}{\|\boldsymbol{v} \times \boldsymbol{w}\|} = |\boldsymbol{u} \cdot (\boldsymbol{v} \times \boldsymbol{w})|\] ◻

Example 4.33 (Volume Using Scalar Triple Product). Find the volume of the parallelepiped spanned by \(\boldsymbol{u} = (1, 2, 3)\), \(\boldsymbol{v} = (2, 1, 0)\), and \(\boldsymbol{w} = (1, 1, 1)\).

\[\text{Volume} = \left|\begin{vmatrix} 1 & 2 & 3 \\ 2 & 1 & 0 \\ 1 & 1 & 1 \end{vmatrix}\right|\]

Expanding along the third row: \[= \left|1 \cdot \begin{vmatrix} 2 & 3 \\ 1 & 0 \end{vmatrix} - 1 \cdot \begin{vmatrix} 1 & 3 \\ 2 & 0 \end{vmatrix} + 1 \cdot \begin{vmatrix} 1 & 2 \\ 2 & 1 \end{vmatrix}\right|\]

\[= |1(0 - 3) - 1(0 - 6) + 1(1 - 4)| = |-3 + 6 - 3| = |0| = 0\]

The volume is 0, which means the three vectors are coplanar (they all lie in the same plane).

4.4 Operators in \(\mathbb{R}^3\)

Linear operators in three-dimensional space have rich geometric interpretations involving rotations, reflections, and other transformations. We explore the most important classes of operators and their properties.

Definition 4.34 (Rotation Matrix). A rotation matrix in \(\mathbb{R}^3\) is an orthogonal matrix with determinant 1. It represents a rotation about an axis through the origin.

Theorem 4.35 (Properties of Rotation Matrices). If \(\mathbf{R}\) is a rotation matrix, then:

1. \(\mathbf{R}^{\mathsf{T}}\mathbf{R} = \mathbf{I}\) (orthogonality)

2. \(\det(\mathbf{R}) = 1\) (orientation preserving)

3. \(\|\mathbf{R}\boldsymbol{v}\| = \|\boldsymbol{v}\|\) for all \(\boldsymbol{v}\) (length preserving)

4. \((\mathbf{R}\boldsymbol{u}) \cdot (\mathbf{R}\boldsymbol{v}) = \boldsymbol{u} \cdot \boldsymbol{v}\) for all \(\boldsymbol{u}, \boldsymbol{v}\) (angle preserving)

Example 4.36 (Rotation About Coordinate Axes). Rotation by angle \(\theta\) about the \(z\)-axis: \[\mathbf{R}_z(\theta) = \begin{pmatrix} \cos\theta & -\sin\theta & 0 \\ \sin\theta & \cos\theta & 0 \\ 0 & 0 & 1 \end{pmatrix}\]

This rotates points in the \(xy\)-plane while leaving the \(z\)-coordinate unchanged.

Rotation by angle \(\theta\) about the \(x\)-axis: \[\mathbf{R}_x(\theta) = \begin{pmatrix} 1 & 0 & 0 \\ 0 & \cos\theta & -\sin\theta \\ 0 & \sin\theta & \cos\theta \end{pmatrix}\]

Rotation by angle \(\theta\) about the \(y\)-axis: \[\mathbf{R}_y(\theta) = \begin{pmatrix} \cos\theta & 0 & \sin\theta \\ 0 & 1 & 0 \\ -\sin\theta & 0 & \cos\theta \end{pmatrix}\]

Note the sign pattern difference in \(\mathbf{R}_y\)—this maintains the right-hand rule convention.

Theorem 4.37 (Rodrigues’ Rotation Formula). Rotation by angle \(\theta\) about a unit vector \(\boldsymbol{n} = (n_x, n_y, n_z)\) is given by: \[\mathbf{R} = \mathbf{I} + (\sin\theta)[\boldsymbol{n}]_{\times} + (1-\cos\theta)[\boldsymbol{n}]_{\times}^2\] where \([\boldsymbol{n}]_{\times}\) is the skew-symmetric matrix: \[[\boldsymbol{n}]_{\times} = \begin{pmatrix} 0 & -n_z & n_y \\ n_z & 0 & -n_x \\ -n_y & n_x & 0 \end{pmatrix}\]

Remark 235. The matrix \([\boldsymbol{n}]_{\times}\) satisfies \([\boldsymbol{n}]_{\times}\boldsymbol{v} = \boldsymbol{n} \times \boldsymbol{v}\) for any vector \(\boldsymbol{v}\). This connection between cross products and matrix multiplication is fundamental in 3D geometry.

Example 4.39 (Rotation About Arbitrary Axis). Find the matrix for rotation by \(90°\) about the axis \(\boldsymbol{n} = \frac{1}{\sqrt{3}}(1, 1, 1)\).

First, we form the skew-symmetric matrix: \[[\boldsymbol{n}]_{\times} = \frac{1}{\sqrt{3}}\begin{pmatrix} 0 & -1 & 1 \\ 1 & 0 & -1 \\ -1 & 1 & 0 \end{pmatrix}\]

For \(\theta = 90°\): \(\sin\theta = 1\) and \(\cos\theta = 0\), so: \[\mathbf{R} = \mathbf{I} + [\boldsymbol{n}]_{\times} + [\boldsymbol{n}]_{\times}^2\]

Computing \([\boldsymbol{n}]_{\times}^2\): \[[\boldsymbol{n}]_{\times}^2 = \frac{1}{3}\begin{pmatrix} 0 & -1 & 1 \\ 1 & 0 & -1 \\ -1 & 1 & 0 \end{pmatrix}^2 = \frac{1}{3}\begin{pmatrix} -2 & 1 & 1 \\ 1 & -2 & 1 \\ 1 & 1 & -2 \end{pmatrix}\]

Therefore: \[\mathbf{R} = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{pmatrix} + \frac{1}{\sqrt{3}}\begin{pmatrix} 0 & -1 & 1 \\ 1 & 0 & -1 \\ -1 & 1 & 0 \end{pmatrix} + \frac{1}{3}\begin{pmatrix} -2 & 1 & 1 \\ 1 & -2 & 1 \\ 1 & 1 & -2 \end{pmatrix}\]

\[= \begin{pmatrix} \frac{1}{3} & \frac{1-\sqrt{3}}{3} & \frac{1+\sqrt{3}}{3} \\ \frac{1+\sqrt{3}}{3} & \frac{1}{3} & \frac{1-\sqrt{3}}{3} \\ \frac{1-\sqrt{3}}{3} & \frac{1+\sqrt{3}}{3} & \frac{1}{3} \end{pmatrix}\]

Definition 4.40 (Reflection Matrix). A reflection matrix represents reflection across a plane through the origin. For reflection across a plane with unit normal vector \(\boldsymbol{n}\): \[\mathbf{F} = \mathbf{I} - 2\boldsymbol{n}\boldsymbol{n}^{\mathsf{T}}\]

Theorem 4.41 (Properties of Reflection Matrices). If \(\mathbf{F}\) is a reflection matrix, then:

1. \(\mathbf{F}^2 = \mathbf{I}\) (involutory: applying twice returns to original)

2. \(\mathbf{F}^{\mathsf{T}} = \mathbf{F}\) (symmetric)

3. \(\det(\mathbf{F}) = -1\) (orientation reversing)

4. \(\|\mathbf{F}\boldsymbol{v}\| = \|\boldsymbol{v}\|\) for all \(\boldsymbol{v}\) (length preserving)

Example 4.42 (Reflection Across a Plane). Find the matrix for reflection across the plane \(x + y + z = 0\).

The unit normal vector is \(\boldsymbol{n} = \frac{1}{\sqrt{3}}(1, 1, 1)\).

\[\boldsymbol{n}\boldsymbol{n}^{\mathsf{T}} = \frac{1}{3}\begin{pmatrix} 1 \\ 1 \\ 1 \end{pmatrix}\begin{pmatrix} 1 & 1 & 1 \end{pmatrix} = \frac{1}{3}\begin{pmatrix} 1 & 1 & 1 \\ 1 & 1 & 1 \\ 1 & 1 & 1 \end{pmatrix}\]

Therefore: \[\mathbf{F} = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{pmatrix} - \frac{2}{3}\begin{pmatrix} 1 & 1 & 1 \\ 1 & 1 & 1 \\ 1 & 1 & 1 \end{pmatrix} = \begin{pmatrix} \frac{1}{3} & -\frac{2}{3} & -\frac{2}{3} \\ -\frac{2}{3} & \frac{1}{3} & -\frac{2}{3} \\ -\frac{2}{3} & -\frac{2}{3} & \frac{1}{3} \end{pmatrix}\]

Verification: The vector \((1, 1, 1)\) should map to its negative: \[\mathbf{F}\begin{pmatrix} 1 \\ 1 \\ 1 \end{pmatrix} = \frac{1}{3}\begin{pmatrix} 1 - 2 - 2 \\ -2 + 1 - 2 \\ -2 - 2 + 1 \end{pmatrix} = \frac{1}{3}\begin{pmatrix} -3 \\ -3 \\ -3 \end{pmatrix} = \begin{pmatrix} -1 \\ -1 \\ -1 \end{pmatrix}\]

Definition 4.43 (Orthogonal Projection Matrix). The matrix for orthogonal projection onto a subspace spanned by orthonormal vectors \(\boldsymbol{u}_1, \boldsymbol{u}_2, \ldots, \boldsymbol{u}_k\) is: \[\mathbf{P} = \boldsymbol{u}_1\boldsymbol{u}_1^{\mathsf{T}} + \boldsymbol{u}_2\boldsymbol{u}_2^{\mathsf{T}} + \cdots + \boldsymbol{u}_k\boldsymbol{u}_k^{\mathsf{T}}\]

Theorem 4.44 (Properties of Projection Matrices). If \(\mathbf{P}\) is an orthogonal projection matrix, then:

1. \(\mathbf{P}^2 = \mathbf{P}\) (idempotent)

2. \(\mathbf{P}^{\mathsf{T}} = \mathbf{P}\) (symmetric)

3. \(0 \leq \|\mathbf{P}\boldsymbol{v}\| \leq \|\boldsymbol{v}\|\) for all \(\boldsymbol{v}\) (contractive)

4.5 Application to Computer Graphics

Computer graphics relies heavily on vector geometry for 3D modeling, rendering, and animation. We explore key applications including transformations, lighting calculations, and 3D rendering pipelines.

Definition 4.45 (Homogeneous Coordinates). In computer graphics, 3D points are often represented using homogeneous coordinates as 4-dimensional vectors: \[\begin{pmatrix} x \\ y \\ z \\ w \end{pmatrix}\] where the 3D point is \((x/w, y/w, z/w)\) when \(w \neq 0\). This allows both points (\(w = 1\)) and directions (\(w = 0\)) to be represented uniformly.

Remark 243. Homogeneous coordinates enable translations to be represented as matrix multiplications, unifying all affine transformations (rotations, translations, scaling, shearing) into a single framework.

Definition 4.47 (Transformation Matrices in Computer Graphics). Common 4×4 transformation matrices include:

Translation by vector \((t_x, t_y, t_z)\): \[\mathbf{T} = \begin{pmatrix} 1 & 0 & 0 & t_x \\ 0 & 1 & 0 & t_y \\ 0 & 0 & 1 & t_z \\ 0 & 0 & 0 & 1 \end{pmatrix}\]

Uniform scaling by factor \(s\): \[\mathbf{S} = \begin{pmatrix} s & 0 & 0 & 0 \\ 0 & s & 0 & 0 \\ 0 & 0 & s & 0 \\ 0 & 0 & 0 & 1 \end{pmatrix}\]

Rotation (embedding 3×3 rotation matrix): \[\mathbf{R} = \begin{pmatrix} & & & 0 \\ & \mathbf{R}_{3 \times 3} & & 0 \\ & & & 0 \\ 0 & 0 & 0 & 1 \end{pmatrix}\]

Example 4.48 (Composite Transformations). To rotate an object about a point \(P\) that is not the origin:

1. Translate so that \(P\) moves to the origin: \(\mathbf{T}_{-P}\)

2. Apply the rotation: \(\mathbf{R}\)

3. Translate back: \(\mathbf{T}_P\)

The composite transformation is: \(\mathbf{M} = \mathbf{T}_P \mathbf{R} \mathbf{T}_{-P}\)

For rotation by \(45°\) about the \(z\)-axis around point \((2, 3, 0)\): \[\mathbf{T}_{-P} = \begin{pmatrix} 1 & 0 & 0 & -2 \\ 0 & 1 & 0 & -3 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{pmatrix}\]

\[\mathbf{R} = \begin{pmatrix} \frac{\sqrt{2}}{2} & -\frac{\sqrt{2}}{2} & 0 & 0 \\ \frac{\sqrt{2}}{2} & \frac{\sqrt{2}}{2} & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{pmatrix}\]

\[\mathbf{T}_P = \begin{pmatrix} 1 & 0 & 0 & 2 \\ 0 & 1 & 0 & 3 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{pmatrix}\]

The final transformation matrix \(\mathbf{M} = \mathbf{T}_P \mathbf{R} \mathbf{T}_{-P}\) can be computed by matrix multiplication.

Definition 4.49 (Surface Normals and Lighting). In 3D graphics, lighting calculations depend on surface normals—vectors perpendicular to surfaces at each point. For a triangle with vertices \(A\), \(B\), \(C\), the normal vector is: \[\boldsymbol{n} = \frac{\overrightarrow{AB} \times \overrightarrow{AC}}{\|\overrightarrow{AB} \times \overrightarrow{AC}\|}\]

Theorem 4.50 (Lambert’s Law (Diffuse Lighting)). The intensity of diffuse reflection from a surface is proportional to the cosine of the angle between the surface normal \(\boldsymbol{n}\) and the light direction \(\boldsymbol{l}\): \[I = I_0 \max(0, \boldsymbol{n} \cdot \boldsymbol{l})\] where \(I_0\) is the light intensity and both \(\boldsymbol{n}\) and \(\boldsymbol{l}\) are unit vectors.

Example 4.51 (Lighting Calculation). Consider a triangular face with normal vector \(\boldsymbol{n} = (0, 0, 1)\) (pointing upward) and a light source in direction \(\boldsymbol{l} = \frac{1}{\sqrt{2}}(1, 0, 1)\) (45° from vertical).

The diffuse lighting intensity is: \[I = I_0 \max(0, \boldsymbol{n} \cdot \boldsymbol{l}) = I_0 \max\left(0, (0, 0, 1) \cdot \frac{1}{\sqrt{2}}(1, 0, 1)\right) = I_0 \max\left(0, \frac{1}{\sqrt{2}}\right) = \frac{I_0}{\sqrt{2}}\]

The surface receives \(\frac{1}{\sqrt{2}} \approx 0.707\) times the maximum possible light intensity.

Definition 4.52 (View and Projection Transformations). Computer graphics uses several coordinate systems:

• World coordinates: Global 3D coordinate system

• View coordinates: Coordinate system relative to the camera/observer

• Projection coordinates: 2D coordinates after perspective projection

The view transformation positions the camera, while the projection transformation maps 3D points to 2D screen coordinates.

Theorem 4.53 (Perspective Projection). Perspective projection from 3D point \((x, y, z)\) to 2D screen coordinates \((u, v)\) is given by: \[u = f \frac{x}{z}, \quad v = f \frac{y}{z}\] where \(f\) is the focal length and \(z > 0\) (points in front of the camera).

Remark 251. This formula explains why distant objects appear smaller: as \(z\) increases, both \(u\) and \(v\) decrease proportionally. The division by \(z\) creates the perspective effect that gives depth to 2D images.

Example 4.55 (Ray Tracing Applications). In ray tracing, we need to find intersections between rays and geometric objects.

Ray-sphere intersection: A ray from point \(\boldsymbol{p}\) in direction \(\boldsymbol{d}\) has parametric form \(\boldsymbol{r}(t) = \boldsymbol{p} + t\boldsymbol{d}\). A sphere centered at \(\boldsymbol{c}\) with radius \(R\) satisfies \(\|\boldsymbol{x} - \boldsymbol{c}\|^2 = R^2\).

Substituting the ray equation: \[\|\boldsymbol{p} + t\boldsymbol{d} - \boldsymbol{c}\|^2 = R^2\]

Expanding: \[(\boldsymbol{p} - \boldsymbol{c}) \cdot (\boldsymbol{p} - \boldsymbol{c}) + 2t\boldsymbol{d} \cdot (\boldsymbol{p} - \boldsymbol{c}) + t^2\boldsymbol{d} \cdot \boldsymbol{d} = R^2\]

This is a quadratic equation in \(t\): \[(\boldsymbol{d} \cdot \boldsymbol{d})t^2 + 2\boldsymbol{d} \cdot (\boldsymbol{p} - \boldsymbol{c})t + [(\boldsymbol{p} - \boldsymbol{c}) \cdot (\boldsymbol{p} - \boldsymbol{c}) - R^2] = 0\]

The discriminant determines the intersection:

• Discriminant > 0: Two intersections (ray passes through sphere)

• Discriminant = 0: One intersection (ray tangent to sphere)

• Discriminant < 0: No intersection (ray misses sphere)

Key Idea

Vector Geometry in Computer Graphics:

• Transformations use matrix multiplication to efficiently apply complex geometric operations

• Lighting models rely on dot products to calculate angles between surface normals and light directions

• Cross products compute surface normals essential for lighting and collision detection

• Projection transformations map 3D worlds to 2D screens using perspective division

• Ray-object intersection algorithms use parametric line equations and geometric properties

The mathematical tools of vector geometry provide the foundation for modern 3D graphics, enabling everything from simple transformations to sophisticated rendering techniques.

This completes our comprehensive study of vector geometry. We have established the geometric interpretation of vector operations, developed tools for describing lines and planes, explored the cross product and its applications, investigated linear operators in three dimensions, and demonstrated how these concepts enable modern computer graphics. The synthesis of algebraic vector operations with geometric intuition provides powerful tools for spatial reasoning and practical applications across many fields.

5 The Vector Space \(\mathbb{R}^n\)

The vector space \(\mathbb{R}^n\) serves as the concrete foundation upon which much of linear algebra is built. While we have explored abstract vector spaces, \(\mathbb{R}^n\) provides the most intuitive and computationally accessible setting for understanding fundamental concepts like subspaces, linear independence, orthogonality, and approximation. This chapter develops the theory of \(\mathbb{R}^n\) in depth, establishing tools that will prove essential for applications ranging from data analysis to optimization theory.

5.1 Subspaces and Spanning Sets

We begin by examining the subspace structure of \(\mathbb{R}^n\), building on our earlier introduction to these concepts while providing more geometric insight and computational techniques.

Definition 5.1 (Subspace of \(\mathbb{R}^n\)). A subset \(H\) of \(\mathbb{R}^n\) is a subspace if it satisfies:

1. \(\boldsymbol{0} \in H\) (contains the zero vector)

2. If \(\boldsymbol{u}, \boldsymbol{v} \in H\), then \(\boldsymbol{u} + \boldsymbol{v} \in H\) (closed under addition)

3. If \(\boldsymbol{v} \in H\) and \(c \in \mathbb{R}\), then \(c\boldsymbol{v} \in H\) (closed under scalar multiplication)

Remark 254. These three conditions ensure that \(H\) inherits the vector space structure from \(\mathbb{R}^n\). The first condition could be omitted since it follows from the third (taking \(c = 0\)), but we include it for clarity. Geometrically, subspaces of \(\mathbb{R}^n\) have a very specific structure: they are either \(\{\boldsymbol{0}\}\), lines through the origin, planes through the origin, or higher-dimensional analogues—all passing through the origin.

Theorem 5.3 (Geometric Classification of Subspaces in Low Dimensions). In \(\mathbb{R}^2\), the subspaces are:

• \(\{\boldsymbol{0}\}\) (the trivial subspace)

• Lines through the origin

• \(\mathbb{R}^2\) itself

In \(\mathbb{R}^3\), the subspaces are:

• \(\{\boldsymbol{0}\}\) (the trivial subspace)

• Lines through the origin

• Planes through the origin

• \(\mathbb{R}^3\) itself

Proof. We prove the \(\mathbb{R}^2\) case; the \(\mathbb{R}^3\) case follows similarly.

Let \(H\) be a subspace of \(\mathbb{R}^2\). If \(H = \{\boldsymbol{0}\}\), we’re done. Otherwise, there exists a non-zero vector \(\boldsymbol{v} \in H\). Since \(H\) is closed under scalar multiplication, \(\operatorname{span}\{\boldsymbol{v}\} \subseteq H\).

If \(H = \operatorname{span}\{\boldsymbol{v}\}\), then \(H\) is a line through the origin. Otherwise, there exists \(\boldsymbol{w} \in H\) with \(\boldsymbol{w} \notin \operatorname{span}\{\boldsymbol{v}\}\). Since \(\boldsymbol{v}\) and \(\boldsymbol{w}\) are linearly independent vectors in \(\mathbb{R}^2\), they span all of \(\mathbb{R}^2\). Since \(H\) is closed under linear combinations, \(\mathbb{R}^2 = \operatorname{span}\{\boldsymbol{v}, \boldsymbol{w}\} \subseteq H \subseteq \mathbb{R}^2\), so \(H = \mathbb{R}^2\). ◻

Example 5.4 (Identifying Subspaces). 1. Lines through the origin: \(H_1 = \{(t, 2t) : t \in \mathbb{R}\} = \operatorname{span}\{(1, 2)\}\) is a subspace of \(\mathbb{R}^2\).

2. Planes through the origin: \(H_2 = \{(x, y, z) : x + y - z = 0\}\) is a subspace of \(\mathbb{R}^3\).

To verify: if \((x_1, y_1, z_1)\) and \((x_2, y_2, z_2)\) satisfy \(x_i + y_i - z_i = 0\), then: \[(x_1 + x_2) + (y_1 + y_2) - (z_1 + z_2) = (x_1 + y_1 - z_1) + (x_2 + y_2 - z_2) = 0 + 0 = 0\]

3. Non-example: \(S = \{(x, y) : x + y = 1\}\) is NOT a subspace because:

• It doesn’t contain \(\boldsymbol{0}\): \((0, 0)\) doesn’t satisfy \(0 + 0 = 1\)

• It’s not closed under addition: \((1, 0)\) and \((0, 1)\) are in \(S\), but \((1, 1)\) is not since \(1 + 1 \neq 1\)

This is a line not passing through the origin, which cannot be a subspace.

Definition 5.5 (Span in \(\mathbb{R}^n\)). Given vectors \(\boldsymbol{v}_1, \boldsymbol{v}_2, \ldots, \boldsymbol{v}_k \in \mathbb{R}^n\), their span is: \[\operatorname{span}\{\boldsymbol{v}_1, \boldsymbol{v}_2, \ldots, \boldsymbol{v}_k\} = \left\{c_1\boldsymbol{v}_1 + c_2\boldsymbol{v}_2 + \cdots + c_k\boldsymbol{v}_k : c_i \in \mathbb{R}\right\}\]

Theorem 5.6 (Span is the Smallest Subspace). For any set of vectors \(\{\boldsymbol{v}_1, \boldsymbol{v}_2, \ldots, \boldsymbol{v}_k\} \subset \mathbb{R}^n\):

1. \(\operatorname{span}\{\boldsymbol{v}_1, \boldsymbol{v}_2, \ldots, \boldsymbol{v}_k\}\) is a subspace of \(\mathbb{R}^n\)

2. It is the smallest subspace containing all the vectors \(\boldsymbol{v}_1, \boldsymbol{v}_2, \ldots, \boldsymbol{v}_k\)

Proof. Part 1: We verify the subspace conditions:

• Zero vector: \(\boldsymbol{0} = 0\boldsymbol{v}_1 + 0\boldsymbol{v}_2 + \cdots + 0\boldsymbol{v}_k \in \operatorname{span}\{\boldsymbol{v}_1, \ldots, \boldsymbol{v}_k\}\)

• Closure under addition: If \(\boldsymbol{u} = \sum a_i\boldsymbol{v}_i\) and \(\boldsymbol{w} = \sum b_i\boldsymbol{v}_i\), then \(\boldsymbol{u} + \boldsymbol{w} = \sum (a_i + b_i)\boldsymbol{v}_i\)

• Closure under scalar multiplication: If \(\boldsymbol{u} = \sum a_i\boldsymbol{v}_i\), then \(c\boldsymbol{u} = \sum (ca_i)\boldsymbol{v}_i\)

Part 2: Any subspace \(H\) containing \(\boldsymbol{v}_1, \ldots, \boldsymbol{v}_k\) must be closed under linear combinations, so it must contain \(\operatorname{span}\{\boldsymbol{v}_1, \ldots, \boldsymbol{v}_k\}\). ◻

Example 5.7 (Computing Spans). 1. Two vectors in \(\mathbb{R}^3\): Let \(\boldsymbol{v}_1 = (1, 0, 1)\) and \(\boldsymbol{v}_2 = (0, 1, 1)\).

\(\operatorname{span}\{\boldsymbol{v}_1, \boldsymbol{v}_2\} = \{s(1, 0, 1) + t(0, 1, 1) : s, t \in \mathbb{R}\} = \{(s, t, s + t) : s, t \in \mathbb{R}\}\)

This describes a plane through the origin with normal vector \(\boldsymbol{n}\) satisfying \(\boldsymbol{n} \cdot \boldsymbol{v}_1 = 0\) and \(\boldsymbol{n} \cdot \boldsymbol{v}_2 = 0\).

We can find \(\boldsymbol{n} = \boldsymbol{v}_1 \times \boldsymbol{v}_2 = (1, 0, 1) \times (0, 1, 1) = (-1, -1, 1)\).

Indeed, the plane has equation \(-x - y + z = 0\), or \(x + y - z = 0\).

2. Three vectors in \(\mathbb{R}^3\): Let \(\boldsymbol{v}_1 = (1, 2, 3)\), \(\boldsymbol{v}_2 = (2, 4, 6)\), and \(\boldsymbol{v}_3 = (1, 1, 1)\).

Since \(\boldsymbol{v}_2 = 2\boldsymbol{v}_1\), we have \(\operatorname{span}\{\boldsymbol{v}_1, \boldsymbol{v}_2, \boldsymbol{v}_3\} = \operatorname{span}\{\boldsymbol{v}_1, \boldsymbol{v}_3\}\).

To determine if this spans all of \(\mathbb{R}^3\), we check if \(\boldsymbol{v}_1\) and \(\boldsymbol{v}_3\) are linearly independent: \[c_1(1, 2, 3) + c_2(1, 1, 1) = (0, 0, 0)\] gives us \(c_1 + c_2 = 0\), \(2c_1 + c_2 = 0\), \(3c_1 + c_2 = 0\).

From the first two equations: \(c_1 = 0\) and \(c_2 = 0\). This satisfies the third equation, so the vectors are linearly independent.

Therefore, \(\operatorname{span}\{\boldsymbol{v}_1, \boldsymbol{v}_3\}\) is a 2-dimensional subspace (a plane) in \(\mathbb{R}^3\).

Definition 5.8 (Column Space and Null Space). For an \(m \times n\) matrix \(\mathbf{A}\):

• The column space is \(\text{Col}(\mathbf{A}) = \operatorname{span}\{\boldsymbol{a}_1, \boldsymbol{a}_2, \ldots, \boldsymbol{a}_n\}\) where \(\boldsymbol{a}_i\) are the columns of \(\mathbf{A}\)

• The null space is \(\text{Nul}(\mathbf{A}) = \{\boldsymbol{x} \in \mathbb{R}^n : \mathbf{A}\boldsymbol{x} = \boldsymbol{0}\}\)

Theorem 5.9 (Column Space and Null Space are Subspaces). For any \(m \times n\) matrix \(\mathbf{A}\):

1. \(\text{Col}(\mathbf{A})\) is a subspace of \(\mathbb{R}^m\)

2. \(\text{Nul}(\mathbf{A})\) is a subspace of \(\mathbb{R}^n\)

Proof. Column space: This follows immediately since the column space is the span of the columns, and we’ve shown spans are subspaces.

Null space: We verify the subspace conditions:

• Zero vector: \(\mathbf{A}\boldsymbol{0} = \boldsymbol{0}\), so \(\boldsymbol{0} \in \text{Nul}(\mathbf{A})\)

• Closure under addition: If \(\mathbf{A}\boldsymbol{u} = \boldsymbol{0}\) and \(\mathbf{A}\boldsymbol{v} = \boldsymbol{0}\), then \(\mathbf{A}(\boldsymbol{u} + \boldsymbol{v}) = \mathbf{A}\boldsymbol{u} + \mathbf{A}\boldsymbol{v} = \boldsymbol{0} + \boldsymbol{0} = \boldsymbol{0}\)

• Closure under scalar multiplication: If \(\mathbf{A}\boldsymbol{u} = \boldsymbol{0}\), then \(\mathbf{A}(c\boldsymbol{u}) = c(\mathbf{A}\boldsymbol{u}) = c\boldsymbol{0} = \boldsymbol{0}\)

 ◻

Example 5.10 (Finding Column and Null Spaces). Consider \(\mathbf{A} = \begin{pmatrix} 1 & 2 & 3 \\ 2 & 4 & 6 \\ 1 & 3 & 4 \end{pmatrix}\).

Column space: The columns are \((1, 2, 1)^T\), \((2, 4, 3)^T\), and \((3, 6, 4)^T\).

Notice that the second column is not a multiple of the first, so these two span a plane. Let’s check if the third column is in this span: \[(3, 6, 4) = a(1, 2, 1) + b(2, 4, 3)\] gives us \(a + 2b = 3\), \(2a + 4b = 6\), \(a + 3b = 4\).

From the first equation: \(a = 3 - 2b\). Substituting into the third: \((3 - 2b) + 3b = 4\), so \(b = 1\) and \(a = 1\). Check the second: \(2(1) + 4(1) = 6\)

So the third column is redundant: \(\text{Col}(\mathbf{A}) = \operatorname{span}\{(1, 2, 1)^T, (2, 4, 3)^T\}\).

Null space: Solve \(\mathbf{A}\boldsymbol{x} = \boldsymbol{0}\): \[\begin{pmatrix} 1 & 2 & 3 \\ 2 & 4 & 6 \\ 1 & 3 & 4 \end{pmatrix} \begin{pmatrix} x_1 \\ x_2 \\ x_3 \end{pmatrix} = \begin{pmatrix} 0 \\ 0 \\ 0 \end{pmatrix}\]

Row reducing: \[\begin{pmatrix} 1 & 2 & 3 \\ 2 & 4 & 6 \\ 1 & 3 & 4 \end{pmatrix} \sim \begin{pmatrix} 1 & 0 & 1 \\ 0 & 1 & 1 \\ 0 & 0 & 0 \end{pmatrix}\]

This gives \(x_1 + x_3 = 0\) and \(x_2 + x_3 = 0\), so \(x_1 = x_2 = -x_3\).

Setting \(x_3 = t\): \(\text{Nul}(\mathbf{A}) = \{(-t, -t, t) : t \in \mathbb{R}\} = \operatorname{span}\{(-1, -1, 1)^T\}\).

5.2 Independence and Dimension

Linear independence and dimension are fundamental concepts that allow us to measure the "size" of subspaces and understand their structure precisely.

Definition 5.11 (Linear Independence in \(\mathbb{R}^n\)). Vectors \(\boldsymbol{v}_1, \boldsymbol{v}_2, \ldots, \boldsymbol{v}_k \in \mathbb{R}^n\) are linearly independent if the only solution to: \[c_1\boldsymbol{v}_1 + c_2\boldsymbol{v}_2 + \cdots + c_k\boldsymbol{v}_k = \boldsymbol{0}\] is \(c_1 = c_2 = \cdots = c_k = 0\).

If there exist scalars \(c_1, c_2, \ldots, c_k\), not all zero, such that the linear combination equals \(\boldsymbol{0}\), then the vectors are linearly dependent.

Theorem 5.12 (Characterizations of Linear Dependence). Vectors \(\boldsymbol{v}_1, \boldsymbol{v}_2, \ldots, \boldsymbol{v}_k \in \mathbb{R}^n\) are linearly dependent if and only if one of the following equivalent conditions holds:

1. One of the vectors can be written as a linear combination of the others

2. One of the vectors is in the span of the preceding vectors

3. The vectors do not all contribute to the span (removing one doesn’t change the span)

Proof. We show equivalence by proving the cycle: dependence \(\Rightarrow\) (1) \(\Rightarrow\) dependence.

Dependence \(\Rightarrow\) (1): If \(c_1\boldsymbol{v}_1 + \cdots + c_k\boldsymbol{v}_k = \boldsymbol{0}\) with some \(c_i \neq 0\), then: \[\boldsymbol{v}_i = -\frac{c_1}{c_i}\boldsymbol{v}_1 - \cdots - \frac{c_{i-1}}{c_i}\boldsymbol{v}_{i-1} - \frac{c_{i+1}}{c_i}\boldsymbol{v}_{i+1} - \cdots - \frac{c_k}{c_i}\boldsymbol{v}_k\]

(1) \(\Rightarrow\) dependence: If \(\boldsymbol{v}_i = a_1\boldsymbol{v}_1 + \cdots + a_{i-1}\boldsymbol{v}_{i-1} + a_{i+1}\boldsymbol{v}_{i+1} + \cdots + a_k\boldsymbol{v}_k\), then: \[a_1\boldsymbol{v}_1 + \cdots + a_{i-1}\boldsymbol{v}_{i-1} - \boldsymbol{v}_i + a_{i+1}\boldsymbol{v}_{i+1} + \cdots + a_k\boldsymbol{v}_k = \boldsymbol{0}\]

This is a nontrivial linear combination equaling zero (the coefficient of \(\boldsymbol{v}_i\) is \(-1 \neq 0\)). ◻

Example 5.13 (Testing Linear Independence). 1. Three vectors in \(\mathbb{R}^3\): Test whether \(\boldsymbol{v}_1 = (1, 2, 3)\), \(\boldsymbol{v}_2 = (4, 5, 6)\), \(\boldsymbol{v}_3 = (7, 8, 9)\) are linearly independent.

Set up: \(c_1(1, 2, 3) + c_2(4, 5, 6) + c_3(7, 8, 9) = (0, 0, 0)\)

This gives the system: \[\begin{align} c_1 + 4c_2 + 7c_3 &= 0\\ 2c_1 + 5c_2 + 8c_3 &= 0\\ 3c_1 + 6c_2 + 9c_3 &= 0 \end{align}\]

Row reducing the coefficient matrix: \[\begin{pmatrix} 1 & 4 & 7 \\ 2 & 5 & 8 \\ 3 & 6 & 9 \end{pmatrix} \sim \begin{pmatrix} 1 & 4 & 7 \\ 0 & -3 & -6 \\ 0 & -6 & -12 \end{pmatrix} \sim \begin{pmatrix} 1 & 4 & 7 \\ 0 & 1 & 2 \\ 0 & 0 & 0 \end{pmatrix}\]

The last row of zeros indicates infinitely many solutions. Back-substituting: From row 2: \(c_2 + 2c_3 = 0\), so \(c_2 = -2c_3\) From row 1: \(c_1 + 4(-2c_3) + 7c_3 = 0\), so \(c_1 = c_3\)

Setting \(c_3 = 1\): we get \((c_1, c_2, c_3) = (1, -2, 1)\).

Verification: \((1, 2, 3) - 2(4, 5, 6) + (7, 8, 9) = (1-8+7, 2-10+8, 3-12+9) = (0, 0, 0)\)

Therefore, the vectors are linearly dependent.

2. Standard basis vectors: The vectors \(\boldsymbol{e}_1 = (1, 0, 0)\), \(\boldsymbol{e}_2 = (0, 1, 0)\), \(\boldsymbol{e}_3 = (0, 0, 1)\) are linearly independent.

If \(c_1\boldsymbol{e}_1 + c_2\boldsymbol{e}_2 + c_3\boldsymbol{e}_3 = \boldsymbol{0}\), then \((c_1, c_2, c_3) = (0, 0, 0)\), so all coefficients must be zero.

Theorem 5.14 (Properties of Linear Independence).

1. Any set containing the zero vector is linearly dependent

2. Any subset of a linearly independent set is linearly independent

3. Any superset of a linearly dependent set is linearly dependent

4. In \(\mathbb{R}^n\), any set of more than \(n\) vectors is linearly dependent

5. In \(\mathbb{R}^n\), any set of fewer than \(n\) linearly independent vectors can be extended to a basis

Proof. Property 1: If \(\boldsymbol{0}\) is among the vectors, then \(1 \cdot \boldsymbol{0} + 0 \cdot \boldsymbol{v}_2 + \cdots = \boldsymbol{0}\) is a nontrivial linear combination.

Property 4: If we have \(k > n\) vectors in \(\mathbb{R}^n\), arrange them as columns of an \(n \times k\) matrix. This gives a homogeneous system with more variables than equations, which must have nontrivial solutions.

The other properties follow from the definitions and basic linear algebra principles. ◻

Definition 5.15 (Basis for a Subspace). A set of vectors \(\{\boldsymbol{v}_1, \boldsymbol{v}_2, \ldots, \boldsymbol{v}_k\}\) is a basis for subspace \(H\) if:

1. The vectors are linearly independent

2. The vectors span \(H\): \(H = \operatorname{span}\{\boldsymbol{v}_1, \boldsymbol{v}_2, \ldots, \boldsymbol{v}_k\}\)

Theorem 5.16 (Properties of Bases). If \(\{\boldsymbol{v}_1, \boldsymbol{v}_2, \ldots, \boldsymbol{v}_k\}\) is a basis for subspace \(H\), then:

1. Every vector in \(H\) can be written uniquely as a linear combination of the basis vectors

2. Any linearly independent set in \(H\) has at most \(k\) vectors

3. Any spanning set for \(H\) has at least \(k\) vectors

4. All bases for \(H\) have exactly \(k\) vectors

Definition 5.17 (Dimension). The dimension of a subspace \(H\), denoted \(\dim(H)\), is the number of vectors in any basis for \(H\).

Example 5.18 (Finding Bases and Dimensions). 1. Column space basis: For \(\mathbf{A} = \begin{pmatrix} 1 & 3 & 3 & 2 \\ 2 & 6 & 9 & 5 \\ -1 & -3 & 3 & 0 \end{pmatrix}\), find a basis for \(\text{Col}(\mathbf{A})\).

Row reduce to identify pivot columns: \[\begin{pmatrix} 1 & 3 & 3 & 2 \\ 2 & 6 & 9 & 5 \\ -1 & -3 & 3 & 0 \end{pmatrix} \sim \begin{pmatrix} 1 & 3 & 0 & 1/3 \\ 0 & 0 & 1 & 1/3 \\ 0 & 0 & 0 & 0 \end{pmatrix}\]

The pivot columns are 1 and 3, so a basis for \(\text{Col}(\mathbf{A})\) is: \[\left\{\begin{pmatrix} 1 \\ 2 \\ -1 \end{pmatrix}, \begin{pmatrix} 3 \\ 9 \\ 3 \end{pmatrix}\right\}\]

Therefore, \(\dim(\text{Col}(\mathbf{A})) = 2\).

2. Null space basis: From the RREF above, the system \(\mathbf{A}\boldsymbol{x} = \boldsymbol{0}\) becomes: \[x_1 + 3x_2 + \frac{1}{3}x_4 = 0, \quad x_3 + \frac{1}{3}x_4 = 0\]

Free variables are \(x_2\) and \(x_4\). Setting \(x_2 = s, x_4 = t\): \[x_1 = -3s - \frac{1}{3}t, \quad x_3 = -\frac{1}{3}t\]

So \(\text{Nul}(\mathbf{A}) = \operatorname{span}\{(-3, 1, 0, 0)^T, (-1/3, 0, -1/3, 1)^T\}\).

Scaling the second vector: \(\text{Nul}(\mathbf{A}) = \operatorname{span}\{(-3, 1, 0, 0)^T, (-1, 0, -1, 3)^T\}\).

Therefore, \(\dim(\text{Nul}(\mathbf{A})) = 2\).

Verification: \(\dim(\text{Col}(\mathbf{A})) + \dim(\text{Nul}(\mathbf{A})) = 2 + 2 = 4 =\) number of columns

Theorem 5.19 (Rank-Nullity Theorem). For any \(m \times n\) matrix \(\mathbf{A}\): \[\operatorname{rank}(\mathbf{A}) + \operatorname{null}(\mathbf{A}) = n\] where \(\operatorname{rank}(\mathbf{A}) = \dim(\text{Col}(\mathbf{A}))\) and \(\operatorname{null}(\mathbf{A}) = \dim(\text{Nul}(\mathbf{A}))\).

Proof. This follows from the general rank-nullity theorem for linear transformations applied to the transformation \(T(\boldsymbol{x}) = \mathbf{A}\boldsymbol{x}\). ◻

5.3 Orthogonality in \(\mathbb{R}^n\)

Orthogonality introduces geometric concepts into linear algebra, providing tools for measuring distances, angles, and projections. These concepts are fundamental for applications ranging from optimization to data analysis.

Definition 5.20 (Dot Product and Orthogonality). For vectors \(\boldsymbol{u}, \boldsymbol{v} \in \mathbb{R}^n\):

• The dot product is \(\boldsymbol{u} \cdot \boldsymbol{v} = u_1v_1 + u_2v_2 + \cdots + u_nv_n\)

• The vectors are orthogonal if \(\boldsymbol{u} \cdot \boldsymbol{v} = 0\)

• The length (or norm) of \(\boldsymbol{v}\) is \(\|\boldsymbol{v}\| = \sqrt{\boldsymbol{v} \cdot \boldsymbol{v}}\)

Theorem 5.21 (Properties of the Dot Product). For vectors \(\boldsymbol{u}, \boldsymbol{v}, \boldsymbol{w} \in \mathbb{R}^n\) and scalar \(c\):

1. \(\boldsymbol{u} \cdot \boldsymbol{v} = \boldsymbol{v} \cdot \boldsymbol{u}\) (symmetry)

2. \((\boldsymbol{u} + \boldsymbol{v}) \cdot \boldsymbol{w} = \boldsymbol{u} \cdot \boldsymbol{w} + \boldsymbol{v} \cdot \boldsymbol{w}\) (linearity in first argument)

3. \((c\boldsymbol{u}) \cdot \boldsymbol{v} = c(\boldsymbol{u} \cdot \boldsymbol{v})\) (homogeneity)

4. \(\boldsymbol{v} \cdot \boldsymbol{v} \geq 0\), with equality if and only if \(\boldsymbol{v} = \boldsymbol{0}\) (positive definiteness)

Theorem 5.22 (Geometric Interpretation of Dot Product). For non-zero vectors \(\boldsymbol{u}, \boldsymbol{v} \in \mathbb{R}^n\): \[\boldsymbol{u} \cdot \boldsymbol{v} = \|\boldsymbol{u}\|\|\boldsymbol{v}\|\cos\theta\] where \(\theta\) is the angle between the vectors.

Proof. This follows from the law of cosines applied to the triangle formed by \(\boldsymbol{u}\), \(\boldsymbol{v}\), and \(\boldsymbol{u} - \boldsymbol{v}\): \[\|\boldsymbol{u} - \boldsymbol{v}\|^2 = \|\boldsymbol{u}\|^2 + \|\boldsymbol{v}\|^2 - 2\|\boldsymbol{u}\|\|\boldsymbol{v}\|\cos\theta\]

Expanding the left side using properties of the dot product: \[(\boldsymbol{u} - \boldsymbol{v}) \cdot (\boldsymbol{u} - \boldsymbol{v}) = \boldsymbol{u} \cdot \boldsymbol{u} - 2\boldsymbol{u} \cdot \boldsymbol{v} + \boldsymbol{v} \cdot \boldsymbol{v}\]

Comparing terms gives the result. ◻

Corollary 5.23 (Cauchy-Schwarz Inequality). For any vectors \(\boldsymbol{u}, \boldsymbol{v} \in \mathbb{R}^n\): \[|\boldsymbol{u} \cdot \boldsymbol{v}| \leq \|\boldsymbol{u}\|\|\boldsymbol{v}\|\] with equality if and only if the vectors are parallel.

Definition 5.24 (Orthogonal and Orthonormal Sets). A set of vectors \(\{\boldsymbol{u}_1, \boldsymbol{u}_2, \ldots, \boldsymbol{u}_k\}\) is:

• Orthogonal if \(\boldsymbol{u}_i \cdot \boldsymbol{u}_j = 0\) for all \(i \neq j\)

• Orthonormal if it is orthogonal and \(\|\boldsymbol{u}_i\| = 1\) for all \(i\)

Theorem 5.25 (Orthogonal Sets are Linearly Independent). Any orthogonal set of non-zero vectors is linearly independent.

Proof. Suppose \(c_1\boldsymbol{u}_1 + c_2\boldsymbol{u}_2 + \cdots + c_k\boldsymbol{u}_k = \boldsymbol{0}\) where the \(\boldsymbol{u}_i\) are orthogonal and non-zero.

Taking the dot product with \(\boldsymbol{u}_j\): \[c_1(\boldsymbol{u}_1 \cdot \boldsymbol{u}_j) + c_2(\boldsymbol{u}_2 \cdot \boldsymbol{u}_j) + \cdots + c_k(\boldsymbol{u}_k \cdot \boldsymbol{u}_j) = 0\]

Since the vectors are orthogonal, all terms vanish except \(c_j(\boldsymbol{u}_j \cdot \boldsymbol{u}_j) = c_j\|\boldsymbol{u}_j\|^2 = 0\).

Since \(\boldsymbol{u}_j \neq \boldsymbol{0}\), we have \(\|\boldsymbol{u}_j\|^2 > 0\), so \(c_j = 0\).

This holds for all \(j\), proving linear independence. ◻

Example 5.26 (Constructing Orthogonal Sets). 1. Standard basis: The standard basis \(\{\boldsymbol{e}_1, \boldsymbol{e}_2, \ldots, \boldsymbol{e}_n\}\) is orthonormal since: \[\boldsymbol{e}_i \cdot \boldsymbol{e}_j = \begin{cases} 1 & \text{if } i = j \\ 0 & \text{if } i \neq j \end{cases}\]

2. Gram-Schmidt process: Start with linearly independent vectors \(\{\boldsymbol{v}_1, \boldsymbol{v}_2, \boldsymbol{v}_3\} = \{(1,1,1), (1,1,0), (1,0,0)\}\).

Step 1: \(\boldsymbol{u}_1 = \boldsymbol{v}_1 = (1,1,1)\)

Step 2: \(\boldsymbol{u}_2 = \boldsymbol{v}_2 - \frac{\boldsymbol{v}_2 \cdot \boldsymbol{u}_1}{\boldsymbol{u}_1 \cdot \boldsymbol{u}_1}\boldsymbol{u}_1\)

\(\boldsymbol{v}_2 \cdot \boldsymbol{u}_1 = (1,1,0) \cdot (1,1,1) = 2\) \(\boldsymbol{u}_1 \cdot \boldsymbol{u}_1 = 3\)

\(\boldsymbol{u}_2 = (1,1,0) - \frac{2}{3}(1,1,1) = (1,1,0) - (\frac{2}{3}, \frac{2}{3}, \frac{2}{3}) = (\frac{1}{3}, \frac{1}{3}, -\frac{2}{3})\)

Step 3: \(\boldsymbol{u}_3 = \boldsymbol{v}_3 - \frac{\boldsymbol{v}_3 \cdot \boldsymbol{u}_1}{\boldsymbol{u}_1 \cdot \boldsymbol{u}_1}\boldsymbol{u}_1 - \frac{\boldsymbol{v}_3 \cdot \boldsymbol{u}_2}{\boldsymbol{u}_2 \cdot \boldsymbol{u}_2}\boldsymbol{u}_2\)

\(\boldsymbol{v}_3 \cdot \boldsymbol{u}_1 = (1,0,0) \cdot (1,1,1) = 1\) \(\boldsymbol{v}_3 \cdot \boldsymbol{u}_2 = (1,0,0) \cdot (\frac{1}{3}, \frac{1}{3}, -\frac{2}{3}) = \frac{1}{3}\) \(\boldsymbol{u}_2 \cdot \boldsymbol{u}_2 = \frac{1}{9} + \frac{1}{9} + \frac{4}{9} = \frac{6}{9} = \frac{2}{3}\)

\(\boldsymbol{u}_3 = (1,0,0) - \frac{1}{3}(1,1,1) - \frac{1/3}{2/3}(\frac{1}{3}, \frac{1}{3}, -\frac{2}{3})\) \(= (1,0,0) - (\frac{1}{3}, \frac{1}{3}, \frac{1}{3}) - \frac{1}{2}(\frac{1}{3}, \frac{1}{3}, -\frac{2}{3})\) \(= (1,0,0) - (\frac{1}{3}, \frac{1}{3}, \frac{1}{3}) - (\frac{1}{6}, \frac{1}{6}, -\frac{1}{3})\) \(= (\frac{1}{2}, -\frac{1}{2}, 0)\)

The orthogonal set is \(\{(1,1,1), (\frac{1}{3}, \frac{1}{3}, -\frac{2}{3}), (\frac{1}{2}, -\frac{1}{2}, 0)\}\).

Definition 5.27 (Orthogonal Complement). For a subspace \(W\) of \(\mathbb{R}^n\), its orthogonal complement is: \[W^{\perp} = \{\boldsymbol{x} \in \mathbb{R}^n : \boldsymbol{x} \cdot \boldsymbol{w} = 0 \text{ for all } \boldsymbol{w} \in W\}\]

Theorem 5.28 (Properties of Orthogonal Complements). For any subspace \(W\) of \(\mathbb{R}^n\):

1. \(W^{\perp}\) is a subspace of \(\mathbb{R}^n\)

2. \(W \cap W^{\perp} = \{\boldsymbol{0}\}\)

3. \((W^{\perp})^{\perp} = W\)

4. \(\dim(W) + \dim(W^{\perp}) = n\)

5. \(\mathbb{R}^n = W \oplus W^{\perp}\) (orthogonal direct sum)

Example 5.29 (Computing Orthogonal Complements). 1. Line through origin: Let \(W = \operatorname{span}\{(1, 2, 3)\}\). Then: \[W^{\perp} = \{\boldsymbol{x} \in \mathbb{R}^3 : \boldsymbol{x} \cdot (1, 2, 3) = 0\} = \{(x, y, z) : x + 2y + 3z = 0\}\]

This is a plane through the origin perpendicular to \((1, 2, 3)\).

2. For a matrix: For matrix \(\mathbf{A}\), we have \((\text{Col}(\mathbf{A}))^{\perp} = \text{Nul}(\mathbf{A}^T)\) and \((\text{Nul}(\mathbf{A}))^{\perp} = \text{Col}(\mathbf{A}^T)\).

This gives us the fundamental theorem of linear algebra: \[\mathbb{R}^m = \text{Col}(\mathbf{A}) \oplus \text{Nul}(\mathbf{A}^T)\] \[\mathbb{R}^n = \text{Col}(\mathbf{A}^T) \oplus \text{Nul}(\mathbf{A})\]

5.4 Rank of a Matrix

The rank of a matrix is a fundamental invariant that encodes essential information about its structure and the linear transformation it represents.

Definition 5.30 (Rank). The rank of an \(m \times n\) matrix \(\mathbf{A}\), denoted \(\operatorname{rank}(\mathbf{A})\), is:

• The dimension of the column space: \(\operatorname{rank}(\mathbf{A}) = \dim(\text{Col}(\mathbf{A}))\)

• Equivalently, the number of linearly independent columns

• Equivalently, the number of pivot positions in any row echelon form of \(\mathbf{A}\)

Theorem 5.31 (Equivalent Characterizations of Rank). For an \(m \times n\) matrix \(\mathbf{A}\), the following are equal:

1. \(\dim(\text{Col}(\mathbf{A}))\) (dimension of column space)

2. \(\dim(\text{Row}(\mathbf{A}))\) (dimension of row space)

3. Number of pivot columns in RREF of \(\mathbf{A}\)

4. Number of non-zero rows in any row echelon form of \(\mathbf{A}\)

Proof. (1) = (3): The pivot columns of the RREF correspond to a maximal linearly independent subset of the original columns.

(3) = (4): In row echelon form, each pivot creates exactly one non-zero row.

(1) = (2): This is the fundamental result that row and column ranks are equal. It follows from the fact that elementary row operations don’t change the column dependencies, and the row space is preserved under these operations. ◻

Example 5.32 (Computing Rank). Find the rank of \(\mathbf{A} = \begin{pmatrix} 1 & 2 & 3 & 4 \\ 2 & 4 & 6 & 8 \\ 1 & 3 & 5 & 7 \\ 0 & 1 & 2 & 3 \end{pmatrix}\).

Row reduce to echelon form: \[\begin{pmatrix} 1 & 2 & 3 & 4 \\ 2 & 4 & 6 & 8 \\ 1 & 3 & 5 & 7 \\ 0 & 1 & 2 & 3 \end{pmatrix} \sim \begin{pmatrix} 1 & 2 & 3 & 4 \\ 0 & 0 & 0 & 0 \\ 0 & 1 & 2 & 3 \\ 0 & 1 & 2 & 3 \end{pmatrix} \sim \begin{pmatrix} 1 & 2 & 3 & 4 \\ 0 & 1 & 2 & 3 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \end{pmatrix}\]

There are 2 non-zero rows, so \(\operatorname{rank}(\mathbf{A}) = 2\).

We can verify: the column space is spanned by columns 1 and 2 (the pivot columns), and these are linearly independent.

Theorem 5.33 (Properties of Rank). For matrices \(\mathbf{A}\) and \(\mathbf{B}\) of appropriate sizes:

1. \(\operatorname{rank}(\mathbf{A}) \leq \min(m, n)\) for \(m \times n\) matrix \(\mathbf{A}\)

2. \(\operatorname{rank}(\mathbf{A}^T) = \operatorname{rank}(\mathbf{A})\)

3. \(\operatorname{rank}(\mathbf{A}\mathbf{B}) \leq \min(\operatorname{rank}(\mathbf{A}), \operatorname{rank}(\mathbf{B}))\)

4. If \(\mathbf{A}\) is \(m \times n\), then \(\operatorname{rank}(\mathbf{A}) + \operatorname{null}(\mathbf{A}) = n\)

5. \(\operatorname{rank}(\mathbf{A}) = \operatorname{rank}(\mathbf{A}^T\mathbf{A}) = \operatorname{rank}(\mathbf{A}\mathbf{A}^T)\)

Definition 5.34 (Full Rank). An \(m \times n\) matrix \(\mathbf{A}\) has:

• Full column rank if \(\operatorname{rank}(\mathbf{A}) = n\) (columns are linearly independent)

• Full row rank if \(\operatorname{rank}(\mathbf{A}) = m\) (rows are linearly independent)

• Full rank if \(\operatorname{rank}(\mathbf{A}) = \min(m, n)\)

Theorem 5.35 (Consequences of Full Rank). For an \(m \times n\) matrix \(\mathbf{A}\):

1. \(\mathbf{A}\) has full column rank if and only if \(\text{Nul}(\mathbf{A}) = \{\boldsymbol{0}\}\)

2. \(\mathbf{A}\) has full row rank if and only if \(\text{Col}(\mathbf{A}) = \mathbb{R}^m\)

3. If \(\mathbf{A}\) is square and has full rank, then \(\mathbf{A}\) is invertible

4. If \(\mathbf{A}\) has full column rank, then \(\mathbf{A}^T\mathbf{A}\) is invertible

5. If \(\mathbf{A}\) has full row rank, then \(\mathbf{A}\mathbf{A}^T\) is invertible

5.5 Similarity and Diagonalization

Similar matrices represent the same linear transformation in different coordinate systems. Diagonalization finds the optimal coordinate system where the transformation has the simplest possible form.

Definition 5.36 (Similar Matrices). Matrices \(\mathbf{A}\) and \(\mathbf{B}\) are similar, denoted \(\mathbf{A} \sim \mathbf{B}\), if there exists an invertible matrix \(\mathbf{P}\) such that: \[\mathbf{B} = \mathbf{P}^{-1}\mathbf{A}\mathbf{P}\]

Theorem 5.37 (Properties of Similarity). Similarity is an equivalence relation:

1. Reflexive: \(\mathbf{A} \sim \mathbf{A}\) (take \(\mathbf{P} = \mathbf{I}\))

2. Symmetric: If \(\mathbf{A} \sim \mathbf{B}\), then \(\mathbf{B} \sim \mathbf{A}\)

3. Transitive: If \(\mathbf{A} \sim \mathbf{B}\) and \(\mathbf{B} \sim \mathbf{C}\), then \(\mathbf{A} \sim \mathbf{C}\)

Theorem 5.38 (Similarity Invariants). If \(\mathbf{A} \sim \mathbf{B}\), then:

1. \(\det(\mathbf{A}) = \det(\mathbf{B})\)

2. \(\operatorname{tr}(\mathbf{A}) = \operatorname{tr}(\mathbf{B})\)

3. \(\operatorname{rank}(\mathbf{A}) = \operatorname{rank}(\mathbf{B})\)

4. \(\mathbf{A}\) and \(\mathbf{B}\) have the same eigenvalues (counting multiplicities)

5. \(\mathbf{A}\) and \(\mathbf{B}\) have the same characteristic polynomial

Proof. For property (1): If \(\mathbf{B} = \mathbf{P}^{-1}\mathbf{A}\mathbf{P}\), then: \[\det(\mathbf{B}) = \det(\mathbf{P}^{-1}\mathbf{A}\mathbf{P}) = \det(\mathbf{P}^{-1})\det(\mathbf{A})\det(\mathbf{P}) = \frac{1}{\det(\mathbf{P})}\det(\mathbf{A})\det(\mathbf{P}) = \det(\mathbf{A})\]

The other properties follow from similar arguments using the multiplicative properties of determinant, trace, and characteristic polynomial. ◻

Definition 5.39 (Diagonalizable Matrix). A matrix \(\mathbf{A}\) is diagonalizable if it is similar to a diagonal matrix. That is, there exists an invertible matrix \(\mathbf{P}\) and a diagonal matrix \(\mathbf{D}\) such that: \[\mathbf{A} = \mathbf{P}\mathbf{D}\mathbf{P}^{-1}\]

Theorem 5.40 (Diagonalization Theorem). An \(n \times n\) matrix \(\mathbf{A}\) is diagonalizable if and only if \(\mathbf{A}\) has \(n\) linearly independent eigenvectors.

Moreover, if \(\mathbf{A}\) is diagonalizable, then:

• \(\mathbf{P}\) can be chosen to have eigenvectors of \(\mathbf{A}\) as its columns

• \(\mathbf{D}\) has the corresponding eigenvalues on its diagonal

• The columns of \(\mathbf{P}\) can be arranged so that \(\mathbf{D} = \text{diag}(\lambda_1, \lambda_2, \ldots, \lambda_n)\)

Proof. (\(\Rightarrow\)) Suppose \(\mathbf{A} = \mathbf{P}\mathbf{D}\mathbf{P}^{-1}\) where \(\mathbf{D} = \text{diag}(\lambda_1, \ldots, \lambda_n)\).

Let \(\boldsymbol{p}_1, \ldots, \boldsymbol{p}_n\) be the columns of \(\mathbf{P}\). Then: \[\mathbf{A}\mathbf{P} = \mathbf{A}[\boldsymbol{p}_1 \mid \cdots \mid \boldsymbol{p}_n] = [\mathbf{A}\boldsymbol{p}_1 \mid \cdots \mid \mathbf{A}\boldsymbol{p}_n]\]

Also: \[\mathbf{P}\mathbf{D} = [\boldsymbol{p}_1 \mid \cdots \mid \boldsymbol{p}_n]\text{diag}(\lambda_1, \ldots, \lambda_n) = [\lambda_1\boldsymbol{p}_1 \mid \cdots \mid \lambda_n\boldsymbol{p}_n]\]

Since \(\mathbf{A}\mathbf{P} = \mathbf{P}\mathbf{D}\), we have \(\mathbf{A}\boldsymbol{p}_i = \lambda_i\boldsymbol{p}_i\) for each \(i\).

Since \(\mathbf{P}\) is invertible, its columns are linearly independent.

(\(\Leftarrow\)) Suppose \(\mathbf{A}\) has linearly independent eigenvectors \(\boldsymbol{p}_1, \ldots, \boldsymbol{p}_n\) with eigenvalues \(\lambda_1, \ldots, \lambda_n\).

Let \(\mathbf{P} = [\boldsymbol{p}_1 \mid \cdots \mid \boldsymbol{p}_n]\) and \(\mathbf{D} = \text{diag}(\lambda_1, \ldots, \lambda_n)\).

Since the eigenvectors are linearly independent, \(\mathbf{P}\) is invertible.

The eigenvalue equations \(\mathbf{A}\boldsymbol{p}_i = \lambda_i\boldsymbol{p}_i\) combine to give \(\mathbf{A}\mathbf{P} = \mathbf{P}\mathbf{D}\).

Multiplying by \(\mathbf{P}^{-1}\) on the right: \(\mathbf{A} = \mathbf{P}\mathbf{D}\mathbf{P}^{-1}\). ◻

Example 5.41 (Diagonalizing a Matrix). Diagonalize \(\mathbf{A} = \begin{pmatrix} 4 & -2 \\ 1 & 1 \end{pmatrix}\).

Step 1: Find eigenvalues \[\det(\mathbf{A} - \lambda\mathbf{I}) = \det\begin{pmatrix} 4-\lambda & -2 \\ 1 & 1-\lambda \end{pmatrix} = (4-\lambda)(1-\lambda) + 2 = \lambda^2 - 5\lambda + 6 = (\lambda-2)(\lambda-3)\]

Eigenvalues: \(\lambda_1 = 2, \lambda_2 = 3\).

Step 2: Find eigenvectors

For \(\lambda_1 = 2\): \[(\mathbf{A} - 2\mathbf{I})\boldsymbol{v} = \begin{pmatrix} 2 & -2 \\ 1 & -1 \end{pmatrix}\boldsymbol{v} = \boldsymbol{0}\]

This gives \(2v_1 - 2v_2 = 0\), so \(v_1 = v_2\). Eigenvector: \(\boldsymbol{v}_1 = \begin{pmatrix} 1 \\ 1 \end{pmatrix}\).

For \(\lambda_2 = 3\): \[(\mathbf{A} - 3\mathbf{I})\boldsymbol{v} = \begin{pmatrix} 1 & -2 \\ 1 & -2 \end{pmatrix}\boldsymbol{v} = \boldsymbol{0}\]

This gives \(v_1 - 2v_2 = 0\), so \(v_1 = 2v_2\). Eigenvector: \(\boldsymbol{v}_2 = \begin{pmatrix} 2 \\ 1 \end{pmatrix}\).

Step 3: Form matrices \[\mathbf{P} = \begin{pmatrix} 1 & 2 \\ 1 & 1 \end{pmatrix}, \quad \mathbf{D} = \begin{pmatrix} 2 & 0 \\ 0 & 3 \end{pmatrix}\]

Step 4: Verify \[\mathbf{P}^{-1} = \frac{1}{1-2}\begin{pmatrix} 1 & -2 \\ -1 & 1 \end{pmatrix} = \begin{pmatrix} -1 & 2 \\ 1 & -1 \end{pmatrix}\]

\[\mathbf{P}\mathbf{D}\mathbf{P}^{-1} = \begin{pmatrix} 1 & 2 \\ 1 & 1 \end{pmatrix}\begin{pmatrix} 2 & 0 \\ 0 & 3 \end{pmatrix}\begin{pmatrix} -1 & 2 \\ 1 & -1 \end{pmatrix} = \begin{pmatrix} 2 & 6 \\ 2 & 3 \end{pmatrix}\begin{pmatrix} -1 & 2 \\ 1 & -1 \end{pmatrix} = \begin{pmatrix} 4 & -2 \\ 1 & 1 \end{pmatrix} = \mathbf{A}\]

Theorem 5.42 (Sufficient Conditions for Diagonalizability).

1. If \(\mathbf{A}\) has \(n\) distinct eigenvalues, then \(\mathbf{A}\) is diagonalizable

2. If \(\mathbf{A}\) is symmetric, then \(\mathbf{A}\) is diagonalizable (with orthogonal eigenvectors)

3. \(\mathbf{A}\) is diagonalizable if and only if the geometric multiplicity equals the algebraic multiplicity for each eigenvalue

5.6 Best Approximation and Least Squares

When exact solutions don’t exist, we seek the "best" approximate solution. This leads to the fundamental problem of least squares approximation, which has applications throughout science and engineering.

Definition 5.43 (Orthogonal Projection). Let \(W\) be a subspace of \(\mathbb{R}^n\) with orthogonal basis \(\{\boldsymbol{u}_1, \boldsymbol{u}_2, \ldots, \boldsymbol{u}_k\}\). The orthogonal projection of \(\boldsymbol{y}\) onto \(W\) is: \[\text{proj}_W(\boldsymbol{y}) = \frac{\boldsymbol{y} \cdot \boldsymbol{u}_1}{\boldsymbol{u}_1 \cdot \boldsymbol{u}_1}\boldsymbol{u}_1 + \frac{\boldsymbol{y} \cdot \boldsymbol{u}_2}{\boldsymbol{u}_2 \cdot \boldsymbol{u}_2}\boldsymbol{u}_2 + \cdots + \frac{\boldsymbol{y} \cdot \boldsymbol{u}_k}{\boldsymbol{u}_k \cdot \boldsymbol{u}_k}\boldsymbol{u}_k\]

Theorem 5.44 (Best Approximation Theorem). Let \(W\) be a subspace of \(\mathbb{R}^n\) and \(\boldsymbol{y} \in \mathbb{R}^n\). Then \(\text{proj}_W(\boldsymbol{y})\) is the closest point in \(W\) to \(\boldsymbol{y}\). That is: \[\|\boldsymbol{y} - \text{proj}_W(\boldsymbol{y})\| < \|\boldsymbol{y} - \boldsymbol{w}\|\] for any \(\boldsymbol{w} \in W\) with \(\boldsymbol{w} \neq \text{proj}_W(\boldsymbol{y})\).

Proof. Let \(\hat{\boldsymbol{y}} = \text{proj}_W(\boldsymbol{y})\) and \(\boldsymbol{w} \in W\) with \(\boldsymbol{w} \neq \hat{\boldsymbol{y}}\).

By the properties of orthogonal projection, \(\boldsymbol{y} - \hat{\boldsymbol{y}} \perp W\), so \((\boldsymbol{y} - \hat{\boldsymbol{y}}) \perp (\hat{\boldsymbol{y}} - \boldsymbol{w})\).

Using the Pythagorean theorem: \[\|\boldsymbol{y} - \boldsymbol{w}\|^2 = \|(\boldsymbol{y} - \hat{\boldsymbol{y}}) + (\hat{\boldsymbol{y}} - \boldsymbol{w})\|^2 = \|\boldsymbol{y} - \hat{\boldsymbol{y}}\|^2 + \|\hat{\boldsymbol{y}} - \boldsymbol{w}\|^2\]

Since \(\hat{\boldsymbol{y}} \neq \boldsymbol{w}\), we have \(\|\hat{\boldsymbol{y}} - \boldsymbol{w}\|^2 > 0\), so: \[\|\boldsymbol{y} - \boldsymbol{w}\|^2 > \|\boldsymbol{y} - \hat{\boldsymbol{y}}\|^2\]

Taking square roots gives the result. ◻

Definition 5.45 (Least Squares Problem). Given an \(m \times n\) matrix \(\mathbf{A}\) and vector \(\boldsymbol{b} \in \mathbb{R}^m\), the least squares solution to \(\mathbf{A}\boldsymbol{x} = \boldsymbol{b}\) is the vector \(\hat{\boldsymbol{x}}\) that minimizes \(\|\mathbf{A}\boldsymbol{x} - \boldsymbol{b}\|^2\).

Theorem 5.46 (Normal Equation). The least squares solution \(\hat{\boldsymbol{x}}\) satisfies the normal equation: \[\mathbf{A}^T\mathbf{A}\hat{\boldsymbol{x}} = \mathbf{A}^T\boldsymbol{b}\]

If \(\mathbf{A}\) has linearly independent columns, then: \[\hat{\boldsymbol{x}} = (\mathbf{A}^T\mathbf{A})^{-1}\mathbf{A}^T\boldsymbol{b}\]

Proof. The least squares solution corresponds to the orthogonal projection of \(\boldsymbol{b}\) onto \(\text{Col}(\mathbf{A})\): \[\text{proj}_{\text{Col}(\mathbf{A})}(\boldsymbol{b}) = \mathbf{A}\hat{\boldsymbol{x}}\]

The error vector \(\boldsymbol{b} - \mathbf{A}\hat{\boldsymbol{x}}\) must be orthogonal to \(\text{Col}(\mathbf{A})\), which means it’s orthogonal to every column of \(\mathbf{A}\). This gives: \[\mathbf{A}^T(\boldsymbol{b} - \mathbf{A}\hat{\boldsymbol{x}}) = \boldsymbol{0}\]

Expanding: \(\mathbf{A}^T\boldsymbol{b} - \mathbf{A}^T\mathbf{A}\hat{\boldsymbol{x}} = \boldsymbol{0}\), which gives the normal equation.

When \(\mathbf{A}\) has linearly independent columns, \(\mathbf{A}^T\mathbf{A}\) is invertible, so we can solve for \(\hat{\boldsymbol{x}}\). ◻

Example 5.47 (Linear Regression). Find the best-fit line \(y = c_0 + c_1x\) for the data points \((1, 1)\), \((2, 3)\), \((3, 2)\), \((4, 5)\).

Set up the system: We want to minimize: \[\sum_{i=1}^4 (c_0 + c_1x_i - y_i)^2\]

This is equivalent to solving \(\mathbf{A}\boldsymbol{c} = \boldsymbol{b}\) in the least squares sense, where: \[\mathbf{A} = \begin{pmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \\ 1 & 4 \end{pmatrix}, \quad \boldsymbol{c} = \begin{pmatrix} c_0 \\ c_1 \end{pmatrix}, \quad \boldsymbol{b} = \begin{pmatrix} 1 \\ 3 \\ 2 \\ 5 \end{pmatrix}\]

Compute the normal equation: \[\mathbf{A}^T\mathbf{A} = \begin{pmatrix} 1 & 1 & 1 & 1 \\ 1 & 2 & 3 & 4 \end{pmatrix}\begin{pmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \\ 1 & 4 \end{pmatrix} = \begin{pmatrix} 4 & 10 \\ 10 & 30 \end{pmatrix}\]

\[\mathbf{A}^T\boldsymbol{b} = \begin{pmatrix} 1 & 1 & 1 & 1 \\ 1 & 2 & 3 & 4 \end{pmatrix}\begin{pmatrix} 1 \\ 3 \\ 2 \\ 5 \end{pmatrix} = \begin{pmatrix} 11 \\ 32 \end{pmatrix}\]

Solve: \[\begin{pmatrix} 4 & 10 \\ 10 & 30 \end{pmatrix}\begin{pmatrix} c_0 \\ c_1 \end{pmatrix} = \begin{pmatrix} 11 \\ 32 \end{pmatrix}\]

Using the inverse: \[\begin{pmatrix} c_0 \\ c_1 \end{pmatrix} = \frac{1}{120-100}\begin{pmatrix} 30 & -10 \\ -10 & 4 \end{pmatrix}\begin{pmatrix} 11 \\ 32 \end{pmatrix} = \frac{1}{20}\begin{pmatrix} 330-320 \\ -110+128 \end{pmatrix} = \begin{pmatrix} 0.5 \\ 0.9 \end{pmatrix}\]

Result: The best-fit line is \(y = 0.5 + 0.9x\).

Verification of fit: \[\begin{align} x = 1: &\quad y = 0.5 + 0.9(1) = 1.4 \quad (\text{actual: } 1)\\ x = 2: &\quad y = 0.5 + 0.9(2) = 2.3 \quad (\text{actual: } 3)\\ x = 3: &\quad y = 0.5 + 0.9(3) = 3.2 \quad (\text{actual: } 2)\\ x = 4: &\quad y = 0.5 + 0.9(4) = 4.1 \quad (\text{actual: } 5) \end{align}\]

5.7 Application to Correlation and Variance

Linear algebra provides powerful tools for analyzing statistical relationships and data structures. We explore applications to correlation analysis and principal component analysis.

Definition 5.48 (Sample Mean and Centered Data). For data vectors \(\boldsymbol{x}_1, \boldsymbol{x}_2, \ldots, \boldsymbol{x}_n \in \mathbb{R}^p\):

• The sample mean is \(\bar{\boldsymbol{x}} = \frac{1}{n}\sum_{i=1}^n \boldsymbol{x}_i\)

• The centered data are \(\tilde{\boldsymbol{x}}_i = \boldsymbol{x}_i - \bar{\boldsymbol{x}}\)

• The data matrix is \(\mathbf{X} = [\boldsymbol{x}_1 \mid \boldsymbol{x}_2 \mid \cdots \mid \boldsymbol{x}_n]\) (each column is an observation)

Definition 5.49 (Covariance Matrix). For centered data matrix \(\tilde{\mathbf{X}}\), the sample covariance matrix is: \[\mathbf{S} = \frac{1}{n-1}\tilde{\mathbf{X}}\tilde{\mathbf{X}}^T\]

The \((i,j)\)-entry is: \[S_{ij} = \frac{1}{n-1}\sum_{k=1}^n (\tilde{x}_{ik})(\tilde{x}_{jk})\]

This measures the covariance between variables \(i\) and \(j\).

Theorem 5.50 (Properties of Covariance Matrix). The covariance matrix \(\mathbf{S}\) is:

1. Symmetric: \(\mathbf{S} = \mathbf{S}^T\)

2. Positive semidefinite: \(\boldsymbol{v}^T\mathbf{S}\boldsymbol{v} \geq 0\) for all \(\boldsymbol{v}\)

3. Has non-negative eigenvalues

4. If data are not collinear, \(\mathbf{S}\) is positive definite

Definition 5.51 (Correlation Matrix). The sample correlation matrix \(\mathbf{R}\) has entries: \[R_{ij} = \frac{S_{ij}}{\sqrt{S_{ii}S_{jj}}}\]

This normalizes covariances to the range \([-1, 1]\).

Example 5.52 (Computing Covariance and Correlation). Consider the data matrix (each column is an observation): \[\mathbf{X} = \begin{pmatrix} 1 & 3 & 2 & 4 \\ 2 & 4 & 1 & 5 \end{pmatrix}\]

Step 1: Compute sample mean \[\bar{\boldsymbol{x}} = \frac{1}{4}\begin{pmatrix} 1+3+2+4 \\ 2+4+1+5 \end{pmatrix} = \begin{pmatrix} 2.5 \\ 3 \end{pmatrix}\]

Step 2: Center the data \[\tilde{\mathbf{X}} = \begin{pmatrix} 1-2.5 & 3-2.5 & 2-2.5 & 4-2.5 \\ 2-3 & 4-3 & 1-3 & 5-3 \end{pmatrix} = \begin{pmatrix} -1.5 & 0.5 & -0.5 & 1.5 \\ -1 & 1 & -2 & 2 \end{pmatrix}\]

Step 3: Compute covariance matrix \[\mathbf{S} = \frac{1}{3}\tilde{\mathbf{X}}\tilde{\mathbf{X}}^T = \frac{1}{3}\begin{pmatrix} -1.5 & 0.5 & -0.5 & 1.5 \\ -1 & 1 & -2 & 2 \end{pmatrix}\begin{pmatrix} -1.5 & -1 \\ 0.5 & 1 \\ -0.5 & -2 \\ 1.5 & 2 \end{pmatrix}\]

\[= \frac{1}{3}\begin{pmatrix} 2.25+0.25+0.25+2.25 & 1.5+0.5+1+3 \\ 1.5+0.5+1+3 & 1+1+4+4 \end{pmatrix} = \frac{1}{3}\begin{pmatrix} 5 & 6 \\ 6 & 10 \end{pmatrix} = \begin{pmatrix} 5/3 & 2 \\ 2 & 10/3 \end{pmatrix}\]

Step 4: Compute correlation matrix \[R_{12} = \frac{S_{12}}{\sqrt{S_{11}S_{22}}} = \frac{2}{\sqrt{(5/3)(10/3)}} = \frac{2}{\sqrt{50/9}} = \frac{2 \cdot 3}{\sqrt{50}} = \frac{6}{5\sqrt{2}} = \frac{6\sqrt{2}}{10} = 0.849\]

So \(\mathbf{R} = \begin{pmatrix} 1 & 0.849 \\ 0.849 & 1 \end{pmatrix}\).

This indicates a strong positive correlation between the two variables.

Definition 5.53 (Principal Component Analysis (PCA)). Given a covariance matrix \(\mathbf{S}\), the principal components are the eigenvectors of \(\mathbf{S}\), ordered by decreasing eigenvalue. The first principal component is the direction of maximum variance in the data.

Theorem 5.54 (Variance Maximization Property). Let \(\mathbf{S}\) be the covariance matrix of centered data. Among all unit vectors \(\boldsymbol{u}\), the variance in the direction \(\boldsymbol{u}\) is maximized when \(\boldsymbol{u}\) is the eigenvector corresponding to the largest eigenvalue of \(\mathbf{S}\).

Proof. The variance in direction \(\boldsymbol{u}\) is \(\boldsymbol{u}^T\mathbf{S}\boldsymbol{u}\). We want to maximize this subject to \(\|\boldsymbol{u}\|^2 = 1\).

Using Lagrange multipliers, we maximize \(f(\boldsymbol{u}) = \boldsymbol{u}^T\mathbf{S}\boldsymbol{u} - \lambda(\boldsymbol{u}^T\boldsymbol{u} - 1)\).

Taking the gradient: \(\nabla f = 2\mathbf{S}\boldsymbol{u} - 2\lambda\boldsymbol{u} = \boldsymbol{0}\).

This gives \(\mathbf{S}\boldsymbol{u} = \lambda\boldsymbol{u}\), so \(\boldsymbol{u}\) must be an eigenvector of \(\mathbf{S}\).

The variance is then \(\boldsymbol{u}^T\mathbf{S}\boldsymbol{u} = \lambda\boldsymbol{u}^T\boldsymbol{u} = \lambda\), which is maximized when \(\lambda\) is the largest eigenvalue. ◻

Example 5.55 (PCA in Action). For the covariance matrix \(\mathbf{S} = \begin{pmatrix} 5/3 & 2 \\ 2 & 10/3 \end{pmatrix}\) from the previous example:

Find eigenvalues: \[\det(\mathbf{S} - \lambda\mathbf{I}) = \det\begin{pmatrix} 5/3-\lambda & 2 \\ 2 & 10/3-\lambda \end{pmatrix} = (5/3-\lambda)(10/3-\lambda) - 4\]

\[= \frac{50}{9} - \frac{15\lambda}{3} + \lambda^2 - 4 = \lambda^2 - 5\lambda + \frac{50-36}{9} = \lambda^2 - 5\lambda + \frac{14}{9}\]

Using the quadratic formula: \[\lambda = \frac{5 \pm \sqrt{25 - 4 \cdot \frac{14}{9}}}{2} = \frac{5 \pm \sqrt{25 - \frac{56}{9}}}{2} = \frac{5 \pm \sqrt{\frac{225-56}{9}}}{2} = \frac{5 \pm \sqrt{\frac{169}{9}}}{2} = \frac{5 \pm \frac{13}{3}}{2}\]

So \(\lambda_1 = \frac{5 + 13/3}{2} = \frac{28/3}{2} = \frac{14}{3}\) and \(\lambda_2 = \frac{5 - 13/3}{2} = \frac{2/3}{2} = \frac{1}{3}\).

Find eigenvectors:

For \(\lambda_1 = 14/3\): \[\begin{pmatrix} 5/3 - 14/3 & 2 \\ 2 & 10/3 - 14/3 \end{pmatrix}\boldsymbol{v} = \begin{pmatrix} -3 & 2 \\ 2 & -4/3 \end{pmatrix}\boldsymbol{v} = \boldsymbol{0}\]

From the first row: \(-3v_1 + 2v_2 = 0\), so \(v_2 = \frac{3}{2}v_1\).

Principal component: \(\boldsymbol{u}_1 = \frac{1}{\sqrt{13}}\begin{pmatrix} 2 \\ 3 \end{pmatrix}\) (normalized).

The first principal component explains \(\frac{14/3}{14/3 + 1/3} = \frac{14}{15} \approx 93.3\%\) of the total variance.

This example shows how PCA can identify the dominant direction of variation in multivariate data.

Key Idea

The Unity of \(\mathbb{R}^n\) Theory:

• Subspace structure: Lines, planes, and hyperplanes through the origin provide the geometric foundation

• Independence and dimension: These concepts quantify the "degrees of freedom" in linear systems

• Orthogonality: Introduces geometric concepts of angle, distance, and projection

• Rank theory: Connects matrix properties to transformation behavior and solvability

• Similarity and diagonalization: Reveals optimal coordinate systems for understanding transformations

• Least squares: Provides practical tools for optimization and data fitting

• Statistical applications: Shows how linear algebra underlies modern data analysis techniques

The theory of \(\mathbb{R}^n\) demonstrates how abstract mathematical concepts translate into powerful practical tools, providing both computational algorithms and conceptual frameworks for understanding complex data and phenomena.

This completes our comprehensive study of the vector space \(\mathbb{R}^n\). We have developed the geometric and algebraic foundations, explored subspace structure and dimension theory, introduced orthogonality and projection methods, examined matrix rank and similarity, and demonstrated applications to optimization and statistics. These tools form the practical core of linear algebra, providing both theoretical understanding and computational methods for solving real-world problems.

6 Abstract Vector Spaces

Our study thus far has focused primarily on the concrete vector spaces \(\mathbb{R}^n\), where vectors are represented as ordered tuples of real numbers. However, the power of linear algebra extends far beyond this setting. Many mathematical objects—polynomials, functions, matrices, solutions to differential equations—exhibit the same fundamental algebraic structure as \(\mathbb{R}^n\). By abstracting the essential properties of vector spaces, we unlock a unified framework that applies across diverse areas of mathematics, physics, and engineering.

6.1 Examples and Properties

We begin by identifying the fundamental axioms that characterize all vector spaces, then explore the rich variety of mathematical structures that satisfy these axioms.

Definition 6.1 (Vector Space). A vector space (or linear space) over a field \(\mathbb{F}\) is a set \(V\) together with two operations:

• Vector addition: \(+: V \times V \to V\)

• Scalar multiplication: \(\cdot: \mathbb{F} \times V \to V\)

such that for all \(\boldsymbol{u}, \boldsymbol{v}, \boldsymbol{w} \in V\) and \(a, b \in \mathbb{F}\), the following axioms hold:

Addition Axioms:

1. \(\boldsymbol{u} + \boldsymbol{v} = \boldsymbol{v} + \boldsymbol{u}\) (commutativity)

2. \((\boldsymbol{u} + \boldsymbol{v}) + \boldsymbol{w} = \boldsymbol{u} + (\boldsymbol{v} + \boldsymbol{w})\) (associativity)

3. There exists \(\boldsymbol{0} \in V\) such that \(\boldsymbol{v} + \boldsymbol{0} = \boldsymbol{v}\) for all \(\boldsymbol{v} \in V\) (additive identity)

4. For each \(\boldsymbol{v} \in V\), there exists \(-\boldsymbol{v} \in V\) such that \(\boldsymbol{v} + (-\boldsymbol{v}) = \boldsymbol{0}\) (additive inverse)

Scalar Multiplication Axioms:

1. \(a(b\boldsymbol{v}) = (ab)\boldsymbol{v}\) (associativity)

2. \(1\boldsymbol{v} = \boldsymbol{v}\) (multiplicative identity)

Distributive Axioms:

1. \(a(\boldsymbol{u} + \boldsymbol{v}) = a\boldsymbol{u} + a\boldsymbol{v}\) (scalar distributivity)

2. \((a + b)\boldsymbol{v} = a\boldsymbol{v} + b\boldsymbol{v}\) (vector distributivity)

Remark 309. These eight axioms capture the essential algebraic structure shared by all vector spaces. While they may seem abstract, each axiom corresponds to familiar properties we’ve observed in \(\mathbb{R}^n\). The power of abstraction lies in recognizing that these same properties appear in many other mathematical contexts.

Example 6.3 (The Fundamental Examples). 1. Euclidean Spaces \(\mathbb{R}^n\): The space of all \(n\)-tuples of real numbers with componentwise addition and scalar multiplication is our prototypical example.

2. Complex Numbers \(\mathbb{C}\): Viewed as a vector space over \(\mathbb{R}\), with \(\mathbb{C} = \{a + bi : a, b \in \mathbb{R}\}\). - Addition: \((a + bi) + (c + di) = (a + c) + (b + d)i\) - Scalar multiplication: \(r(a + bi) = ra + rbi\) for \(r \in \mathbb{R}\)

3. Zero Vector Space \(\{\boldsymbol{0}\}\): The trivial vector space containing only the zero vector. All operations are uniquely determined.

4. Matrix Spaces \(\mathbb{R}^{m \times n}\): The space of all \(m \times n\) real matrices with matrix addition and scalar multiplication.

Now we explore more sophisticated examples that demonstrate the breadth of vector space theory:

Example 6.4 (Polynomial Spaces). 1. \(\mathcal{P}_n\): Polynomials of degree at most \(n\)

Let \(\mathcal{P}_n = \{a_0 + a_1x + a_2x^2 + \cdots + a_nx^n : a_i \in \mathbb{R}\}\).

Addition: \((a_0 + a_1x + \cdots + a_nx^n) + (b_0 + b_1x + \cdots + b_nx^n) = (a_0 + b_0) + (a_1 + b_1)x + \cdots + (a_n + b_n)x^n\)

Scalar multiplication: \(c(a_0 + a_1x + \cdots + a_nx^n) = ca_0 + ca_1x + \cdots + ca_nx^n\)

Zero vector: The zero polynomial \(0(x) = 0\)

Additive inverse: \(-(a_0 + a_1x + \cdots + a_nx^n) = (-a_0) + (-a_1)x + \cdots + (-a_n)x^n\)

Verification of axioms: The vector space axioms follow from the corresponding properties of real numbers. For instance: - Commutativity: Polynomial addition is commutative because real number addition is commutative - Distributivity: \(c(p(x) + q(x)) = cp(x) + cq(x)\) follows from the distributive property of real numbers applied to each coefficient

2. \(\mathcal{P}\): All polynomials

\(\mathcal{P} = \{a_0 + a_1x + a_2x^2 + \cdots + a_nx^n : n \geq 0, a_i \in \mathbb{R}\}\)

This is the union of all \(\mathcal{P}_n\) and forms an infinite-dimensional vector space.

Example 6.5 (Function Spaces). 1. \(\mathcal{F}(\mathbb{R})\): All real-valued functions

Let \(\mathcal{F}(\mathbb{R}) = \{f : \mathbb{R} \to \mathbb{R}\}\) be the set of all real-valued functions defined on \(\mathbb{R}\).

Addition: \((f + g)(x) = f(x) + g(x)\) for all \(x \in \mathbb{R}\)

Scalar multiplication: \((cf)(x) = c \cdot f(x)\) for all \(x \in \mathbb{R}\)

Zero vector: The zero function \(\mathbf{0}(x) = 0\) for all \(x\)

2. \(C[a,b]\): Continuous functions on an interval

\(C[a,b] = \{f : [a,b] \to \mathbb{R} : f \text{ is continuous}\}\)

This is a subspace of \(\mathcal{F}(\mathbb{R})\) since: - The sum of continuous functions is continuous - A scalar multiple of a continuous function is continuous - The zero function is continuous

3. \(C^\infty(\mathbb{R})\): Infinitely differentiable functions

Functions that have derivatives of all orders form a vector space under pointwise operations.

Theorem 6.6 (Basic Properties of Vector Spaces). In any vector space \(V\), the following properties hold:

1. The zero vector is unique

2. Each vector has a unique additive inverse

3. \(0\boldsymbol{v} = \boldsymbol{0}\) for all \(\boldsymbol{v} \in V\)

4. \((-1)\boldsymbol{v} = -\boldsymbol{v}\) for all \(\boldsymbol{v} \in V\)

5. If \(a\boldsymbol{v} = \boldsymbol{0}\), then either \(a = 0\) or \(\boldsymbol{v} = \boldsymbol{0}\)

Proof. Property 1 (Uniqueness of zero vector): Suppose \(\boldsymbol{0}\) and \(\boldsymbol{0}'\) are both zero vectors. Then: \[\boldsymbol{0}' = \boldsymbol{0}' + \boldsymbol{0} = \boldsymbol{0} + \boldsymbol{0}' = \boldsymbol{0}\] where we used the fact that \(\boldsymbol{0}\) acts as an identity for \(\boldsymbol{0}'\), commutativity, and \(\boldsymbol{0}'\) acts as an identity for \(\boldsymbol{0}\).

Property 2 (Uniqueness of additive inverse): Suppose \(\boldsymbol{w}\) and \(\boldsymbol{w}'\) are both additive inverses of \(\boldsymbol{v}\). Then: \[\boldsymbol{w}' = \boldsymbol{w}' + \boldsymbol{0} = \boldsymbol{w}' + (\boldsymbol{v} + \boldsymbol{w}) = (\boldsymbol{w}' + \boldsymbol{v}) + \boldsymbol{w} = \boldsymbol{0} + \boldsymbol{w} = \boldsymbol{w}\]

Property 3: We have: \[0\boldsymbol{v} + 0\boldsymbol{v} = (0 + 0)\boldsymbol{v} = 0\boldsymbol{v}\] Adding \(-(0\boldsymbol{v})\) to both sides: \[0\boldsymbol{v} = (0\boldsymbol{v} + 0\boldsymbol{v}) + (-(0\boldsymbol{v})) = 0\boldsymbol{v} + (0\boldsymbol{v} + (-(0\boldsymbol{v}))) = 0\boldsymbol{v} + \boldsymbol{0} = 0\boldsymbol{v}\]

But also: \(0\boldsymbol{v} + (-(0\boldsymbol{v})) = \boldsymbol{0}\), so we must have \(0\boldsymbol{v} = \boldsymbol{0}\).

Property 4: We need to show that \((-1)\boldsymbol{v}\) is the additive inverse of \(\boldsymbol{v}\): \[\boldsymbol{v} + (-1)\boldsymbol{v} = 1\boldsymbol{v} + (-1)\boldsymbol{v} = (1 + (-1))\boldsymbol{v} = 0\boldsymbol{v} = \boldsymbol{0}\]

Property 5: Suppose \(a\boldsymbol{v} = \boldsymbol{0}\) and \(a \neq 0\). Then \(a\) has a multiplicative inverse \(a^{-1}\), so: \[\boldsymbol{v} = 1\boldsymbol{v} = (a^{-1}a)\boldsymbol{v} = a^{-1}(a\boldsymbol{v}) = a^{-1}\boldsymbol{0} = \boldsymbol{0}\] ◻

Example 6.7 (Non-Examples: When Vector Space Axioms Fail). Understanding what is not a vector space helps clarify the importance of each axiom.

1. Positive real numbers under usual multiplication Let \(V = \{x \in \mathbb{R} : x > 0\}\) with "addition" defined as multiplication: \(x \oplus y = xy\). - This fails axiom A3: there’s no additive identity since we need \(x \oplus e = x\), so \(xe = x\), which gives \(e = 1\). But then \(1 \oplus 1 = 1 \cdot 1 = 1 \neq 1\) (well, this actually works), but \(2 \oplus 1 = 2 \cdot 1 = 2\), which is correct.

Actually, let me reconsider this. If we define \(x \oplus y = xy\) and scalar multiplication as \(c \odot x = x^c\), then: - Additive identity: \(e = 1\) since \(x \cdot 1 = x\) - Additive inverse of \(x\): \(x^{-1}\) since \(x \cdot x^{-1} = 1\) - Scalar multiplication: \(c \odot x = x^c\)

Let’s check: \((c + d) \odot x = x^{c+d} = x^c \cdot x^d = (c \odot x) \oplus (d \odot x)\)? But \((c \odot x) \oplus (d \odot x) = x^c \cdot x^d = x^{c+d}\), so this works.

Let me try a different non-example:

1. The first quadrant with usual operations Let \(V = \{(x,y) : x \geq 0, y \geq 0\}\) with usual vector addition and scalar multiplication. This fails because it’s not closed under additive inverses: \((1,1) \in V\) but \(-(1,1) = (-1,-1) \notin V\).

2. Integers under usual operations \(\mathbb{Z}\) with usual addition and multiplication by rational numbers fails to be a vector space over \(\mathbb{Q}\) because it’s not closed under scalar multiplication: \(\frac{1}{2} \cdot 1 = \frac{1}{2} \notin \mathbb{Z}\).

Definition 6.8 (Vector Subspace). A subset \(W\) of a vector space \(V\) is called a subspace of \(V\) if \(W\) is itself a vector space under the operations inherited from \(V\).

Theorem 6.9 (Subspace Test). A non-empty subset \(W\) of a vector space \(V\) is a subspace if and only if:

1. \(W\) is closed under addition: if \(\boldsymbol{u}, \boldsymbol{v} \in W\), then \(\boldsymbol{u} + \boldsymbol{v} \in W\)

2. \(W\) is closed under scalar multiplication: if \(\boldsymbol{v} \in W\) and \(c\) is a scalar, then \(c\boldsymbol{v} \in W\)

Proof. (\(\Rightarrow\)) If \(W\) is a subspace, then by definition it satisfies all vector space axioms, including closure under both operations.

(\(\Leftarrow\)) Assume \(W\) satisfies the two conditions. We verify that \(W\) inherits all vector space axioms from \(V\):

Axioms A1, A2, M1, M2, D1, D2: These are inherited from \(V\) since they involve only elements already known to be in \(W\).

Axiom A3 (zero vector): Since \(W\) is non-empty, there exists some \(\boldsymbol{w} \in W\). By closure under scalar multiplication, \(0\boldsymbol{w} = \boldsymbol{0} \in W\).

Axiom A4 (additive inverses): For any \(\boldsymbol{w} \in W\), we have \((-1)\boldsymbol{w} = -\boldsymbol{w} \in W\) by closure under scalar multiplication. ◻

Example 6.10 (Important Subspaces). 1. Solution spaces of homogeneous linear systems For any \(m \times n\) matrix \(\mathbf{A}\), the set \(\{\boldsymbol{x} \in \mathbb{R}^n : \mathbf{A}\boldsymbol{x} = \boldsymbol{0}\}\) is a subspace of \(\mathbb{R}^n\).

2. Even polynomials \(W = \{p(x) \in \mathcal{P} : p(-x) = p(x)\}\) is a subspace of \(\mathcal{P}\) since: - If \(p(-x) = p(x)\) and \(q(-x) = q(x)\), then \((p+q)(-x) = p(-x) + q(-x) = p(x) + q(x) = (p+q)(x)\) - If \(p(-x) = p(x)\), then \((cp)(-x) = cp(-x) = cp(x) = (cp)(x)\)

3. Symmetric matrices \(W = \{\mathbf{A} \in \mathbb{R}^{n \times n} : \mathbf{A}^T = \mathbf{A}\}\) is a subspace of \(\mathbb{R}^{n \times n}\).

6.2 Subspaces and Spanning Sets

Subspaces arise naturally as the sets of all linear combinations of given vectors. This leads to the fundamental concept of spanning, which describes how finite sets of vectors can generate entire subspaces.

Definition 6.11 (Linear Combination). A linear combination of vectors \(\boldsymbol{v}_1, \boldsymbol{v}_2, \ldots, \boldsymbol{v}_k\) in a vector space \(V\) is a vector of the form: \[c_1\boldsymbol{v}_1 + c_2\boldsymbol{v}_2 + \cdots + c_k\boldsymbol{v}_k\] where \(c_1, c_2, \ldots, c_k\) are scalars.

Definition 6.12 (Span). The span of a set of vectors \(\{\boldsymbol{v}_1, \boldsymbol{v}_2, \ldots, \boldsymbol{v}_k\}\) is the set of all linear combinations of these vectors: \[\operatorname{span}\{\boldsymbol{v}_1, \boldsymbol{v}_2, \ldots, \boldsymbol{v}_k\} = \{c_1\boldsymbol{v}_1 + c_2\boldsymbol{v}_2 + \cdots + c_k\boldsymbol{v}_k : c_i \in \mathbb{F}\}\]

Theorem 6.13 (Span is a Subspace). For any finite set of vectors \(\{\boldsymbol{v}_1, \boldsymbol{v}_2, \ldots, \boldsymbol{v}_k\}\) in a vector space \(V\), the span \(\operatorname{span}\{\boldsymbol{v}_1, \boldsymbol{v}_2, \ldots, \boldsymbol{v}_k\}\) is a subspace of \(V\).

Proof. Let \(W = \operatorname{span}\{\boldsymbol{v}_1, \boldsymbol{v}_2, \ldots, \boldsymbol{v}_k\}\). We apply the subspace test:

Non-empty: \(W\) contains \(\boldsymbol{0} = 0\boldsymbol{v}_1 + 0\boldsymbol{v}_2 + \cdots + 0\boldsymbol{v}_k\).

Closed under addition: Let \(\boldsymbol{u}, \boldsymbol{w} \in W\). Then: \[\boldsymbol{u} = a_1\boldsymbol{v}_1 + a_2\boldsymbol{v}_2 + \cdots + a_k\boldsymbol{v}_k\] \[\boldsymbol{w} = b_1\boldsymbol{v}_1 + b_2\boldsymbol{v}_2 + \cdots + b_k\boldsymbol{v}_k\]

Therefore: \[\boldsymbol{u} + \boldsymbol{w} = (a_1 + b_1)\boldsymbol{v}_1 + (a_2 + b_2)\boldsymbol{v}_2 + \cdots + (a_k + b_k)\boldsymbol{v}_k \in W\]

Closed under scalar multiplication: For \(\boldsymbol{u} \in W\) and scalar \(c\): \[c\boldsymbol{u} = c(a_1\boldsymbol{v}_1 + a_2\boldsymbol{v}_2 + \cdots + a_k\boldsymbol{v}_k) = (ca_1)\boldsymbol{v}_1 + (ca_2)\boldsymbol{v}_2 + \cdots + (ca_k)\boldsymbol{v}_k \in W\] ◻

Example 6.14 (Spans in Different Vector Spaces). 1. In \(\mathbb{R}^3\): - \(\operatorname{span}\{(1,0,0)\} = \{(t,0,0) : t \in \mathbb{R}\}\) (the \(x\)-axis) - \(\operatorname{span}\{(1,0,0), (0,1,0)\} = \{(s,t,0) : s,t \in \mathbb{R}\}\) (the \(xy\)-plane) - \(\operatorname{span}\{(1,2,3), (2,4,6)\} = \operatorname{span}\{(1,2,3)\}\) since \((2,4,6) = 2(1,2,3)\)

2. In \(\mathcal{P}_2\): - \(\operatorname{span}\{1, x, x^2\} = \mathcal{P}_2\) (all polynomials of degree at most 2) - \(\operatorname{span}\{1, x^2\} = \{a + bx^2 : a,b \in \mathbb{R}\}\) (polynomials with no linear term) - \(\operatorname{span}\{1 + x, x + x^2\} = \operatorname{span}\{1, x, x^2\} = \mathcal{P}_2\)

To verify the last equality, note that: \((1 + x) + (x + x^2) - 2x = 1 + x^2\), so \(1 + x^2 \in \operatorname{span}\{1 + x, x + x^2\}\) \((1 + x) - (1 + x^2) = x - x^2\), and \((x + x^2) - (x - x^2) = 2x^2\), so \(x^2 \in \operatorname{span}\{1 + x, x + x^2\}\) \((1 + x) - (1 + x^2) + x^2 = x\), so \(x \in \operatorname{span}\{1 + x, x + x^2\}\) \((1 + x) - x = 1\), so \(1 \in \operatorname{span}\{1 + x, x + x^2\}\)

3. In matrix spaces: \[\operatorname{span}\left\{\begin{pmatrix} 1 & 0 \\ 0 & 0 \end{pmatrix}, \begin{pmatrix} 0 & 1 \\ 0 & 0 \end{pmatrix}\right\} = \left\{\begin{pmatrix} a & b \\ 0 & 0 \end{pmatrix} : a,b \in \mathbb{R}\right\}\]

Theorem 6.15 (Properties of Span). Let \(S\) and \(T\) be sets of vectors in a vector space \(V\). Then:

1. \(S \subseteq \operatorname{span}(S)\)

2. If \(S \subseteq T\), then \(\operatorname{span}(S) \subseteq \operatorname{span}(T)\)

3. \(\operatorname{span}(\operatorname{span}(S)) = \operatorname{span}(S)\)

4. \(\operatorname{span}(S \cup T) = \operatorname{span}(S) + \operatorname{span}(T)\), where \(\operatorname{span}(S) + \operatorname{span}(T) = \{\boldsymbol{s} + \boldsymbol{t} : \boldsymbol{s} \in \operatorname{span}(S), \boldsymbol{t} \in \operatorname{span}(T)\}\)

Proof. Property 1: Each \(\boldsymbol{v} \in S\) can be written as \(1 \cdot \boldsymbol{v}\), which is a linear combination, so \(\boldsymbol{v} \in \operatorname{span}(S)\).

Property 2: Any linear combination of vectors from \(S\) is also a linear combination of vectors from \(T\) (since \(S \subseteq T\)).

Property 3: Since \(\operatorname{span}(S)\) is a subspace (hence closed under linear combinations), any linear combination of elements from \(\operatorname{span}(S)\) is already in \(\operatorname{span}(S)\).

Property 4: This follows from the distributive properties of vector addition and scalar multiplication. ◻

Definition 6.16 (Spanning Set). A set of vectors \(\{\boldsymbol{v}_1, \boldsymbol{v}_2, \ldots, \boldsymbol{v}_k\}\) spans a vector space \(V\) if \(\operatorname{span}\{\boldsymbol{v}_1, \boldsymbol{v}_2, \ldots, \boldsymbol{v}_k\} = V\). In this case, we say the set is a spanning set for \(V\).

Example 6.17 (Standard Spanning Sets). 1. Standard basis vectors span \(\mathbb{R}^n\): \[\operatorname{span}\{\boldsymbol{e}_1, \boldsymbol{e}_2, \ldots, \boldsymbol{e}_n\} = \mathbb{R}^n\] since any vector \((x_1, x_2, \ldots, x_n) = x_1\boldsymbol{e}_1 + x_2\boldsymbol{e}_2 + \cdots + x_n\boldsymbol{e}_n\).

2. Monomials span polynomial spaces: \[\operatorname{span}\{1, x, x^2, \ldots, x^n\} = \mathcal{P}_n\]

3. Elementary matrices span matrix spaces: The set of all \(m \times n\) matrices with exactly one entry equal to 1 and all others equal to 0 spans \(\mathbb{R}^{m \times n}\).

4. Trigonometric functions: The set \(\{1, \cos x, \sin x, \cos 2x, \sin 2x, \ldots\}\) spans a dense subspace of the space of continuous periodic functions on \([0, 2\pi]\) (this is the foundation of Fourier analysis).

Theorem 6.18 (Characterization of Subspaces via Spanning). A subset \(W\) of a vector space \(V\) is a subspace if and only if \(W = \operatorname{span}(S)\) for some subset \(S \subseteq V\).

Proof. (\(\Leftarrow\)) We’ve already shown that spans are subspaces.

(\(\Rightarrow\)) If \(W\) is a subspace, then \(W = \operatorname{span}(W)\) since: - Every element of \(W\) is a linear combination of elements in \(W\) (namely, itself with coefficient 1) - Every linear combination of elements in \(W\) belongs to \(W\) (by closure properties of subspaces) ◻

Definition 6.19 (Sum of Subspaces). Given subspaces \(U\) and \(W\) of a vector space \(V\), their sum is: \[U + W = \{\boldsymbol{u} + \boldsymbol{w} : \boldsymbol{u} \in U, \boldsymbol{w} \in W\}\]

Theorem 6.20 (Sum of Subspaces is a Subspace). If \(U\) and \(W\) are subspaces of \(V\), then \(U + W\) is also a subspace of \(V\). Moreover, \(U + W\) is the smallest subspace containing both \(U\) and \(W\).

Proof. Subspace verification: - \(\boldsymbol{0} = \boldsymbol{0} + \boldsymbol{0} \in U + W\) (non-empty) - If \(\boldsymbol{v}_1 = \boldsymbol{u}_1 + \boldsymbol{w}_1\) and \(\boldsymbol{v}_2 = \boldsymbol{u}_2 + \boldsymbol{w}_2\) are in \(U + W\), then: \[\boldsymbol{v}_1 + \boldsymbol{v}_2 = (\boldsymbol{u}_1 + \boldsymbol{u}_2) + (\boldsymbol{w}_1 + \boldsymbol{w}_2) \in U + W\] since \(\boldsymbol{u}_1 + \boldsymbol{u}_2 \in U\) and \(\boldsymbol{w}_1 + \boldsymbol{w}_2 \in W\). - If \(\boldsymbol{v} = \boldsymbol{u} + \boldsymbol{w} \in U + W\) and \(c\) is a scalar, then: \[c\boldsymbol{v} = c\boldsymbol{u} + c\boldsymbol{w} \in U + W\] since \(c\boldsymbol{u} \in U\) and \(c\boldsymbol{w} \in W\).

Minimality: Any subspace containing both \(U\) and \(W\) must contain all sums \(\boldsymbol{u} + \boldsymbol{w}\), hence must contain \(U + W\). ◻

6.3 Linear Independence and Basis

The concepts of linear independence and basis are fundamental to understanding the structure of vector spaces. They allow us to identify minimal spanning sets and establish coordinate systems for abstract vector spaces.

Definition 6.21 (Linear Independence). A set of vectors \(\{\boldsymbol{v}_1, \boldsymbol{v}_2, \ldots, \boldsymbol{v}_k\}\) in a vector space \(V\) is linearly independent if the only solution to the equation: \[c_1\boldsymbol{v}_1 + c_2\boldsymbol{v}_2 + \cdots + c_k\boldsymbol{v}_k = \boldsymbol{0}\] is \(c_1 = c_2 = \cdots = c_k = 0\).

If there exist scalars \(c_1, c_2, \ldots, c_k\), not all zero, such that \(c_1\boldsymbol{v}_1 + c_2\boldsymbol{v}_2 + \cdots + c_k\boldsymbol{v}_k = \boldsymbol{0}\), then the set is linearly dependent.

Remark 329. Linear independence captures the idea that no vector in the set can be expressed as a linear combination of the others. Each vector contributes something "new" to the span that cannot be obtained from the remaining vectors.

Theorem 6.23 (Characterizations of Linear Dependence). A set of vectors \(\{\boldsymbol{v}_1, \boldsymbol{v}_2, \ldots, \boldsymbol{v}_k\}\) is linearly dependent if and only if one of the following equivalent conditions holds:

1. Some vector \(\boldsymbol{v}_i\) can be written as a linear combination of the others

2. Some vector \(\boldsymbol{v}_i\) is in the span of the preceding vectors \(\{\boldsymbol{v}_1, \ldots, \boldsymbol{v}_{i-1}\}\)

3. Removing some vector from the set does not change the span

Proof. Linear dependence \(\Rightarrow\) (1): If \(c_1\boldsymbol{v}_1 + \cdots + c_k\boldsymbol{v}_k = \boldsymbol{0}\) with some \(c_i \neq 0\), then: \[\boldsymbol{v}_i = -\frac{c_1}{c_i}\boldsymbol{v}_1 - \cdots - \frac{c_{i-1}}{c_i}\boldsymbol{v}_{i-1} - \frac{c_{i+1}}{c_i}\boldsymbol{v}_{i+1} - \cdots - \frac{c_k}{c_i}\boldsymbol{v}_k\]

(1) \(\Rightarrow\) Linear dependence: If \(\boldsymbol{v}_i = a_1\boldsymbol{v}_1 + \cdots + a_{i-1}\boldsymbol{v}_{i-1} + a_{i+1}\boldsymbol{v}_{i+1} + \cdots + a_k\boldsymbol{v}_k\), then: \[a_1\boldsymbol{v}_1 + \cdots + a_{i-1}\boldsymbol{v}_{i-1} - 1\boldsymbol{v}_i + a_{i+1}\boldsymbol{v}_{i+1} + \cdots + a_k\boldsymbol{v}_k = \boldsymbol{0}\] This is a non-trivial linear combination equal to zero.

The equivalence with conditions (2) and (3) follows similarly. ◻

Example 6.24 (Testing Linear Independence). 1. In \(\mathbb{R}^3\): Test whether \(\{(1,2,3), (2,1,0), (1,1,1)\}\) is linearly independent.

We solve \(c_1(1,2,3) + c_2(2,1,0) + c_3(1,1,1) = (0,0,0)\): \[\begin{align} c_1 + 2c_2 + c_3 &= 0\\ 2c_1 + c_2 + c_3 &= 0\\ 3c_1 + 0c_2 + c_3 &= 0 \end{align}\]

From equation (3): \(c_3 = -3c_1\) Substituting into equation (1): \(c_1 + 2c_2 - 3c_1 = 0 \Rightarrow c_2 = c_1\) Substituting into equation (2): \(2c_1 + c_1 - 3c_1 = 0 \Rightarrow 0 = 0\)

The system has infinitely many solutions (parameterized by \(c_1\)), so the vectors are linearly dependent.

2. In \(\mathcal{P}_2\): Test whether \(\{1 + x, x + x^2, 1 + x^2\}\) is linearly independent.

We solve \(c_1(1 + x) + c_2(x + x^2) + c_3(1 + x^2) = 0\): \[c_1 + c_2x + c_3 + c_1x + c_2x^2 + c_3x^2 = 0\] \[(c_1 + c_3) + (c_1 + c_2)x + (c_2 + c_3)x^2 = 0\]

Since the zero polynomial has all coefficients equal to zero: \[\begin{align} c_1 + c_3 &= 0\\ c_1 + c_2 &= 0\\ c_2 + c_3 &= 0 \end{align}\]

From equations (1) and (3): \(c_1 = -c_3\) and \(c_2 = -c_3\) From equation (2): \(-c_3 + (-c_3) = 0 \Rightarrow c_3 = 0\)

Therefore \(c_1 = c_2 = c_3 = 0\), so the set is linearly independent.

3. In function spaces: The functions \(\{1, \cos x, \sin x\}\) are linearly independent in \(C[0, 2\pi]\). If \(c_1 \cdot 1 + c_2\cos x + c_3\sin x = 0\) for all \(x \in [0, 2\pi]\), then: - At \(x = 0\): \(c_1 + c_2 = 0\) - At \(x = \pi/2\): \(c_1 + c_3 = 0\) - At \(x = \pi\): \(c_1 - c_2 = 0\)

From the first and third equations: \(c_2 = -c_1\) and \(c_2 = c_1\), so \(c_1 = c_2 = 0\). From the second equation: \(c_3 = 0\).

Definition 6.25 (Basis). A set of vectors \(\mathcal{B} = \{\boldsymbol{v}_1, \boldsymbol{v}_2, \ldots, \boldsymbol{v}_n\}\) in a vector space \(V\) is a basis for \(V\) if:

1. \(\mathcal{B}\) is linearly independent

2. \(\mathcal{B}\) spans \(V\)

Theorem 6.26 (Uniqueness of Representation). If \(\mathcal{B} = \{\boldsymbol{v}_1, \boldsymbol{v}_2, \ldots, \boldsymbol{v}_n\}\) is a basis for vector space \(V\), then every vector \(\boldsymbol{v} \in V\) can be written uniquely as: \[\boldsymbol{v} = c_1\boldsymbol{v}_1 + c_2\boldsymbol{v}_2 + \cdots + c_n\boldsymbol{v}_n\] The scalars \(c_1, c_2, \ldots, c_n\) are called the coordinates of \(\boldsymbol{v}\) with respect to the basis \(\mathcal{B}\).

Proof. Existence: Since \(\mathcal{B}\) spans \(V\), every vector in \(V\) can be written as a linear combination of basis vectors.

Uniqueness: Suppose \(\boldsymbol{v} = c_1\boldsymbol{v}_1 + \cdots + c_n\boldsymbol{v}_n = d_1\boldsymbol{v}_1 + \cdots + d_n\boldsymbol{v}_n\). Then: \[(c_1 - d_1)\boldsymbol{v}_1 + (c_2 - d_2)\boldsymbol{v}_2 + \cdots + (c_n - d_n)\boldsymbol{v}_n = \boldsymbol{0}\]

Since \(\mathcal{B}\) is linearly independent, all coefficients must be zero: \(c_i - d_i = 0\) for all \(i\), so \(c_i = d_i\). ◻

Example 6.27 (Standard Bases). 1. Standard basis for \(\mathbb{R}^n\): \[\mathcal{E} = \{\boldsymbol{e}_1, \boldsymbol{e}_2, \ldots, \boldsymbol{e}_n\} = \{(1,0,\ldots,0), (0,1,\ldots,0), \ldots, (0,0,\ldots,1)\}\]

2. Standard basis for \(\mathcal{P}_n\): \[\mathcal{B} = \{1, x, x^2, \ldots, x^n\}\]

Every polynomial \(p(x) = a_0 + a_1x + \cdots + a_nx^n\) has unique coordinates \((a_0, a_1, \ldots, a_n)\).

3. Standard basis for \(\mathbb{R}^{2 \times 2}\): \[\mathcal{B} = \left\{\begin{pmatrix} 1 & 0 \\ 0 & 0 \end{pmatrix}, \begin{pmatrix} 0 & 1 \\ 0 & 0 \end{pmatrix}, \begin{pmatrix} 0 & 0 \\ 1 & 0 \end{pmatrix}, \begin{pmatrix} 0 & 0 \\ 0 & 1 \end{pmatrix}\right\}\]

Any \(2 \times 2\) matrix \(\begin{pmatrix} a & b \\ c & d \end{pmatrix}\) has coordinates \((a, b, c, d)\).

Example 6.28 (Non-Standard Bases). 1. Alternative basis for \(\mathbb{R}^2\): \(\mathcal{B} = \{(1,1), (1,-1)\}\) is a basis for \(\mathbb{R}^2\).

To find coordinates of \((3,1)\) with respect to this basis: \[c_1(1,1) + c_2(1,-1) = (3,1)\] \[c_1 + c_2 = 3, \quad c_1 - c_2 = 1\] Solving: \(c_1 = 2, c_2 = 1\), so \((3,1) = 2(1,1) + 1(1,-1)\).

2. Basis for the subspace of symmetric \(2 \times 2\) matrices: \[\mathcal{B} = \left\{\begin{pmatrix} 1 & 0 \\ 0 & 0 \end{pmatrix}, \begin{pmatrix} 0 & 0 \\ 0 & 1 \end{pmatrix}, \begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix}\right\}\]

Any symmetric matrix \(\begin{pmatrix} a & b \\ b & c \end{pmatrix}\) has coordinates \((a, c, b)\).

Theorem 6.29 (Basis Extension Theorem). Any linearly independent set in a finite-dimensional vector space can be extended to a basis.

Theorem 6.30 (Basis Reduction Theorem). Any spanning set for a finite-dimensional vector space contains a basis.

Remark 338. These theorems ensure that bases always exist in finite-dimensional spaces and can be constructed from either linearly independent sets or spanning sets.

6.4 Finite-Dimensional Spaces

The concept of dimension provides a measure of the "size" of a vector space and establishes fundamental relationships between bases, spanning sets, and linearly independent sets.

Definition 6.32 (Finite-Dimensional Vector Space). A vector space \(V\) is finite-dimensional if it has a finite spanning set. Otherwise, \(V\) is infinite-dimensional.

Theorem 6.33 (Fundamental Theorem of Finite-Dimensional Vector Spaces). In a finite-dimensional vector space, all bases have the same number of vectors.

Proof. Let \(V\) be finite-dimensional with bases \(\mathcal{B} = \{\boldsymbol{v}_1, \ldots, \boldsymbol{v}_m\}\) and \(\mathcal{C} = \{\boldsymbol{w}_1, \ldots, \boldsymbol{w}_n\}\).

Since \(\mathcal{B}\) spans \(V\) and \(\mathcal{C}\) is linearly independent, we have \(n \leq m\) (a linearly independent set cannot have more vectors than a spanning set).

Similarly, since \(\mathcal{C}\) spans \(V\) and \(\mathcal{B}\) is linearly independent, we have \(m \leq n\).

Therefore, \(m = n\). ◻

Definition 6.34 (Dimension). The dimension of a finite-dimensional vector space \(V\), denoted \(\dim(V)\), is the number of vectors in any basis for \(V\). By convention, \(\dim(\{\boldsymbol{0}\}) = 0\).

Example 6.35 (Dimensions of Standard Spaces).

• \(\dim(\mathbb{R}^n) = n\) (basis: standard basis vectors)

• \(\dim(\mathcal{P}_n) = n + 1\) (basis: \(\{1, x, x^2, \ldots, x^n\}\))

• \(\dim(\mathbb{R}^{m \times n}) = mn\) (basis: matrices with single 1 and rest 0s)

• \(\dim(\mathcal{P}) = \infty\) (infinite-dimensional)

• \(\dim(C[a,b]) = \infty\) (infinite-dimensional)

Theorem 6.36 (Dimension Theorem for Subspaces). If \(W\) is a subspace of a finite-dimensional vector space \(V\), then:

1. \(W\) is finite-dimensional

2. \(\dim(W) \leq \dim(V)\)

3. \(\dim(W) = \dim(V)\) if and only if \(W = V\)

Proof. (1) and (2): Any linearly independent set in \(W\) is also linearly independent in \(V\), so it can have at most \(\dim(V)\) vectors. This bounds the size of linearly independent sets in \(W\), ensuring \(W\) is finite-dimensional with \(\dim(W) \leq \dim(V)\).

(3): If \(\dim(W) = \dim(V)\) and \(\mathcal{B}\) is a basis for \(W\), then \(\mathcal{B}\) has \(\dim(V)\) vectors and is linearly independent in \(V\). By the dimension theorem, \(\mathcal{B}\) must span \(V\), so \(W = \operatorname{span}(\mathcal{B}) = V\).

Conversely, if \(W = V\), then clearly \(\dim(W) = \dim(V)\). ◻

Theorem 6.37 (Characterization of Bases in Finite-Dimensional Spaces). In an \(n\)-dimensional vector space \(V\):

1. Any set of \(n\) linearly independent vectors is a basis

2. Any set of \(n\) vectors that spans \(V\) is a basis

3. Any linearly independent set with fewer than \(n\) vectors can be extended to a basis

4. Any spanning set with more than \(n\) vectors contains a basis

Proof. (1): A linearly independent set of \(n = \dim(V)\) vectors in \(V\) must span \(V\) (otherwise, we could extend it to a larger linearly independent set, contradicting the dimension).

(2): A spanning set of \(n = \dim(V)\) vectors must be linearly independent (otherwise, we could remove a vector and still have a spanning set, contradicting minimality of bases).

(3) and (4): These follow from the basis extension and reduction theorems. ◻

Example 6.38 (Using Dimension to Test for Bases). 1. In \(\mathbb{R}^3\) (dimension 3): The set \(\{(1,2,3), (0,1,2), (1,0,1)\}\) has 3 vectors. To check if it’s a basis, we test linear independence:

\[c_1(1,2,3) + c_2(0,1,2) + c_3(1,0,1) = (0,0,0)\]

This gives the system: \[\begin{align} c_1 + c_3 &= 0\\ 2c_1 + c_2 &= 0\\ 3c_1 + 2c_2 + c_3 &= 0 \end{align}\]

The coefficient matrix has determinant \(\begin{vmatrix} 1 & 0 & 1 \\ 2 & 1 & 0 \\ 3 & 2 & 1 \end{vmatrix} = 1(1) - 0 + 1(4-3) = 2 \neq 0\).

Since the system has only the trivial solution, the vectors are linearly independent, hence form a basis.

2. In \(\mathcal{P}_2\) (dimension 3): The set \(\{1 + x, x + x^2, 1 + x^2\}\) has 3 polynomials. We showed earlier that they are linearly independent, so they form a basis for \(\mathcal{P}_2\).

Theorem 6.39 (Rank-Nullity Theorem for Abstract Vector Spaces). Let \(T: V \to W\) be a linear transformation between finite-dimensional vector spaces. Then: \[\dim(V) = \dim(\ker(T)) + \dim(\text{range}(T))\] where \(\ker(T) = \{\boldsymbol{v} \in V : T(\boldsymbol{v}) = \boldsymbol{0}\}\) is the kernel (null space) and \(\text{range}(T) = \{T(\boldsymbol{v}) : \boldsymbol{v} \in V\}\) is the range (column space).

Definition 6.40 (Coordinates and Coordinate Mapping). If \(\mathcal{B} = \{\boldsymbol{v}_1, \boldsymbol{v}_2, \ldots, \boldsymbol{v}_n\}\) is a basis for vector space \(V\), then the coordinate mapping \([\cdot]_{\mathcal{B}}: V \to \mathbb{R}^n\) is defined by: \[[\boldsymbol{v}]_{\mathcal{B}} = \begin{pmatrix} c_1 \\ c_2 \\ \vdots \\ c_n \end{pmatrix}\] where \(\boldsymbol{v} = c_1\boldsymbol{v}_1 + c_2\boldsymbol{v}_2 + \cdots + c_n\boldsymbol{v}_n\).

Theorem 6.41 (Properties of Coordinate Mapping). The coordinate mapping \([\cdot]_{\mathcal{B}}: V \to \mathbb{R}^n\) is:

1. Linear: \([a\boldsymbol{u} + b\boldsymbol{v}]_{\mathcal{B}} = a[\boldsymbol{u}]_{\mathcal{B}} + b[\boldsymbol{v}]_{\mathcal{B}}\)

2. Bijective (one-to-one and onto)

3. A vector space isomorphism

Corollary 6.42 (Finite-Dimensional Spaces are Isomorphic to \(\mathbb{R}^n\)). Every \(n\)-dimensional vector space is isomorphic to \(\mathbb{R}^n\).

Remark 350. This fundamental result means that, from an algebraic perspective, all finite-dimensional vector spaces of the same dimension are essentially identical. The coordinate mapping allows us to translate problems in abstract vector spaces into concrete problems in \(\mathbb{R}^n\).

6.5 Applications

Abstract vector space theory finds profound applications across mathematics. We explore two major areas: polynomial spaces and differential equations, demonstrating how the abstract framework illuminates concrete problems.

6.5.1 Polynomial Spaces

Polynomial spaces provide a natural setting for approximation theory, interpolation, and the study of algebraic structures.

Definition 6.44 (Polynomial Interpolation Problem). Given \(n+1\) distinct points \((x_0, y_0), (x_1, y_1), \ldots, (x_n, y_n)\), find a polynomial \(p(x) \in \mathcal{P}_n\) such that \(p(x_i) = y_i\) for \(i = 0, 1, \ldots, n\).

Theorem 6.45 (Existence and Uniqueness of Polynomial Interpolation). For any \(n+1\) distinct points \((x_0, y_0), (x_1, y_1), \ldots, (x_n, y_n)\), there exists a unique polynomial \(p(x) \in \mathcal{P}_n\) such that \(p(x_i) = y_i\) for all \(i\).

Proof. Existence: Define the linear transformation \(T: \mathcal{P}_n \to \mathbb{R}^{n+1}\) by: \[T(p) = (p(x_0), p(x_1), \ldots, p(x_n))\]

We want to show that \((y_0, y_1, \ldots, y_n)\) is in the range of \(T\).

Injectivity: If \(T(p) = \boldsymbol{0}\), then \(p(x_i) = 0\) for \(i = 0, 1, \ldots, n\). A non-zero polynomial of degree at most \(n\) can have at most \(n\) roots, but \(p\) has \(n+1\) roots. Therefore \(p = 0\), proving \(T\) is injective.

Surjectivity: Since \(\dim(\mathcal{P}_n) = n+1 = \dim(\mathbb{R}^{n+1})\) and \(T\) is injective, it must also be surjective.

Uniqueness: If \(p\) and \(q\) both satisfy the interpolation conditions, then \((p-q)(x_i) = 0\) for all \(i\). By the injectivity argument, \(p - q = 0\), so \(p = q\). ◻

Example 6.46 (Lagrange Interpolation). The Lagrange interpolating polynomial provides an explicit formula. Define the Lagrange basis polynomials: \[L_i(x) = \prod_{j=0, j \neq i}^n \frac{x - x_j}{x_i - x_j}\]

These satisfy \(L_i(x_j) = \delta_{ij}\) (Kronecker delta), forming a basis for \(\mathcal{P}_n\) adapted to the interpolation points.

The interpolating polynomial is: \[p(x) = \sum_{i=0}^n y_i L_i(x)\]

Example: Interpolate through points \((0,1)\), \((1,4)\), \((2,1)\).

\[L_0(x) = \frac{(x-1)(x-2)}{(0-1)(0-2)} = \frac{(x-1)(x-2)}{2}\] \[L_1(x) = \frac{(x-0)(x-2)}{(1-0)(1-2)} = \frac{x(x-2)}{-1} = -x(x-2)\] \[L_2(x) = \frac{(x-0)(x-1)}{(2-0)(2-1)} = \frac{x(x-1)}{2}\]

Therefore: \[p(x) = 1 \cdot \frac{(x-1)(x-2)}{2} + 4 \cdot (-x(x-2)) + 1 \cdot \frac{x(x-1)}{2}\] \[= \frac{(x-1)(x-2)}{2} - 4x(x-2) + \frac{x(x-1)}{2}\] \[= \frac{x^2 - 3x + 2 - 8x^2 + 16x + x^2 - x}{2} = \frac{-6x^2 + 12x + 2}{2} = -3x^2 + 6x + 1\]

Verification: \(p(0) = 1\), \(p(1) = -3 + 6 + 1 = 4\), \(p(2) = -12 + 12 + 1 = 1\)

Definition 6.47 (Chebyshev Polynomials). The Chebyshev polynomials of the first kind are defined by: \[T_n(\cos\theta) = \cos(n\theta)\] or equivalently by the recurrence: \[T_0(x) = 1, \quad T_1(x) = x, \quad T_{n+1}(x) = 2xT_n(x) - T_{n-1}(x)\]

Theorem 6.48 (Properties of Chebyshev Polynomials). The Chebyshev polynomials have remarkable approximation properties:

1. \(T_n(x)\) is a polynomial of degree \(n\) with leading coefficient \(2^{n-1}\) (for \(n \geq 1\))

2. Among all monic polynomials of degree \(n\), \(2^{1-n}T_n(x)\) has the smallest maximum absolute value on \([-1,1]\)

3. The roots of \(T_n(x)\) are \(x_k = \cos\left(\frac{(2k-1)\pi}{2n}\right)\) for \(k = 1, 2, \ldots, n\)

Example 6.49 (Polynomial Approximation Spaces). Consider the subspace \(W\) of \(\mathcal{P}_4\) consisting of polynomials that are even functions: \[W = \{p(x) \in \mathcal{P}_4 : p(-x) = p(x)\}\]

A polynomial \(p(x) = a_0 + a_1x + a_2x^2 + a_3x^3 + a_4x^4\) is even if and only if all odd-powered terms have zero coefficients: \(a_1 = a_3 = 0\).

Therefore: \(W = \{a_0 + a_2x^2 + a_4x^4 : a_0, a_2, a_4 \in \mathbb{R}\}\)

A basis for \(W\) is \(\{1, x^2, x^4\}\), so \(\dim(W) = 3\).

This demonstrates how symmetry conditions create interesting subspaces with reduced dimension.

6.5.2 Differential Equations

Linear differential equations provide one of the most important applications of vector space theory, where solution spaces form vector spaces and fundamental solution sets correspond to bases.

Definition 6.50 (Linear Differential Equation). An \(n\)-th order linear homogeneous differential equation has the form: \[y^{(n)} + a_{n-1}(x)y^{(n-1)} + \cdots + a_1(x)y' + a_0(x)y = 0\] where \(a_0(x), a_1(x), \ldots, a_{n-1}(x)\) are given functions.

Theorem 6.51 (Solution Space is a Vector Space). The set of all solutions to a linear homogeneous differential equation forms a vector space under pointwise addition and scalar multiplication.

Proof. Let \(\mathcal{S}\) be the solution set. We verify the vector space axioms:

Closure under addition: If \(y_1\) and \(y_2\) are solutions, then: \[(y_1 + y_2)^{(n)} + a_{n-1}(x)(y_1 + y_2)^{(n-1)} + \cdots + a_0(x)(y_1 + y_2)\] \[= [y_1^{(n)} + a_{n-1}(x)y_1^{(n-1)} + \cdots + a_0(x)y_1] + [y_2^{(n)} + a_{n-1}(x)y_2^{(n-1)} + \cdots + a_0(x)y_2]\] \[= 0 + 0 = 0\]

Closure under scalar multiplication: If \(y\) is a solution and \(c\) is a scalar: \[(cy)^{(n)} + a_{n-1}(x)(cy)^{(n-1)} + \cdots + a_0(x)(cy) = c[y^{(n)} + a_{n-1}(x)y^{(n-1)} + \cdots + a_0(x)y] = c \cdot 0 = 0\]

Zero vector: The zero function \(y(x) = 0\) is clearly a solution.

Other axioms: Inherited from the vector space of functions. ◻

Theorem 6.52 (Dimension of Solution Space). For an \(n\)-th order linear homogeneous differential equation with continuous coefficients on an interval \(I\), the solution space has dimension \(n\).

Definition 6.53 (Wronskian). For functions \(y_1, y_2, \ldots, y_n\), the Wronskian is the determinant: \[W(y_1, y_2, \ldots, y_n)(x) = \begin{vmatrix} y_1 & y_2 & \cdots & y_n\\ y_1' & y_2' & \cdots & y_n'\\ \vdots & \vdots & \ddots & \vdots\\ y_1^{(n-1)} & y_2^{(n-1)} & \cdots & y_n^{(n-1)} \end{vmatrix}\]

Theorem 6.54 (Wronskian Test for Linear Independence). Solutions \(y_1, y_2, \ldots, y_n\) of an \(n\)-th order linear differential equation are linearly independent if and only if their Wronskian is non-zero at some (equivalently, every) point in the interval.

Example 6.55 (Second-Order Constant Coefficient Equations). Consider \(y'' + ay' + by = 0\) where \(a\) and \(b\) are constants.

The characteristic equation is \(r^2 + ar + b = 0\) with discriminant \(\Delta = a^2 - 4b\).

Case 1: \(\Delta > 0\) (distinct real roots \(r_1, r_2\)) \[\mathcal{B} = \{e^{r_1x}, e^{r_2x}\}\]

Case 2: \(\Delta = 0\) (repeated root \(r\)) \[\mathcal{B} = \{e^{rx}, xe^{rx}\}\]

Case 3: \(\Delta < 0\) (complex roots \(\alpha \pm \beta i\)) \[\mathcal{B} = \{e^{\alpha x}\cos(\beta x), e^{\alpha x}\sin(\beta x)\}\]

In each case, \(\mathcal{B}\) forms a basis for the 2-dimensional solution space.

Example: Solve \(y'' - 3y' + 2y = 0\).

Characteristic equation: \(r^2 - 3r + 2 = 0 \Rightarrow (r-1)(r-2) = 0\) Roots: \(r_1 = 1, r_2 = 2\)

General solution: \(y = c_1e^x + c_2e^{2x}\)

Verification of linear independence: \[W(e^x, e^{2x}) = \begin{vmatrix} e^x & e^{2x} \\ e^x & 2e^{2x} \end{vmatrix} = e^x \cdot 2e^{2x} - e^{2x} \cdot e^x = e^{3x} \neq 0\]

Example 6.56 (Boundary Value Problems). Consider the differential equation \(y'' + \lambda y = 0\) on \([0, \pi]\) with boundary conditions \(y(0) = y(\pi) = 0\).

Case 1: \(\lambda \leq 0\) General solution: \(y = c_1e^{\sqrt{-\lambda}x} + c_2e^{-\sqrt{-\lambda}x}\) (for \(\lambda < 0\)) or \(y = c_1 + c_2x\) (for \(\lambda = 0\)).

Boundary conditions force \(c_1 = c_2 = 0\), giving only the trivial solution \(y = 0\).

Case 2: \(\lambda > 0\) General solution: \(y = c_1\cos(\sqrt{\lambda}x) + c_2\sin(\sqrt{\lambda}x)\)

Boundary conditions: - \(y(0) = c_1 = 0\) - \(y(\pi) = c_2\sin(\sqrt{\lambda}\pi) = 0\)

For non-trivial solutions, we need \(\sin(\sqrt{\lambda}\pi) = 0\), which gives \(\sqrt{\lambda}\pi = n\pi\) for positive integers \(n\).

Therefore: \(\lambda_n = n^2\) with corresponding eigenfunctions \(y_n(x) = \sin(nx)\).

The set \(\{\sin(x), \sin(2x), \sin(3x), \ldots\}\) forms an orthogonal basis for a certain function space, fundamental to Fourier series theory.

Key Idea

The Power of Abstract Vector Spaces:

• Unification: The same theoretical framework applies to polynomial interpolation, differential equations, and many other areas

• Dimension theory: Provides powerful tools for counting solutions and understanding structural properties

• Basis theory: Enables coordinate representations and transforms abstract problems into concrete linear algebra

• Linear independence: Captures fundamental concepts like the Wronskian test and fundamental solution sets

• Subspace structure: Reveals how constraints (boundary conditions, symmetries) create interesting subspaces

The abstraction of vector space theory pays dividends by providing a unified language for diverse mathematical phenomena, enabling techniques from one area to illuminate problems in another.

This completes our comprehensive study of abstract vector spaces. We have developed the axiomatic foundation, explored the rich variety of examples, established the fundamental concepts of linear independence and basis, investigated finite-dimensional spaces and their properties, and demonstrated applications to polynomial approximation and differential equations. The abstract framework provides both theoretical elegance and practical power, serving as a cornerstone for advanced mathematics and its applications.

7 Linear Transformations

Linear transformations are the fundamental morphisms in linear algebra, providing the natural way to relate different vector spaces while preserving their linear structure. Just as continuous functions are the natural maps between topological spaces and group homomorphisms are the natural maps between groups, linear transformations are the natural maps between vector spaces. This chapter develops the theory of linear transformations between abstract vector spaces, extending our previous work with matrix transformations to the general setting.

7.1 Examples and Properties

We begin by establishing the definition and fundamental properties of linear transformations, then explore the rich variety of examples that arise across different areas of mathematics.

Definition 7.1 (Linear Transformation). Let \(V\) and \(W\) be vector spaces over the same field \(\mathbb{F}\). A function \(T: V \to W\) is called a linear transformation (or linear map) if it satisfies:

1. Additivity: \(T(\boldsymbol{u} + \boldsymbol{v}) = T(\boldsymbol{u}) + T(\boldsymbol{v})\) for all \(\boldsymbol{u}, \boldsymbol{v} \in V\)

2. Homogeneity: \(T(c\boldsymbol{v}) = cT(\boldsymbol{v})\) for all \(\boldsymbol{v} \in V\) and \(c \in \mathbb{F}\)

Remark 365. These two conditions can be combined into a single equivalent condition: \[T(a\boldsymbol{u} + b\boldsymbol{v}) = aT(\boldsymbol{u}) + bT(\boldsymbol{v})\] for all \(\boldsymbol{u}, \boldsymbol{v} \in V\) and \(a, b \in \mathbb{F}\). This property is called linearity, and it captures the essence of what it means to "preserve linear combinations."

Theorem 7.3 (Basic Properties of Linear Transformations). If \(T: V \to W\) is a linear transformation, then:

1. \(T(\boldsymbol{0}_V) = \boldsymbol{0}_W\) (zero vector is mapped to zero vector)

2. \(T(-\boldsymbol{v}) = -T(\boldsymbol{v})\) for all \(\boldsymbol{v} \in V\) (negatives are preserved)

3. \(T(c_1\boldsymbol{v}_1 + c_2\boldsymbol{v}_2 + \cdots + c_n\boldsymbol{v}_n) = c_1T(\boldsymbol{v}_1) + c_2T(\boldsymbol{v}_2) + \cdots + c_nT(\boldsymbol{v}_n)\) (linearity extends to finite linear combinations)

Proof. Property 1: Using homogeneity with \(c = 0\): \[T(\boldsymbol{0}_V) = T(0 \cdot \boldsymbol{v}) = 0 \cdot T(\boldsymbol{v}) = \boldsymbol{0}_W\] for any \(\boldsymbol{v} \in V\).

Property 2: We have \(-\boldsymbol{v} = (-1)\boldsymbol{v}\), so: \[T(-\boldsymbol{v}) = T((-1)\boldsymbol{v}) = (-1)T(\boldsymbol{v}) = -T(\boldsymbol{v})\]

Property 3: This follows by induction on the number of terms, using additivity and homogeneity repeatedly. ◻

Example 7.4 (Matrix Transformations). Every \(m \times n\) matrix \(\mathbf{A}\) defines a linear transformation \(T_{\mathbf{A}}: \mathbb{R}^n \to \mathbb{R}^m\) by: \[T_{\mathbf{A}}(\boldsymbol{x}) = \mathbf{A}\boldsymbol{x}\]

We’ve already verified that matrix multiplication is linear. This provides our fundamental class of examples and, as we’ll see, every linear transformation between finite-dimensional spaces can be represented this way.

Specific examples:

• Identity transformation: \(I_V: V \to V\) defined by \(I_V(\boldsymbol{v}) = \boldsymbol{v}\)

• Zero transformation: \(\mathcal{O}: V \to W\) defined by \(\mathcal{O}(\boldsymbol{v}) = \boldsymbol{0}_W\)

• Scaling transformation: \(S_c: \mathbb{R}^n \to \mathbb{R}^n\) defined by \(S_c(\boldsymbol{v}) = c\boldsymbol{v}\)

• Projection transformations: Orthogonal projection onto subspaces

• Rotation transformations: Rotations in \(\mathbb{R}^2\) and \(\mathbb{R}^3\)

Example 7.5 (Differentiation and Integration). 1. Differentiation Operator: Define \(D: \mathcal{P}_n \to \mathcal{P}_{n-1}\) by \(D(p) = p'\) (the derivative of polynomial \(p\)).

Verification of linearity: \[D(af + bg) = (af + bg)' = af' + bg' = aD(f) + bD(g)\]

Explicit action: If \(p(x) = a_0 + a_1x + a_2x^2 + \cdots + a_nx^n\), then: \[D(p)(x) = a_1 + 2a_2x + 3a_3x^2 + \cdots + na_nx^{n-1}\]

2. Integration Operator: Define \(I: \mathcal{P}_n \to \mathcal{P}_{n+1}\) by \(I(p)(x) = \int_0^x p(t) \, dt\).

This is linear because: \[I(af + bg)(x) = \int_0^x (af(t) + bg(t)) \, dt = a\int_0^x f(t) \, dt + b\int_0^x g(t) \, dt = aI(f)(x) + bI(g)(x)\]

3. Higher-Order Derivatives: The \(k\)-th derivative operator \(D^k: C^k(\mathbb{R}) \to C(\mathbb{R})\) defined by \(D^k(f) = f^{(k)}\) is linear.

Example 7.6 (Evaluation and Point Functionals). 1. Evaluation at a Point: For a fixed \(a \in \mathbb{R}\), define \(\text{ev}_a: \mathcal{P} \to \mathbb{R}\) by \(\text{ev}_a(p) = p(a)\).

This is linear: \[\text{ev}_a(cp + dq) = (cp + dq)(a) = cp(a) + dq(a) = c \cdot \text{ev}_a(p) + d \cdot \text{ev}_a(q)\]

2. Linear Functionals: More generally, any linear transformation \(T: V \to \mathbb{F}\) (where \(\mathbb{F}\) is the scalar field) is called a linear functional. These form the dual space \(V^*\).

Examples of linear functionals:

• \(T(x_1, x_2, x_3) = 2x_1 - 3x_2 + x_3\) on \(\mathbb{R}^3\)

• \(T(p) = \int_0^1 p(x) \, dx\) on \(\mathcal{P}\)

• \(T(\mathbf{A}) = \operatorname{tr}(\mathbf{A})\) (trace) on \(\mathbb{R}^{n \times n}\)

Example 7.7 (Coordinate Projections and Inclusions). 1. Coordinate Projections: Define \(\pi_i: \mathbb{R}^n \to \mathbb{R}\) by \(\pi_i(x_1, x_2, \ldots, x_n) = x_i\).

These extract the \(i\)-th coordinate and are clearly linear.

2. Canonical Inclusion: If \(W\) is a subspace of \(V\), the inclusion map \(\iota: W \to V\) defined by \(\iota(\boldsymbol{w}) = \boldsymbol{w}\) is linear.

3. Direct Sum Projections: If \(V = U \oplus W\) (direct sum), there are natural projection maps \(\pi_U: V \to U\) and \(\pi_W: V \to W\) such that every \(\boldsymbol{v} \in V\) can be uniquely written as \(\boldsymbol{v} = \pi_U(\boldsymbol{v}) + \pi_W(\boldsymbol{v})\).

Theorem 7.8 (Linear Transformations Determined by Basis Images). Let \(V\) and \(W\) be vector spaces, and let \(\mathcal{B} = \{\boldsymbol{v}_1, \boldsymbol{v}_2, \ldots, \boldsymbol{v}_n\}\) be a basis for \(V\). For any choice of vectors \(\boldsymbol{w}_1, \boldsymbol{w}_2, \ldots, \boldsymbol{w}_n \in W\), there exists a unique linear transformation \(T: V \to W\) such that \(T(\boldsymbol{v}_i) = \boldsymbol{w}_i\) for \(i = 1, 2, \ldots, n\).

Proof. Existence: Since \(\mathcal{B}\) is a basis, every \(\boldsymbol{v} \in V\) can be uniquely written as: \[\boldsymbol{v} = c_1\boldsymbol{v}_1 + c_2\boldsymbol{v}_2 + \cdots + c_n\boldsymbol{v}_n\]

Define \(T: V \to W\) by: \[T(\boldsymbol{v}) = c_1\boldsymbol{w}_1 + c_2\boldsymbol{w}_2 + \cdots + c_n\boldsymbol{w}_n\]

Well-defined: Since the representation of \(\boldsymbol{v}\) in terms of the basis is unique, \(T(\boldsymbol{v})\) is uniquely determined.

Linearity: Let \(\boldsymbol{u} = a_1\boldsymbol{v}_1 + \cdots + a_n\boldsymbol{v}_n\) and \(\boldsymbol{v} = b_1\boldsymbol{v}_1 + \cdots + b_n\boldsymbol{v}_n\). Then: \[\boldsymbol{u} + \boldsymbol{v} = (a_1 + b_1)\boldsymbol{v}_1 + \cdots + (a_n + b_n)\boldsymbol{v}_n\]

So: \[T(\boldsymbol{u} + \boldsymbol{v}) = (a_1 + b_1)\boldsymbol{w}_1 + \cdots + (a_n + b_n)\boldsymbol{w}_n = T(\boldsymbol{u}) + T(\boldsymbol{v})\]

Homogeneity follows similarly.

Correctness: By construction, \(T(\boldsymbol{v}_i) = 0\boldsymbol{w}_1 + \cdots + 1\boldsymbol{w}_i + \cdots + 0\boldsymbol{w}_n = \boldsymbol{w}_i\).

Uniqueness: If \(S: V \to W\) is another linear transformation with \(S(\boldsymbol{v}_i) = \boldsymbol{w}_i\), then for any \(\boldsymbol{v} = c_1\boldsymbol{v}_1 + \cdots + c_n\boldsymbol{v}_n\): \[S(\boldsymbol{v}) = c_1S(\boldsymbol{v}_1) + \cdots + c_nS(\boldsymbol{v}_n) = c_1\boldsymbol{w}_1 + \cdots + c_n\boldsymbol{w}_n = T(\boldsymbol{v})\] ◻

Remark 372. This theorem is fundamental: it shows that linear transformations are completely determined by where they send basis vectors. This gives us enormous flexibility in constructing linear transformations and provides the foundation for matrix representations.

Example 7.10 (Constructing Linear Transformations via Basis Images). 1. From \(\mathbb{R}^2\) to \(\mathbb{R}^3\): Define \(T: \mathbb{R}^2 \to \mathbb{R}^3\) by specifying: \[T(1, 0) = (1, 2, 3), \quad T(0, 1) = (4, 5, 6)\]

For any \((x, y) \in \mathbb{R}^2\): \[T(x, y) = xT(1, 0) + yT(0, 1) = x(1, 2, 3) + y(4, 5, 6) = (x + 4y, 2x + 5y, 3x + 6y)\]

2. From \(\mathcal{P}_2\) to \(\mathcal{P}_3\): Define \(T: \mathcal{P}_2 \to \mathcal{P}_3\) by: \[T(1) = x, \quad T(x) = x^2, \quad T(x^2) = x^3\]

For any \(p(x) = a + bx + cx^2\): \[T(p) = aT(1) + bT(x) + cT(x^2) = ax + bx^2 + cx^3\]

This transformation "shifts up" the powers of \(x\).

3. Non-example: We cannot define a linear transformation \(T: \mathbb{R}^2 \to \mathbb{R}^2\) by \(T(x, y) = (x^2, y)\) because this violates linearity: \[T(2 \cdot (1, 0)) = T(2, 0) = (4, 0) \neq 2 \cdot (1, 0) = 2T(1, 0)\]

Theorem 7.11 (Operations on Linear Transformations). Let \(S, T: V \to W\) be linear transformations and \(c\) be a scalar. Then:

1. The sum \((S + T): V \to W\) defined by \((S + T)(\boldsymbol{v}) = S(\boldsymbol{v}) + T(\boldsymbol{v})\) is linear

2. The scalar multiple \((cT): V \to W\) defined by \((cT)(\boldsymbol{v}) = cT(\boldsymbol{v})\) is linear

3. The set \(\mathcal{L}(V, W)\) of all linear transformations from \(V\) to \(W\) forms a vector space under these operations

Proof. Sum is linear: \[(S + T)(a\boldsymbol{u} + b\boldsymbol{v}) = S(a\boldsymbol{u} + b\boldsymbol{v}) + T(a\boldsymbol{u} + b\boldsymbol{v})\] \[= aS(\boldsymbol{u}) + bS(\boldsymbol{v}) + aT(\boldsymbol{u}) + bT(\boldsymbol{v}) = a(S + T)(\boldsymbol{u}) + b(S + T)(\boldsymbol{v})\]

Scalar multiple is linear: \[(cT)(a\boldsymbol{u} + b\boldsymbol{v}) = cT(a\boldsymbol{u} + b\boldsymbol{v}) = c(aT(\boldsymbol{u}) + bT(\boldsymbol{v})) = a(cT)(\boldsymbol{u}) + b(cT)(\boldsymbol{v})\]

Vector space structure: All vector space axioms can be verified directly. The zero transformation serves as the additive identity, and \((-T)(\boldsymbol{v}) = -T(\boldsymbol{v})\) provides additive inverses. ◻

Definition 7.12 (Matrix Representation of Linear Transformations). Let \(T: V \to W\) be a linear transformation where \(V\) and \(W\) are finite-dimensional with bases \(\mathcal{B} = \{\boldsymbol{v}_1, \ldots, \boldsymbol{v}_n\}\) and \(\mathcal{C} = \{\boldsymbol{w}_1, \ldots, \boldsymbol{w}_m\}\) respectively. The matrix representation of \(T\) with respect to these bases is the \(m \times n\) matrix \([\mathbf{T}]_{\mathcal{C}}^{\mathcal{B}}\) whose \(j\)-th column is the coordinate vector \([T(\boldsymbol{v}_j)]_{\mathcal{C}}\).

Theorem 7.13 (Matrix Representation Theorem). If \(T: V \to W\) is linear and \([\mathbf{T}]_{\mathcal{C}}^{\mathcal{B}}\) is its matrix representation, then for any \(\boldsymbol{v} \in V\): \[[T(\boldsymbol{v})]_{\mathcal{C}} = [\mathbf{T}]_{\mathcal{C}}^{\mathcal{B}} [\boldsymbol{v}]_{\mathcal{B}}\]

Moreover, the map \(T \mapsto [\mathbf{T}]_{\mathcal{C}}^{\mathcal{B}}\) is a vector space isomorphism between \(\mathcal{L}(V, W)\) and \(\mathbb{R}^{m \times n}\).

Example 7.14 (Computing Matrix Representations). 1. Differentiation operator on \(\mathcal{P}_3\): Let \(D: \mathcal{P}_3 \to \mathcal{P}_2\) be the differentiation operator with standard bases \(\mathcal{B} = \{1, x, x^2, x^3\}\) and \(\mathcal{C} = \{1, x, x^2\}\).

\[D(1) = 0 = 0 \cdot 1 + 0 \cdot x + 0 \cdot x^2 \Rightarrow [D(1)]_{\mathcal{C}} = \begin{pmatrix} 0 \\ 0 \\ 0 \end{pmatrix}\]

\[D(x) = 1 = 1 \cdot 1 + 0 \cdot x + 0 \cdot x^2 \Rightarrow [D(x)]_{\mathcal{C}} = \begin{pmatrix} 1 \\ 0 \\ 0 \end{pmatrix}\]

\[D(x^2) = 2x = 0 \cdot 1 + 2 \cdot x + 0 \cdot x^2 \Rightarrow [D(x^2)]_{\mathcal{C}} = \begin{pmatrix} 0 \\ 2 \\ 0 \end{pmatrix}\]

\[D(x^3) = 3x^2 = 0 \cdot 1 + 0 \cdot x + 3 \cdot x^2 \Rightarrow [D(x^3)]_{\mathcal{C}} = \begin{pmatrix} 0 \\ 0 \\ 3 \end{pmatrix}\]

Therefore: \[[\mathbf{D}]_{\mathcal{C}}^{\mathcal{B}} = \begin{pmatrix} 0 & 1 & 0 & 0 \\ 0 & 0 & 2 & 0 \\ 0 & 0 & 0 & 3 \end{pmatrix}\]

Verification: For \(p(x) = 2 + 3x - x^2 + 4x^3\), we have \([p]_{\mathcal{B}} = (2, 3, -1, 4)^T\) and \(p'(x) = 3 - 2x + 12x^2\), so \([p']_{\mathcal{C}} = (3, -2, 12)^T\).

\[[\mathbf{D}]_{\mathcal{C}}^{\mathcal{B}} [p]_{\mathcal{B}} = \begin{pmatrix} 0 & 1 & 0 & 0 \\ 0 & 0 & 2 & 0 \\ 0 & 0 & 0 & 3 \end{pmatrix} \begin{pmatrix} 2 \\ 3 \\ -1 \\ 4 \end{pmatrix} = \begin{pmatrix} 3 \\ -2 \\ 12 \end{pmatrix}\]

7.2 Kernel and Image

Every linear transformation naturally gives rise to two important subspaces: the kernel (which generalizes the null space) and the image (which generalizes the column space). These subspaces encode essential information about the transformation’s behavior.

Definition 7.15 (Kernel and Image). Let \(T: V \to W\) be a linear transformation.

1. The kernel (or null space) of \(T\) is: \[\ker(T) = \{\boldsymbol{v} \in V : T(\boldsymbol{v}) = \boldsymbol{0}_W\}\]

2. The image (or range) of \(T\) is: \[\text{im}(T) = \{T(\boldsymbol{v}) : \boldsymbol{v} \in V\} = \{T(\boldsymbol{v}) \in W : \boldsymbol{v} \in V\}\]

Theorem 7.16 (Kernel and Image are Subspaces). For any linear transformation \(T: V \to W\):

1. \(\ker(T)\) is a subspace of \(V\)

2. \(\text{im}(T)\) is a subspace of \(W\)

Proof. Kernel is a subspace:

• Non-empty: \(T(\boldsymbol{0}_V) = \boldsymbol{0}_W\), so \(\boldsymbol{0}_V \in \ker(T)\)

• Closed under addition: If \(\boldsymbol{u}, \boldsymbol{v} \in \ker(T)\), then: \[T(\boldsymbol{u} + \boldsymbol{v}) = T(\boldsymbol{u}) + T(\boldsymbol{v}) = \boldsymbol{0}_W + \boldsymbol{0}_W = \boldsymbol{0}_W\] So \(\boldsymbol{u} + \boldsymbol{v} \in \ker(T)\).

• Closed under scalar multiplication: If \(\boldsymbol{v} \in \ker(T)\) and \(c\) is a scalar: \[T(c\boldsymbol{v}) = cT(\boldsymbol{v}) = c\boldsymbol{0}_W = \boldsymbol{0}_W\] So \(c\boldsymbol{v} \in \ker(T)\).

Image is a subspace:

• Non-empty: \(\boldsymbol{0}_W = T(\boldsymbol{0}_V) \in \text{im}(T)\)

• Closed under addition: If \(\boldsymbol{w}_1, \boldsymbol{w}_2 \in \text{im}(T)\), then \(\boldsymbol{w}_1 = T(\boldsymbol{v}_1)\) and \(\boldsymbol{w}_2 = T(\boldsymbol{v}_2)\) for some \(\boldsymbol{v}_1, \boldsymbol{v}_2 \in V\). Therefore: \[\boldsymbol{w}_1 + \boldsymbol{w}_2 = T(\boldsymbol{v}_1) + T(\boldsymbol{v}_2) = T(\boldsymbol{v}_1 + \boldsymbol{v}_2) \in \text{im}(T)\]

• Closed under scalar multiplication: If \(\boldsymbol{w} = T(\boldsymbol{v}) \in \text{im}(T)\) and \(c\) is a scalar: \[c\boldsymbol{w} = cT(\boldsymbol{v}) = T(c\boldsymbol{v}) \in \text{im}(T)\]

 ◻

Definition 7.17 (Rank and Nullity). For a linear transformation \(T: V \to W\):

• The rank of \(T\) is \(\operatorname{rank}(T) = \dim(\text{im}(T))\)

• The nullity of \(T\) is \(\operatorname{null}(T) = \dim(\ker(T))\)

Theorem 7.18 (Fundamental Theorem of Linear Algebra (Rank-Nullity Theorem)). If \(T: V \to W\) is a linear transformation and \(V\) is finite-dimensional, then: \[\dim(V) = \operatorname{null}(T) + \operatorname{rank}(T)\]

Proof. Let \(n = \dim(V)\) and \(k = \operatorname{null}(T) = \dim(\ker(T))\).

Step 1: Choose a basis \(\{\boldsymbol{u}_1, \boldsymbol{u}_2, \ldots, \boldsymbol{u}_k\}\) for \(\ker(T)\).

Step 2: Extend this to a basis \(\{\boldsymbol{u}_1, \boldsymbol{u}_2, \ldots, \boldsymbol{u}_k, \boldsymbol{v}_1, \boldsymbol{v}_2, \ldots, \boldsymbol{v}_{n-k}\}\) for \(V\).

Step 3: We claim that \(\{T(\boldsymbol{v}_1), T(\boldsymbol{v}_2), \ldots, T(\boldsymbol{v}_{n-k})\}\) is a basis for \(\text{im}(T)\).

Spanning: Any \(T(\boldsymbol{v}) \in \text{im}(T)\) can be written as: \[T(\boldsymbol{v}) = T\left(\sum_{i=1}^k a_i\boldsymbol{u}_i + \sum_{j=1}^{n-k} b_j\boldsymbol{v}_j\right) = \sum_{i=1}^k a_iT(\boldsymbol{u}_i) + \sum_{j=1}^{n-k} b_jT(\boldsymbol{v}_j) = \sum_{j=1}^{n-k} b_jT(\boldsymbol{v}_j)\] since \(T(\boldsymbol{u}_i) = \boldsymbol{0}\) for all \(i\).

Linear independence: Suppose \(\sum_{j=1}^{n-k} c_jT(\boldsymbol{v}_j) = \boldsymbol{0}\). Then: \[T\left(\sum_{j=1}^{n-k} c_j\boldsymbol{v}_j\right) = \boldsymbol{0}\]

This means \(\sum_{j=1}^{n-k} c_j\boldsymbol{v}_j \in \ker(T)\), so it can be written as: \[\sum_{j=1}^{n-k} c_j\boldsymbol{v}_j = \sum_{i=1}^k d_i\boldsymbol{u}_i\]

Rearranging: \[\sum_{i=1}^k (-d_i)\boldsymbol{u}_i + \sum_{j=1}^{n-k} c_j\boldsymbol{v}_j = \boldsymbol{0}\]

Since the full set is a basis for \(V\), all coefficients must be zero: \(c_j = 0\) for all \(j\).

Conclusion: \(\operatorname{rank}(T) = \dim(\text{im}(T)) = n - k = \dim(V) - \operatorname{null}(T)\). ◻

Remark 382. The Rank-Nullity Theorem is one of the most important results in linear algebra. It provides a fundamental constraint relating the dimension of the domain, the size of the kernel, and the size of the image. This theorem has profound implications for understanding when linear transformations are injective, surjective, or bijective.

Example 7.20 (Computing Kernels and Images). 1. Differentiation Operator: Consider \(D: \mathcal{P}_3 \to \mathcal{P}_2\) defined by \(D(p) = p'\).

Kernel: \(\ker(D) = \{p \in \mathcal{P}_3 : p' = 0\} = \{c : c \in \mathbb{R}\}\) (constant polynomials) So \(\operatorname{null}(D) = 1\).

Image: Since \(D(a + bx + cx^2 + dx^3) = b + 2cx + 3dx^2\), we can achieve any polynomial in \(\mathcal{P}_2\) by choosing appropriate coefficients. Therefore \(\text{im}(D) = \mathcal{P}_2\), so \(\operatorname{rank}(D) = 3\).

Verification: \(\dim(\mathcal{P}_3) = 4 = 1 + 3 = \operatorname{null}(D) + \operatorname{rank}(D)\)

2. Linear Functional: Consider \(T: \mathbb{R}^3 \to \mathbb{R}\) defined by \(T(x, y, z) = x + 2y - z\).

Kernel: \(\ker(T) = \{(x, y, z) : x + 2y - z = 0\}\) (a plane through the origin) A basis for \(\ker(T)\) can be found by solving \(x + 2y - z = 0\): Setting \(y = s, z = t\), we get \(x = -2s + t\), so: \[\ker(T) = \operatorname{span}\{(-2, 1, 0), (1, 0, 1)\}\] Therefore \(\operatorname{null}(T) = 2\).

Image: \(\text{im}(T) = \mathbb{R}\) since \(T\) is surjective (e.g., \(T(a, 0, 0) = a\) for any \(a \in \mathbb{R}\)). So \(\operatorname{rank}(T) = 1\).

Verification: \(\dim(\mathbb{R}^3) = 3 = 2 + 1 = \operatorname{null}(T) + \operatorname{rank}(T)\)

3. Matrix Transformation: Consider \(T: \mathbb{R}^4 \to \mathbb{R}^3\) represented by: \[\mathbf{A} = \begin{pmatrix} 1 & 2 & 0 & 1 \\ 0 & 1 & 1 & 2 \\ 1 & 3 & 1 & 3 \end{pmatrix}\]

Row reducing the augmented matrix \([\mathbf{A}|\boldsymbol{0}]\): \[\begin{pmatrix} 1 & 2 & 0 & 1 & 0 \\ 0 & 1 & 1 & 2 & 0 \\ 1 & 3 & 1 & 3 & 0 \end{pmatrix} \sim \begin{pmatrix} 1 & 0 & -2 & -3 & 0 \\ 0 & 1 & 1 & 2 & 0 \\ 0 & 0 & 0 & 0 & 0 \end{pmatrix}\]

From this RREF, we see that \(x_3\) and \(x_4\) are free variables. Setting \(x_3 = s, x_4 = t\): \[x_1 = 2s + 3t, \quad x_2 = -s - 2t\]

So \(\ker(T) = \operatorname{span}\{(2, -1, 1, 0), (3, -2, 0, 1)\}\) and \(\operatorname{null}(T) = 2\).

The rank is \(\operatorname{rank}(T) = 4 - 2 = 2\), which we can verify by noting that the RREF has 2 pivot columns.

Theorem 7.21 (Characterizations via Kernel and Image). Let \(T: V \to W\) be a linear transformation. Then:

1. \(T\) is injective (one-to-one) if and only if \(\ker(T) = \{\boldsymbol{0}\}\)

2. \(T\) is surjective (onto) if and only if \(\text{im}(T) = W\)

3. If \(\dim(V) = \dim(W)\), then \(T\) is bijective if and only if \(T\) is injective if and only if \(T\) is surjective

Proof. Property 1: (\(\Rightarrow\)) If \(T\) is injective and \(\boldsymbol{v} \in \ker(T)\), then \(T(\boldsymbol{v}) = \boldsymbol{0} = T(\boldsymbol{0})\). By injectivity, \(\boldsymbol{v} = \boldsymbol{0}\).

(\(\Leftarrow\)) Suppose \(\ker(T) = \{\boldsymbol{0}\}\) and \(T(\boldsymbol{u}) = T(\boldsymbol{v})\). Then \(T(\boldsymbol{u} - \boldsymbol{v}) = T(\boldsymbol{u}) - T(\boldsymbol{v}) = \boldsymbol{0}\), so \(\boldsymbol{u} - \boldsymbol{v} \in \ker(T) = \{\boldsymbol{0}\}\). Therefore \(\boldsymbol{u} = \boldsymbol{v}\).

Property 2: This is immediate from the definition of surjectivity.

Property 3: When \(\dim(V) = \dim(W) = n\), the Rank-Nullity Theorem gives: \[n = \operatorname{null}(T) + \operatorname{rank}(T)\]

\(T\) is injective \(\Leftrightarrow \ker(T) = \{\boldsymbol{0}\} \Leftrightarrow \operatorname{null}(T) = 0 \Leftrightarrow \operatorname{rank}(T) = n \Leftrightarrow \text{im}(T) = W \Leftrightarrow T\) is surjective. ◻

Definition 7.22 (Change of Basis for Linear Transformations). If \(T: V \to V\) is a linear transformation and \(\mathcal{B}, \mathcal{C}\) are two bases for \(V\), then the matrix representations \([\mathbf{T}]_{\mathcal{B}}\) and \([\mathbf{T}]_{\mathcal{C}}\) are related by: \[[\mathbf{T}]_{\mathcal{C}} = [\mathbf{P}]_{\mathcal{C}}^{\mathcal{B}} [\mathbf{T}]_{\mathcal{B}} [\mathbf{P}]_{\mathcal{B}}^{\mathcal{C}}\] where \([\mathbf{P}]_{\mathcal{B}}^{\mathcal{C}}\) is the change of basis matrix from \(\mathcal{C}\) to \(\mathcal{B}\).

Remark 386. This formula shows that matrix representations of the same linear transformation with respect to different bases are similar. This is why similar matrices represent the same linear transformation in different coordinate systems.

7.3 Isomorphisms and Composition

Isomorphisms are the "perfect" linear transformations—bijective maps that preserve all linear structure. They allow us to identify different vector spaces as being "essentially the same" from a linear algebra perspective.

Definition 7.24 (Isomorphism). A linear transformation \(T: V \to W\) is called an isomorphism if it is bijective (both injective and surjective). In this case, we say that \(V\) and \(W\) are isomorphic, denoted \(V \cong W\).

Theorem 7.25 (Properties of Isomorphisms). If \(T: V \to W\) is an isomorphism, then:

1. \(T^{-1}: W \to V\) exists and is also an isomorphism

2. \(T\) preserves linear combinations: \(T(c_1\boldsymbol{v}_1 + \cdots + c_n\boldsymbol{v}_n) = c_1T(\boldsymbol{v}_1) + \cdots + c_nT(\boldsymbol{v}_n)\)

3. \(T\) maps linearly independent sets to linearly independent sets

4. \(T\) maps spanning sets to spanning sets

5. \(T\) maps bases to bases

Proof. Property 1: Since \(T\) is bijective, \(T^{-1}\) exists. To show linearity of \(T^{-1}\): Let \(\boldsymbol{w}_1, \boldsymbol{w}_2 \in W\) with \(T^{-1}(\boldsymbol{w}_1) = \boldsymbol{v}_1\) and \(T^{-1}(\boldsymbol{w}_2) = \boldsymbol{v}_2\). Then: \[T^{-1}(a\boldsymbol{w}_1 + b\boldsymbol{w}_2) = T^{-1}(aT(\boldsymbol{v}_1) + bT(\boldsymbol{v}_2)) = T^{-1}(T(a\boldsymbol{v}_1 + b\boldsymbol{v}_2)) = a\boldsymbol{v}_1 + b\boldsymbol{v}_2 = aT^{-1}(\boldsymbol{w}_1) + bT^{-1}(\boldsymbol{w}_2)\]

Property 3: Suppose \(\{\boldsymbol{v}_1, \ldots, \boldsymbol{v}_k\}\) is linearly independent in \(V\). If \(c_1T(\boldsymbol{v}_1) + \cdots + c_kT(\boldsymbol{v}_k) = \boldsymbol{0}\), then: \[T(c_1\boldsymbol{v}_1 + \cdots + c_k\boldsymbol{v}_k) = \boldsymbol{0}\] Since \(T\) is injective, \(c_1\boldsymbol{v}_1 + \cdots + c_k\boldsymbol{v}_k = \boldsymbol{0}\). By linear independence, all \(c_i = 0\).

Properties 4 and 5: Follow from Properties 1-3 and the fact that \(T\) is surjective. ◻

Theorem 7.26 (Fundamental Isomorphism Theorem). Two finite-dimensional vector spaces are isomorphic if and only if they have the same dimension.

Proof. (\(\Rightarrow\)) If \(T: V \to W\) is an isomorphism, then \(T\) maps a basis of \(V\) to a basis of \(W\), so \(\dim(V) = \dim(W)\).

(\(\Leftarrow\)) Suppose \(\dim(V) = \dim(W) = n\). Let \(\mathcal{B} = \{\boldsymbol{v}_1, \ldots, \boldsymbol{v}_n\}\) be a basis for \(V\) and \(\mathcal{C} = \{\boldsymbol{w}_1, \ldots, \boldsymbol{w}_n\}\) be a basis for \(W\). Define \(T: V \to W\) by \(T(\boldsymbol{v}_i) = \boldsymbol{w}_i\) and extend linearly.

\(T\) is injective because if \(T(\boldsymbol{v}) = \boldsymbol{0}\) for \(\boldsymbol{v} = c_1\boldsymbol{v}_1 + \cdots + c_n\boldsymbol{v}_n\), then \(c_1\boldsymbol{w}_1 + \cdots + c_n\boldsymbol{w}_n = \boldsymbol{0}\). Since \(\mathcal{C}\) is a basis, all \(c_i = 0\), so \(\boldsymbol{v} = \boldsymbol{0}\).

\(T\) is surjective because any \(\boldsymbol{w} = d_1\boldsymbol{w}_1 + \cdots + d_n\boldsymbol{w}_n \in W\) is the image of \(\boldsymbol{v} = d_1\boldsymbol{v}_1 + \cdots + d_n\boldsymbol{v}_n \in V\). ◻

Corollary 7.27 (Classification of Finite-Dimensional Vector Spaces). Every \(n\)-dimensional vector space over \(\mathbb{F}\) is isomorphic to \(\mathbb{F}^n\). Thus, up to isomorphism, there is exactly one vector space of each finite dimension over a given field.

Example 7.28 (Standard Isomorphisms). 1. Coordinate Isomorphism: For any \(n\)-dimensional vector space \(V\) with basis \(\mathcal{B}\), the coordinate map \([\cdot]_{\mathcal{B}}: V \to \mathbb{R}^n\) is an isomorphism.

2. Polynomial-Vector Isomorphism: \(\mathcal{P}_n \cong \mathbb{R}^{n+1}\) via the map \(T(a_0 + a_1x + \cdots + a_nx^n) = (a_0, a_1, \ldots, a_n)\).

3. Matrix-Vector Isomorphism: \(\mathbb{R}^{m \times n} \cong \mathbb{R}^{mn}\) by vectorizing matrices (stacking columns).

4. Function Space Isomorphism (Finite-Dimensional): The space of polynomials of degree at most \(n\) is isomorphic to the space of sequences \((a_0, a_1, \ldots, a_n)\), connecting function spaces to coordinate spaces.

Now we turn to composition of linear transformations:

Definition 7.29 (Composition of Linear Transformations). If \(T: U \to V\) and \(S: V \to W\) are linear transformations, their composition \(S \circ T: U \to W\) is defined by: \[(S \circ T)(\boldsymbol{u}) = S(T(\boldsymbol{u}))\]

Theorem 7.30 (Composition Preserves Linearity). If \(T: U \to V\) and \(S: V \to W\) are linear transformations, then \(S \circ T: U \to W\) is also linear.

Proof. For any \(\boldsymbol{u}_1, \boldsymbol{u}_2 \in U\) and scalars \(a, b\): \[(S \circ T)(a\boldsymbol{u}_1 + b\boldsymbol{u}_2) = S(T(a\boldsymbol{u}_1 + b\boldsymbol{u}_2)) = S(aT(\boldsymbol{u}_1) + bT(\boldsymbol{u}_2))\] \[= aS(T(\boldsymbol{u}_1)) + bS(T(\boldsymbol{u}_2)) = a(S \circ T)(\boldsymbol{u}_1) + b(S \circ T)(\boldsymbol{u}_2)\] ◻

Theorem 7.31 (Matrix Representation of Composition). If \(T: U \to V\) and \(S: V \to W\) are linear transformations with matrix representations \([\mathbf{T}]\) and \([\mathbf{S}]\) respectively, then: \[[\mathbf{S \circ T}] = [\mathbf{S}][\mathbf{T}]\]

Remark 395. This theorem explains why matrix multiplication is defined the way it is: it corresponds exactly to composition of the associated linear transformations. The "backwards" order (reading right to left) reflects the fact that in function composition \((S \circ T)(\boldsymbol{u})\), we apply \(T\) first, then \(S\).

Example 7.33 (Composition in Action). 1. Successive Rotations: Consider rotations \(R_\alpha: \mathbb{R}^2 \to \mathbb{R}^2\) by angle \(\alpha\) and \(R_\beta: \mathbb{R}^2 \to \mathbb{R}^2\) by angle \(\beta\). Their composition \(R_\beta \circ R_\alpha\) is rotation by angle \(\alpha + \beta\).

This corresponds to matrix multiplication: \[\begin{pmatrix} \cos\beta & -\sin\beta \\ \sin\beta & \cos\beta \end{pmatrix}\begin{pmatrix} \cos\alpha & -\sin\alpha \\ \sin\alpha & \cos\alpha \end{pmatrix} = \begin{pmatrix} \cos(\alpha+\beta) & -\sin(\alpha+\beta) \\ \sin(\alpha+\beta) & \cos(\alpha+\beta) \end{pmatrix}\]

2. Differentiation followed by Integration: Consider \(D: \mathcal{P}_2 \to \mathcal{P}_1\) (differentiation) and \(I: \mathcal{P}_1 \to \mathcal{P}_2\) (integration from 0 to \(x\)).

For \(p(x) = ax^2 + bx + c\): \[D(p) = 2ax + b\] \[(I \circ D)(p)(x) = \int_0^x (2at + b) \, dt = ax^2 + bx\]

So \((I \circ D)(p) = p - c\) (integration of derivative loses the constant term).

Matrix representations: \[[\mathbf{D}] = \begin{pmatrix} 0 & 1 & 0 \\ 0 & 0 & 2 \end{pmatrix}, \quad [\mathbf{I}] = \begin{pmatrix} 0 & 0 \\ 1 & 0 \\ 0 & \frac{1}{2} \end{pmatrix}\]

\[[\mathbf{I}][\mathbf{D}] = \begin{pmatrix} 0 & 0 \\ 1 & 0 \\ 0 & \frac{1}{2} \end{pmatrix}\begin{pmatrix} 0 & 1 & 0 \\ 0 & 0 & 2 \end{pmatrix} = \begin{pmatrix} 0 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{pmatrix}\]

This matrix representation confirms that \((I \circ D)\) acts as \(p(x) \mapsto p(x) - p(0)\).

Theorem 7.34 (Properties of Composition). Composition of linear transformations satisfies:

1. Associativity: \((R \circ S) \circ T = R \circ (S \circ T)\) when defined

2. Identity: \(I_V \circ T = T\) and \(T \circ I_U = T\) for \(T: U \to V\)

3. Distributivity: \((S_1 + S_2) \circ T = S_1 \circ T + S_2 \circ T\) and \(S \circ (T_1 + T_2) = S \circ T_1 + S \circ T_2\) when defined

Definition 7.35 (Invertible Linear Transformation). A linear transformation \(T: V \to W\) is invertible if there exists a linear transformation \(S: W \to V\) such that \(S \circ T = I_V\) and \(T \circ S = I_W\). In this case, \(S = T^{-1}\) is unique and called the inverse of \(T\).

Theorem 7.36 (Characterization of Invertible Transformations). A linear transformation \(T: V \to W\) is invertible if and only if \(T\) is an isomorphism.

Proof. (\(\Rightarrow\)) If \(T\) is invertible with inverse \(S\), then \(T\) is bijective: - Injective: If \(T(\boldsymbol{v}_1) = T(\boldsymbol{v}_2)\), then \(\boldsymbol{v}_1 = S(T(\boldsymbol{v}_1)) = S(T(\boldsymbol{v}_2)) = \boldsymbol{v}_2\). - Surjective: For any \(\boldsymbol{w} \in W\), we have \(\boldsymbol{w} = T(S(\boldsymbol{w}))\), so \(\boldsymbol{w}\) is in the image of \(T\).

(\(\Leftarrow\)) If \(T\) is an isomorphism, then \(T^{-1}\) exists as a function and is linear (as proved earlier). The compositions \(T^{-1} \circ T\) and \(T \circ T^{-1}\) are the appropriate identity transformations. ◻

7.4 Applications

The abstract theory of linear transformations finds powerful applications across mathematics. We explore two significant areas: the fundamental dimension theorem and applications to recurrence relations and differential equations.

7.4.1 The Dimension Theorem

The dimension theorem provides deep insights into the structure of vector spaces and their subspaces. We explore several important consequences and applications.

Theorem 7.37 (Dimension Theorem for Subspace Sums). If \(U\) and \(W\) are finite-dimensional subspaces of a vector space \(V\), then: \[\dim(U + W) = \dim(U) + \dim(W) - \dim(U \cap W)\] where \(U + W = \{\boldsymbol{u} + \boldsymbol{w} : \boldsymbol{u} \in U, \boldsymbol{w} \in W\}\) is the sum of the subspaces.

Proof. Let \(\dim(U \cap W) = k\), \(\dim(U) = m\), and \(\dim(W) = n\).

Step 1: Choose a basis \(\{\boldsymbol{v}_1, \ldots, \boldsymbol{v}_k\}\) for \(U \cap W\).

Step 2: Extend this to a basis \(\{\boldsymbol{v}_1, \ldots, \boldsymbol{v}_k, \boldsymbol{u}_1, \ldots, \boldsymbol{u}_{m-k}\}\) for \(U\).

Step 3: Extend the intersection basis to a basis \(\{\boldsymbol{v}_1, \ldots, \boldsymbol{v}_k, \boldsymbol{w}_1, \ldots, \boldsymbol{w}_{n-k}\}\) for \(W\).

Step 4: We claim that \(\mathcal{B} = \{\boldsymbol{v}_1, \ldots, \boldsymbol{v}_k, \boldsymbol{u}_1, \ldots, \boldsymbol{u}_{m-k}, \boldsymbol{w}_1, \ldots, \boldsymbol{w}_{n-k}\}\) is a basis for \(U + W\).

Spanning: Any element of \(U + W\) has the form \(\boldsymbol{u} + \boldsymbol{w}\) where: \[\boldsymbol{u} = \sum_{i=1}^k a_i\boldsymbol{v}_i + \sum_{j=1}^{m-k} b_j\boldsymbol{u}_j, \quad \boldsymbol{w} = \sum_{i=1}^k c_i\boldsymbol{v}_i + \sum_{\ell=1}^{n-k} d_\ell\boldsymbol{w}_\ell\]

Therefore: \[\boldsymbol{u} + \boldsymbol{w} = \sum_{i=1}^k (a_i + c_i)\boldsymbol{v}_i + \sum_{j=1}^{m-k} b_j\boldsymbol{u}_j + \sum_{\ell=1}^{n-k} d_\ell\boldsymbol{w}_\ell \in \operatorname{span}(\mathcal{B})\]

Linear independence: Suppose: \[\sum_{i=1}^k \alpha_i\boldsymbol{v}_i + \sum_{j=1}^{m-k} \beta_j\boldsymbol{u}_j + \sum_{\ell=1}^{n-k} \gamma_\ell\boldsymbol{w}_\ell = \boldsymbol{0}\]

Rearranging: \[\sum_{i=1}^k \alpha_i\boldsymbol{v}_i + \sum_{j=1}^{m-k} \beta_j\boldsymbol{u}_j = -\sum_{\ell=1}^{n-k} \gamma_\ell\boldsymbol{w}_\ell\]

The left side is in \(U\), the right side is in \(W\), so both sides are in \(U \cap W\). Therefore: \[-\sum_{\ell=1}^{n-k} \gamma_\ell\boldsymbol{w}_\ell = \sum_{i=1}^k \delta_i\boldsymbol{v}_i\]

for some scalars \(\delta_i\). This gives: \[\sum_{i=1}^k \delta_i\boldsymbol{v}_i + \sum_{\ell=1}^{n-k} \gamma_\ell\boldsymbol{w}_\ell = \boldsymbol{0}\]

Since \(\{\boldsymbol{v}_1, \ldots, \boldsymbol{v}_k, \boldsymbol{w}_1, \ldots, \boldsymbol{w}_{n-k}\}\) is a basis for \(W\), all coefficients are zero: \(\delta_i = 0\) and \(\gamma_\ell = 0\).

Similarly, substituting back shows \(\alpha_i = 0\) and \(\beta_j = 0\).

Conclusion: \(\dim(U + W) = k + (m-k) + (n-k) = m + n - k = \dim(U) + \dim(W) - \dim(U \cap W)\). ◻

Example 7.38 (Applying the Dimension Theorem). In \(\mathbb{R}^4\), consider: \[U = \operatorname{span}\{(1,1,0,0), (0,1,1,0)\}, \quad W = \operatorname{span}\{(1,0,1,1), (0,1,0,1)\}\]

Find \(\dim(U \cap W)\): A vector \((a,b,c,d) \in U \cap W\) satisfies: \[\begin{cases} a = s \\ b = s + t \\ c = t \\ d = 0 \end{cases} \quad \text{and} \quad \begin{cases} a = r \\ b = q \\ c = r \\ d = q \end{cases}\]

From these conditions: \(a = s = r\), \(c = t = r\), \(b = s + t = q\), \(d = 0 = q\). So \(s = r\), \(t = r\), \(q = 0\), giving us \(a = c = s\) and \(b = 2s\), \(d = 0\).

Therefore \(U \cap W = \operatorname{span}\{(1,2,1,0)\}\) and \(\dim(U \cap W) = 1\).

Apply the dimension theorem: \[\dim(U + W) = \dim(U) + \dim(W) - \dim(U \cap W) = 2 + 2 - 1 = 3\]

Verification: The set \(\{(1,1,0,0), (0,1,1,0), (1,0,1,1), (0,1,0,1)\}\) spans \(U + W\). We can verify that three of these vectors are linearly independent, confirming \(\dim(U + W) = 3\).

Definition 7.39 (Direct Sum of Subspaces). Subspaces \(U\) and \(W\) of a vector space \(V\) form a direct sum, denoted \(V = U \oplus W\), if:

1. \(V = U + W\) (every vector in \(V\) can be written as \(\boldsymbol{u} + \boldsymbol{w}\))

2. \(U \cap W = \{\boldsymbol{0}\}\) (the intersection is trivial)

Theorem 7.40 (Characterization of Direct Sums). The following are equivalent for subspaces \(U\) and \(W\) of \(V\):

1. \(V = U \oplus W\)

2. Every \(\boldsymbol{v} \in V\) can be uniquely written as \(\boldsymbol{v} = \boldsymbol{u} + \boldsymbol{w}\) with \(\boldsymbol{u} \in U, \boldsymbol{w} \in W\)

3. \(\dim(V) = \dim(U) + \dim(W)\) and \(U \cap W = \{\boldsymbol{0}\}\)

4. There exist bases \(\mathcal{B}_U\) for \(U\) and \(\mathcal{B}_W\) for \(W\) such that \(\mathcal{B}_U \cup \mathcal{B}_W\) is a basis for \(V\)

7.4.2 Linear Recurrences and Differential Equations

Linear transformations provide a unifying framework for understanding linear recurrence relations and systems of differential equations.

Definition 7.41 (Linear Recurrence Relation). A \(k\)-th order linear recurrence relation has the form: \[a_n = c_1a_{n-1} + c_2a_{n-2} + \cdots + c_ka_{n-k} + f(n)\] where \(c_1, c_2, \ldots, c_k\) are constants and \(f(n)\) is a given function. When \(f(n) = 0\), the recurrence is homogeneous.

Theorem 7.42 (Vector Space of Solutions to Homogeneous Recurrences). The set of all solutions to a \(k\)-th order homogeneous linear recurrence relation forms a \(k\)-dimensional vector space.

Proof. Define the state vector \(\boldsymbol{s}_n = (a_n, a_{n-1}, \ldots, a_{n-k+1})^T\). The recurrence relation becomes: \[\boldsymbol{s}_{n+1} = \mathbf{A}\boldsymbol{s}_n\] where \[\mathbf{A} = \begin{pmatrix} c_1 & c_2 & c_3 & \cdots & c_k \\ 1 & 0 & 0 & \cdots & 0 \\ 0 & 1 & 0 & \cdots & 0 \\ \vdots & \vdots & \ddots & \ddots & \vdots \\ 0 & 0 & 0 & \cdots & 0 \end{pmatrix}\]

The solution is determined by the initial conditions \((a_0, a_1, \ldots, a_{k-1})\), which form a \(k\)-dimensional space. The linearity of the recurrence ensures that the solution space is a vector space of dimension \(k\). ◻

Example 7.43 (Fibonacci Sequence via Linear Transformations). The Fibonacci recurrence \(F_n = F_{n-1} + F_{n-2}\) with \(F_0 = 0, F_1 = 1\) becomes: \[\begin{pmatrix} F_{n+1} \\ F_n \end{pmatrix} = \begin{pmatrix} 1 & 1 \\ 1 & 0 \end{pmatrix}\begin{pmatrix} F_n \\ F_{n-1} \end{pmatrix}\]

The matrix \(\mathbf{A} = \begin{pmatrix} 1 & 1 \\ 1 & 0 \end{pmatrix}\) has characteristic polynomial \(\lambda^2 - \lambda - 1 = 0\) with roots \(\phi = \frac{1+\sqrt{5}}{2}\) and \(\hat{\phi} = \frac{1-\sqrt{5}}{2}\).

The general solution is \(F_n = c_1\phi^n + c_2\hat{\phi}^n\). Using initial conditions: \[F_0 = c_1 + c_2 = 0, \quad F_1 = c_1\phi + c_2\hat{\phi} = 1\]

Solving: \(c_1 = \frac{1}{\sqrt{5}}, c_2 = -\frac{1}{\sqrt{5}}\), giving Binet’s formula: \[F_n = \frac{\phi^n - \hat{\phi}^n}{\sqrt{5}}\]

For systems of differential equations, linear transformations provide the natural framework:

Definition 7.44 (System of Linear Differential Equations). A system of linear differential equations has the form: \[\frac{d\boldsymbol{x}}{dt} = \mathbf{A}\boldsymbol{x} + \boldsymbol{f}(t)\] where \(\boldsymbol{x}(t) \in \mathbb{R}^n\), \(\mathbf{A}\) is an \(n \times n\) constant matrix, and \(\boldsymbol{f}(t)\) is a given vector function. When \(\boldsymbol{f}(t) = \boldsymbol{0}\), the system is homogeneous.

Theorem 7.45 (Solution Space for Homogeneous Systems). The solution space of the homogeneous system \(\frac{d\boldsymbol{x}}{dt} = \mathbf{A}\boldsymbol{x}\) forms an \(n\)-dimensional vector space. If \(\mathbf{A}\) has \(n\) linearly independent eigenvectors with eigenvalues \(\lambda_1, \ldots, \lambda_n\), then the general solution is: \[\boldsymbol{x}(t) = c_1e^{\lambda_1 t}\boldsymbol{v}_1 + c_2e^{\lambda_2 t}\boldsymbol{v}_2 + \cdots + c_ne^{\lambda_n t}\boldsymbol{v}_n\] where \(\boldsymbol{v}_i\) are the corresponding eigenvectors.

Example 7.46 (Coupled Oscillator System). Consider the system: \[\frac{d}{dt}\begin{pmatrix} x \\ y \end{pmatrix} = \begin{pmatrix} 0 & 1 \\ -4 & -4 \end{pmatrix}\begin{pmatrix} x \\ y \end{pmatrix}\]

Find eigenvalues: \[\det\begin{pmatrix} -\lambda & 1 \\ -4 & -4-\lambda \end{pmatrix} = \lambda(\lambda + 4) + 4 = \lambda^2 + 4\lambda + 4 = (\lambda + 2)^2\]

So \(\lambda = -2\) is a repeated eigenvalue.

Find eigenvectors: \[\begin{pmatrix} 2 & 1 \\ -4 & -2 \end{pmatrix}\begin{pmatrix} v_1 \\ v_2 \end{pmatrix} = \begin{pmatrix} 0 \\ 0 \end{pmatrix}\]

This gives \(2v_1 + v_2 = 0\), so \(v_2 = -2v_1\). The eigenspace is one-dimensional: \(\operatorname{span}\{(1, -2)^T\}\).

Since we have a repeated eigenvalue with geometric multiplicity 1, we need a generalized eigenvector. The general solution involves terms of the form \(e^{-2t}(c_1\boldsymbol{v} + c_2(t\boldsymbol{v} + \boldsymbol{w}))\) where \(\boldsymbol{w}\) satisfies \((\mathbf{A} + 2\mathbf{I})\boldsymbol{w} = \boldsymbol{v}\).

This example illustrates how linear transformation theory guides the analysis of complex dynamical systems.

Key Idea

The Unity of Linear Transformation Theory:

• Abstract framework: Linear transformations provide a unified language for diverse mathematical phenomena

• Kernel and image: These fundamental subspaces encode essential information about transformation behavior

• Rank-Nullity Theorem: This fundamental constraint governs the relationship between domain, kernel, and image

• Isomorphisms: These perfect transformations reveal when different mathematical objects are "essentially the same"

• Composition: The natural way to combine transformations, corresponding to matrix multiplication

• Applications: From recurrence relations to differential equations, linear transformations provide both theoretical insight and computational power

The abstract theory pays dividends by revealing deep connections between seemingly disparate areas of mathematics and providing powerful tools for analysis and computation.

This completes our comprehensive study of linear transformations. We have developed the general theory, explored kernels and images, investigated isomorphisms and composition, and demonstrated applications to fundamental problems in mathematics. The framework of linear transformations serves as a bridge between abstract mathematical theory and concrete computational practice, providing both conceptual clarity and practical tools for solving real problems.

8 Orthogonality and Decomposition

Orthogonality is one of the most powerful and beautiful concepts in linear algebra, extending our geometric intuition to abstract vector spaces and providing the foundation for many of the most important decompositions and algorithms in modern mathematics. This chapter develops the theory of orthogonal complements, orthogonal diagonalization, and matrix decompositions that have revolutionized computational linear algebra and found applications across science, engineering, and data analysis.

8.1 Orthogonal Complements and Projections

We begin by establishing the fundamental relationship between subspaces and their orthogonal complements, extending our earlier work on projections to provide a complete theoretical framework.

Definition 8.1 (Orthogonal Complement). Let \(W\) be a subspace of \(\mathbb{R}^n\). The orthogonal complement of \(W\) is: \[W^{\perp} = \{\boldsymbol{x} \in \mathbb{R}^n : \boldsymbol{x} \cdot \boldsymbol{w} = 0 \text{ for all } \boldsymbol{w} \in W\}\]

Remark 411. The orthogonal complement captures all vectors that are perpendicular to every vector in the subspace \(W\). This geometric concept has profound algebraic implications and provides the foundation for understanding projections, least squares problems, and matrix decompositions.

Theorem 8.3 (Properties of Orthogonal Complements). Let \(W\) be a subspace of \(\mathbb{R}^n\). Then:

1. \(W^{\perp}\) is a subspace of \(\mathbb{R}^n\)

2. \(W \cap W^{\perp} = \{\boldsymbol{0}\}\)

3. \((W^{\perp})^{\perp} = W\)

4. \(\dim(W) + \dim(W^{\perp}) = n\)

5. \(\mathbb{R}^n = W \oplus W^{\perp}\) (orthogonal direct sum)

Proof. Property 1: We verify the subspace conditions for \(W^{\perp}\):

• Zero vector: \(\boldsymbol{0} \cdot \boldsymbol{w} = 0\) for all \(\boldsymbol{w} \in W\), so \(\boldsymbol{0} \in W^{\perp}\)

• Closure under addition: If \(\boldsymbol{x}, \boldsymbol{y} \in W^{\perp}\), then for any \(\boldsymbol{w} \in W\): \[(\boldsymbol{x} + \boldsymbol{y}) \cdot \boldsymbol{w} = \boldsymbol{x} \cdot \boldsymbol{w} + \boldsymbol{y} \cdot \boldsymbol{w} = 0 + 0 = 0\]

• Closure under scalar multiplication: If \(\boldsymbol{x} \in W^{\perp}\) and \(c \in \mathbb{R}\): \[(c\boldsymbol{x}) \cdot \boldsymbol{w} = c(\boldsymbol{x} \cdot \boldsymbol{w}) = c \cdot 0 = 0\]

Property 2: If \(\boldsymbol{v} \in W \cap W^{\perp}\), then \(\boldsymbol{v} \in W\) and \(\boldsymbol{v} \cdot \boldsymbol{w} = 0\) for all \(\boldsymbol{w} \in W\). In particular, \(\boldsymbol{v} \cdot \boldsymbol{v} = 0\), which implies \(\boldsymbol{v} = \boldsymbol{0}\).

Property 3: We show \((W^{\perp})^{\perp} = W\) by proving both inclusions.

• \(W \subseteq (W^{\perp})^{\perp}\): If \(\boldsymbol{w} \in W\) and \(\boldsymbol{x} \in W^{\perp}\), then \(\boldsymbol{w} \cdot \boldsymbol{x} = 0\), so \(\boldsymbol{w} \in (W^{\perp})^{\perp}\).

• \((W^{\perp})^{\perp} \subseteq W\): This follows from Property 5 below and dimensional considerations.

Property 4: This follows from the rank-nullity theorem applied to the orthogonal projection operator.

Property 5: From Properties 2 and 4, every vector can be uniquely decomposed as the sum of its projections onto \(W\) and \(W^{\perp}\). ◻

Example 8.4 (Computing Orthogonal Complements). 1. Line through the origin: Let \(W = \operatorname{span}\{(1, 2, 3)\}\). To find \(W^{\perp}\), we need all vectors \((x, y, z)\) such that: \[(x, y, z) \cdot (1, 2, 3) = x + 2y + 3z = 0\]

Therefore: \(W^{\perp} = \{(x, y, z) : x + 2y + 3z = 0\}\)

This is a plane through the origin with normal vector \((1, 2, 3)\). We can find a basis by setting two variables freely: - Setting \(y = 1, z = 0\): \(x = -2\), giving \((-2, 1, 0)\) - Setting \(y = 0, z = 1\): \(x = -3\), giving \((-3, 0, 1)\)

So \(W^{\perp} = \operatorname{span}\{(-2, 1, 0), (-3, 0, 1)\}\) and \(\dim(W^{\perp}) = 2\).

Verification: \(\dim(W) + \dim(W^{\perp}) = 1 + 2 = 3 = \dim(\mathbb{R}^3)\)

2. Matrix orthogonal complements: For an \(m \times n\) matrix \(\mathbf{A}\):

• \((\text{Col}(\mathbf{A}))^{\perp} = \text{Nul}(\mathbf{A}^T)\) (left null space)

• \((\text{Nul}(\mathbf{A}))^{\perp} = \text{Col}(\mathbf{A}^T)\) (row space)

This gives us the Fundamental Theorem of Linear Algebra: \[\mathbb{R}^m = \text{Col}(\mathbf{A}) \oplus \text{Nul}(\mathbf{A}^T)\] \[\mathbb{R}^n = \text{Col}(\mathbf{A}^T) \oplus \text{Nul}(\mathbf{A})\]

Definition 8.5 (Orthogonal Projection onto a Subspace). Let \(W\) be a subspace of \(\mathbb{R}^n\) with orthonormal basis \(\{\boldsymbol{u}_1, \boldsymbol{u}_2, \ldots, \boldsymbol{u}_k\}\). The orthogonal projection of \(\boldsymbol{y}\) onto \(W\) is: \[\text{proj}_W(\boldsymbol{y}) = (\boldsymbol{y} \cdot \boldsymbol{u}_1)\boldsymbol{u}_1 + (\boldsymbol{y} \cdot \boldsymbol{u}_2)\boldsymbol{u}_2 + \cdots + (\boldsymbol{y} \cdot \boldsymbol{u}_k)\boldsymbol{u}_k\]

If \(\mathbf{U} = [\boldsymbol{u}_1 \mid \boldsymbol{u}_2 \mid \cdots \mid \boldsymbol{u}_k]\), then: \[\text{proj}_W(\boldsymbol{y}) = \mathbf{U}\mathbf{U}^T\boldsymbol{y}\]

Theorem 8.6 (Properties of Orthogonal Projection). Let \(\mathbf{P} = \mathbf{U}\mathbf{U}^T\) be the orthogonal projection matrix onto subspace \(W\). Then:

1. \(\mathbf{P}^2 = \mathbf{P}\) (idempotent)

2. \(\mathbf{P}^T = \mathbf{P}\) (symmetric)

3. \(\text{rank}(\mathbf{P}) = \dim(W)\)

4. \(\text{Col}(\mathbf{P}) = W\) and \(\text{Nul}(\mathbf{P}) = W^{\perp}\)

5. For any \(\boldsymbol{y} \in \mathbb{R}^n\): \(\boldsymbol{y} = \text{proj}_W(\boldsymbol{y}) + \text{proj}_{W^{\perp}}(\boldsymbol{y})\)

Proof. Property 1: Since \(\mathbf{U}^T\mathbf{U} = \mathbf{I}_k\) (orthonormal columns): \[\mathbf{P}^2 = (\mathbf{U}\mathbf{U}^T)(\mathbf{U}\mathbf{U}^T) = \mathbf{U}(\mathbf{U}^T\mathbf{U})\mathbf{U}^T = \mathbf{U}\mathbf{I}_k\mathbf{U}^T = \mathbf{U}\mathbf{U}^T = \mathbf{P}\]

Property 2: \(\mathbf{P}^T = (\mathbf{U}\mathbf{U}^T)^T = (\mathbf{U}^T)^T\mathbf{U}^T = \mathbf{U}\mathbf{U}^T = \mathbf{P}\)

Property 3: Since \(\mathbf{U}\) has \(k\) orthonormal columns, \(\text{rank}(\mathbf{U}) = k\), and since \(\mathbf{P} = \mathbf{U}\mathbf{U}^T\), we have \(\text{rank}(\mathbf{P}) = k = \dim(W)\).

Property 4: \(\text{Col}(\mathbf{P}) = \text{Col}(\mathbf{U}) = W\) by construction. For the null space, if \(\mathbf{P}\boldsymbol{x} = \boldsymbol{0}\), then \(\mathbf{U}\mathbf{U}^T\boldsymbol{x} = \boldsymbol{0}\), which means \(\mathbf{U}^T\boldsymbol{x} = \boldsymbol{0}\), so \(\boldsymbol{x} \perp W\). ◻

Example 8.7 (Projection Matrix Construction). Find the projection matrix onto the plane \(W = \{(x, y, z) : x + y + z = 0\}\).

Step 1: Find an orthonormal basis for \(W\). We need vectors perpendicular to \((1, 1, 1)\). Start with two linearly independent vectors: \(\boldsymbol{v}_1 = (1, -1, 0)\) and \(\boldsymbol{v}_2 = (1, 0, -1)\)

Step 2: Apply Gram-Schmidt: \[\boldsymbol{u}_1 = \frac{\boldsymbol{v}_1}{\|\boldsymbol{v}_1\|} = \frac{(1, -1, 0)}{\sqrt{2}} = \left(\frac{1}{\sqrt{2}}, -\frac{1}{\sqrt{2}}, 0\right)\]

\[\boldsymbol{u}_2' = \boldsymbol{v}_2 - (\boldsymbol{v}_2 \cdot \boldsymbol{u}_1)\boldsymbol{u}_1\]

\[\boldsymbol{v}_2 \cdot \boldsymbol{u}_1 = (1, 0, -1) \cdot \left(\frac{1}{\sqrt{2}}, -\frac{1}{\sqrt{2}}, 0\right) = \frac{1}{\sqrt{2}}\]

\[\boldsymbol{u}_2' = (1, 0, -1) - \frac{1}{\sqrt{2}}\left(\frac{1}{\sqrt{2}}, -\frac{1}{\sqrt{2}}, 0\right) = (1, 0, -1) - \left(\frac{1}{2}, -\frac{1}{2}, 0\right) = \left(\frac{1}{2}, \frac{1}{2}, -1\right)\]

\[\boldsymbol{u}_2 = \frac{\boldsymbol{u}_2'}{\|\boldsymbol{u}_2'\|} = \frac{(1/2, 1/2, -1)}{\sqrt{1/4 + 1/4 + 1}} = \frac{(1/2, 1/2, -1)}{\sqrt{3/2}} = \left(\frac{1}{\sqrt{6}}, \frac{1}{\sqrt{6}}, -\frac{2}{\sqrt{6}}\right)\]

Step 3: Form the projection matrix: \[\mathbf{U} = \begin{pmatrix} \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{6}} \\ -\frac{1}{\sqrt{2}} & \frac{1}{\sqrt{6}} \\ 0 & -\frac{2}{\sqrt{6}} \end{pmatrix}\]

\[\mathbf{P} = \mathbf{U}\mathbf{U}^T = \begin{pmatrix} \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{6}} \\ -\frac{1}{\sqrt{2}} & \frac{1}{\sqrt{6}} \\ 0 & -\frac{2}{\sqrt{6}} \end{pmatrix}\begin{pmatrix} \frac{1}{\sqrt{2}} & -\frac{1}{\sqrt{2}} & 0 \\ \frac{1}{\sqrt{6}} & \frac{1}{\sqrt{6}} & -\frac{2}{\sqrt{6}} \end{pmatrix}\]

\[= \begin{pmatrix} \frac{2}{3} & -\frac{1}{3} & -\frac{1}{3} \\ -\frac{1}{3} & \frac{2}{3} & -\frac{1}{3} \\ -\frac{1}{3} & -\frac{1}{3} & \frac{2}{3} \end{pmatrix}\]

Alternative method: Since \(W^{\perp} = \operatorname{span}\{(1, 1, 1)\}\), we can use: \[\mathbf{P} = \mathbf{I} - \text{proj}_{W^{\perp}} = \mathbf{I} - \frac{\boldsymbol{n}\boldsymbol{n}^T}{\boldsymbol{n}^T\boldsymbol{n}}\]

where \(\boldsymbol{n} = (1, 1, 1)\): \[\mathbf{P} = \mathbf{I} - \frac{1}{3}\begin{pmatrix} 1 \\ 1 \\ 1 \end{pmatrix}\begin{pmatrix} 1 & 1 & 1 \end{pmatrix} = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{pmatrix} - \frac{1}{3}\begin{pmatrix} 1 & 1 & 1 \\ 1 & 1 & 1 \\ 1 & 1 & 1 \end{pmatrix}\]

This gives the same result.

Theorem 8.8 (Best Approximation Theorem). Let \(W\) be a subspace of \(\mathbb{R}^n\) and \(\boldsymbol{y} \in \mathbb{R}^n\). Then \(\text{proj}_W(\boldsymbol{y})\) is the unique vector in \(W\) that minimizes \(\|\boldsymbol{y} - \boldsymbol{w}\|\) for \(\boldsymbol{w} \in W\).

Proof. Let \(\hat{\boldsymbol{y}} = \text{proj}_W(\boldsymbol{y})\) and \(\boldsymbol{w} \in W\). We can write: \[\boldsymbol{y} - \boldsymbol{w} = (\boldsymbol{y} - \hat{\boldsymbol{y}}) + (\hat{\boldsymbol{y}} - \boldsymbol{w})\]

Since \(\boldsymbol{y} - \hat{\boldsymbol{y}} \in W^{\perp}\) and \(\hat{\boldsymbol{y}} - \boldsymbol{w} \in W\), these vectors are orthogonal. By the Pythagorean theorem: \[\|\boldsymbol{y} - \boldsymbol{w}\|^2 = \|\boldsymbol{y} - \hat{\boldsymbol{y}}\|^2 + \|\hat{\boldsymbol{y}} - \boldsymbol{w}\|^2\]

This is minimized when \(\|\hat{\boldsymbol{y}} - \boldsymbol{w}\|^2 = 0\), i.e., when \(\boldsymbol{w} = \hat{\boldsymbol{y}}\). ◻

8.2 Orthogonal Diagonalization

Orthogonal diagonalization provides the optimal way to diagonalize symmetric matrices, using orthogonal matrices that preserve lengths and angles.

Definition 8.9 (Orthogonal Matrix). A square matrix \(\mathbf{Q}\) is orthogonal if its columns form an orthonormal set. Equivalently: \[\mathbf{Q}^T\mathbf{Q} = \mathbf{I} \quad \text{or} \quad \mathbf{Q}^T = \mathbf{Q}^{-1}\]

Theorem 8.10 (Properties of Orthogonal Matrices). If \(\mathbf{Q}\) is an orthogonal matrix, then:

1. \(\det(\mathbf{Q}) = \pm 1\)

2. \(\|\mathbf{Q}\boldsymbol{x}\| = \|\boldsymbol{x}\|\) for all \(\boldsymbol{x}\) (length preserving)

3. \((\mathbf{Q}\boldsymbol{x}) \cdot (\mathbf{Q}\boldsymbol{y}) = \boldsymbol{x} \cdot \boldsymbol{y}\) for all \(\boldsymbol{x}, \boldsymbol{y}\) (inner product preserving)

4. The eigenvalues of \(\mathbf{Q}\) have absolute value 1

5. If \(\det(\mathbf{Q}) = 1\), then \(\mathbf{Q}\) represents a rotation; if \(\det(\mathbf{Q}) = -1\), then \(\mathbf{Q}\) represents a rotation combined with a reflection

Proof. Property 1: From \(\mathbf{Q}^T\mathbf{Q} = \mathbf{I}\), taking determinants: \[\det(\mathbf{Q}^T)\det(\mathbf{Q}) = \det(\mathbf{I}) = 1\] Since \(\det(\mathbf{Q}^T) = \det(\mathbf{Q})\), we have \((\det(\mathbf{Q}))^2 = 1\), so \(\det(\mathbf{Q}) = \pm 1\).

Property 2: \[\|\mathbf{Q}\boldsymbol{x}\|^2 = (\mathbf{Q}\boldsymbol{x}) \cdot (\mathbf{Q}\boldsymbol{x}) = (\mathbf{Q}\boldsymbol{x})^T(\mathbf{Q}\boldsymbol{x}) = \boldsymbol{x}^T\mathbf{Q}^T\mathbf{Q}\boldsymbol{x} = \boldsymbol{x}^T\boldsymbol{x} = \|\boldsymbol{x}\|^2\]

Property 3: Similar to Property 2. ◻

Definition 8.11 (Orthogonally Diagonalizable). A matrix \(\mathbf{A}\) is orthogonally diagonalizable if there exists an orthogonal matrix \(\mathbf{Q}\) and a diagonal matrix \(\mathbf{D}\) such that: \[\mathbf{A} = \mathbf{Q}\mathbf{D}\mathbf{Q}^T\]

Theorem 8.12 (Spectral Theorem for Symmetric Matrices). A matrix \(\mathbf{A}\) is orthogonally diagonalizable if and only if \(\mathbf{A}\) is symmetric. Moreover, if \(\mathbf{A}\) is symmetric, then:

1. All eigenvalues of \(\mathbf{A}\) are real

2. Eigenvectors corresponding to different eigenvalues are orthogonal

3. The geometric multiplicity equals the algebraic multiplicity for each eigenvalue

4. There exists an orthonormal basis of eigenvectors for \(\mathbb{R}^n\)

Proof. (\(\Rightarrow\)) If \(\mathbf{A} = \mathbf{Q}\mathbf{D}\mathbf{Q}^T\), then: \[\mathbf{A}^T = (\mathbf{Q}\mathbf{D}\mathbf{Q}^T)^T = (\mathbf{Q}^T)^T\mathbf{D}^T\mathbf{Q}^T = \mathbf{Q}\mathbf{D}\mathbf{Q}^T = \mathbf{A}\]

(\(\Leftarrow\)) This requires a more advanced proof using the fact that symmetric matrices always have real eigenvalues and orthogonal eigenvectors.

Real eigenvalues: If \(\mathbf{A}\boldsymbol{v} = \lambda\boldsymbol{v}\) for complex \(\lambda\) and \(\boldsymbol{v}\), then: \[\overline{\lambda}\overline{\boldsymbol{v}}^T\boldsymbol{v} = \overline{\boldsymbol{v}}^T\overline{\mathbf{A}}\boldsymbol{v} = \overline{\boldsymbol{v}}^T\mathbf{A}^T\boldsymbol{v} = \overline{\boldsymbol{v}}^T\mathbf{A}\boldsymbol{v} = \lambda\overline{\boldsymbol{v}}^T\boldsymbol{v}\]

Since \(\overline{\boldsymbol{v}}^T\boldsymbol{v} = \|\boldsymbol{v}\|^2 > 0\), we have \(\overline{\lambda} = \lambda\), so \(\lambda\) is real.

Orthogonal eigenvectors: If \(\mathbf{A}\boldsymbol{u} = \lambda_1\boldsymbol{u}\) and \(\mathbf{A}\boldsymbol{v} = \lambda_2\boldsymbol{v}\) with \(\lambda_1 \neq \lambda_2\): \[\lambda_1(\boldsymbol{u} \cdot \boldsymbol{v}) = (\mathbf{A}\boldsymbol{u}) \cdot \boldsymbol{v} = \boldsymbol{u} \cdot (\mathbf{A}\boldsymbol{v}) = \lambda_2(\boldsymbol{u} \cdot \boldsymbol{v})\]

Since \(\lambda_1 \neq \lambda_2\), we must have \(\boldsymbol{u} \cdot \boldsymbol{v} = 0\). ◻

Example 8.13 (Orthogonal Diagonalization Process). Orthogonally diagonalize \(\mathbf{A} = \begin{pmatrix} 3 & 1 \\ 1 & 3 \end{pmatrix}\).

Step 1: Find eigenvalues \[\det(\mathbf{A} - \lambda\mathbf{I}) = \det\begin{pmatrix} 3-\lambda & 1 \\ 1 & 3-\lambda \end{pmatrix} = (3-\lambda)^2 - 1 = \lambda^2 - 6\lambda + 8 = (\lambda-2)(\lambda-4)\]

Eigenvalues: \(\lambda_1 = 2, \lambda_2 = 4\).

Step 2: Find eigenvectors

For \(\lambda_1 = 2\): \[(\mathbf{A} - 2\mathbf{I})\boldsymbol{v} = \begin{pmatrix} 1 & 1 \\ 1 & 1 \end{pmatrix}\boldsymbol{v} = \boldsymbol{0}\]

This gives \(v_1 + v_2 = 0\), so \(v_2 = -v_1\). Eigenvector: \(\boldsymbol{v}_1 = \begin{pmatrix} 1 \\ -1 \end{pmatrix}\).

For \(\lambda_2 = 4\): \[(\mathbf{A} - 4\mathbf{I})\boldsymbol{v} = \begin{pmatrix} -1 & 1 \\ 1 & -1 \end{pmatrix}\boldsymbol{v} = \boldsymbol{0}\]

This gives \(-v_1 + v_2 = 0\), so \(v_2 = v_1\). Eigenvector: \(\boldsymbol{v}_2 = \begin{pmatrix} 1 \\ 1 \end{pmatrix}\).

Step 3: Normalize eigenvectors \[\boldsymbol{u}_1 = \frac{\boldsymbol{v}_1}{\|\boldsymbol{v}_1\|} = \frac{1}{\sqrt{2}}\begin{pmatrix} 1 \\ -1 \end{pmatrix}, \quad \boldsymbol{u}_2 = \frac{\boldsymbol{v}_2}{\|\boldsymbol{v}_2\|} = \frac{1}{\sqrt{2}}\begin{pmatrix} 1 \\ 1 \end{pmatrix}\]

Step 4: Form orthogonal matrix \[\mathbf{Q} = \frac{1}{\sqrt{2}}\begin{pmatrix} 1 & 1 \\ -1 & 1 \end{pmatrix}, \quad \mathbf{D} = \begin{pmatrix} 2 & 0 \\ 0 & 4 \end{pmatrix}\]

Verification: \[\mathbf{Q}\mathbf{D}\mathbf{Q}^T = \frac{1}{\sqrt{2}}\begin{pmatrix} 1 & 1 \\ -1 & 1 \end{pmatrix}\begin{pmatrix} 2 & 0 \\ 0 & 4 \end{pmatrix}\frac{1}{\sqrt{2}}\begin{pmatrix} 1 & -1 \\ 1 & 1 \end{pmatrix}\]

\[= \frac{1}{2}\begin{pmatrix} 2 & 4 \\ -2 & 4 \end{pmatrix}\begin{pmatrix} 1 & -1 \\ 1 & 1 \end{pmatrix} = \frac{1}{2}\begin{pmatrix} 6 & 2 \\ 2 & 6 \end{pmatrix} = \begin{pmatrix} 3 & 1 \\ 1 & 3 \end{pmatrix} = \mathbf{A}\]

Verification of orthogonality: \[\boldsymbol{u}_1 \cdot \boldsymbol{u}_2 = \frac{1}{\sqrt{2}}\begin{pmatrix} 1 \\ -1 \end{pmatrix} \cdot \frac{1}{\sqrt{2}}\begin{pmatrix} 1 \\ 1 \end{pmatrix} = \frac{1}{2}(1 - 1) = 0\]

Theorem 8.14 (Principal Axis Theorem). For a symmetric matrix \(\mathbf{A}\), the orthogonal diagonalization \(\mathbf{A} = \mathbf{Q}\mathbf{D}\mathbf{Q}^T\) provides:

1. The principal axes of the quadratic form \(\boldsymbol{x}^T\mathbf{A}\boldsymbol{x}\) (columns of \(\mathbf{Q}\))

2. The principal values (eigenvalues on diagonal of \(\mathbf{D}\))

3. A coordinate transformation that eliminates cross terms in the quadratic form

Example 8.15 (Quadratic Form Diagonalization). Consider the quadratic form \(Q(x, y) = 3x^2 + 2xy + 3y^2\).

This corresponds to the matrix \(\mathbf{A} = \begin{pmatrix} 3 & 1 \\ 1 & 3 \end{pmatrix}\) (from the previous example).

After diagonalization with \(\mathbf{Q} = \frac{1}{\sqrt{2}}\begin{pmatrix} 1 & 1 \\ -1 & 1 \end{pmatrix}\):

The change of variables \(\begin{pmatrix} x \\ y \end{pmatrix} = \mathbf{Q}\begin{pmatrix} u \\ v \end{pmatrix}\) gives: \[x = \frac{u + v}{\sqrt{2}}, \quad y = \frac{-u + v}{\sqrt{2}}\]

Substituting into the quadratic form: \[Q(u, v) = 2u^2 + 4v^2\]

The cross term \(xy\) has been eliminated, and the quadratic form is now in standard form along the principal axes.

8.3 Positive Definite Matrices

Positive definite matrices are the symmetric matrices that behave most like positive real numbers. They arise naturally in optimization, statistics, and many physical applications.

Definition 8.16 (Positive Definite Matrix). A symmetric matrix \(\mathbf{A}\) is:

• Positive definite if \(\boldsymbol{x}^T\mathbf{A}\boldsymbol{x} > 0\) for all \(\boldsymbol{x} \neq \boldsymbol{0}\)

• Positive semidefinite if \(\boldsymbol{x}^T\mathbf{A}\boldsymbol{x} \geq 0\) for all \(\boldsymbol{x}\)

• Negative definite if \(\boldsymbol{x}^T\mathbf{A}\boldsymbol{x} < 0\) for all \(\boldsymbol{x} \neq \boldsymbol{0}\)

• Indefinite if \(\boldsymbol{x}^T\mathbf{A}\boldsymbol{x}\) takes both positive and negative values

Theorem 8.17 (Characterizations of Positive Definiteness). For a symmetric matrix \(\mathbf{A}\), the following are equivalent:

1. \(\mathbf{A}\) is positive definite

2. All eigenvalues of \(\mathbf{A}\) are positive

3. All principal minors of \(\mathbf{A}\) are positive

4. There exists an invertible matrix \(\mathbf{B}\) such that \(\mathbf{A} = \mathbf{B}^T\mathbf{B}\)

5. The Cholesky decomposition \(\mathbf{A} = \mathbf{L}\mathbf{L}^T\) exists (where \(\mathbf{L}\) is lower triangular with positive diagonal entries)

Proof. (1) \(\Leftrightarrow\) (2): From the spectral theorem, \(\mathbf{A} = \mathbf{Q}\mathbf{D}\mathbf{Q}^T\) where \(\mathbf{D} = \text{diag}(\lambda_1, \ldots, \lambda_n)\). For any \(\boldsymbol{x} \neq \boldsymbol{0}\): \[\boldsymbol{x}^T\mathbf{A}\boldsymbol{x} = \boldsymbol{x}^T\mathbf{Q}\mathbf{D}\mathbf{Q}^T\boldsymbol{x} = \boldsymbol{y}^T\mathbf{D}\boldsymbol{y} = \sum_{i=1}^n \lambda_i y_i^2\] where \(\boldsymbol{y} = \mathbf{Q}^T\boldsymbol{x} \neq \boldsymbol{0}\) (since \(\mathbf{Q}\) is invertible). This is positive for all \(\boldsymbol{x} \neq \boldsymbol{0}\) if and only if all \(\lambda_i > 0\).

(2) \(\Rightarrow\) (4): If all eigenvalues are positive, set \(\mathbf{B} = \mathbf{Q}\mathbf{D}^{1/2}\) where \(\mathbf{D}^{1/2} = \text{diag}(\sqrt{\lambda_1}, \ldots, \sqrt{\lambda_n})\). Then: \[\mathbf{B}^T\mathbf{B} = (\mathbf{D}^{1/2})^T\mathbf{Q}^T\mathbf{Q}\mathbf{D}^{1/2} = \mathbf{D}^{1/2}\mathbf{D}^{1/2} = \mathbf{D} = \mathbf{A}\]

(4) \(\Rightarrow\) (1): If \(\mathbf{A} = \mathbf{B}^T\mathbf{B}\) and \(\boldsymbol{x} \neq \boldsymbol{0}\): \[\boldsymbol{x}^T\mathbf{A}\boldsymbol{x} = \boldsymbol{x}^T\mathbf{B}^T\mathbf{B}\boldsymbol{x} = (\mathbf{B}\boldsymbol{x})^T(\mathbf{B}\boldsymbol{x}) = \|\mathbf{B}\boldsymbol{x}\|^2\]

Since \(\mathbf{B}\) is invertible, \(\mathbf{B}\boldsymbol{x} \neq \boldsymbol{0}\) when \(\boldsymbol{x} \neq \boldsymbol{0}\), so \(\|\mathbf{B}\boldsymbol{x}\|^2 > 0\).

The equivalence with (3) and (5) requires more detailed analysis of principal minors and the Cholesky algorithm. ◻

Example 8.18 (Testing for Positive Definiteness). 1. Using eigenvalues: Test whether \(\mathbf{A} = \begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix}\) is positive definite.

Eigenvalues: \(\det(\mathbf{A} - \lambda\mathbf{I}) = (2-\lambda)^2 - 1 = \lambda^2 - 4\lambda + 3 = (\lambda-1)(\lambda-3)\)

So \(\lambda_1 = 1 > 0\) and \(\lambda_2 = 3 > 0\). Therefore \(\mathbf{A}\) is positive definite.

2. Using principal minors: The principal minors are: - \(M_1 = 2 > 0\) - \(M_2 = \det(\mathbf{A}) = 4 - 1 = 3 > 0\)

Since all principal minors are positive, \(\mathbf{A}\) is positive definite.

3. Direct verification: For \(\boldsymbol{x} = (x, y)^T\): \[\boldsymbol{x}^T\mathbf{A}\boldsymbol{x} = 2x^2 + 2xy + 2y^2 = 2(x^2 + xy + y^2) = 2\left(\left(x + \frac{y}{2}\right)^2 + \frac{3y^2}{4}\right)\]

This is positive for all \((x, y) \neq (0, 0)\), confirming positive definiteness.

Definition 8.19 (Cholesky Decomposition). If \(\mathbf{A}\) is positive definite, it has a unique Cholesky decomposition: \[\mathbf{A} = \mathbf{L}\mathbf{L}^T\] where \(\mathbf{L}\) is lower triangular with positive diagonal entries.

Example 8.20 (Cholesky Decomposition Computation). Find the Cholesky decomposition of \(\mathbf{A} = \begin{pmatrix} 4 & 2 & 1 \\ 2 & 3 & 0.5 \\ 1 & 0.5 & 1 \end{pmatrix}\).

Step 1: \(i = 1\) - \(\ell_{11} = \sqrt{a_{11}} = \sqrt{4} = 2\) - \(\ell_{21} = \frac{a_{21}}{\ell_{11}} = \frac{2}{2} = 1\) - \(\ell_{31} = \frac{a_{31}}{\ell_{11}} = \frac{1}{2} = 0.5\)

Step 2: \(i = 2\) - \(\ell_{22} = \sqrt{a_{22} - \ell_{21}^2} = \sqrt{3 - 1} = \sqrt{2}\) - \(\ell_{32} = \frac{a_{32} - \ell_{31}\ell_{21}}{\ell_{22}} = \frac{0.5 - 0.5 \cdot 1}{\sqrt{2}} = 0\)

Step 3: \(i = 3\) - \(\ell_{33} = \sqrt{a_{33} - \ell_{31}^2 - \ell_{32}^2} = \sqrt{1 - 0.25 - 0} = \sqrt{0.75} = \frac{\sqrt{3}}{2}\)

Result: \[\mathbf{L} = \begin{pmatrix} 2 & 0 & 0 \\ 1 & \sqrt{2} & 0 \\ 0.5 & 0 & \frac{\sqrt{3}}{2} \end{pmatrix}\]

Verification: \[\mathbf{L}\mathbf{L}^T = \begin{pmatrix} 2 & 0 & 0 \\ 1 & \sqrt{2} & 0 \\ 0.5 & 0 & \frac{\sqrt{3}}{2} \end{pmatrix}\begin{pmatrix} 2 & 1 & 0.5 \\ 0 & \sqrt{2} & 0 \\ 0 & 0 & \frac{\sqrt{3}}{2} \end{pmatrix} = \begin{pmatrix} 4 & 2 & 1 \\ 2 & 3 & 0.5 \\ 1 & 0.5 & 1 \end{pmatrix} = \mathbf{A}\]

Theorem 8.21 (Applications of Positive Definite Matrices). Positive definite matrices arise in:

1. Optimization: The Hessian matrix of a convex function is positive semidefinite

2. Statistics: Covariance matrices are positive semidefinite

3. Least squares: The normal equation matrix \(\mathbf{A}^T\mathbf{A}\) is positive semidefinite (positive definite if \(\mathbf{A}\) has full column rank)

4. Numerical analysis: Systems with positive definite matrices are always uniquely solvable and numerically stable

8.4 QR-Factorization

The QR factorization decomposes any matrix into the product of an orthogonal matrix and an upper triangular matrix. This decomposition is fundamental for solving least squares problems and computing eigenvalues.

Definition 8.22 (QR Factorization). An \(m \times n\) matrix \(\mathbf{A}\) with linearly independent columns has a QR factorization: \[\mathbf{A} = \mathbf{Q}\mathbf{R}\] where \(\mathbf{Q}\) is \(m \times n\) with orthonormal columns and \(\mathbf{R}\) is \(n \times n\) upper triangular with positive diagonal entries.

Theorem 8.23 (Existence and Uniqueness of QR Factorization). If \(\mathbf{A}\) is an \(m \times n\) matrix with linearly independent columns, then:

1. The QR factorization exists

2. If we require \(\mathbf{R}\) to have positive diagonal entries, the factorization is unique

3. \(\mathbf{Q}\) has orthonormal columns: \(\mathbf{Q}^T\mathbf{Q} = \mathbf{I}_n\)

4. \(\text{Col}(\mathbf{Q}) = \text{Col}(\mathbf{A})\)

Example 8.24 (QR Factorization Computation). Find the QR factorization of \(\mathbf{A} = \begin{pmatrix} 1 & 1 & 0 \\ 1 & 0 & 1 \\ 0 & 1 & 1 \end{pmatrix}\).

Step 1: Apply Gram-Schmidt

Column 1: \(\boldsymbol{a}_1 = \begin{pmatrix} 1 \\ 1 \\ 0 \end{pmatrix}\) \[\boldsymbol{v}_1 = \boldsymbol{a}_1 = \begin{pmatrix} 1 \\ 1 \\ 0 \end{pmatrix}, \quad \boldsymbol{q}_1 = \frac{\boldsymbol{v}_1}{\|\boldsymbol{v}_1\|} = \frac{1}{\sqrt{2}}\begin{pmatrix} 1 \\ 1 \\ 0 \end{pmatrix}\]

Column 2: \(\boldsymbol{a}_2 = \begin{pmatrix} 1 \\ 0 \\ 1 \end{pmatrix}\) \[\boldsymbol{v}_2 = \boldsymbol{a}_2 - (\boldsymbol{a}_2 \cdot \boldsymbol{q}_1)\boldsymbol{q}_1\]

\[\boldsymbol{a}_2 \cdot \boldsymbol{q}_1 = \begin{pmatrix} 1 \\ 0 \\ 1 \end{pmatrix} \cdot \frac{1}{\sqrt{2}}\begin{pmatrix} 1 \\ 1 \\ 0 \end{pmatrix} = \frac{1}{\sqrt{2}}\]

\[\boldsymbol{v}_2 = \begin{pmatrix} 1 \\ 0 \\ 1 \end{pmatrix} - \frac{1}{\sqrt{2}} \cdot \frac{1}{\sqrt{2}}\begin{pmatrix} 1 \\ 1 \\ 0 \end{pmatrix} = \begin{pmatrix} 1 \\ 0 \\ 1 \end{pmatrix} - \frac{1}{2}\begin{pmatrix} 1 \\ 1 \\ 0 \end{pmatrix} = \begin{pmatrix} 1/2 \\ -1/2 \\ 1 \end{pmatrix}\]

\[\boldsymbol{q}_2 = \frac{\boldsymbol{v}_2}{\|\boldsymbol{v}_2\|} = \frac{\begin{pmatrix} 1/2 \\ -1/2 \\ 1 \end{pmatrix}}{\sqrt{1/4 + 1/4 + 1}} = \frac{\begin{pmatrix} 1/2 \\ -1/2 \\ 1 \end{pmatrix}}{\sqrt{3/2}} = \frac{1}{\sqrt{6}}\begin{pmatrix} 1 \\ -1 \\ 2 \end{pmatrix}\]

Column 3: \(\boldsymbol{a}_3 = \begin{pmatrix} 0 \\ 1 \\ 1 \end{pmatrix}\) \[\boldsymbol{v}_3 = \boldsymbol{a}_3 - (\boldsymbol{a}_3 \cdot \boldsymbol{q}_1)\boldsymbol{q}_1 - (\boldsymbol{a}_3 \cdot \boldsymbol{q}_2)\boldsymbol{q}_2\]

\[\boldsymbol{a}_3 \cdot \boldsymbol{q}_1 = \begin{pmatrix} 0 \\ 1 \\ 1 \end{pmatrix} \cdot \frac{1}{\sqrt{2}}\begin{pmatrix} 1 \\ 1 \\ 0 \end{pmatrix} = \frac{1}{\sqrt{2}}\]

\[\boldsymbol{a}_3 \cdot \boldsymbol{q}_2 = \begin{pmatrix} 0 \\ 1 \\ 1 \end{pmatrix} \cdot \frac{1}{\sqrt{6}}\begin{pmatrix} 1 \\ -1 \\ 2 \end{pmatrix} = \frac{1}{\sqrt{6}}(0 - 1 + 2) = \frac{1}{\sqrt{6}}\]

\[\boldsymbol{v}_3 = \begin{pmatrix} 0 \\ 1 \\ 1 \end{pmatrix} - \frac{1}{\sqrt{2}} \cdot \frac{1}{\sqrt{2}}\begin{pmatrix} 1 \\ 1 \\ 0 \end{pmatrix} - \frac{1}{\sqrt{6}} \cdot \frac{1}{\sqrt{6}}\begin{pmatrix} 1 \\ -1 \\ 2 \end{pmatrix}\]

\[= \begin{pmatrix} 0 \\ 1 \\ 1 \end{pmatrix} - \frac{1}{2}\begin{pmatrix} 1 \\ 1 \\ 0 \end{pmatrix} - \frac{1}{6}\begin{pmatrix} 1 \\ -1 \\ 2 \end{pmatrix} = \begin{pmatrix} 0 - 1/2 - 1/6 \\ 1 - 1/2 + 1/6 \\ 1 - 0 - 1/3 \end{pmatrix} = \begin{pmatrix} -2/3 \\ 2/3 \\ 2/3 \end{pmatrix}\]

\[\boldsymbol{q}_3 = \frac{\boldsymbol{v}_3}{\|\boldsymbol{v}_3\|} = \frac{\begin{pmatrix} -2/3 \\ 2/3 \\ 2/3 \end{pmatrix}}{\sqrt{4/9 + 4/9 + 4/9}} = \frac{\begin{pmatrix} -2/3 \\ 2/3 \\ 2/3 \end{pmatrix}}{2\sqrt{3}/3} = \frac{1}{\sqrt{3}}\begin{pmatrix} -1 \\ 1 \\ 1 \end{pmatrix}\]

Step 2: Construct Q and R \[\mathbf{Q} = \begin{pmatrix} \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{6}} & -\frac{1}{\sqrt{3}} \\ \frac{1}{\sqrt{2}} & -\frac{1}{\sqrt{6}} & \frac{1}{\sqrt{3}} \\ 0 & \frac{2}{\sqrt{6}} & \frac{1}{\sqrt{3}} \end{pmatrix}\]

\[r_{11} = \boldsymbol{q}_1 \cdot \boldsymbol{a}_1 = \frac{1}{\sqrt{2}} \cdot 2 = \sqrt{2}\] \[r_{12} = \boldsymbol{q}_1 \cdot \boldsymbol{a}_2 = \frac{1}{\sqrt{2}}\] \[r_{13} = \boldsymbol{q}_1 \cdot \boldsymbol{a}_3 = \frac{1}{\sqrt{2}}\] \[r_{22} = \boldsymbol{q}_2 \cdot \boldsymbol{a}_2 = \frac{1}{\sqrt{6}} \cdot 3 = \frac{\sqrt{6}}{2}\] \[r_{23} = \boldsymbol{q}_2 \cdot \boldsymbol{a}_3 = \frac{1}{\sqrt{6}}\] \[r_{33} = \boldsymbol{q}_3 \cdot \boldsymbol{a}_3 = \frac{2}{\sqrt{3}}\]

\[\mathbf{R} = \begin{pmatrix} \sqrt{2} & \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}} \\ 0 & \frac{\sqrt{6}}{2} & \frac{1}{\sqrt{6}} \\ 0 & 0 & \frac{2}{\sqrt{3}} \end{pmatrix}\]

Verification: \(\mathbf{Q}\mathbf{R} = \mathbf{A}\) (computation omitted for brevity).

Theorem 8.25 (Applications of QR Factorization). The QR factorization provides efficient algorithms for:

1. Least squares problems: If \(\mathbf{A} = \mathbf{Q}\mathbf{R}\), then \(\hat{\boldsymbol{x}} = \mathbf{R}^{-1}\mathbf{Q}^T\boldsymbol{b}\)

2. Computing orthonormal bases: The columns of \(\mathbf{Q}\) form an orthonormal basis for \(\text{Col}(\mathbf{A})\)

3. Matrix inversion: If \(\mathbf{A}\) is square and invertible, then \(\mathbf{A}^{-1} = \mathbf{R}^{-1}\mathbf{Q}^T\)

4. Eigenvalue computation: The QR algorithm iteratively applies QR factorization to compute eigenvalues

Example 8.26 (Solving Least Squares with QR). Solve the least squares problem \(\min_{\boldsymbol{x}} \|\mathbf{A}\boldsymbol{x} - \boldsymbol{b}\|^2\) where: \[\mathbf{A} = \begin{pmatrix} 1 & 1 \\ 1 & 0 \\ 0 & 1 \end{pmatrix}, \quad \boldsymbol{b} = \begin{pmatrix} 3 \\ 1 \\ 1 \end{pmatrix}\]

Step 1: QR factorization of A \[\boldsymbol{q}_1 = \frac{1}{\sqrt{2}}\begin{pmatrix} 1 \\ 1 \\ 0 \end{pmatrix}, \quad \boldsymbol{q}_2 = \frac{1}{\sqrt{6}}\begin{pmatrix} 1 \\ -1 \\ 2 \end{pmatrix}\]

\[\mathbf{Q} = \begin{pmatrix} \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{6}} \\ \frac{1}{\sqrt{2}} & -\frac{1}{\sqrt{6}} \\ 0 & \frac{2}{\sqrt{6}} \end{pmatrix}, \quad \mathbf{R} = \begin{pmatrix} \sqrt{2} & \frac{1}{\sqrt{2}} \\ 0 & \frac{\sqrt{6}}{2} \end{pmatrix}\]

Step 2: Solve using QR \[\hat{\boldsymbol{x}} = \mathbf{R}^{-1}\mathbf{Q}^T\boldsymbol{b}\]

\[\mathbf{Q}^T\boldsymbol{b} = \begin{pmatrix} \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}} & 0 \\ \frac{1}{\sqrt{6}} & -\frac{1}{\sqrt{6}} & \frac{2}{\sqrt{6}} \end{pmatrix}\begin{pmatrix} 3 \\ 1 \\ 1 \end{pmatrix} = \begin{pmatrix} \frac{4}{\sqrt{2}} \\ \frac{4}{\sqrt{6}} \end{pmatrix}\]

Solving \(\mathbf{R}\hat{\boldsymbol{x}} = \mathbf{Q}^T\boldsymbol{b}\): \[\begin{pmatrix} \sqrt{2} & \frac{1}{\sqrt{2}} \\ 0 & \frac{\sqrt{6}}{2} \end{pmatrix}\begin{pmatrix} x_1 \\ x_2 \end{pmatrix} = \begin{pmatrix} \frac{4}{\sqrt{2}} \\ \frac{4}{\sqrt{6}} \end{pmatrix}\]

From the second equation: \(x_2 = \frac{4/\sqrt{6}}{\sqrt{6}/2} = \frac{8}{6} = \frac{4}{3}\)

From the first equation: \(x_1 = \frac{4/\sqrt{2} - (1/\sqrt{2})(4/3)}{\sqrt{2}} = \frac{4 - 4/3}{2} = \frac{8/3}{2} = \frac{4}{3}\)

Therefore: \(\hat{\boldsymbol{x}} = \begin{pmatrix} 4/3 \\ 4/3 \end{pmatrix}\)

8.5 Computing Eigenvalues

Computing eigenvalues of large matrices is one of the most important problems in numerical linear algebra. We explore the fundamental algorithms that make eigenvalue computation practical.

Theorem 8.27 (Convergence of Power Method). If \(\mathbf{A}\) has eigenvalues \(|\lambda_1| > |\lambda_2| \geq |\lambda_3| \geq \cdots \geq |\lambda_n|\) and the initial vector \(\boldsymbol{x}_0\) has a non-zero component in the direction of the dominant eigenvector, then the power method converges to \(\lambda_1\) and the corresponding eigenvector with convergence rate \(|\lambda_2/\lambda_1|\).

Example 8.28 (Power Method Application). Apply the power method to \(\mathbf{A} = \begin{pmatrix} 3 & 1 \\ 1 & 3 \end{pmatrix}\) with \(\boldsymbol{x}_0 = \begin{pmatrix} 1 \\ 1 \end{pmatrix}\).

Iteration 1: \[\boldsymbol{y}_1 = \mathbf{A}\boldsymbol{x}_0 = \begin{pmatrix} 3 & 1 \\ 1 & 3 \end{pmatrix}\begin{pmatrix} 1 \\ 1 \end{pmatrix} = \begin{pmatrix} 4 \\ 4 \end{pmatrix}\] \[\mu_1 = 4, \quad \boldsymbol{x}_1 = \begin{pmatrix} 1 \\ 1 \end{pmatrix}\]

Iteration 2: \[\boldsymbol{y}_2 = \mathbf{A}\boldsymbol{x}_1 = \begin{pmatrix} 4 \\ 4 \end{pmatrix}, \quad \mu_2 = 4, \quad \boldsymbol{x}_2 = \begin{pmatrix} 1 \\ 1 \end{pmatrix}\]

The method converges immediately because we started with the dominant eigenvector! The exact eigenvalue is 4.

Starting with different initial vector: \(\boldsymbol{x}_0 = \begin{pmatrix} 1 \\ 0 \end{pmatrix}\)

Iteration 1: \[\boldsymbol{y}_1 = \begin{pmatrix} 3 \\ 1 \end{pmatrix}, \quad \mu_1 = 3, \quad \boldsymbol{x}_1 = \begin{pmatrix} 1 \\ 1/3 \end{pmatrix}\]

Iteration 2: \[\boldsymbol{y}_2 = \begin{pmatrix} 3 & 1 \\ 1 & 3 \end{pmatrix}\begin{pmatrix} 1 \\ 1/3 \end{pmatrix} = \begin{pmatrix} 10/3 \\ 4/3 \end{pmatrix}, \quad \mu_2 = 10/3 \approx 3.33\]

The sequence converges to the dominant eigenvalue \(\lambda_1 = 4\) with eigenvector \((1, 1)^T\).

Theorem 8.29 (Convergence of QR Algorithm). Under suitable conditions (no repeated eigenvalues, certain ordering of eigenvalue magnitudes), the QR algorithm converges:

1. The diagonal entries of \(\mathbf{A}_k\) converge to the eigenvalues of \(\mathbf{A}\)

2. The entries below the diagonal converge to zero

3. The convergence rate depends on the ratios \(|\lambda_i/\lambda_j|\) for \(i > j\)

Example 8.30 (QR Algorithm Steps). Apply one step of the QR algorithm to \(\mathbf{A} = \begin{pmatrix} 3 & 1 \\ 1 & 3 \end{pmatrix}\).

Step 1: QR factorization of A From earlier computations: \[\mathbf{Q}_0 = \frac{1}{\sqrt{2}}\begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix}, \quad \mathbf{R}_0 = \sqrt{2}\begin{pmatrix} 2 & 2 \\ 0 & 1 \end{pmatrix}\]

Step 2: Update \[\mathbf{A}_1 = \mathbf{R}_0\mathbf{Q}_0 = \sqrt{2}\begin{pmatrix} 2 & 2 \\ 0 & 1 \end{pmatrix} \cdot \frac{1}{\sqrt{2}}\begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix} = \begin{pmatrix} 2 & 2 \\ 0 & 1 \end{pmatrix}\begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix}\]

\[= \begin{pmatrix} 4 & 0 \\ 1 & -1 \end{pmatrix}\]

Wait, this doesn’t look right. Let me recalculate the QR factorization:

For \(\mathbf{A} = \begin{pmatrix} 3 & 1 \\ 1 & 3 \end{pmatrix}\): - \(\boldsymbol{q}_1 = \frac{(3,1)^T}{\|(3,1)^T\|} = \frac{1}{\sqrt{10}}(3,1)^T\) - \(\boldsymbol{v}_2 = (1,3)^T - \frac{(1,3) \cdot \boldsymbol{q}_1}{\|\boldsymbol{q}_1\|^2}\boldsymbol{q}_1 = (1,3)^T - \frac{6}{\sqrt{10}} \cdot \frac{1}{\sqrt{10}}(3,1)^T = (1,3)^T - \frac{6}{10}(3,1)^T = (-4/5, 12/5)^T\) - \(\boldsymbol{q}_2 = \frac{(-4/5, 12/5)^T}{\|(-4/5, 12/5)^T\|} = \frac{1}{\sqrt{160/25}}(-4/5, 12/5)^T = \frac{1}{\sqrt{10}}(-1,3)^T\)

So \(\mathbf{Q}_0 = \frac{1}{\sqrt{10}}\begin{pmatrix} 3 & -1 \\ 1 & 3 \end{pmatrix}\) and \(\mathbf{R}_0 = \begin{pmatrix} \sqrt{10} & 6/\sqrt{10} \\ 0 & 4/\sqrt{10} \end{pmatrix}\)

This example illustrates the computational complexity involved in the QR algorithm.

Remark 440. In practice, the QR algorithm is enhanced with several improvements:

• Hessenberg reduction: First reduce the matrix to upper Hessenberg form

• Shifts: Use spectral shifts to accelerate convergence

• Deflation: Remove converged eigenvalues to reduce the problem size

• Implicit methods: Avoid explicit QR factorization for better numerical stability

These refinements make the QR algorithm practical for matrices of size thousands by thousands.

8.6 Singular Value Decomposition (SVD)

The Singular Value Decomposition is arguably the most important matrix decomposition in modern applied mathematics. It provides a complete analysis of any matrix and has applications from data compression to machine learning.

Definition 8.32 (Singular Value Decomposition). Every \(m \times n\) matrix \(\mathbf{A}\) has a singular value decomposition: \[\mathbf{A} = \mathbf{U}\mathbf{\Sigma}\mathbf{V}^T\] where:

• \(\mathbf{U}\) is \(m \times m\) orthogonal (left singular vectors)

• \(\mathbf{V}\) is \(n \times n\) orthogonal (right singular vectors)

• \(\mathbf{\Sigma}\) is \(m \times n\) diagonal with non-negative entries \(\sigma_1 \geq \sigma_2 \geq \cdots \geq \sigma_r > 0\) (singular values)

Theorem 8.33 (Existence and Properties of SVD).

1. Every matrix has an SVD

2. The singular values are uniquely determined

3. If the singular values are distinct, the singular vectors are unique up to sign

4. The rank of \(\mathbf{A}\) equals the number of non-zero singular values

5. \(\|\mathbf{A}\|_2 = \sigma_1\) (operator norm equals largest singular value)

Theorem 8.34 (Relationship to Eigenvalue Decomposition). The SVD is closely related to eigenvalue decompositions:

1. The singular values of \(\mathbf{A}\) are the square roots of the eigenvalues of \(\mathbf{A}^T\mathbf{A}\) (and \(\mathbf{A}\mathbf{A}^T\))

2. The right singular vectors are eigenvectors of \(\mathbf{A}^T\mathbf{A}\)

3. The left singular vectors are eigenvectors of \(\mathbf{A}\mathbf{A}^T\)

4. \(\sigma_i^2 = \lambda_i(\mathbf{A}^T\mathbf{A}) = \lambda_i(\mathbf{A}\mathbf{A}^T)\) for \(i \leq r = \operatorname{rank}(\mathbf{A})\)

Example 8.35 (SVD Computation). Find the SVD of \(\mathbf{A} = \begin{pmatrix} 3 & 1 \\ 1 & 3 \\ 1 & 1 \end{pmatrix}\).

Step 1: Compute \(\mathbf{A}^T\mathbf{A}\) \[\mathbf{A}^T\mathbf{A} = \begin{pmatrix} 3 & 1 & 1 \\ 1 & 3 & 1 \end{pmatrix}\begin{pmatrix} 3 & 1 \\ 1 & 3 \\ 1 & 1 \end{pmatrix} = \begin{pmatrix} 11 & 7 \\ 7 & 11 \end{pmatrix}\]

Step 2: Find eigenvalues and eigenvectors \[\det(\mathbf{A}^T\mathbf{A} - \lambda\mathbf{I}) = (11-\lambda)^2 - 49 = \lambda^2 - 22\lambda + 72 = (\lambda-18)(\lambda-4)\]

Eigenvalues: \(\lambda_1 = 18, \lambda_2 = 4\) Singular values: \(\sigma_1 = \sqrt{18} = 3\sqrt{2}, \sigma_2 = 2\)

For \(\lambda_1 = 18\): \[(\mathbf{A}^T\mathbf{A} - 18\mathbf{I})\boldsymbol{v} = \begin{pmatrix} -7 & 7 \\ 7 & -7 \end{pmatrix}\boldsymbol{v} = \boldsymbol{0}\] This gives \(v_1 = v_2\), so \(\boldsymbol{v}_1 = \frac{1}{\sqrt{2}}\begin{pmatrix} 1 \\ 1 \end{pmatrix}\).

For \(\lambda_2 = 4\): \[(\mathbf{A}^T\mathbf{A} - 4\mathbf{I})\boldsymbol{v} = \begin{pmatrix} 7 & 7 \\ 7 & 7 \end{pmatrix}\boldsymbol{v} = \boldsymbol{0}\] This gives \(v_1 = -v_2\), so \(\boldsymbol{v}_2 = \frac{1}{\sqrt{2}}\begin{pmatrix} 1 \\ -1 \end{pmatrix}\).

Therefore: \(\mathbf{V} = \frac{1}{\sqrt{2}}\begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix}\)

Step 3: Compute left singular vectors \[\boldsymbol{u}_1 = \frac{\mathbf{A}\boldsymbol{v}_1}{\sigma_1} = \frac{1}{3\sqrt{2}}\begin{pmatrix} 3 & 1 \\ 1 & 3 \\ 1 & 1 \end{pmatrix}\frac{1}{\sqrt{2}}\begin{pmatrix} 1 \\ 1 \end{pmatrix} = \frac{1}{6}\begin{pmatrix} 4 \\ 4 \\ 2 \end{pmatrix} = \frac{1}{3}\begin{pmatrix} 2 \\ 2 \\ 1 \end{pmatrix}\]

\[\boldsymbol{u}_2 = \frac{\mathbf{A}\boldsymbol{v}_2}{\sigma_2} = \frac{1}{2}\begin{pmatrix} 3 & 1 \\ 1 & 3 \\ 1 & 1 \end{pmatrix}\frac{1}{\sqrt{2}}\begin{pmatrix} 1 \\ -1 \end{pmatrix} = \frac{1}{2\sqrt{2}}\begin{pmatrix} 2 \\ -2 \\ 0 \end{pmatrix} = \frac{1}{\sqrt{2}}\begin{pmatrix} 1 \\ -1 \\ 0 \end{pmatrix}\]

For the third left singular vector, we need a vector orthogonal to both \(\boldsymbol{u}_1\) and \(\boldsymbol{u}_2\): Using the cross product or solving the orthogonality conditions, we get: \[\boldsymbol{u}_3 = \frac{1}{\sqrt{18}}\begin{pmatrix} 1 \\ 1 \\ -4 \end{pmatrix}\]

Step 4: Final SVD \[\mathbf{U} = \begin{pmatrix} \frac{2}{3} & \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{18}} \\ \frac{2}{3} & -\frac{1}{\sqrt{2}} & \frac{1}{\sqrt{18}} \\ \frac{1}{3} & 0 & -\frac{4}{\sqrt{18}} \end{pmatrix}, \quad \mathbf{\Sigma} = \begin{pmatrix} 3\sqrt{2} & 0 \\ 0 & 2 \\ 0 & 0 \end{pmatrix}, \quad \mathbf{V} = \frac{1}{\sqrt{2}}\begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix}\]

8.6.1 Fundamental Subspaces

The SVD provides a complete characterization of the four fundamental subspaces associated with any matrix.

Theorem 8.36 (Four Fundamental Subspaces via SVD). If \(\mathbf{A} = \mathbf{U}\mathbf{\Sigma}\mathbf{V}^T\) with rank \(r\), then:

1. Column space: \(\text{Col}(\mathbf{A}) = \operatorname{span}\{\boldsymbol{u}_1, \boldsymbol{u}_2, \ldots, \boldsymbol{u}_r\}\)

2. Row space: \(\text{Col}(\mathbf{A}^T) = \operatorname{span}\{\boldsymbol{v}_1, \boldsymbol{v}_2, \ldots, \boldsymbol{v}_r\}\)

3. Null space: \(\text{Nul}(\mathbf{A}) = \operatorname{span}\{\boldsymbol{v}_{r+1}, \boldsymbol{v}_{r+2}, \ldots, \boldsymbol{v}_n\}\)

4. Left null space: \(\text{Nul}(\mathbf{A}^T) = \operatorname{span}\{\boldsymbol{u}_{r+1}, \boldsymbol{u}_{r+2}, \ldots, \boldsymbol{u}_m\}\)

Moreover:

• \(\mathbb{R}^n = \text{Col}(\mathbf{A}^T) \oplus \text{Nul}(\mathbf{A})\)

• \(\mathbb{R}^m = \text{Col}(\mathbf{A}) \oplus \text{Nul}(\mathbf{A}^T)\)

Proof. These results follow directly from the SVD structure. For example: \[\mathbf{A}\boldsymbol{v}_i = \mathbf{U}\mathbf{\Sigma}\mathbf{V}^T\boldsymbol{v}_i = \mathbf{U}\mathbf{\Sigma}\boldsymbol{e}_i = \sigma_i\boldsymbol{u}_i\]

So for \(i \leq r\): \(\mathbf{A}\boldsymbol{v}_i = \sigma_i\boldsymbol{u}_i \neq \boldsymbol{0}\), meaning \(\boldsymbol{v}_i \notin \text{Nul}(\mathbf{A})\). For \(i > r\): \(\mathbf{A}\boldsymbol{v}_i = 0 \cdot \boldsymbol{u}_i = \boldsymbol{0}\), meaning \(\boldsymbol{v}_i \in \text{Nul}(\mathbf{A})\). ◻

8.6.2 Polar Decomposition

The SVD leads naturally to the polar decomposition, which is the matrix analogue of writing a complex number as \(re^{i\theta}\).

Definition 8.37 (Polar Decomposition). Every \(m \times n\) matrix \(\mathbf{A}\) can be written as: \[\mathbf{A} = \mathbf{Q}\mathbf{P}\] where \(\mathbf{Q}\) has orthonormal columns and \(\mathbf{P}\) is positive semidefinite.

Theorem 8.38 (Polar Decomposition via SVD). If \(\mathbf{A} = \mathbf{U}\mathbf{\Sigma}\mathbf{V}^T\) is the SVD, then:

1. Right polar decomposition: \(\mathbf{A} = \mathbf{Q}_R\mathbf{P}_R\) where \(\mathbf{Q}_R = \mathbf{U}\mathbf{V}^T\) and \(\mathbf{P}_R = \mathbf{V}\mathbf{\Sigma}\mathbf{V}^T\)

2. Left polar decomposition: \(\mathbf{A} = \mathbf{P}_L\mathbf{Q}_L\) where \(\mathbf{P}_L = \mathbf{U}\mathbf{\Sigma}\mathbf{U}^T\) and \(\mathbf{Q}_L = \mathbf{U}\mathbf{V}^T\)

Proof. For the right polar decomposition: \[\mathbf{Q}_R\mathbf{P}_R = (\mathbf{U}\mathbf{V}^T)(\mathbf{V}\mathbf{\Sigma}\mathbf{V}^T) = \mathbf{U}\mathbf{\Sigma}\mathbf{V}^T = \mathbf{A}\]

\(\mathbf{Q}_R\) has orthonormal columns since both \(\mathbf{U}\) and \(\mathbf{V}\) are orthogonal. \(\mathbf{P}_R\) is positive semidefinite since it has the form \(\mathbf{V}\mathbf{D}\mathbf{V}^T\) where \(\mathbf{D}\) has non-negative diagonal entries. ◻

8.6.3 Pseudoinverse

The SVD provides the natural generalization of matrix inversion to non-square and singular matrices.

Definition 8.39 (Moore-Penrose Pseudoinverse). For a matrix \(\mathbf{A} = \mathbf{U}\mathbf{\Sigma}\mathbf{V}^T\) with SVD, the Moore-Penrose pseudoinverse is: \[\mathbf{A}^+ = \mathbf{V}\mathbf{\Sigma}^+\mathbf{U}^T\] where \(\mathbf{\Sigma}^+\) is formed by taking the reciprocal of each non-zero singular value and transposing.

Theorem 8.40 (Properties of the Pseudoinverse). The pseudoinverse \(\mathbf{A}^+\) satisfies:

1. \(\mathbf{A}\mathbf{A}^+\mathbf{A} = \mathbf{A}\)

2. \(\mathbf{A}^+\mathbf{A}\mathbf{A}^+ = \mathbf{A}^+\)

3. \((\mathbf{A}\mathbf{A}^+)^T = \mathbf{A}\mathbf{A}^+\)

4. \((\mathbf{A}^+\mathbf{A})^T = \mathbf{A}^+\mathbf{A}\)

5. If \(\mathbf{A}\) is invertible, then \(\mathbf{A}^+ = \mathbf{A}^{-1}\)

6. \(\mathbf{A}^+\) provides the minimum norm solution to \(\mathbf{A}\boldsymbol{x} = \boldsymbol{b}\) when solutions exist

7. \(\mathbf{A}^+\boldsymbol{b}\) provides the least squares solution when \(\mathbf{A}\boldsymbol{x} = \boldsymbol{b}\) has no exact solution

Example 8.41 (Pseudoinverse Computation). Find the pseudoinverse of \(\mathbf{A} = \begin{pmatrix} 1 & 2 \\ 2 & 4 \end{pmatrix}\).

Step 1: Note that \(\mathbf{A}\) has rank 1 The second row is twice the first, so \(\operatorname{rank}(\mathbf{A}) = 1\).

Step 2: Compute SVD \[\mathbf{A}^T\mathbf{A} = \begin{pmatrix} 1 & 2 \\ 2 & 4 \end{pmatrix}\begin{pmatrix} 1 & 2 \\ 2 & 4 \end{pmatrix} = \begin{pmatrix} 5 & 10 \\ 10 & 20 \end{pmatrix}\]

Eigenvalues: \(\lambda_1 = 25, \lambda_2 = 0\) (since \(\det = 0\) and \(\operatorname{tr}= 25\)) Singular values: \(\sigma_1 = 5, \sigma_2 = 0\)

Right singular vectors: - For \(\lambda_1 = 25\): \(\boldsymbol{v}_1 = \frac{1}{\sqrt{5}}\begin{pmatrix} 1 \\ 2 \end{pmatrix}\) - For \(\lambda_2 = 0\): \(\boldsymbol{v}_2 = \frac{1}{\sqrt{5}}\begin{pmatrix} 2 \\ -1 \end{pmatrix}\)

Left singular vectors: \[\boldsymbol{u}_1 = \frac{\mathbf{A}\boldsymbol{v}_1}{\sigma_1} = \frac{1}{5}\begin{pmatrix} 1 & 2 \\ 2 & 4 \end{pmatrix}\frac{1}{\sqrt{5}}\begin{pmatrix} 1 \\ 2 \end{pmatrix} = \frac{1}{5\sqrt{5}}\begin{pmatrix} 5 \\ 10 \end{pmatrix} = \frac{1}{\sqrt{5}}\begin{pmatrix} 1 \\ 2 \end{pmatrix}\]

\[\boldsymbol{u}_2 = \frac{1}{\sqrt{5}}\begin{pmatrix} 2 \\ -1 \end{pmatrix}\] (orthogonal to \(\boldsymbol{u}_1\))

Step 3: Form pseudoinverse \[\mathbf{\Sigma}^+ = \begin{pmatrix} 1/5 & 0 \\ 0 & 0 \end{pmatrix}\]

\[\mathbf{A}^+ = \mathbf{V}\mathbf{\Sigma}^+\mathbf{U}^T = \frac{1}{\sqrt{5}}\begin{pmatrix} 1 & 2 \\ 2 & -1 \end{pmatrix}\begin{pmatrix} 1/5 & 0 \\ 0 & 0 \end{pmatrix}\frac{1}{\sqrt{5}}\begin{pmatrix} 1 & 2 \\ 2 & -1 \end{pmatrix}\]

\[= \frac{1}{25}\begin{pmatrix} 1 & 2 \\ 2 & -1 \end{pmatrix}\begin{pmatrix} 1 & 0 \\ 0 & 0 \end{pmatrix}\begin{pmatrix} 1 & 2 \\ 2 & -1 \end{pmatrix} = \frac{1}{25}\begin{pmatrix} 1 \\ 2 \end{pmatrix}\begin{pmatrix} 1 & 2 \end{pmatrix} = \frac{1}{25}\begin{pmatrix} 1 & 2 \\ 2 & 4 \end{pmatrix}\]

Verification: \[\mathbf{A}\mathbf{A}^+ = \begin{pmatrix} 1 & 2 \\ 2 & 4 \end{pmatrix}\frac{1}{25}\begin{pmatrix} 1 & 2 \\ 2 & 4 \end{pmatrix} = \frac{1}{25}\begin{pmatrix} 5 & 10 \\ 10 & 20 \end{pmatrix} = \frac{1}{5}\begin{pmatrix} 1 & 2 \\ 2 & 4 \end{pmatrix}\]

And indeed: \(\mathbf{A}\mathbf{A}^+\mathbf{A} = \frac{1}{5}\begin{pmatrix} 1 & 2 \\ 2 & 4 \end{pmatrix}\begin{pmatrix} 1 & 2 \\ 2 & 4 \end{pmatrix} = \frac{1}{5}\begin{pmatrix} 5 & 10 \\ 10 & 20 \end{pmatrix} = \begin{pmatrix} 1 & 2 \\ 2 & 4 \end{pmatrix} = \mathbf{A}\)

8.7 Complex Matrices

Complex matrices arise naturally in many applications, from quantum mechanics to signal processing. The theory of orthogonality extends naturally to the complex setting with important modifications.

Definition 8.42 (Complex Inner Product). For vectors \(\boldsymbol{u}, \boldsymbol{v} \in \mathbb{C}^n\), the complex inner product is: \[\langle \boldsymbol{u}, \boldsymbol{v} \rangle = \boldsymbol{u}^*\boldsymbol{v} = \sum_{i=1}^n \overline{u_i}v_i\] where \(\boldsymbol{u}^* = \overline{\boldsymbol{u}}^T\) is the conjugate transpose.

Definition 8.43 (Unitary Matrix). A complex matrix \(\mathbf{U}\) is unitary if \(\mathbf{U}^*\mathbf{U} = \mathbf{I}\), i.e., \(\mathbf{U}^* = \mathbf{U}^{-1}\).

Properties of unitary matrices:

• They preserve the complex inner product: \(\langle \mathbf{U}\boldsymbol{x}, \mathbf{U}\boldsymbol{y} \rangle = \langle \boldsymbol{x}, \boldsymbol{y} \rangle\)

• \(|\det(\mathbf{U})| = 1\)

• All eigenvalues have absolute value 1

Definition 8.44 (Hermitian Matrix). A complex matrix \(\mathbf{A}\) is Hermitian if \(\mathbf{A}^* = \mathbf{A}\), i.e., \(a_{ij} = \overline{a_{ji}}\) for all \(i, j\).

Properties of Hermitian matrices:

• All eigenvalues are real

• Eigenvectors corresponding to different eigenvalues are orthogonal (with respect to the complex inner product)

• Every Hermitian matrix is unitarily diagonalizable

Theorem 8.45 (Spectral Theorem for Hermitian Matrices). A complex matrix \(\mathbf{A}\) is unitarily diagonalizable if and only if \(\mathbf{A}\) is Hermitian. Moreover, there exists a unitary matrix \(\mathbf{U}\) such that: \[\mathbf{A} = \mathbf{U}\mathbf{D}\mathbf{U}^*\] where \(\mathbf{D}\) is real diagonal.

Example 8.46 (Complex Matrix Diagonalization). Diagonalize the Hermitian matrix \(\mathbf{A} = \begin{pmatrix} 2 & 1-i \\ 1+i & 3 \end{pmatrix}\).

Verification of Hermitian property: \[\mathbf{A}^* = \begin{pmatrix} 2 & \overline{1+i} \\ \overline{1-i} & 3 \end{pmatrix} = \begin{pmatrix} 2 & 1-i \\ 1+i & 3 \end{pmatrix} = \mathbf{A}\]

Step 1: Find eigenvalues \[\det(\mathbf{A} - \lambda\mathbf{I}) = \det\begin{pmatrix} 2-\lambda & 1-i \\ 1+i & 3-\lambda \end{pmatrix} = (2-\lambda)(3-\lambda) - (1-i)(1+i)\]

\[= (2-\lambda)(3-\lambda) - (1-i^2) = (2-\lambda)(3-\lambda) - 2 = \lambda^2 - 5\lambda + 4 = (\lambda-1)(\lambda-4)\]

Eigenvalues: \(\lambda_1 = 1, \lambda_2 = 4\) (both real, as expected for Hermitian matrices).

Step 2: Find eigenvectors

For \(\lambda_1 = 1\): \[(\mathbf{A} - \mathbf{I})\boldsymbol{v} = \begin{pmatrix} 1 & 1-i \\ 1+i & 2 \end{pmatrix}\boldsymbol{v} = \boldsymbol{0}\]

From the first row: \(v_1 + (1-i)v_2 = 0\), so \(v_1 = -(1-i)v_2\). Choosing \(v_2 = 1\): \(\boldsymbol{v}_1 = \begin{pmatrix} -(1-i) \\ 1 \end{pmatrix} = \begin{pmatrix} -1+i \\ 1 \end{pmatrix}\).

Normalize: \(\|\boldsymbol{v}_1\|^2 = |-1+i|^2 + |1|^2 = 2 + 1 = 3\) \[\boldsymbol{u}_1 = \frac{1}{\sqrt{3}}\begin{pmatrix} -1+i \\ 1 \end{pmatrix}\]

For \(\lambda_2 = 4\): \[(\mathbf{A} - 4\mathbf{I})\boldsymbol{v} = \begin{pmatrix} -2 & 1-i \\ 1+i & -1 \end{pmatrix}\boldsymbol{v} = \boldsymbol{0}\]

From the first row: \(-2v_1 + (1-i)v_2 = 0\), so \(v_1 = \frac{1-i}{2}v_2\). Choosing \(v_2 = 2\): \(\boldsymbol{v}_2 = \begin{pmatrix} 1-i \\ 2 \end{pmatrix}\).

Normalize: \(\|\boldsymbol{v}_2\|^2 = |1-i|^2 + |2|^2 = 2 + 4 = 6\) \[\boldsymbol{u}_2 = \frac{1}{\sqrt{6}}\begin{pmatrix} 1-i \\ 2 \end{pmatrix}\]

Step 3: Form unitary matrix \[\mathbf{U} = \begin{pmatrix} \frac{-1+i}{\sqrt{3}} & \frac{1-i}{\sqrt{6}} \\ \frac{1}{\sqrt{3}} & \frac{2}{\sqrt{6}} \end{pmatrix}, \quad \mathbf{D} = \begin{pmatrix} 1 & 0 \\ 0 & 4 \end{pmatrix}\]

Verification of orthogonality: \[\langle \boldsymbol{u}_1, \boldsymbol{u}_2 \rangle = \boldsymbol{u}_1^*\boldsymbol{u}_2 = \frac{1}{\sqrt{18}}\begin{pmatrix} -1-i & 1 \end{pmatrix}\begin{pmatrix} 1-i \\ 2 \end{pmatrix}\] \[= \frac{1}{\sqrt{18}}((-1-i)(1-i) + 2) = \frac{1}{\sqrt{18}}(-1+i-i-1+2) = \frac{1}{\sqrt{18}} \cdot 0 = 0\]

Definition 8.47 (Normal Matrix). A complex matrix \(\mathbf{A}\) is normal if \(\mathbf{A}\mathbf{A}^* = \mathbf{A}^*\mathbf{A}\).

Important classes of normal matrices:

• Hermitian matrices: \(\mathbf{A}^* = \mathbf{A}\)

• Unitary matrices: \(\mathbf{A}^*\mathbf{A} = \mathbf{I}\)

• Skew-Hermitian matrices: \(\mathbf{A}^* = -\mathbf{A}\)

Theorem 8.48 (Spectral Theorem for Normal Matrices). A complex matrix \(\mathbf{A}\) is unitarily diagonalizable if and only if \(\mathbf{A}\) is normal.

8.8 Applications of Matrix Decompositions

The decompositions we have studied have far-reaching applications across science and engineering. We conclude by exploring several key application areas.

8.8.1 Principal Component Analysis (PCA)

Principal Component Analysis uses the SVD to find the most important directions of variation in high-dimensional data.

Example 8.49 (PCA Application). Consider data points representing measurements of height and weight. If the data matrix (after centering) has SVD with singular values \(\sigma_1 = 10, \sigma_2 = 2\), then:

• The first principal component explains \(\frac{100}{104} \approx 96.2\%\) of the variance

• The second principal component explains \(\frac{4}{104} \approx 3.8\%\) of the variance

• For dimension reduction, we might keep only the first principal component, losing only 3.8% of the information

8.8.2 Image Compression

The SVD provides an optimal low-rank approximation for any matrix, making it ideal for image compression.

Theorem 8.50 (Eckart-Young Theorem). If \(\mathbf{A} = \mathbf{U}\mathbf{\Sigma}\mathbf{V}^T\) is the SVD with singular values \(\sigma_1 \geq \sigma_2 \geq \cdots \geq \sigma_r > 0\), then the best rank-\(k\) approximation to \(\mathbf{A}\) in both the Frobenius and operator norms is: \[\mathbf{A}_k = \sum_{i=1}^k \sigma_i \boldsymbol{u}_i \boldsymbol{v}_i^T\]

Moreover: \[\|\mathbf{A} - \mathbf{A}_k\|_2 = \sigma_{k+1}, \quad \|\mathbf{A} - \mathbf{A}_k\|_F = \sqrt{\sum_{i=k+1}^r \sigma_i^2}\]

Example 8.51 (Image Compression with SVD). For a grayscale image represented as an \(m \times n\) matrix \(\mathbf{A}\):

Original storage: \(mn\) values Rank-\(k\) approximation storage: \(k(m + n + 1)\) values (storing \(\boldsymbol{u}_i, \boldsymbol{v}_i, \sigma_i\) for \(i = 1, \ldots, k\))

Compression ratio: \(\frac{mn}{k(m+n+1)}\)

For a \(1000 \times 1000\) image with \(k = 50\): - Original: \(1,000,000\) values - Compressed: \(50 \times 2001 = 100,050\) values - Compression ratio: \(\approx 10:1\)

The quality depends on how quickly the singular values decay.

8.8.3 Recommendation Systems

Matrix factorization techniques based on SVD are fundamental to modern recommendation systems.

8.8.4 Solving Linear Systems

Different decompositions are optimal for different types of linear systems.

Example 8.52 (Condition Number Analysis). Consider the Hilbert matrix \(H_n\) with entries \(h_{ij} = \frac{1}{i+j-1}\):

For \(n = 3\): \(\mathbf{H}_3 = \begin{pmatrix} 1 & 1/2 & 1/3 \\ 1/2 & 1/3 & 1/4 \\ 1/3 & 1/4 & 1/5 \end{pmatrix}\)

The condition numbers grow exponentially:

• \(\kappa(\mathbf{H}_3) \approx 524\)

• \(\kappa(\mathbf{H}_5) \approx 476,607\)

• \(\kappa(\mathbf{H}_{10}) \approx 1.6 \times 10^{13}\)

This makes Hilbert matrices notoriously difficult for numerical computation.

8.8.5 Quantum Mechanics Applications

In quantum mechanics, Hermitian matrices represent observable quantities, and their eigenvalues correspond to possible measurement outcomes.

Example 8.53 (Quantum Spin System). The Pauli matrices represent spin-1/2 particles: \[\sigma_x = \begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix}, \quad \sigma_y = \begin{pmatrix} 0 & -i \\ i & 0 \end{pmatrix}, \quad \sigma_z = \begin{pmatrix} 1 & 0 \\ 0 & -1 \end{pmatrix}\]

Each is Hermitian with eigenvalues \(\pm 1\), corresponding to spin measurements of \(+\hbar/2\) and \(-\hbar/2\).

For \(\sigma_x\): - Eigenvalues: \(\pm 1\) - Eigenvectors: \(\frac{1}{\sqrt{2}}\begin{pmatrix} 1 \\ 1 \end{pmatrix}\) and \(\frac{1}{\sqrt{2}}\begin{pmatrix} 1 \\ -1 \end{pmatrix}\)

The measurement probabilities are given by \(|\langle \psi | \boldsymbol{v}_i \rangle|^2\) where \(\psi\) is the quantum state and \(\boldsymbol{v}_i\) are the eigenvectors.

Key Idea

The Power of Orthogonality and Decomposition:

• Orthogonal complements: Provide the geometric foundation for understanding subspace relationships and projections

• Orthogonal diagonalization: Reveals the natural coordinate systems where linear transformations have the simplest form

• Positive definite matrices: Characterize optimization problems and statistical covariance structures

• QR factorization: Provides numerically stable algorithms for least squares and eigenvalue computation

• Singular Value Decomposition: Offers the most comprehensive matrix analysis tool, revealing rank, fundamental subspaces, and optimal approximations

• Complex extensions: Show how the theory generalizes to quantum mechanics and signal processing applications

These concepts form the computational and theoretical backbone of modern applied mathematics. From data science to quantum computing, from optimization to machine learning, the tools developed in this chapter provide both conceptual understanding and practical algorithms for solving complex real-world problems.

The beauty of this theory lies in how abstract geometric concepts translate into powerful computational tools. The orthogonality that began as a simple geometric notion of perpendicularity has blossomed into decompositions that can compress images, predict preferences, solve optimization problems, and describe quantum mechanical systems.

Moreover, the interplay between different decompositions—how the QR factorization leads to eigenvalue algorithms, how the SVD generalizes eigenvalue decomposition, how Cholesky decomposition optimizes positive definite systems—demonstrates the deep unity underlying linear algebra. Each technique illuminates different aspects of the same fundamental geometric and algebraic structures.

This unified perspective not only aids understanding but also guides practical choices. Knowing when to use which decomposition, understanding the trade-offs between computational cost and numerical stability, recognizing the geometric meaning behind algebraic manipulations—these insights transform linear algebra from a collection of techniques into a coherent framework for mathematical modeling and computation.

This completes our comprehensive exploration of orthogonality and decomposition. We have journeyed from basic concepts of orthogonal complements through sophisticated matrix decompositions, arriving at applications that shape modern technology and scientific understanding. The theory developed here provides both the mathematical foundation and practical tools for tackling complex problems across diverse fields, demonstrating the remarkable power and elegance of linear algebraic methods.

9 Change of Basis

The concept of change of basis is fundamental to understanding how mathematical objects can be represented in different coordinate systems. Just as the same physical vector can be described using different coordinate axes, the same abstract vector can be represented using different bases. This flexibility is crucial for simplifying computations, revealing geometric structure, and connecting different mathematical perspectives. This chapter develops the complete theory of basis changes and their applications to linear transformations.

9.1 Coordinate Vectors and Matrix Representations

We begin by formalizing how vectors and linear transformations are represented with respect to different bases, establishing the foundation for understanding basis changes.

Definition 9.1 (Coordinate Vector). Let \(V\) be a finite-dimensional vector space with basis \(\mathcal{B} = \{\boldsymbol{b}_1, \boldsymbol{b}_2, \ldots, \boldsymbol{b}_n\}\). For any vector \(\boldsymbol{v} \in V\), there exist unique scalars \(c_1, c_2, \ldots, c_n\) such that: \[\boldsymbol{v} = c_1\boldsymbol{b}_1 + c_2\boldsymbol{b}_2 + \cdots + c_n\boldsymbol{b}_n\]

The coordinate vector of \(\boldsymbol{v}\) with respect to basis \(\mathcal{B}\) is: \[[\boldsymbol{v}]_{\mathcal{B}} = \begin{pmatrix} c_1 \\ c_2 \\ \vdots \\ c_n \end{pmatrix}\]

Remark 464. The coordinate vector \([\boldsymbol{v}]_{\mathcal{B}}\) is the "recipe" for constructing \(\boldsymbol{v}\) from the basis vectors. It transforms the abstract vector \(\boldsymbol{v}\) into a concrete column vector in \(\mathbb{R}^n\), allowing us to perform computations using familiar matrix operations. The uniqueness of this representation is guaranteed by the linear independence of the basis vectors.

Theorem 9.3 (Properties of Coordinate Representations). Let \(\mathcal{B}\) be a basis for vector space \(V\). The coordinate mapping \(\boldsymbol{v} \mapsto [\boldsymbol{v}]_{\mathcal{B}}\) is:

1. Linear: \([a\boldsymbol{u} + b\boldsymbol{v}]_{\mathcal{B}} = a[\boldsymbol{u}]_{\mathcal{B}} + b[\boldsymbol{v}]_{\mathcal{B}}\)

2. Bijective: It establishes a one-to-one correspondence between \(V\) and \(\mathbb{R}^n\)

3. Basis-preserving: \([\boldsymbol{b}_i]_{\mathcal{B}} = \boldsymbol{e}_i\) (the \(i\)-th standard basis vector)

Proof. Linearity: If \(\boldsymbol{u} = \sum u_i\boldsymbol{b}_i\) and \(\boldsymbol{v} = \sum v_i\boldsymbol{b}_i\), then: \[a\boldsymbol{u} + b\boldsymbol{v} = \sum (au_i + bv_i)\boldsymbol{b}_i\]

Therefore: \[[a\boldsymbol{u} + b\boldsymbol{v}]_{\mathcal{B}} = \begin{pmatrix} au_1 + bv_1 \\ \vdots \\ au_n + bv_n \end{pmatrix} = a\begin{pmatrix} u_1 \\ \vdots \\ u_n \end{pmatrix} + b\begin{pmatrix} v_1 \\ \vdots \\ v_n \end{pmatrix} = a[\boldsymbol{u}]_{\mathcal{B}} + b[\boldsymbol{v}]_{\mathcal{B}}\]

Bijectivity: - Injective: If \([\boldsymbol{u}]_{\mathcal{B}} = [\boldsymbol{v}]_{\mathcal{B}}\), then \(\boldsymbol{u}\) and \(\boldsymbol{v}\) have the same coordinates, so \(\boldsymbol{u} = \boldsymbol{v}\). - Surjective: Any vector in \(\mathbb{R}^n\) corresponds to some linear combination of the basis vectors.

Basis-preserving: \(\boldsymbol{b}_i = 0\boldsymbol{b}_1 + \cdots + 1\boldsymbol{b}_i + \cdots + 0\boldsymbol{b}_n\), so \([\boldsymbol{b}_i]_{\mathcal{B}} = \boldsymbol{e}_i\). ◻

Example 9.4 (Computing Coordinate Vectors). 1. Polynomial space: In \(\mathcal{P}_2\) with basis \(\mathcal{B} = \{1, x, x^2\}\), find the coordinate vector of \(p(x) = 3 - 2x + 5x^2\).

Since \(p(x) = 3 \cdot 1 + (-2) \cdot x + 5 \cdot x^2\): \[[p(x)]_{\mathcal{B}} = \begin{pmatrix} 3 \\ -2 \\ 5 \end{pmatrix}\]

2. Alternative basis: With basis \(\mathcal{C} = \{1, 1+x, 1+x+x^2\}\), find \([p(x)]_{\mathcal{C}}\) for the same polynomial.

We need: \(3 - 2x + 5x^2 = c_1 \cdot 1 + c_2(1+x) + c_3(1+x+x^2)\)

Expanding: \(3 - 2x + 5x^2 = (c_1 + c_2 + c_3) + (c_2 + c_3)x + c_3x^2\)

Comparing coefficients: \[\begin{align} c_1 + c_2 + c_3 &= 3 \\ c_2 + c_3 &= -2 \\ c_3 &= 5 \end{align}\]

From equation (3): \(c_3 = 5\) From equation (2): \(c_2 = -2 - 5 = -7\) From equation (1): \(c_1 = 3 - (-7) - 5 = 5\)

Therefore: \([p(x)]_{\mathcal{C}} = \begin{pmatrix} 5 \\ -7 \\ 5 \end{pmatrix}\)

Verification: \(5 \cdot 1 + (-7)(1+x) + 5(1+x+x^2) = 5 - 7 - 7x + 5 + 5x + 5x^2 = 3 - 2x + 5x^2\)

Definition 9.5 (Matrix Representation of Linear Transformations). Let \(T: V \to W\) be a linear transformation, where \(V\) has basis \(\mathcal{B} = \{\boldsymbol{b}_1, \ldots, \boldsymbol{b}_n\}\) and \(W\) has basis \(\mathcal{C} = \{\boldsymbol{c}_1, \ldots, \boldsymbol{c}_m\}\). The matrix representation of \(T\) with respect to these bases is the \(m \times n\) matrix \([\mathbf{T}]_{\mathcal{C}}^{\mathcal{B}}\) whose \(j\)-th column is \([T(\boldsymbol{b}_j)]_{\mathcal{C}}\).

In other words: \[[\mathbf{T}]_{\mathcal{C}}^{\mathcal{B}} = \begin{bmatrix} [T(\boldsymbol{b}_1)]_{\mathcal{C}} & [T(\boldsymbol{b}_2)]_{\mathcal{C}} & \cdots & [T(\boldsymbol{b}_n)]_{\mathcal{C}} \end{bmatrix}\]

Theorem 9.6 (Fundamental Matrix Representation Theorem). If \(T: V \to W\) is a linear transformation with matrix representation \([\mathbf{T}]_{\mathcal{C}}^{\mathcal{B}}\), then for any \(\boldsymbol{v} \in V\): \[[T(\boldsymbol{v})]_{\mathcal{C}} = [\mathbf{T}]_{\mathcal{C}}^{\mathcal{B}} [\boldsymbol{v}]_{\mathcal{B}}\]

This equation is the bridge between abstract linear transformations and concrete matrix operations.

Proof. Let \(\boldsymbol{v} = \sum_{j=1}^n v_j \boldsymbol{b}_j\), so \([\boldsymbol{v}]_{\mathcal{B}} = (v_1, \ldots, v_n)^T\).

By linearity of \(T\): \[T(\boldsymbol{v}) = T\left(\sum_{j=1}^n v_j \boldsymbol{b}_j\right) = \sum_{j=1}^n v_j T(\boldsymbol{b}_j)\]

Taking coordinates with respect to \(\mathcal{C}\): \[[T(\boldsymbol{v})]_{\mathcal{C}} = \left[\sum_{j=1}^n v_j T(\boldsymbol{b}_j)\right]_{\mathcal{C}} = \sum_{j=1}^n v_j [T(\boldsymbol{b}_j)]_{\mathcal{C}}\]

This is precisely the matrix-vector product \([\mathbf{T}]_{\mathcal{C}}^{\mathcal{B}} [\boldsymbol{v}]_{\mathcal{B}}\). ◻

Example 9.7 (Matrix Representation Computation). Find the matrix representation of the differentiation operator \(D: \mathcal{P}_2 \to \mathcal{P}_1\) with respect to the standard bases \(\mathcal{B} = \{1, x, x^2\}\) and \(\mathcal{C} = \{1, x\}\).

Step 1: Apply \(D\) to each basis vector \[\begin{align} D(1) &= 0 = 0 \cdot 1 + 0 \cdot x \\ D(x) &= 1 = 1 \cdot 1 + 0 \cdot x \\ D(x^2) &= 2x = 0 \cdot 1 + 2 \cdot x \end{align}\]

Step 2: Write coordinate vectors \[[D(1)]_{\mathcal{C}} = \begin{pmatrix} 0 \\ 0 \end{pmatrix}, \quad [D(x)]_{\mathcal{C}} = \begin{pmatrix} 1 \\ 0 \end{pmatrix}, \quad [D(x^2)]_{\mathcal{C}} = \begin{pmatrix} 0 \\ 2 \end{pmatrix}\]

Step 3: Form matrix \[[\mathbf{D}]_{\mathcal{C}}^{\mathcal{B}} = \begin{pmatrix} 0 & 1 & 0 \\ 0 & 0 & 2 \end{pmatrix}\]

Verification: For \(p(x) = a + bx + cx^2\) with \([p]_{\mathcal{B}} = (a, b, c)^T\): \[[\mathbf{D}]_{\mathcal{C}}^{\mathcal{B}} [p]_{\mathcal{B}} = \begin{pmatrix} 0 & 1 & 0 \\ 0 & 0 & 2 \end{pmatrix}\begin{pmatrix} a \\ b \\ c \end{pmatrix} = \begin{pmatrix} b \\ 2c \end{pmatrix}\]

This corresponds to \(D(p) = b + 2cx = p'(x)\)

9.2 Change of Basis Matrix

The change of basis matrix is the fundamental tool for converting between different coordinate representations of the same vector.

Definition 9.8 (Change of Basis Matrix). Let \(\mathcal{B} = \{\boldsymbol{b}_1, \ldots, \boldsymbol{b}_n\}\) and \(\mathcal{C} = \{\boldsymbol{c}_1, \ldots, \boldsymbol{c}_n\}\) be two bases for vector space \(V\). The change of basis matrix from \(\mathcal{C}\) to \(\mathcal{B}\) is: \[[\mathbf{P}]_{\mathcal{B}}^{\mathcal{C}} = \begin{bmatrix} [\boldsymbol{c}_1]_{\mathcal{B}} & [\boldsymbol{c}_2]_{\mathcal{B}} & \cdots & [\boldsymbol{c}_n]_{\mathcal{B}} \end{bmatrix}\]

This matrix transforms coordinate vectors from \(\mathcal{C}\)-coordinates to \(\mathcal{B}\)-coordinates.

Remark 471. The notation may seem backwards at first, but it follows the standard convention: \([\mathbf{P}]_{\mathcal{B}}^{\mathcal{C}}\) has its "input basis" \(\mathcal{C}\) as a superscript and its "output basis" \(\mathcal{B}\) as a subscript. This matches the matrix representation notation for linear transformations.

Theorem 9.10 (Change of Basis Formula). If \([\mathbf{P}]_{\mathcal{B}}^{\mathcal{C}}\) is the change of basis matrix from \(\mathcal{C}\) to \(\mathcal{B}\), then for any vector \(\boldsymbol{v} \in V\): \[[\boldsymbol{v}]_{\mathcal{B}} = [\mathbf{P}]_{\mathcal{B}}^{\mathcal{C}} [\boldsymbol{v}]_{\mathcal{C}}\]

Moreover:

1. \([\mathbf{P}]_{\mathcal{B}}^{\mathcal{C}}\) is invertible

2. \(([\mathbf{P}]_{\mathcal{B}}^{\mathcal{C}})^{-1} = [\mathbf{P}]_{\mathcal{C}}^{\mathcal{B}}\)

3. \([\mathbf{P}]_{\mathcal{B}}^{\mathcal{B}} = \mathbf{I}\) (identity transformation)

Proof. Main formula: Let \(\boldsymbol{v} = \sum_{j=1}^n v_j^{(\mathcal{C})} \boldsymbol{c}_j\) where \([\boldsymbol{v}]_{\mathcal{C}} = (v_1^{(\mathcal{C})}, \ldots, v_n^{(\mathcal{C})})^T\).

Expressing each \(\boldsymbol{c}_j\) in terms of the \(\mathcal{B}\) basis: \[\boldsymbol{v} = \sum_{j=1}^n v_j^{(\mathcal{C})} \boldsymbol{c}_j = \sum_{j=1}^n v_j^{(\mathcal{C})} \left(\sum_{i=1}^n p_{ij} \boldsymbol{b}_i\right) = \sum_{i=1}^n \left(\sum_{j=1}^n p_{ij} v_j^{(\mathcal{C})}\right) \boldsymbol{b}_i\]

Therefore \([\boldsymbol{v}]_{\mathcal{B}} = [\mathbf{P}]_{\mathcal{B}}^{\mathcal{C}} [\boldsymbol{v}]_{\mathcal{C}}\).

Invertibility: Since both \(\mathcal{B}\) and \(\mathcal{C}\) are bases, the coordinate mappings are bijective. The change of basis matrix, being the composition of bijective linear maps, is also bijective and hence invertible.

Inverse relationship: Applying the change of basis formula twice: \[[\boldsymbol{v}]_{\mathcal{C}} = [\mathbf{P}]_{\mathcal{C}}^{\mathcal{B}} [\boldsymbol{v}]_{\mathcal{B}} = [\mathbf{P}]_{\mathcal{C}}^{\mathcal{B}} [\mathbf{P}]_{\mathcal{B}}^{\mathcal{C}} [\boldsymbol{v}]_{\mathcal{C}}\]

Since this holds for all \(\boldsymbol{v}\), we have \([\mathbf{P}]_{\mathcal{C}}^{\mathcal{B}} [\mathbf{P}]_{\mathcal{B}}^{\mathcal{C}} = \mathbf{I}\). ◻

Example 9.11 (Computing Change of Basis Matrices). In \(\mathbb{R}^2\), let \(\mathcal{B} = \{(1,0), (0,1)\}\) (standard basis) and \(\mathcal{C} = \{(1,1), (1,-1)\}\).

Find \([\mathbf{P}]_{\mathcal{B}}^{\mathcal{C}}\): We need to express each vector in \(\mathcal{C}\) as coordinates relative to \(\mathcal{B}\): \[\begin{align} (1,1) &= 1 \cdot (1,0) + 1 \cdot (0,1) \Rightarrow [(1,1)]_{\mathcal{B}} = \begin{pmatrix} 1 \\ 1 \end{pmatrix} \\ (1,-1) &= 1 \cdot (1,0) + (-1) \cdot (0,1) \Rightarrow [(1,-1)]_{\mathcal{B}} = \begin{pmatrix} 1 \\ -1 \end{pmatrix} \end{align}\]

Therefore: \[[\mathbf{P}]_{\mathcal{B}}^{\mathcal{C}} = \begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix}\]

Find \([\mathbf{P}]_{\mathcal{C}}^{\mathcal{B}}\): We need to express each vector in \(\mathcal{B}\) as coordinates relative to \(\mathcal{C}\):

For \((1,0) = a(1,1) + b(1,-1) = (a+b, a-b)\): \(a + b = 1\) and \(a - b = 0\), so \(a = 1/2, b = 1/2\). Thus \([(1,0)]_{\mathcal{C}} = (1/2, 1/2)^T\).

For \((0,1) = c(1,1) + d(1,-1) = (c+d, c-d)\): \(c + d = 0\) and \(c - d = 1\), so \(c = 1/2, d = -1/2\). Thus \([(0,1)]_{\mathcal{C}} = (1/2, -1/2)^T\).

Therefore: \[[\mathbf{P}]_{\mathcal{C}}^{\mathcal{B}} = \begin{pmatrix} 1/2 & 1/2 \\ 1/2 & -1/2 \end{pmatrix}\]

Verification: \[[\mathbf{P}]_{\mathcal{B}}^{\mathcal{C}} [\mathbf{P}]_{\mathcal{C}}^{\mathcal{B}} = \begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix}\begin{pmatrix} 1/2 & 1/2 \\ 1/2 & -1/2 \end{pmatrix} = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix} = \mathbf{I}\]

Example application: Convert \(\boldsymbol{v} = (3, 1)\) from \(\mathcal{B}\)-coordinates to \(\mathcal{C}\)-coordinates: \[[\boldsymbol{v}]_{\mathcal{C}} = [\mathbf{P}]_{\mathcal{C}}^{\mathcal{B}} [\boldsymbol{v}]_{\mathcal{B}} = \begin{pmatrix} 1/2 & 1/2 \\ 1/2 & -1/2 \end{pmatrix}\begin{pmatrix} 3 \\ 1 \end{pmatrix} = \begin{pmatrix} 2 \\ 1 \end{pmatrix}\]

Verification: \(2(1,1) + 1(1,-1) = (2,2) + (1,-1) = (3,1)\)

Theorem 9.12 (Composition of Changes of Basis). If \(\mathcal{B}\), \(\mathcal{C}\), and \(\mathcal{D}\) are three bases for vector space \(V\), then: \[[\mathbf{P}]_{\mathcal{B}}^{\mathcal{D}} = [\mathbf{P}]_{\mathcal{B}}^{\mathcal{C}} [\mathbf{P}]_{\mathcal{C}}^{\mathcal{D}}\]

In other words, change of basis matrices compose in the natural way.

Proof. For any vector \(\boldsymbol{v} \in V\): \[[\boldsymbol{v}]_{\mathcal{B}} = [\mathbf{P}]_{\mathcal{B}}^{\mathcal{C}} [\boldsymbol{v}]_{\mathcal{C}} = [\mathbf{P}]_{\mathcal{B}}^{\mathcal{C}} ([\mathbf{P}]_{\mathcal{C}}^{\mathcal{D}} [\boldsymbol{v}]_{\mathcal{D}}) = ([\mathbf{P}]_{\mathcal{B}}^{\mathcal{C}} [\mathbf{P}]_{\mathcal{C}}^{\mathcal{D}}) [\boldsymbol{v}]_{\mathcal{D}}\]

Since this holds for all \(\boldsymbol{v}\), the matrices must be equal. ◻

9.3 Change of Basis for Linear Transformations

Understanding how matrix representations change when we choose different bases is crucial for finding optimal representations of linear transformations.

Theorem 9.13 (Change of Basis for Linear Transformations). Let \(T: V \to V\) be a linear transformation, and let \(\mathcal{B}\) and \(\mathcal{C}\) be two bases for \(V\). If \([\mathbf{P}]_{\mathcal{B}}^{\mathcal{C}}\) is the change of basis matrix from \(\mathcal{C}\) to \(\mathcal{B}\), then: \[[\mathbf{T}]_{\mathcal{C}} = ([\mathbf{P}]_{\mathcal{B}}^{\mathcal{C}})^{-1} [\mathbf{T}]_{\mathcal{B}} [\mathbf{P}]_{\mathcal{B}}^{\mathcal{C}}\]

In other words, the matrix representations are similar.

Proof. For any vector \(\boldsymbol{v} \in V\), we have the following commutative diagram of relationships:

Following the path through \(\mathcal{B}\)-coordinates: \[[T(\boldsymbol{v})]_{\mathcal{C}} = ([\mathbf{P}]_{\mathcal{B}}^{\mathcal{C}})^{-1} [T(\boldsymbol{v})]_{\mathcal{B}} = ([\mathbf{P}]_{\mathcal{B}}^{\mathcal{C}})^{-1} [\mathbf{T}]_{\mathcal{B}} [\boldsymbol{v}]_{\mathcal{B}}\] \[= ([\mathbf{P}]_{\mathcal{B}}^{\mathcal{C}})^{-1} [\mathbf{T}]_{\mathcal{B}} [\mathbf{P}]_{\mathcal{B}}^{\mathcal{C}} [\boldsymbol{v}]_{\mathcal{C}}\]

Since this holds for all \(\boldsymbol{v}\), we have the desired relationship. ◻

Definition 9.14 (Similar Matrices). Two \(n \times n\) matrices \(\mathbf{A}\) and \(\mathbf{B}\) are similar if there exists an invertible matrix \(\mathbf{P}\) such that: \[\mathbf{B} = \mathbf{P}^{-1}\mathbf{A}\mathbf{P}\]

We write \(\mathbf{A} \sim \mathbf{B}\) to denote similarity.

Theorem 9.15 (Properties of Similar Matrices). If matrices \(\mathbf{A}\) and \(\mathbf{B}\) are similar, then they have:

1. The same characteristic polynomial: \(\det(\mathbf{A} - \lambda\mathbf{I}) = \det(\mathbf{B} - \lambda\mathbf{I})\)

2. The same eigenvalues (counting multiplicities)

3. The same trace: \(\operatorname{tr}(\mathbf{A}) = \operatorname{tr}(\mathbf{B})\)

4. The same determinant: \(\det(\mathbf{A}) = \det(\mathbf{B})\)

5. The same rank: \(\operatorname{rank}(\mathbf{A}) = \operatorname{rank}(\mathbf{B})\)

Proof. Let \(\mathbf{B} = \mathbf{P}^{-1}\mathbf{A}\mathbf{P}\).

Characteristic polynomial: \[\det(\mathbf{B} - \lambda\mathbf{I}) = \det(\mathbf{P}^{-1}\mathbf{A}\mathbf{P} - \lambda\mathbf{I}) = \det(\mathbf{P}^{-1}(\mathbf{A} - \lambda\mathbf{I})\mathbf{P})\] \[= \det(\mathbf{P}^{-1})\det(\mathbf{A} - \lambda\mathbf{I})\det(\mathbf{P}) = \det(\mathbf{A} - \lambda\mathbf{I})\]

The other properties follow from the characteristic polynomial or direct computation. ◻

Example 9.16 (Finding Optimal Representations). Consider the linear transformation \(T: \mathbb{R}^2 \to \mathbb{R}^2\) that reflects vectors across the line \(y = x\).

Standard basis representation: \(T(1,0) = (0,1)\) and \(T(0,1) = (1,0)\), so: \[[\mathbf{T}]_{\text{std}} = \begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix}\]

Finding a better basis: The line \(y = x\) corresponds to the vector \((1,1)\), and the line \(y = -x\) (perpendicular to it) corresponds to \((1,-1)\). Vectors along these lines should be eigenspaces.

Check: \(T(1,1) = (1,1)\) and \(T(1,-1) = (-1,1) = -(1,-1)\)

So \(\lambda_1 = 1\) with eigenvector \((1,1)\) and \(\lambda_2 = -1\) with eigenvector \((1,-1)\).

Diagonal representation: Using basis \(\mathcal{C} = \{(1,1), (1,-1)\}\): \[[\mathbf{T}]_{\mathcal{C}} = \begin{pmatrix} 1 & 0 \\ 0 & -1 \end{pmatrix}\]

Verification via change of basis: \[[\mathbf{P}]_{\text{std}}^{\mathcal{C}} = \begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix}, \quad [\mathbf{P}]_{\mathcal{C}}^{\text{std}} = \frac{1}{2}\begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix}\]

\[[\mathbf{P}]_{\mathcal{C}}^{\text{std}} [\mathbf{T}]_{\text{std}} [\mathbf{P}]_{\text{std}}^{\mathcal{C}} = \frac{1}{2}\begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix}\begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix}\begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix}\]

\[= \frac{1}{2}\begin{pmatrix} 1 & 1 \\ -1 & 1 \end{pmatrix}\begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix} = \frac{1}{2}\begin{pmatrix} 2 & 0 \\ 0 & -2 \end{pmatrix} = \begin{pmatrix} 1 & 0 \\ 0 & -1 \end{pmatrix}\]

The diagonal form makes the geometric meaning of the transformation crystal clear: it preserves vectors along \(y = x\) and reverses vectors along \(y = -x\).

9.4 Applications of Change of Basis

Change of basis is not merely a theoretical tool—it has profound practical applications in solving differential equations, optimization problems, and data analysis.

9.4.1 Diagonalization and Systems of Differential Equations

Example 9.17 (System of Linear ODEs). Solve the system of differential equations: \[\frac{d}{dt}\begin{pmatrix} x \\ y \end{pmatrix} = \begin{pmatrix} 1 & 2 \\ 2 & 1 \end{pmatrix}\begin{pmatrix} x \\ y \end{pmatrix}\]

Step 1: Diagonalize the coefficient matrix \[\mathbf{A} = \begin{pmatrix} 1 & 2 \\ 2 & 1 \end{pmatrix}\]

Eigenvalues: \(\det(\mathbf{A} - \lambda\mathbf{I}) = (1-\lambda)^2 - 4 = \lambda^2 - 2\lambda - 3 = (\lambda-3)(\lambda+1)\)

So \(\lambda_1 = 3, \lambda_2 = -1\).

Eigenvectors: - For \(\lambda_1 = 3\): \((\mathbf{A} - 3\mathbf{I})\boldsymbol{v} = \begin{pmatrix} -2 & 2 \\ 2 & -2 \end{pmatrix}\boldsymbol{v} = \boldsymbol{0}\) gives \(\boldsymbol{v}_1 = \begin{pmatrix} 1 \\ 1 \end{pmatrix}\) - For \(\lambda_2 = -1\): \((\mathbf{A} + \mathbf{I})\boldsymbol{v} = \begin{pmatrix} 2 & 2 \\ 2 & 2 \end{pmatrix}\boldsymbol{v} = \boldsymbol{0}\) gives \(\boldsymbol{v}_2 = \begin{pmatrix} 1 \\ -1 \end{pmatrix}\)

Step 2: Change to eigencoordinates Let \(\mathbf{P} = \begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix}\) and \(\begin{pmatrix} u \\ v \end{pmatrix} = \mathbf{P}^{-1}\begin{pmatrix} x \\ y \end{pmatrix}\).

Then \(\begin{pmatrix} x \\ y \end{pmatrix} = \mathbf{P}\begin{pmatrix} u \\ v \end{pmatrix}\), and the system becomes: \[\frac{d}{dt}\begin{pmatrix} u \\ v \end{pmatrix} = \mathbf{P}^{-1}\mathbf{A}\mathbf{P}\begin{pmatrix} u \\ v \end{pmatrix} = \begin{pmatrix} 3 & 0 \\ 0 & -1 \end{pmatrix}\begin{pmatrix} u \\ v \end{pmatrix}\]

Step 3: Solve decoupled system \[\frac{du}{dt} = 3u \Rightarrow u(t) = c_1 e^{3t}\] \[\frac{dv}{dt} = -v \Rightarrow v(t) = c_2 e^{-t}\]

Step 4: Transform back \[\begin{pmatrix} x(t) \\ y(t) \end{pmatrix} = \mathbf{P}\begin{pmatrix} u(t) \\ v(t) \end{pmatrix} = \begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix}\begin{pmatrix} c_1 e^{3t} \\ c_2 e^{-t} \end{pmatrix} = \begin{pmatrix} c_1 e^{3t} + c_2 e^{-t} \\ c_1 e^{3t} - c_2 e^{-t} \end{pmatrix}\]

The change of basis has converted a coupled system into two independent scalar equations!

9.4.2 Principal Component Analysis Revisited

Example 9.18 (PCA via Change of Basis). Given data with covariance matrix \(\mathbf{C} = \begin{pmatrix} 5 & 3 \\ 3 & 5 \end{pmatrix}\), find the principal components.

Step 1: Find eigenvalues and eigenvectors of \(\mathbf{C}\) \(\det(\mathbf{C} - \lambda\mathbf{I}) = (5-\lambda)^2 - 9 = \lambda^2 - 10\lambda + 16 = (\lambda-8)(\lambda-2)\)

Eigenvalues: \(\lambda_1 = 8, \lambda_2 = 2\)

Eigenvectors: \(\boldsymbol{v}_1 = \frac{1}{\sqrt{2}}\begin{pmatrix} 1 \\ 1 \end{pmatrix}, \boldsymbol{v}_2 = \frac{1}{\sqrt{2}}\begin{pmatrix} 1 \\ -1 \end{pmatrix}\)

Step 2: Interpret geometrically - First principal component: direction \((1,1)\) with variance 8 - Second principal component: direction \((1,-1)\) with variance 2 - The first PC explains \(\frac{8}{8+2} = 80\%\) of total variance

Step 3: Change of basis In the principal component coordinate system \(\mathcal{P} = \{\boldsymbol{v}_1, \boldsymbol{v}_2\}\), the covariance matrix becomes: \[[\mathbf{C}]_{\mathcal{P}} = \begin{pmatrix} 8 & 0 \\ 0 & 2 \end{pmatrix}\]

This diagonal form immediately reveals the variance along each principal direction and shows that the components are uncorrelated.

9.4.3 Quadratic Forms and Optimization

Example 9.19 (Quadratic Form Analysis). Analyze the quadratic form \(Q(x,y) = 5x^2 + 6xy + 5y^2\).

Step 1: Write in matrix form \[Q(\boldsymbol{x}) = \boldsymbol{x}^T \begin{pmatrix} 5 & 3 \\ 3 & 5 \end{pmatrix} \boldsymbol{x}\]

Step 2: Diagonalize From the previous example, we know the eigenvalues are 8 and 2, with eigenvectors \((1,1)^T/\sqrt{2}\) and \((1,-1)^T/\sqrt{2}\).

Step 3: Change variables Let \(\begin{pmatrix} u \\ v \end{pmatrix} = \frac{1}{\sqrt{2}}\begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix}\begin{pmatrix} x \\ y \end{pmatrix}\)

Then: \(u = \frac{x+y}{\sqrt{2}}, v = \frac{x-y}{\sqrt{2}}\)

Step 4: Simplified form \[Q(u,v) = 8u^2 + 2v^2\]

Geometric interpretation: - The level curves \(Q(\boldsymbol{x}) = c\) are ellipses - The principal axes are along the directions \((1,1)\) and \((1,-1)\) - The ellipse is stretched more in the \((1,1)\) direction (eigenvalue 8) than in the \((1,-1)\) direction (eigenvalue 2)

10 Inner Product Spaces

Inner product spaces generalize the familiar concepts of length, angle, and orthogonality from Euclidean space to abstract vector spaces. This abstraction reveals the geometric structure underlying many mathematical phenomena, from function spaces in analysis to quantum mechanical state spaces in physics. By developing the theory of general inner products, we gain powerful tools for approximation, optimization, and understanding infinite-dimensional spaces.

10.1 Definition and Examples

Definition 10.1 (Inner Product). Let \(V\) be a vector space over \(\mathbb{R}\). An inner product on \(V\) is a function \(\langle \cdot, \cdot \rangle: V \times V \to \mathbb{R}\) satisfying:

1. Symmetry: \(\langle \boldsymbol{u}, \boldsymbol{v} \rangle = \langle \boldsymbol{v}, \boldsymbol{u} \rangle\) for all \(\boldsymbol{u}, \boldsymbol{v} \in V\)

2. Linearity in first argument: \(\langle a\boldsymbol{u} + b\boldsymbol{v}, \boldsymbol{w} \rangle = a\langle \boldsymbol{u}, \boldsymbol{w} \rangle + b\langle \boldsymbol{v}, \boldsymbol{w} \rangle\)

3. Positive definiteness: \(\langle \boldsymbol{v}, \boldsymbol{v} \rangle \geq 0\) for all \(\boldsymbol{v} \in V\), with equality if and only if \(\boldsymbol{v} = \boldsymbol{0}\)

A vector space equipped with an inner product is called an inner product space.

Remark 483. Properties (1) and (2) together imply that the inner product is also linear in the second argument (bilinearity). The inner product generalizes the dot product from \(\mathbb{R}^n\), capturing the essential properties needed to define geometric concepts like length and angle in abstract spaces.

Example 10.3 (Standard Inner Products). 1. Euclidean inner product on \(\mathbb{R}^n\): \[\langle \boldsymbol{u}, \boldsymbol{v} \rangle = \boldsymbol{u} \cdot \boldsymbol{v} = \sum_{i=1}^n u_i v_i\]

2. Weighted inner product on \(\mathbb{R}^n\): For positive weights \(w_1, w_2, \ldots, w_n\): \[\langle \boldsymbol{u}, \boldsymbol{v} \rangle_w = \sum_{i=1}^n w_i u_i v_i\]

Verification: We check positive definiteness: \[\langle \boldsymbol{v}, \boldsymbol{v} \rangle_w = \sum_{i=1}^n w_i v_i^2 \geq 0\] with equality iff all \(v_i = 0\) (since \(w_i > 0\)), i.e., iff \(\boldsymbol{v} = \boldsymbol{0}\).

3. Function inner product on \(C[a,b]\): For continuous functions on \([a,b]\): \[\langle f, g \rangle = \int_a^b f(x)g(x) \, dx\]

Verification: - Symmetry: \(\int_a^b f(x)g(x) \, dx = \int_a^b g(x)f(x) \, dx\) - Linearity: \(\int_a^b (af + bg)(x)h(x) \, dx = a\int_a^b f(x)h(x) \, dx + b\int_a^b g(x)h(x) \, dx\) - Positive definiteness: \(\langle f, f \rangle = \int_a^b f(x)^2 \, dx \geq 0\), with equality iff \(f(x) = 0\) for all \(x \in [a,b]\)

4. Matrix inner product (Frobenius): For \(m \times n\) matrices: \[\langle \mathbf{A}, \mathbf{B} \rangle = \operatorname{tr}(\mathbf{A}^T\mathbf{B}) = \sum_{i=1}^m \sum_{j=1}^n a_{ij}b_{ij}\]

5. Polynomial inner product: On \(\mathcal{P}_n\): \[\langle p, q \rangle = \int_{-1}^1 p(x)q(x) \, dx\]

or with weight function: \[\langle p, q \rangle_w = \int_{-1}^1 p(x)q(x)w(x) \, dx\] where \(w(x) > 0\) is a weight function.

Definition 10.4 (Induced Norm and Distance). If \(V\) is an inner product space, then:

• The norm of \(\boldsymbol{v}\) is \(\|\boldsymbol{v}\| = \sqrt{\langle \boldsymbol{v}, \boldsymbol{v} \rangle}\)

• The distance between \(\boldsymbol{u}\) and \(\boldsymbol{v}\) is \(d(\boldsymbol{u}, \boldsymbol{v}) = \|\boldsymbol{u} - \boldsymbol{v}\|\)

• Vectors \(\boldsymbol{u}\) and \(\boldsymbol{v}\) are orthogonal if \(\langle \boldsymbol{u}, \boldsymbol{v} \rangle = 0\)

Theorem 10.5 (Fundamental Inequalities). In any inner product space:

1. Cauchy-Schwarz Inequality: \[|\langle \boldsymbol{u}, \boldsymbol{v} \rangle| \leq \|\boldsymbol{u}\| \|\boldsymbol{v}\|\] with equality if and only if \(\boldsymbol{u}\) and \(\boldsymbol{v}\) are linearly dependent.

2. Triangle Inequality: \[\|\boldsymbol{u} + \boldsymbol{v}\| \leq \|\boldsymbol{u}\| + \|\boldsymbol{v}\|\]

3. Parallelogram Law: \[\|\boldsymbol{u} + \boldsymbol{v}\|^2 + \|\boldsymbol{u} - \boldsymbol{v}\|^2 = 2(\|\boldsymbol{u}\|^2 + \|\boldsymbol{v}\|^2)\]

Proof. Cauchy-Schwarz: For any \(t \in \mathbb{R}\), consider: \[0 \leq \langle \boldsymbol{u} - t\boldsymbol{v}, \boldsymbol{u} - t\boldsymbol{v} \rangle = \langle \boldsymbol{u}, \boldsymbol{u} \rangle - 2t\langle \boldsymbol{u}, \boldsymbol{v} \rangle + t^2\langle \boldsymbol{v}, \boldsymbol{v} \rangle\]

If \(\boldsymbol{v} \neq \boldsymbol{0}\), choose \(t = \frac{\langle \boldsymbol{u}, \boldsymbol{v} \rangle}{\langle \boldsymbol{v}, \boldsymbol{v} \rangle}\): \[0 \leq \|\boldsymbol{u}\|^2 - 2\frac{(\langle \boldsymbol{u}, \boldsymbol{v} \rangle)^2}{\|\boldsymbol{v}\|^2} + \frac{(\langle \boldsymbol{u}, \boldsymbol{v} \rangle)^2}{\|\boldsymbol{v}\|^2} = \|\boldsymbol{u}\|^2 - \frac{(\langle \boldsymbol{u}, \boldsymbol{v} \rangle)^2}{\|\boldsymbol{v}\|^2}\]

Rearranging: \((\langle \boldsymbol{u}, \boldsymbol{v} \rangle)^2 \leq \|\boldsymbol{u}\|^2 \|\boldsymbol{v}\|^2\).

Triangle Inequality: \[\|\boldsymbol{u} + \boldsymbol{v}\|^2 = \langle \boldsymbol{u} + \boldsymbol{v}, \boldsymbol{u} + \boldsymbol{v} \rangle = \|\boldsymbol{u}\|^2 + 2\langle \boldsymbol{u}, \boldsymbol{v} \rangle + \|\boldsymbol{v}\|^2\] \[\leq \|\boldsymbol{u}\|^2 + 2|\langle \boldsymbol{u}, \boldsymbol{v} \rangle| + \|\boldsymbol{v}\|^2 \leq \|\boldsymbol{u}\|^2 + 2\|\boldsymbol{u}\|\|\boldsymbol{v}\| + \|\boldsymbol{v}\|^2 = (\|\boldsymbol{u}\| + \|\boldsymbol{v}\|)^2\]

Parallelogram Law: \[\|\boldsymbol{u} + \boldsymbol{v}\|^2 = \|\boldsymbol{u}\|^2 + 2\langle \boldsymbol{u}, \boldsymbol{v} \rangle + \|\boldsymbol{v}\|^2\] \[\|\boldsymbol{u} - \boldsymbol{v}\|^2 = \|\boldsymbol{u}\|^2 - 2\langle \boldsymbol{u}, \boldsymbol{v} \rangle + \|\boldsymbol{v}\|^2\] Adding these equations gives the result. ◻

Example 10.6 (Applying Cauchy-Schwarz to Functions). For functions \(f, g \in C[0,1]\) with inner product \(\langle f, g \rangle = \int_0^1 f(x)g(x) \, dx\):

\[\left|\int_0^1 f(x)g(x) \, dx\right| \leq \sqrt{\int_0^1 f(x)^2 \, dx} \sqrt{\int_0^1 g(x)^2 \, dx}\]

Specific example: Let \(f(x) = x\) and \(g(x) = x^2\). \[\int_0^1 x \cdot x^2 \, dx = \int_0^1 x^3 \, dx = \frac{1}{4}\] \[\sqrt{\int_0^1 x^2 \, dx} = \sqrt{\frac{1}{3}}, \quad \sqrt{\int_0^1 x^4 \, dx} = \sqrt{\frac{1}{5}}\]

Cauchy-Schwarz gives: \(\frac{1}{4} \leq \sqrt{\frac{1}{3}} \sqrt{\frac{1}{5}} = \sqrt{\frac{1}{15}} \approx 0.258\)

Indeed: \(0.25 < 0.258\)

10.2 Orthogonality in Inner Product Spaces

Definition 10.7 (Orthogonal and Orthonormal Sets). Let \(V\) be an inner product space.

• Vectors \(\boldsymbol{u}\) and \(\boldsymbol{v}\) are orthogonal if \(\langle \boldsymbol{u}, \boldsymbol{v} \rangle = 0\). We write \(\boldsymbol{u} \perp \boldsymbol{v}\).

• A set \(\{\boldsymbol{v}_1, \boldsymbol{v}_2, \ldots, \boldsymbol{v}_k\}\) is orthogonal if \(\langle \boldsymbol{v}_i, \boldsymbol{v}_j \rangle = 0\) for all \(i \neq j\).

• An orthogonal set is orthonormal if additionally \(\|\boldsymbol{v}_i\| = 1\) for all \(i\).

Theorem 10.8 (Orthogonal Sets are Linearly Independent). Any orthogonal set of non-zero vectors in an inner product space is linearly independent.

Proof. Let \(\{\boldsymbol{v}_1, \ldots, \boldsymbol{v}_k\}\) be orthogonal with each \(\boldsymbol{v}_i \neq \boldsymbol{0}\). Suppose: \[c_1\boldsymbol{v}_1 + c_2\boldsymbol{v}_2 + \cdots + c_k\boldsymbol{v}_k = \boldsymbol{0}\]

Take the inner product with \(\boldsymbol{v}_j\): \[\langle c_1\boldsymbol{v}_1 + \cdots + c_k\boldsymbol{v}_k, \boldsymbol{v}_j \rangle = \langle \boldsymbol{0}, \boldsymbol{v}_j \rangle = 0\]

By linearity and orthogonality: \[c_1\langle \boldsymbol{v}_1, \boldsymbol{v}_j \rangle + \cdots + c_j\langle \boldsymbol{v}_j, \boldsymbol{v}_j \rangle + \cdots + c_k\langle \boldsymbol{v}_k, \boldsymbol{v}_j \rangle = c_j\|\boldsymbol{v}_j\|^2 = 0\]

Since \(\boldsymbol{v}_j \neq \boldsymbol{0}\), we have \(\|\boldsymbol{v}_j\|^2 > 0\), so \(c_j = 0\). This holds for all \(j\). ◻

Theorem 10.9 (Pythagorean Theorem). If \(\boldsymbol{u}_1, \boldsymbol{u}_2, \ldots, \boldsymbol{u}_k\) are mutually orthogonal, then: \[\left\|\sum_{i=1}^k \boldsymbol{u}_i\right\|^2 = \sum_{i=1}^k \|\boldsymbol{u}_i\|^2\]

Proof. \[\left\|\sum_{i=1}^k \boldsymbol{u}_i\right\|^2 = \left\langle \sum_{i=1}^k \boldsymbol{u}_i, \sum_{j=1}^k \boldsymbol{u}_j \right\rangle = \sum_{i=1}^k \sum_{j=1}^k \langle \boldsymbol{u}_i, \boldsymbol{u}_j \rangle\]

By orthogonality, \(\langle \boldsymbol{u}_i, \boldsymbol{u}_j \rangle = 0\) when \(i \neq j\), so only the diagonal terms survive: \[= \sum_{i=1}^k \langle \boldsymbol{u}_i, \boldsymbol{u}_i \rangle = \sum_{i=1}^k \|\boldsymbol{u}_i\|^2\] ◻

10.3 Orthogonal Projections and Gram-Schmidt Process

Definition 10.10 (Orthogonal Projection). Let \(W\) be a finite-dimensional subspace of inner product space \(V\), and let \(\{\boldsymbol{u}_1, \boldsymbol{u}_2, \ldots, \boldsymbol{u}_k\}\) be an orthonormal basis for \(W\). The orthogonal projection of \(\boldsymbol{v} \in V\) onto \(W\) is: \[\text{proj}_W(\boldsymbol{v}) = \sum_{i=1}^k \langle \boldsymbol{v}, \boldsymbol{u}_i \rangle \boldsymbol{u}_i\]

Theorem 10.11 (Best Approximation Property). For any \(\boldsymbol{v} \in V\) and finite-dimensional subspace \(W\), the orthogonal projection \(\text{proj}_W(\boldsymbol{v})\) is the unique vector in \(W\) that minimizes \(\|\boldsymbol{v} - \boldsymbol{w}\|\) for \(\boldsymbol{w} \in W\).

Proof. Let \(\hat{\boldsymbol{v}} = \text{proj}_W(\boldsymbol{v})\) and \(\boldsymbol{w} \in W\) with \(\boldsymbol{w} \neq \hat{\boldsymbol{v}}\). We can write: \[\boldsymbol{v} - \boldsymbol{w} = (\boldsymbol{v} - \hat{\boldsymbol{v}}) + (\hat{\boldsymbol{v}} - \boldsymbol{w})\]

Since \(\boldsymbol{v} - \hat{\boldsymbol{v}} \perp W\) and \(\hat{\boldsymbol{v}} - \boldsymbol{w} \in W\), these vectors are orthogonal. By the Pythagorean theorem: \[\|\boldsymbol{v} - \boldsymbol{w}\|^2 = \|\boldsymbol{v} - \hat{\boldsymbol{v}}\|^2 + \|\hat{\boldsymbol{v}} - \boldsymbol{w}\|^2\]

Since \(\|\hat{\boldsymbol{v}} - \boldsymbol{w}\|^2 > 0\) when \(\boldsymbol{w} \neq \hat{\boldsymbol{v}}\), we have \(\|\boldsymbol{v} - \boldsymbol{w}\| > \|\boldsymbol{v} - \hat{\boldsymbol{v}}\|\). ◻

Example 10.12 (Gram-Schmidt in Function Space). Apply Gram-Schmidt to \(\{1, x, x^2\}\) in \(C[-1,1]\) with inner product \(\langle f, g \rangle = \int_{-1}^1 f(x)g(x) \, dx\). Step 1: \(v_1(x) = 1\) \[\|v_1\|^2 = \int_{-1}^1 1^2 \, dx = 2, \quad \text{so } u_1(x) = \frac{1}{\sqrt{2}}\] Step 2: \(v_2(x) = x\) \[\langle v_2, u_1 \rangle = \int_{-1}^1 x \cdot \frac{1}{\sqrt{2}} \, dx = \frac{1}{\sqrt{2}} \int_{-1}^1 x \, dx = 0\] Since \(v_2 \perp u_1\), we have \(w_2 = v_2 = x\). \[\|w_2\|^2 = \int_{-1}^1 x^2 \, dx = \frac{2}{3}\] Therefore: \(u_2(x) = \frac{x}{\sqrt{2/3}} = x \sqrt{\frac{3}{2}} = \frac{\sqrt{6}}{2} x\) Step 3: \(v_3(x) = x^2\) \[\langle v_3, u_1 \rangle = \int_{-1}^1 x^2 \cdot \frac{1}{\sqrt{2}} \, dx = \frac{1}{\sqrt{2}} \int_{-1}^1 x^2 \, dx = \frac{1}{\sqrt{2}} \cdot \frac{2}{3} = \frac{\sqrt{2}}{3}\] \[\langle v_3, u_2 \rangle = \int_{-1}^1 x^2 \cdot \frac{\sqrt{6}}{2} x \, dx = \frac{\sqrt{6}}{2} \int_{-1}^1 x^3 \, dx = 0\] \[w_3(x) = x^2 - \frac{\sqrt{2}}{3} \cdot \frac{1}{\sqrt{2}} = x^2 - \frac{1}{3}\] \[\|w_3\|^2 = \int_{-1}^1 \left(x^2 - \frac{1}{3}\right)^2 \, dx = \int_{-1}^1 \left(x^4 - \frac{2x^2}{3} + \frac{1}{9}\right) \, dx\] \[= \frac{2}{5} - \frac{2}{3} \cdot \frac{2}{3} + \frac{2}{9} = \frac{2}{5} - \frac{4}{9} + \frac{2}{9} = \frac{2}{5} - \frac{2}{9} = \frac{18-10}{45} = \frac{8}{45}\] Therefore: \(u_3(x) = \sqrt{\frac{45}{8}} \left(x^2 - \frac{1}{3}\right) = \frac{3\sqrt{10}}{4} \left(x^2 - \frac{1}{3}\right)\) Result: The orthonormal polynomials are: \[u_1(x) = \frac{1}{\sqrt{2}}, \quad u_2(x) = \frac{\sqrt{6}}{2} x, \quad u_3(x) = \frac{3\sqrt{10}}{4} \left(x^2 - \frac{1}{3}\right)\] These are (scaled versions of) the first three Legendre polynomials!

10.4 Orthogonal Complements and Direct Sums

Definition 10.13 (Orthogonal Complement). Let \(W\) be a subset of inner product space \(V\). The orthogonal complement of \(W\) is: \[W^{\perp} = \{\boldsymbol{v} \in V : \langle \boldsymbol{v}, \boldsymbol{w} \rangle = 0 \text{ for all } \boldsymbol{w} \in W\}\]

Theorem 10.14 (Properties of Orthogonal Complements). Let \(W\) be a subspace of finite-dimensional inner product space \(V\). Then:

1. \(W^{\perp}\) is a subspace of \(V\)

2. \(W \cap W^{\perp} = \{\boldsymbol{0}\}\)

3. \((W^{\perp})^{\perp} = W\)

4. \(V = W \oplus W^{\perp}\) (orthogonal direct sum)

5. \(\dim(W) + \dim(W^{\perp}) = \dim(V)\)

Proof. Property 1: Standard verification of subspace conditions.

Property 4: For any \(\boldsymbol{v} \in V\), let \(\boldsymbol{w} = \text{proj}_W(\boldsymbol{v})\). Then: \[\boldsymbol{v} = \boldsymbol{w} + (\boldsymbol{v} - \boldsymbol{w})\]

where \(\boldsymbol{w} \in W\) and \(\boldsymbol{v} - \boldsymbol{w} \in W^{\perp}\) by the definition of orthogonal projection.

For uniqueness: if \(\boldsymbol{v} = \boldsymbol{w}_1 + \boldsymbol{w}_1^{\perp} = \boldsymbol{w}_2 + \boldsymbol{w}_2^{\perp}\) with \(\boldsymbol{w}_i \in W\) and \(\boldsymbol{w}_i^{\perp} \in W^{\perp}\), then: \[(\boldsymbol{w}_1 - \boldsymbol{w}_2) = (\boldsymbol{w}_2^{\perp} - \boldsymbol{w}_1^{\perp})\]

The left side is in \(W\), the right side is in \(W^{\perp}\), so both equal \(\boldsymbol{0}\).

The remaining properties follow from the orthogonal direct sum decomposition. ◻

Example 10.15 (Orthogonal Complement in Function Space). In \(C[0, 2\pi]\) with inner product \(\langle f, g \rangle = \int_0^{2\pi} f(x)g(x) \, dx\), let: \[W = \text{span}\{\sin x, \cos x\}\]

Find \(W^{\perp}\) within the space of trigonometric polynomials.

Solution: A function \(f\) is in \(W^{\perp}\) if: \[\int_0^{2\pi} f(x) \sin x \, dx = 0 \quad \text{and} \quad \int_0^{2\pi} f(x) \cos x \, dx = 0\]

These are the orthogonality conditions for the first Fourier coefficients! Functions satisfying these conditions include:

• Constants (except the constant function, which isn’t orthogonal to \(\cos x\))

• \(\sin(2x), \cos(2x), \sin(3x), \cos(3x), \ldots\)

• Any trigonometric polynomial with no first harmonic terms

More precisely, within the space of trigonometric polynomials of degree \(\leq n\): \[W^{\perp} = \text{span}\{1, \sin(2x), \cos(2x), \sin(3x), \cos(3x), \ldots, \sin(nx), \cos(nx)\}\]

10.5 Applications to Approximation Theory

Inner product spaces provide the natural framework for solving approximation problems, from least squares fitting to Fourier analysis.

10.5.1 Least Squares Approximation

Theorem 10.16 (Least Squares in Inner Product Spaces). Let \(W\) be a finite-dimensional subspace of inner product space \(V\), and let \(\{\boldsymbol{w}_1, \boldsymbol{w}_2, \ldots, \boldsymbol{w}_k\}\) be a basis for \(W\). For any \(\boldsymbol{v} \in V\), the least squares approximation is: \[\boldsymbol{v}^* = \sum_{i=1}^k c_i \boldsymbol{w}_i\] where the coefficients satisfy the normal equations: \[\sum_{j=1}^k \langle \boldsymbol{w}_i, \boldsymbol{w}_j \rangle c_j = \langle \boldsymbol{v}, \boldsymbol{w}_i \rangle, \quad i = 1, 2, \ldots, k\]

In matrix form: \(\mathbf{G}\boldsymbol{c} = \boldsymbol{b}\) where \(G_{ij} = \langle \boldsymbol{w}_i, \boldsymbol{w}_j \rangle\) and \(b_i = \langle \boldsymbol{v}, \boldsymbol{w}_i \rangle\).

Remark 498. The matrix \(\mathbf{G}\) is called the Gram matrix of the basis. It’s positive definite when the basis vectors are linearly independent, ensuring a unique solution to the normal equations.

Example 10.18 (Polynomial Least Squares). Find the best quadratic approximation to \(f(x) = e^x\) on \([-1, 1]\) using the inner product \(\langle f, g \rangle = \int_{-1}^1 f(x)g(x) \, dx\).

Step 1: Set up the problem We want \(p(x) = c_0 + c_1 x + c_2 x^2\) minimizing \(\|f - p\|^2\).

Step 2: Compute the Gram matrix With basis \(\{1, x, x^2\}\): \[G_{11} = \langle 1, 1 \rangle = \int_{-1}^1 1 \, dx = 2\] \[G_{12} = G_{21} = \langle 1, x \rangle = \int_{-1}^1 x \, dx = 0\] \[G_{13} = G_{31} = \langle 1, x^2 \rangle = \int_{-1}^1 x^2 \, dx = \frac{2}{3}\] \[G_{22} = \langle x, x \rangle = \int_{-1}^1 x^2 \, dx = \frac{2}{3}\] \[G_{23} = G_{32} = \langle x, x^2 \rangle = \int_{-1}^1 x^3 \, dx = 0\] \[G_{33} = \langle x^2, x^2 \rangle = \int_{-1}^1 x^4 \, dx = \frac{2}{5}\]

\[\mathbf{G} = \begin{pmatrix} 2 & 0 & 2/3 \\ 0 & 2/3 & 0 \\ 2/3 & 0 & 2/5 \end{pmatrix}\]

Step 3: Compute the right-hand side \[b_1 = \langle e^x, 1 \rangle = \int_{-1}^1 e^x \, dx = e - e^{-1} = 2\sinh(1)\] \[b_2 = \langle e^x, x \rangle = \int_{-1}^1 xe^x \, dx = 2\sinh(1) - 2e^{-1} = 2(\sinh(1) - e^{-1})\] \[b_3 = \langle e^x, x^2 \rangle = \int_{-1}^1 x^2e^x \, dx = 2\sinh(1) - 4e^{-1}\]

Step 4: Solve the normal equations From the structure of \(\mathbf{G}\), we can solve: - \(c_1 = \frac{b_2}{2/3} = 3b_2\) - From \(2c_0 + \frac{2}{3}c_2 = b_1\) and \(\frac{2}{3}c_0 + \frac{2}{5}c_2 = b_3\)

This gives the best quadratic approximation to \(e^x\) on \([-1,1]\).

10.5.2 Fourier Series and Orthogonal Functions

Example 10.19 (Fourier Series as Orthogonal Projection). In \(L^2[0, 2\pi]\), the trigonometric functions \(\{1, \cos x, \sin x, \cos 2x, \sin 2x, \ldots\}\) form an orthogonal set under the inner product \(\langle f, g \rangle = \int_0^{2\pi} f(x)g(x) \, dx\).

Orthogonality relations: \[\int_0^{2\pi} \cos(mx) \cos(nx) \, dx = \begin{cases} \pi & \text{if } m = n \neq 0 \\ 2\pi & \text{if } m = n = 0 \\ 0 & \text{if } m \neq n \end{cases}\]

\[\int_0^{2\pi} \sin(mx) \sin(nx) \, dx = \begin{cases} \pi & \text{if } m = n \neq 0 \\ 0 & \text{if } m \neq n \text{ or } m = n = 0 \end{cases}\]

\[\int_0^{2\pi} \cos(mx) \sin(nx) \, dx = 0 \text{ for all } m, n\]

Fourier coefficients as projections: For a function \(f\), the Fourier series is: \[f(x) \sim \frac{a_0}{2} + \sum_{n=1}^{\infty} (a_n \cos nx + b_n \sin nx)\]

where: \[a_n = \frac{\langle f, \cos nx \rangle}{\|\cos nx\|^2}, \quad b_n = \frac{\langle f, \sin nx \rangle}{\|\sin nx\|^2}\]

Each partial sum is the orthogonal projection onto the span of the first \(n\) trigonometric functions, making it the best approximation in the \(L^2\) sense.

11 Canonical Forms

Canonical forms represent the culmination of linear algebra’s quest to find the simplest possible representations of linear transformations. While diagonalization provides the ideal simplification when sufficient eigenspaces exist, many transformations cannot be diagonalized. Jordan canonical form extends this theory to handle all matrices, revealing the fundamental structure underlying any linear transformation. This chapter develops the complete theory of canonical forms, exploring generalized eigenspaces, minimal polynomials, and the profound connections between algebra and geometry.

11.1 Generalized Eigenvectors and Eigenspaces

When a matrix fails to be diagonalizable, it’s because some eigenvalues don’t have enough linearly independent eigenvectors. Generalized eigenvectors extend the notion of eigenspace to account for this deficiency.

Definition 11.1 (Generalized Eigenvector). Let \(\mathbf{A}\) be an \(n \times n\) matrix with eigenvalue \(\lambda\). A non-zero vector \(\boldsymbol{v}\) is a generalized eigenvector of rank \(k\) for eigenvalue \(\lambda\) if: \[(\mathbf{A} - \lambda\mathbf{I})^k \boldsymbol{v} = \boldsymbol{0} \quad \text{but} \quad (\mathbf{A} - \lambda\mathbf{I})^{k-1} \boldsymbol{v} \neq \boldsymbol{0}\]

The set of all generalized eigenvectors for \(\lambda\), together with the zero vector, forms the generalized eigenspace: \[E_{\lambda} = \{\boldsymbol{v} : (\mathbf{A} - \lambda\mathbf{I})^k \boldsymbol{v} = \boldsymbol{0} \text{ for some } k \geq 1\}\]

Remark 502. Regular eigenvectors are precisely the generalized eigenvectors of rank 1. The generalized eigenspace contains all vectors that eventually become zero when we repeatedly apply \((\mathbf{A} - \lambda\mathbf{I})\). This captures the idea that these vectors are "almost" eigenvectors—they become eigenvectors after some iteration.

Theorem 11.3 (Properties of Generalized Eigenspaces). For an \(n \times n\) matrix \(\mathbf{A}\) with eigenvalue \(\lambda\):

1. \(E_{\lambda}\) is a subspace of \(\mathbb{C}^n\)

2. If \((\mathbf{A} - \lambda\mathbf{I})^k \boldsymbol{v} = \boldsymbol{0}\), then \((\mathbf{A} - \lambda\mathbf{I})^j \boldsymbol{v} = \boldsymbol{0}\) for all \(j \geq k\)

3. \(E_{\lambda} = \text{Nul}((\mathbf{A} - \lambda\mathbf{I})^n)\)

4. \(\dim(E_{\lambda})\) equals the algebraic multiplicity of \(\lambda\)

5. If \(\lambda_1, \lambda_2, \ldots, \lambda_k\) are distinct eigenvalues, then \(E_{\lambda_1}, E_{\lambda_2}, \ldots, E_{\lambda_k}\) are linearly independent

Proof. Property 1: Standard verification that \(E_{\lambda}\) satisfies the subspace conditions.

Property 3: Since \(\mathbf{A}\) is \(n \times n\), the sequence \(\text{Nul}(\mathbf{A} - \lambda\mathbf{I}) \subseteq \text{Nul}((\mathbf{A} - \lambda\mathbf{I})^2) \subseteq \cdots\) must stabilize by the \(n\)-th power due to dimensional constraints.

Property 4: This is a deep result that requires the Jordan canonical form to prove completely. The intuition is that the generalized eigenspace "captures" all the algebraic multiplicity of the eigenvalue.

Property 5: If \(\sum \boldsymbol{v}_i = \boldsymbol{0}\) with \(\boldsymbol{v}_i \in E_{\lambda_i}\), then applying appropriate polynomials in \(\mathbf{A}\) that annihilate all but one of the generalized eigenspaces shows each \(\boldsymbol{v}_i = \boldsymbol{0}\). ◻

Example 11.4 (Computing Generalized Eigenspaces). Find the generalized eigenspaces for \(\mathbf{A} = \begin{pmatrix} 2 & 1 & 0 \\ 0 & 2 & 1 \\ 0 & 0 & 2 \end{pmatrix}\).

Step 1: Find eigenvalues \(\det(\mathbf{A} - \lambda\mathbf{I}) = (2 - \lambda)^3\), so \(\lambda = 2\) with algebraic multiplicity 3.

Step 2: Find regular eigenspace \[\mathbf{A} - 2\mathbf{I} = \begin{pmatrix} 0 & 1 & 0 \\ 0 & 0 & 1 \\ 0 & 0 & 0 \end{pmatrix}\]

\[\text{Nul}(\mathbf{A} - 2\mathbf{I}) = \text{span}\left\{\begin{pmatrix} 1 \\ 0 \\ 0 \end{pmatrix}\right\}\]

So the geometric multiplicity is 1, meaning \(\mathbf{A}\) is not diagonalizable.

Step 3: Find generalized eigenspace \[(\mathbf{A} - 2\mathbf{I})^2 = \begin{pmatrix} 0 & 1 & 0 \\ 0 & 0 & 1 \\ 0 & 0 & 0 \end{pmatrix}\begin{pmatrix} 0 & 1 & 0 \\ 0 & 0 & 1 \\ 0 & 0 & 0 \end{pmatrix} = \begin{pmatrix} 0 & 0 & 1 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{pmatrix}\]

\[\text{Nul}((\mathbf{A} - 2\mathbf{I})^2) = \text{span}\left\{\begin{pmatrix} 1 \\ 0 \\ 0 \end{pmatrix}, \begin{pmatrix} 0 \\ 1 \\ 0 \end{pmatrix}\right\}\]

Step 4: Check the third power \[(\mathbf{A} - 2\mathbf{I})^3 = \begin{pmatrix} 0 & 0 & 1 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{pmatrix}\begin{pmatrix} 0 & 1 & 0 \\ 0 & 0 & 1 \\ 0 & 0 & 0 \end{pmatrix} = \begin{pmatrix} 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{pmatrix}\]

\[\text{Nul}((\mathbf{A} - 2\mathbf{I})^3) = \mathbb{R}^3\]

Therefore, the generalized eigenspace is all of \(\mathbb{R}^3\), confirming that its dimension equals the algebraic multiplicity.

Generalized eigenvector structure: - \(\boldsymbol{v}_1 = (1, 0, 0)^T\) is a regular eigenvector (rank 1) - \(\boldsymbol{v}_2 = (0, 1, 0)^T\) is a generalized eigenvector of rank 2: \((\mathbf{A} - 2\mathbf{I})\boldsymbol{v}_2 = (1, 0, 0)^T \neq \boldsymbol{0}\) - \(\boldsymbol{v}_3 = (0, 0, 1)^T\) is a generalized eigenvector of rank 3: \((\mathbf{A} - 2\mathbf{I})\boldsymbol{v}_3 = (0, 1, 0)^T \neq \boldsymbol{0}\)

11.2 Jordan Canonical Form

The Jordan canonical form provides the complete classification of matrices up to similarity, extending diagonalization to all square matrices.

Definition 11.5 (Jordan Block). A Jordan block of size \(k\) corresponding to eigenvalue \(\lambda\) is the \(k \times k\) matrix: \[\mathbf{J}_k(\lambda) = \begin{pmatrix} \lambda & 1 & 0 & \cdots & 0 \\ 0 & \lambda & 1 & \cdots & 0 \\ 0 & 0 & \lambda & \cdots & 0 \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & 0 & \cdots & \lambda \end{pmatrix}\]

The Jordan canonical form of a matrix is a block diagonal matrix consisting of Jordan blocks: \[\mathbf{J} = \begin{pmatrix} \mathbf{J}_{k_1}(\lambda_1) & & & \\ & \mathbf{J}_{k_2}(\lambda_2) & & \\ & & \ddots & \\ & & & \mathbf{J}_{k_r}(\lambda_r) \end{pmatrix}\]

Theorem 11.6 (Jordan Canonical Form Theorem). Every square matrix \(\mathbf{A}\) over \(\mathbb{C}\) is similar to a unique Jordan canonical form \(\mathbf{J}\) (up to rearrangement of blocks). That is, there exists an invertible matrix \(\mathbf{P}\) such that: \[\mathbf{A} = \mathbf{P}\mathbf{J}\mathbf{P}^{-1}\]

The Jordan form is uniquely determined by:

1. The eigenvalues of \(\mathbf{A}\) (which appear on the diagonal)

2. The sizes of the Jordan blocks (which are determined by the dimensions of the null spaces of powers of \((\mathbf{A} - \lambda\mathbf{I})\))

Example 11.7 (Finding Jordan Canonical Form). Find the Jordan canonical form of \(\mathbf{A} = \begin{pmatrix} 3 & 1 & 0 & 0 \\ 0 & 3 & 0 & 0 \\ 0 & 0 & 3 & 1 \\ 0 & 0 & 0 & 3 \end{pmatrix}\).

Step 1: Eigenvalues \(\det(\mathbf{A} - \lambda\mathbf{I}) = (3 - \lambda)^4\), so \(\lambda = 3\) with algebraic multiplicity 4.

Step 2: Compute null space dimensions \[\mathbf{A} - 3\mathbf{I} = \begin{pmatrix} 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 1 \\ 0 & 0 & 0 & 0 \end{pmatrix}\]

This has rank 2, so \(d_1 = \dim(\text{Nul}(\mathbf{A} - 3\mathbf{I})) = 4 - 2 = 2\).

\[(\mathbf{A} - 3\mathbf{I})^2 = \begin{pmatrix} 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \end{pmatrix}\]

So \(d_2 = \dim(\text{Nul}((\mathbf{A} - 3\mathbf{I})^2)) = 4\).

Step 3: Determine Jordan block structure - Number of Jordan blocks = \(d_1 = 2\) - Since \(d_2 = 4\) and we have 2 blocks total, and the algebraic multiplicity is 4, the blocks must have sizes that sum to 4 - The difference \(d_2 - d_1 = 4 - 2 = 2\) tells us about the structure

Since \((\mathbf{A} - 3\mathbf{I})^2 = \mathbf{0}\) but \((\mathbf{A} - 3\mathbf{I}) \neq \mathbf{0}\), the largest Jordan block has size 2.

With 2 blocks total and sum 4, we must have blocks of sizes 2 and 2.

Jordan canonical form: \[\mathbf{J} = \begin{pmatrix} 3 & 1 & 0 & 0 \\ 0 & 3 & 0 & 0 \\ 0 & 0 & 3 & 1 \\ 0 & 0 & 0 & 3 \end{pmatrix}\]

In this case, \(\mathbf{A}\) is already in Jordan form!

Example 11.8 (More Complex Jordan Form). Find the Jordan form of \(\mathbf{B} = \begin{pmatrix} 2 & 1 & 0 & 0 & 0 \\ 0 & 2 & 1 & 0 & 0 \\ 0 & 0 & 2 & 0 & 0 \\ 0 & 0 & 0 & 2 & 1 \\ 0 & 0 & 0 & 0 & 2 \end{pmatrix}\).

Analysis: \(\lambda = 2\) with algebraic multiplicity 5.

\[\mathbf{B} - 2\mathbf{I} = \begin{pmatrix} 0 & 1 & 0 & 0 & 0 \\ 0 & 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 1 \\ 0 & 0 & 0 & 0 & 0 \end{pmatrix}\]

\(\text{rank}(\mathbf{B} - 2\mathbf{I}) = 3\), so \(d_1 = 5 - 3 = 2\).

\[(\mathbf{B} - 2\mathbf{I})^2 = \begin{pmatrix} 0 & 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 \end{pmatrix}\]

\(\text{rank}((\mathbf{B} - 2\mathbf{I})^2) = 1\), so \(d_2 = 5 - 1 = 4\).

\[(\mathbf{B} - 2\mathbf{I})^3 = \mathbf{0}\]

So \(d_3 = 5\).

Jordan block analysis: - Number of blocks = \(d_1 = 2\) - Since \(d_2 - d_1 = 2\) and \(d_3 - d_2 = 1\), we have blocks of sizes 3 and 2

Jordan canonical form: \[\mathbf{J} = \begin{pmatrix} 2 & 1 & 0 & 0 & 0 \\ 0 & 2 & 1 & 0 & 0 \\ 0 & 0 & 2 & 0 & 0 \\ 0 & 0 & 0 & 2 & 1 \\ 0 & 0 & 0 & 0 & 2 \end{pmatrix}\]

This consists of one \(3 \times 3\) Jordan block and one \(2 \times 2\) Jordan block.

11.3 Minimal Polynomial

The minimal polynomial provides deep insight into the structure of a linear transformation, revealing the simplest polynomial relationship that the transformation satisfies.

Definition 11.9 (Minimal Polynomial). The minimal polynomial \(m_{\mathbf{A}}(x)\) of an \(n \times n\) matrix \(\mathbf{A}\) is the monic polynomial of lowest degree such that \(m_{\mathbf{A}}(\mathbf{A}) = \mathbf{0}\).

Equivalently, \(m_{\mathbf{A}}(x)\) is the monic polynomial that generates the ideal: \[\{p(x) \in \mathbb{F}[x] : p(\mathbf{A}) = \mathbf{0}\}\]

Remark 510. The minimal polynomial always exists and is unique. It divides every polynomial that annihilates \(\mathbf{A}\), including the characteristic polynomial. While the characteristic polynomial has degree \(n\), the minimal polynomial may have smaller degree, capturing the essential algebraic relationships more efficiently.

Theorem 11.11 (Properties of the Minimal Polynomial). For an \(n \times n\) matrix \(\mathbf{A}\):

1. \(m_{\mathbf{A}}(x)\) divides the characteristic polynomial \(p_{\mathbf{A}}(x)\)

2. \(m_{\mathbf{A}}(x)\) and \(p_{\mathbf{A}}(x)\) have the same irreducible factors (but possibly different multiplicities)

3. \(\deg(m_{\mathbf{A}}) \leq n\)

4. \(\mathbf{A}\) is diagonalizable if and only if \(m_{\mathbf{A}}(x)\) has no repeated roots

5. The minimal polynomial is invariant under similarity: if \(\mathbf{B} = \mathbf{P}^{-1}\mathbf{A}\mathbf{P}\), then \(m_{\mathbf{B}}(x) = m_{\mathbf{A}}(x)\)

Proof. Property 1: By the Cayley-Hamilton theorem, \(p_{\mathbf{A}}(\mathbf{A}) = \mathbf{0}\). Since \(m_{\mathbf{A}}(x)\) is the minimal polynomial annihilating \(\mathbf{A}\), it must divide any polynomial that annihilates \(\mathbf{A}\).

Property 4: \(\mathbf{A}\) is diagonalizable if and only if \(\mathbf{A}\) can be written as \(\mathbf{A} = \mathbf{P}\mathbf{D}\mathbf{P}^{-1}\) where \(\mathbf{D}\) is diagonal. If \(\mathbf{D} = \text{diag}(\lambda_1, \lambda_2, \ldots, \lambda_k)\) where \(\lambda_1, \ldots, \lambda_k\) are the distinct eigenvalues, then the minimal polynomial is \(m_{\mathbf{A}}(x) = (x - \lambda_1)(x - \lambda_2) \cdots (x - \lambda_k)\), which has no repeated roots. Conversely, if the minimal polynomial has no repeated roots, then each eigenspace equals the corresponding generalized eigenspace, ensuring diagonalizability. ◻

Example 11.12 (Computing Minimal Polynomials). 1. Diagonal matrix: For \(\mathbf{A} = \begin{pmatrix} 2 & 0 & 0 \\ 0 & 3 & 0 \\ 0 & 0 & 2 \end{pmatrix}\):

The eigenvalues are 2 (with multiplicity 2) and 3 (with multiplicity 1). Since the matrix is diagonal, it’s already diagonalized with no repeated eigenvalues in any Jordan block.

Minimal polynomial: \(m_{\mathbf{A}}(x) = (x - 2)(x - 3)\)

Verification: \((A - 2I)(A - 3I) = \begin{pmatrix} 0 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0 \end{pmatrix}\begin{pmatrix} -1 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & -1 \end{pmatrix} = \begin{pmatrix} 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{pmatrix}\)

2. Jordan block: For \(\mathbf{B} = \begin{pmatrix} 2 & 1 & 0 \\ 0 & 2 & 1 \\ 0 & 0 & 2 \end{pmatrix}\):

This is a single \(3 \times 3\) Jordan block with eigenvalue 2. Minimal polynomial: \(m_{\mathbf{B}}(x) = (x - 2)^3\)

Verification: We need to check that \((B - 2I)^2 \neq 0\) but \((B - 2I)^3 = 0\). \[\mathbf{B} - 2\mathbf{I} = \begin{pmatrix} 0 & 1 & 0 \\ 0 & 0 & 1 \\ 0 & 0 & 0 \end{pmatrix}\]

\[(\mathbf{B} - 2\mathbf{I})^2 = \begin{pmatrix} 0 & 0 & 1 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{pmatrix} \neq \mathbf{0}\]

\[(\mathbf{B} - 2\mathbf{I})^3 = \begin{pmatrix} 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{pmatrix} = \mathbf{0}\]

3. Mixed Jordan structure: For a matrix with Jordan form \(\begin{pmatrix} 1 & 1 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 2 & 1 \\ 0 & 0 & 0 & 2 \end{pmatrix}\):

This has two \(2 \times 2\) Jordan blocks, one for eigenvalue 1 and one for eigenvalue 2. Minimal polynomial: \(m_{\mathbf{A}}(x) = (x - 1)^2(x - 2)^2\)

The minimal polynomial captures the largest Jordan block size for each eigenvalue.

Theorem 11.13 (Relationship Between Minimal and Characteristic Polynomials). Let \(p_{\mathbf{A}}(x) = (x - \lambda_1)^{a_1}(x - \lambda_2)^{a_2} \cdots (x - \lambda_k)^{a_k}\) be the characteristic polynomial and \(m_{\mathbf{A}}(x) = (x - \lambda_1)^{m_1}(x - \lambda_2)^{m_2} \cdots (x - \lambda_k)^{m_k}\) be the minimal polynomial. Then:

1. \(1 \leq m_i \leq a_i\) for all \(i\)

2. \(m_i\) equals the size of the largest Jordan block corresponding to eigenvalue \(\lambda_i\)

3. \(a_i\) equals the sum of sizes of all Jordan blocks corresponding to eigenvalue \(\lambda_i\)

4. \(\mathbf{A}\) is diagonalizable if and only if \(m_i = 1\) for all \(i\)

Example 11.14 (Minimal Polynomial and Diagonalizability). Question: Determine which of these matrices are diagonalizable by examining their minimal polynomials:

Matrix 1: Characteristic polynomial \(p(x) = (x - 2)^3(x - 5)^2\) If the minimal polynomial is \(m(x) = (x - 2)(x - 5)\), then the matrix is diagonalizable. If the minimal polynomial is \(m(x) = (x - 2)^2(x - 5)\) or \(m(x) = (x - 2)(x - 5)^2\) or higher powers, then it’s not diagonalizable.

Matrix 2: With Jordan form \(\begin{pmatrix} 3 & 1 & 0 \\ 0 & 3 & 0 \\ 0 & 0 & 4 \end{pmatrix}\): Minimal polynomial: \(m(x) = (x - 3)^2(x - 4)\) Since there’s a repeated root with multiplicity > 1, the matrix is not diagonalizable.

Matrix 3: With eigenvalues \(\{1, 2, 3, 4\}\) all simple: Minimal polynomial: \(m(x) = (x - 1)(x - 2)(x - 3)(x - 4)\) Since all roots are simple, the matrix is diagonalizable.

11.4 Rational Canonical Form

While Jordan canonical form requires working over an algebraically closed field (like \(\mathbb{C}\)), the rational canonical form works over any field and provides an alternative canonical form based on companion matrices.

Definition 11.15 (Companion Matrix). The companion matrix of a monic polynomial \(p(x) = x^n + a_{n-1}x^{n-1} + \cdots + a_1x + a_0\) is: \[C(p) = \begin{pmatrix} 0 & 0 & 0 & \cdots & 0 & -a_0 \\ 1 & 0 & 0 & \cdots & 0 & -a_1 \\ 0 & 1 & 0 & \cdots & 0 & -a_2 \\ \vdots & \vdots & \ddots & \ddots & \vdots & \vdots \\ 0 & 0 & 0 & \cdots & 1 & -a_{n-1} \end{pmatrix}\]

The characteristic polynomial of \(C(p)\) is exactly \(p(x)\).

Definition 11.16 (Rational Canonical Form). Every square matrix over a field \(\mathbb{F}\) is similar to a unique matrix in rational canonical form: \[\mathbf{R} = \begin{pmatrix} C(p_1) & & & \\ & C(p_2) & & \\ & & \ddots & \\ & & & C(p_k) \end{pmatrix}\] where each \(p_i(x)\) is a monic polynomial, and \(p_1 | p_2 | \cdots | p_k\) (each polynomial divides the next).

The polynomials \(p_1, p_2, \ldots, p_k\) are called the invariant factors of the matrix.

Theorem 11.17 (Properties of Rational Canonical Form). For a matrix \(\mathbf{A}\) with rational canonical form having invariant factors \(p_1, p_2, \ldots, p_k\):

1. The minimal polynomial is \(m_{\mathbf{A}}(x) = p_k(x)\) (the largest invariant factor)

2. The characteristic polynomial is \(\chi_{\mathbf{A}}(x) = p_1(x)p_2(x) \cdots p_k(x)\)

3. The rational canonical form is unique and depends only on the similarity class of \(\mathbf{A}\)

4. Over \(\mathbb{C}\), the rational canonical form can be transformed to Jordan canonical form

5. The rational canonical form works over any field, not requiring algebraic closure

Example 11.18 (From Jordan to Rational Form). Consider a matrix with Jordan canonical form over \(\mathbb{C}\): \[\mathbf{J} = \begin{pmatrix} 2 & 1 & 0 & 0 \\ 0 & 2 & 0 & 0 \\ 0 & 0 & 3 & 1 \\ 0 & 0 & 0 & 3 \end{pmatrix}\]

The minimal polynomial is \(m(x) = (x-2)^2(x-3)^2\).

To find the rational canonical form, we need to determine the invariant factors. Since the minimal polynomial has degree 4 and we have a \(4 \times 4\) matrix, and both Jordan blocks have the same size, the rational canonical form has a single \(4 \times 4\) companion matrix block:

\[\mathbf{R} = C((x-2)^2(x-3)^2) = C(x^4 - 10x^3 + 37x^2 - 60x + 36)\]

\[= \begin{pmatrix} 0 & 0 & 0 & -36 \\ 1 & 0 & 0 & 60 \\ 0 & 1 & 0 & -37 \\ 0 & 0 & 1 & 10 \end{pmatrix}\]

Over \(\mathbb{R}\), if the eigenvalues were complex, this would be the natural canonical form since we cannot factor complex eigenvalues over the reals.

11.5 Applications and Computational Aspects

Canonical forms have profound applications across mathematics and provide insight into the computational complexity of various linear algebra problems.

11.5.1 Matrix Functions and Differential Equations

Theorem 11.19 (Matrix Exponential via Jordan Form). If \(\mathbf{A} = \mathbf{P}\mathbf{J}\mathbf{P}^{-1}\) where \(\mathbf{J}\) is in Jordan canonical form, then: \[e^{\mathbf{A}} = \mathbf{P}e^{\mathbf{J}}\mathbf{P}^{-1}\]

For a Jordan block \(\mathbf{J}_k(\lambda)\): \[e^{\mathbf{J}_k(\lambda)} = e^{\lambda}\begin{pmatrix} 1 & 1 & \frac{1}{2!} & \frac{1}{3!} & \cdots & \frac{1}{(k-1)!} \\ 0 & 1 & 1 & \frac{1}{2!} & \cdots & \frac{1}{(k-2)!} \\ 0 & 0 & 1 & 1 & \cdots & \frac{1}{(k-3)!} \\ \vdots & \vdots & \vdots & \ddots & \ddots & \vdots \\ 0 & 0 & 0 & 0 & \cdots & 1 \end{pmatrix}\]

Example 11.20 (Solving Matrix Differential Equations). Solve \(\frac{d\boldsymbol{x}}{dt} = \mathbf{A}\boldsymbol{x}\) where \(\mathbf{A} = \begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix}\).

Step 1: Identify the Jordan form \(\mathbf{A}\) is already in Jordan form with a \(2 \times 2\) Jordan block for eigenvalue 1.

Step 2: Compute the matrix exponential \[e^{\mathbf{A}t} = e^t\begin{pmatrix} 1 & t \\ 0 & 1 \end{pmatrix}\]

Step 3: General solution \[\boldsymbol{x}(t) = e^{\mathbf{A}t}\boldsymbol{x}_0 = e^t\begin{pmatrix} 1 & t \\ 0 & 1 \end{pmatrix}\begin{pmatrix} x_0 \\ y_0 \end{pmatrix} = e^t\begin{pmatrix} x_0 + ty_0 \\ y_0 \end{pmatrix}\]

The \(t\) term in the solution reflects the Jordan structure—it wouldn’t appear if the matrix were diagonalizable.

11.5.2 Computational Complexity and Numerical Considerations

Theorem 11.21 (Computational Aspects of Canonical Forms).

1. Characteristic polynomial: Can be computed in \(O(n^3)\) operations using Hessenberg reduction

2. Eigenvalues: Generally require iterative methods; no closed-form solution for \(n \geq 5\)

3. Jordan form: Numerically unstable—small perturbations can change the Jordan structure dramatically

4. Schur form: Numerically stable alternative: \(\mathbf{A} = \mathbf{Q}\mathbf{T}\mathbf{Q}^T\) where \(\mathbf{Q}\) is orthogonal and \(\mathbf{T}\) is upper triangular

Remark 522. The Jordan canonical form, while theoretically beautiful, is rarely computed in practice due to numerical instability. Instead, the Schur decomposition provides a numerically stable way to triangularize matrices, and the singular value decomposition handles the general rectangular case.

11.5.3 Spectral Theory and Functional Analysis

Definition 11.23 (Spectrum of an Operator). For a linear operator \(T: V \to V\), the spectrum \(\sigma(T)\) is the set of complex numbers \(\lambda\) such that \((T - \lambda I)\) is not invertible. In finite dimensions, this is precisely the set of eigenvalues.

Theorem 11.24 (Spectral Radius). The spectral radius of a matrix \(\mathbf{A}\) is: \[\rho(\mathbf{A}) = \max\{|\lambda| : \lambda \text{ is an eigenvalue of } \mathbf{A}\}\]

Properties:

1. \(\rho(\mathbf{A}) \leq \|\mathbf{A}\|\) for any matrix norm

2. \(\lim_{k \to \infty} \|\mathbf{A}^k\|^{1/k} = \rho(\mathbf{A})\)

3. \(\lim_{k \to \infty} \mathbf{A}^k = \mathbf{0}\) if and only if \(\rho(\mathbf{A}) < 1\)

Example 11.25 (Applications to Dynamical Systems). Consider the discrete dynamical system \(\boldsymbol{x}_{n+1} = \mathbf{A}\boldsymbol{x}_n\).

Case 1: Diagonalizable with \(\rho(\mathbf{A}) < 1\) The system converges to the origin exponentially fast.

Case 2: Jordan blocks with eigenvalue of magnitude 1 The system may exhibit polynomial growth or oscillation, depending on the Jordan structure.

Case 3: \(\rho(\mathbf{A}) > 1\) The system diverges, with the rate determined by the largest eigenvalue.

The Jordan canonical form reveals the long-term behavior more precisely than eigenvalues alone.

11.6 Advanced Topics and Connections

11.6.1 Primary Decomposition Theorem

Theorem 11.26 (Primary Decomposition). Let \(\mathbf{A}\) be an \(n \times n\) matrix with minimal polynomial \(m(x) = p_1(x)^{k_1} p_2(x)^{k_2} \cdots p_r(x)^{k_r}\) where the \(p_i(x)\) are distinct irreducible polynomials. Then: \[\mathbb{C}^n = \bigoplus_{i=1}^r \text{Ker}(p_i(\mathbf{A})^{k_i})\]

Each subspace \(\text{Ker}(p_i(\mathbf{A})^{k_i})\) is \(\mathbf{A}\)-invariant and corresponds to the generalized eigenspace(s) associated with the roots of \(p_i(x)\).

Remark 527. This theorem provides the theoretical foundation for understanding how the Jordan canonical form arises from the factorization of the minimal polynomial. Each irreducible factor corresponds to a block-diagonal component of the Jordan form.

11.6.2 Relation to Module Theory

Definition 11.28 (Matrix as Module Action). Every \(n \times n\) matrix \(\mathbf{A}\) defines a \(\mathbb{F}[x]\)-module structure on \(\mathbb{F}^n\) by setting: \[p(x) \cdot \boldsymbol{v} = p(\mathbf{A})\boldsymbol{v}\]

The canonical forms correspond to decomposing this module into cyclic components.

Theorem 11.29 (Structure Theorem for Finitely Generated Modules). Every finitely generated module over a principal ideal domain decomposes as: \[M \cong \bigoplus_{i=1}^k \mathbb{F}[x]/(p_i(x))\]

For the \(\mathbb{F}[x]\)-module \(\mathbb{F}^n\) defined by matrix \(\mathbf{A}\), this decomposition gives:

• Jordan canonical form when working over \(\mathbb{C}\) (where irreducible polynomials are linear)

• Rational canonical form over arbitrary fields

11.6.3 Infinite-Dimensional Extensions

Definition 11.30 (Compact Operators). In infinite-dimensional spaces, compact operators generalize matrices with finite rank. The spectral theorem for compact self-adjoint operators provides: \[T = \sum_{i=1}^{\infty} \lambda_i \langle \cdot, \boldsymbol{e}_i \rangle \boldsymbol{e}_i\]

where \(\lambda_i \to 0\) and \(\{\boldsymbol{e}_i\}\) is an orthonormal basis of eigenvectors.

Example 11.31 (Integral Operators). Consider the integral operator on \(L^2[0,1]\): \[(Tf)(x) = \int_0^1 k(x,y)f(y) \, dy\]

When the kernel \(k(x,y)\) is continuous, \(T\) is compact, and the spectral theorem applies. This connects canonical forms to Fourier analysis and partial differential equations.

Key Idea

The Unifying Power of Canonical Forms:

• Generalized eigenspaces: Extend the eigenspace concept to capture all algebraic structure, not just geometric structure

• Jordan canonical form: Provides the complete classification of matrices up to similarity, revealing the finest algebraic structure

• Minimal polynomial: Captures the essential polynomial relationships, determining diagonalizability and spectral properties

• Rational canonical form: Works over arbitrary fields, connecting linear algebra to abstract algebra through module theory

• Computational reality: While theoretically fundamental, numerical implementations require stable alternatives like Schur decomposition

• Broader connections: Links linear algebra to differential equations, dynamical systems, functional analysis, and algebraic geometry

Canonical forms represent the culmination of linear algebra’s quest for the simplest representations. They reveal that behind every linear transformation lies a fundamental algebraic structure that determines its behavior completely. While diagonalization provides the ideal simplification when possible, the Jordan form shows that even non-diagonalizable transformations have a beautiful and computable canonical structure.

The theory demonstrates the deep connections between: - Algebra (polynomials and their factorizations) - Geometry (eigenspaces and their dimensions) - Analysis (matrix functions and differential equations) - Computation (numerical algorithms and stability)

These connections make canonical forms not just a theoretical capstone, but a practical tool for understanding and computing with linear transformations across mathematics and its applications. From solving differential equations to analyzing dynamical systems, from quantum mechanics to data analysis, the insights provided by canonical forms continue to illuminate the structure underlying linear phenomena.

Most importantly, canonical forms exemplify mathematics at its best: taking a complex, seemingly intractable problem (classifying all possible linear transformations) and revealing its elegant underlying structure through the systematic development of theory. The journey from basic matrix operations to Jordan canonical form illustrates how abstract mathematical thinking can illuminate concrete computational problems, providing both theoretical understanding and practical tools.

This completes our comprehensive exploration of canonical forms and, with it, our journey through the fundamental structures of linear algebra. From vector spaces and linear transformations through orthogonality and decompositions to change of basis and canonical forms, we have developed a complete theoretical framework that unifies geometric intuition, algebraic computation, and practical application. The tools and insights developed throughout these topics provide the foundation for advanced study in virtually every area of mathematics and its applications.

Footnotes

1. If you adapt material from Nicholson’s Linear Algebra with Applications (Open Edition, 2023), keep CC BY-NC-SA terms in mind.