IndianTechnoEra: Linear Algebra For AIML - Chapter 7 Eigenvalues & Eigenvectors

Chapter 7 — Eigenvalues & Eigenvectors

Eigenvalues and eigenvectors are central to understanding linear transformations. They reveal directions that remain unchanged under transformation and their scaling factor, which is crucial in AI & ML for dimensionality reduction and model stability.

7.1 What is an Eigenvector?

An eigenvector of a square matrix A is a non-zero vector v such that multiplication by A only scales the vector and does not change its direction:

A * v = λ * v

Here, λ is the eigenvalue corresponding to v.

AI/ML Relevance: Eigenvectors define principal directions in data, which is the foundation of PCA for reducing dimensions.

7.2 What is an Eigenvalue?

An eigenvalue indicates how much its corresponding eigenvector is stretched or shrunk during the transformation.

If v is an eigenvector and λ its eigenvalue:
A * v = λ * v
λ > 1 → vector stretches
0 < λ < 1 → vector shrinks
λ = 1 → vector unchanged
λ = 0 → vector collapses to zero

ML Importance: Eigenvalues help identify significant directions in data (large eigenvalues) and less important directions (small eigenvalues) — useful for dimensionality reduction.

7.3 Diagonalization

A matrix A is diagonalizable if it can be written as:

A = P * D * P^-1

- P contains the eigenvectors as columns.
- D is a diagonal matrix of eigenvalues.
Diagonalization simplifies computations like powers of matrices (A^n).

AI/ML Context: Simplifies covariance matrix analysis in PCA, helps in computing SVD, and improves stability in optimization of deep learning models.

7.4 Quick NumPy Examples (Practical)

import numpy as np

# Define a matrix
A = np.array([[4, 2],
              [1, 3]])

# Compute eigenvalues and eigenvectors
eigenvalues, eigenvectors = np.linalg.eig(A)

print("Eigenvalues:", eigenvalues)
print("Eigenvectors:\n", eigenvectors)

Interpretation: Columns of eigenvectors are directions unchanged by A, scaled by corresponding eigenvalues.

7.5 Geometric Intuition

- Eigenvectors point in directions that remain on the same line after transformation.
- Eigenvalues tell you how much the vector stretches or compresses along that direction.

7.6 AI/ML Use Cases (Why Eigenvectors & Eigenvalues Matter)

• PCA (Principal Component Analysis): Eigenvectors of the covariance matrix define principal directions; eigenvalues determine their importance.
• Dimensionality reduction: Keep directions with large eigenvalues and drop less important ones.
• Stability in deep learning: Eigenvalues of weight matrices affect convergence speed and stability.
• Graph analysis: Eigenvectors of adjacency matrices help in spectral clustering.

7.7 Exercises

1. Compute eigenvalues and eigenvectors of [[2,1],[1,2]] by hand and verify in Python.
2. For a 3x3 covariance matrix, identify which eigenvectors correspond to the most variance.
3. Diagonalize [[5,2],[2,1]] and verify A = P * D * P^-1.
4. Explain why dropping small eigenvalue directions in PCA is justified.

Answers / Hints

1. Eigenvalues: 3,1; Eigenvectors: [1,1]/sqrt(2), [1,-1]/sqrt(2)
2. Largest eigenvalue eigenvector represents the direction of maximum variance.
3. Compute P with eigenvectors as columns, D as diagonal matrix of eigenvalues, then verify matrix multiplication.
4. Small eigenvalues correspond to low-variance directions, adding little information; dropping them reduces dimensionality without significant loss.

7.8 Practice Projects / Mini Tasks

• Perform PCA on the Iris dataset using NumPy eigen decomposition and visualize the top 2 components.
• Compute eigenvectors of adjacency matrices for a small graph and see clustering effects.
• Experiment with diagonalization to compute powers of a matrix efficiently.

7.9 Further Reading & Videos

• 3Blue1Brown — Essence of Linear Algebra (eigenvectors & eigenvalues intuition).
• NumPy documentation — np.linalg.eig().
• Textbooks: Gilbert Strang's Linear Algebra and applications in ML.

Next chapter: Singular Value Decomposition (SVD) — understand how matrices can be decomposed into U, Σ, V^T, and applications in dimensionality reduction and recommender systems.