IndianTechnoEra: Linear Algebra For AIML - Chapter 9 Singular Value Decomposition (SVD)

Chapter 9 — Singular Value Decomposition (SVD)

SVD is a fundamental matrix factorization technique used in AI & ML to extract important features, reduce dimensions, and analyze patterns in data.

9.1 What is Singular Value Decomposition?

Singular Value Decomposition (SVD) factorizes any real or complex matrix A into three matrices:

A = U Σ Vᵀ

Where:

• U: orthogonal matrix (columns are left singular vectors)
• Σ: diagonal matrix with singular values (non-negative)
• Vᵀ: transpose of an orthogonal matrix (columns of V are right singular vectors)

ML Context: SVD helps in compressing data while preserving essential patterns, crucial in dimensionality reduction and recommender systems.

9.2 Relation with Eigen Decomposition

For a square matrix A:

• The eigenvectors of AᵀA are the right singular vectors V.
• The eigenvectors of AAᵀ are the left singular vectors U.
• The square roots of eigenvalues of AᵀA or AAᵀ give the singular values in Σ.

This connection is why SVD generalizes eigen decomposition to non-square matrices.

9.3 Why SVD is Important in AI/ML

- Reduces high-dimensional data while preserving most variance (dimensionality reduction).
- Used in recommender systems (Netflix, Amazon) to find latent features from user-item matrices.
- Helps solve linear least squares problems and pseudoinverse calculations.
- Improves stability and efficiency in ML pipelines.

9.4 Quick NumPy Example (Practical)

import numpy as np

# create a matrix
A = np.array([[3, 1, 1],
              [-1, 3, 1]])

# perform SVD
U, S, Vt = np.linalg.svd(A, full_matrices=False)

print("U (left singular vectors):")
print(U)

print("Σ (singular values):")
print(S)

print("Vᵀ (right singular vectors):")
print(Vt)

# reconstruct original matrix
Sigma = np.diag(S)
A_reconstructed = U @ Sigma @ Vt
print("Reconstructed A:")
print(A_reconstructed)

9.5 Geometric Intuition

Think of SVD as a sequence of three transformations applied to the unit circle in 2D (or a unit sphere in higher dimensions):

1. Rotate space using Vᵀ.
2. Scale along orthogonal axes using Σ.
3. Rotate again using U.

This transforms any arbitrary shape into a stretched and rotated version while preserving key structure.

9.6 AI/ML Use Cases (Why SVD Matters)

• Dimensionality Reduction: Reduce features in datasets, speeding up training and visualization (e.g., PCA uses SVD).
• Recommendation Systems: Factorize user-item rating matrices to predict missing ratings.
• Latent Semantic Analysis (LSA): Extract concepts from text by applying SVD on term-document matrices.
• Image Compression: Represent images using fewer singular values for storage efficiency.

9.7 Exercises

1. Compute the SVD of [[4, 0], [3, -5]] by hand for practice and verify using NumPy.
2. Apply SVD on a small user-item matrix and predict a missing rating.
3. Reconstruct an image using only the top 2 singular values and compare with original.
4. Explain why singular values indicate the “importance” of each dimension.

Answers / Hints

1. Use np.linalg.svd() to verify decomposition.
2. Top singular values capture the most variance/features.
3. Reconstruction using fewer singular values gives approximate representation.

9.8 Practice Projects / Mini Tasks

• Implement a mini movie recommender using a small ratings dataset and SVD factorization.
• Perform dimensionality reduction on MNIST dataset using SVD and visualize top 2 components.
• Compress a grayscale image using top-k singular values and compare visual quality.

9.9 Further Reading & Videos

• 3Blue1Brown — Essence of Linear Algebra (SVD intuition)
• NumPy documentation — np.linalg.svd
• Papers on collaborative filtering and matrix factorization in recommender systems

Next chapter: Principal Component Analysis (PCA) — applying SVD for dimensionality reduction and feature extraction in ML pipelines.