Linear Algebra For AIML - Chapter 6 Vector Spaces & Subspaces

Chapter 6 — Vector Spaces & Subspaces

Vector spaces and subspaces are foundational concepts in linear algebra that help us understand the structure of data in AI & ML. They define how vectors relate, combine, and form meaningful representations for computations.

6.1 What is a Vector Space?

A vector space is a collection of vectors that can be added together and multiplied by scalars while still remaining in the same space.
Formally, a set V with vector addition and scalar multiplication is a vector space if it satisfies closure, associativity, commutativity, and other axioms.

Importance in AI/ML: Understanding the structure of vector spaces allows us to manipulate data efficiently, perform dimensionality reduction, and identify independent directions in features.

6.2 Span of Vectors

The span of a set of vectors is all possible linear combinations of those vectors. Essentially, it defines the space that can be reached by combining the vectors.

v1 = [1,0,0]
v2 = [0,1,0]

# span of v1 and v2 includes all vectors [a,b,0] for scalars a,b

AI/ML Context: Span shows how features combine to form new meaningful directions in data space.

6.3 Linear Independence

Vectors are linearly independent if no vector in the set can be written as a combination of the others. Otherwise, they are dependent.

v1 = [1,0]
v2 = [0,1]       # independent
v3 = [1,1]       # dependent on v1 and v2

AI/ML Importance: Linear independence ensures features provide unique information. Dependent features can be redundant and reduce model efficiency.

6.4 Basis and Dimension

A basis is a set of linearly independent vectors that span the vector space.
The dimension of the space is the number of vectors in the basis.

V = R^3
Basis = { [1,0,0], [0,1,0], [0,0,1] }
Dimension = 3

ML Relevance: Basis vectors represent the minimum set of independent features needed to describe all data without redundancy.

6.5 Rank of a Matrix

The rank of a matrix is the maximum number of linearly independent rows or columns. It measures the "information content" of a matrix.

A = [[1,2,3],
     [2,4,6],
     [1,1,0]]

# rank(A) = 2, because two rows are independent

AI/ML Context: Rank helps detect redundant features and informs dimensionality reduction methods like PCA.

6.6 Quick NumPy Examples (Practical)

import numpy as np

# Define a matrix
A = np.array([[1,2,3],
              [2,4,6],
              [1,1,0]])

# Compute rank
rank = np.linalg.matrix_rank(A)
print("Rank of A:", rank)

# Check linear independence of columns
cols = A.T
independent = np.linalg.matrix_rank(cols) == cols.shape[1]
print("Columns independent?", independent)

6.7 Geometric Intuition

- Span shows all directions reachable by combining vectors.
- Linear independence ensures vectors point in unique directions.
- Basis vectors form the “coordinate axes” of the space.
- Rank tells us how many independent directions exist in a matrix.

6.8 AI/ML Use Cases (Why Vector Spaces Matter)

Feature reduction: Remove redundant features while keeping all essential information.
PCA: Projects data into lower-dimensional subspace formed by principal components.
Embedding spaces: Word vectors, image embeddings, and latent spaces are all vector spaces with basis directions.
Optimization: Many algorithms assume features are linearly independent to converge efficiently.

6.9 Exercises

Determine if the vectors [1,2,0], [2,4,0], [0,1,1] are linearly independent.
Find a basis for the column space of [[1,2],[3,4]].
Compute the rank of a random 3x3 matrix using NumPy.
Explain why PCA chooses directions of maximum variance in the context of vector spaces.

Answers / Hints

Vectors 1 & 2 are dependent (second = 2 * first), so not all three are independent.
Basis of [[1,2],[3,4]] columns: [[1,3],[2,4]] (after removing dependent vectors).
Use np.linalg.matrix_rank() for rank.
PCA chooses orthogonal basis vectors along directions of maximum variance to reduce dimensionality while preserving information.

6.10 Practice Projects / Mini Tasks

Take a dataset of features and check for linear dependence/redundancy.
Visualize the span of two 2D vectors using matplotlib.
Use NumPy to compute rank and basis vectors for real-world datasets.

6.11 Further Reading & Videos

3Blue1Brown — Essence of Linear Algebra (vectors, span, and basis intuition).
NumPy documentation — matrix_rank() and linear algebra utilities.
Textbooks on linear algebra and ML feature engineering.

Next chapter: Eigenvalues & Eigenvectors — learn how to compute them, their geometric meaning, and their importance in PCA, SVD, and ML models.

IndianTechnoEra