IndianTechnoEra: Calculus For AI/ML - Chapter 10 Practical Calculus in Python

Chapter 10 — Practical Calculus in Python

This chapter focuses on implementing calculus concepts in Python, including symbolic differentiation, numerical derivatives, and gradient descent. Practical examples show how these tools are used in machine learning model training.

10.1 Symbolic Differentiation with SymPy

SymPy is a Python library for symbolic mathematics. It allows exact differentiation of functions, which is useful for theoretical analysis and verifying gradients.

import sympy as sp

x, y = sp.symbols('x y')
f = x**2 + 3*x*y + y**2

# Partial derivatives
df_dx = sp.diff(f, x)
df_dy = sp.diff(f, y)

print("∂f/∂x =", df_dx)
print("∂f/∂y =", df_dy)

ML Context: Symbolic differentiation is useful for deriving gradients in custom loss functions or verifying implementations.

10.2 Numerical Derivatives with NumPy

When analytical derivatives are hard, numerical derivatives approximate gradients using finite differences.

import numpy as np

def f(theta):
    x, y = theta
    return x**2 + 3*x*y + y**2

def numerical_gradient(f, theta, eps=1e-5):
    grad = np.zeros_like(theta)
    for i in range(len(theta)):
        theta_eps_plus = theta.copy()
        theta_eps_minus = theta.copy()
        theta_eps_plus[i] += eps
        theta_eps_minus[i] -= eps
        grad[i] = (f(theta_eps_plus) - f(theta_eps_minus)) / (2*eps)
    return grad

theta = np.array([1.0, 2.0])
grad = numerical_gradient(f, theta)
print("Numerical gradient =", grad)

ML Context: Useful for gradient checking in neural networks to verify backpropagation.

10.3 Implementing Gradient Descent from Scratch

Combining calculus and Python, we can implement gradient descent to minimize functions.

def gradient_descent(f_grad, theta_init, lr=0.1, steps=50):
    theta = theta_init.copy()
    for i in range(steps):
        grad = f_grad(theta)
        theta -= lr * grad
        print(f"Step {i+1}: theta={theta}, f(theta)={f(theta)}")
    return theta

# Using numerical gradient
theta_init = np.array([3.0, 4.0])
gradient_descent(lambda t: numerical_gradient(f, t), theta_init)

ML Context: This shows how to train simple models like linear regression using gradient descent, without relying on deep learning frameworks.

10.4 Exercises

1. Use SymPy to compute the gradient of f(x, y, z) = x*y + y*z + z*x.
2. Implement numerical gradient for f(x, y) = sin(x) + cos(y) and verify against analytical gradient.
3. Implement gradient descent for a simple linear regression problem using numerical derivatives.
4. Experiment with different learning rates and observe convergence behavior.

Hints / Answers

1. Partial derivatives using SymPy are obtained using sp.diff.
2. Numerical gradients should approximate analytical gradients closely for small epsilon.
3. Gradient descent converges faster for moderate learning rates; too high causes overshooting.

10.5 Further Reading & Resources

• SymPy documentation — https://www.sympy.org/en/index.html
• NumPy documentation — for numerical operations and array handling.
• Deep Learning Book (Goodfellow et al.) — Chapters on optimization and gradients.
• Hands-on ML tutorials — implement custom gradient descent using Python and NumPy.

Next topic: Advanced Optimization Techniques — exploring adaptive learning rates, stochastic gradient descent, and their applications in deep learning models.