Python Package Management & PyPI

Complete guide to finding, installing, and publishing Python packages

1. Python Package Index (PyPI) Overview
2. Finding & Installing Packages
3. Publishing Packages to PyPI
4. Package Maintenance Best Practices
5. Troubleshooting Common Issues
6. Advanced Topics

1. Python Package Index (PyPI) Overview

What is PyPI?

The Python Package Index (PyPI) is the official repository for Python packages, serving as:

The primary source for pip-installable packages
A hosting platform for over 400,000 packages (as of 2024)
The central distribution point for the Python ecosystem

PyPI Versions

Environment	URL	Purpose
Production	pypi.org	Main package repository
Test	test.pypi.org	Testing/staging environment

Key Components

Package Metadata

Every Python package contains metadata files that describe the package:

pyproject.toml (modern standard)
setup.py (legacy, still widely used)
setup.cfg (legacy configuration)

Distribution Formats

Format	Extension	Description
Source Distribution	`.tar.gz`	Contains raw source code, requires build
Built Distribution (Wheel)	`.whl`	Pre-built package, faster installation

Package Classifiers

Standardized tags that describe package compatibility:

classifiers = [
    "Programming Language :: Python :: 3",
    "License :: OSI Approved :: MIT License",
    "Operating System :: OS Independent",
    "Intended Audience :: Developers",
    "Topic :: Software Development :: Libraries",
]

2. Finding & Installing Packages

Searching PyPI

Web Interface

The primary way to search is via pypi.org/search which offers:

Advanced search filters
Package statistics and metadata
Download history and popularity metrics

Command Line

Note: pip search was disabled due to performance issues. Use web search instead.

# Inspect a package before installing
pip download --no-deps <package>

# View package metadata
pip show <package>

Installation Methods

Command	Purpose
`pip install package`	Latest stable version
`pip install package==1.0.0`	Specific version
`pip install 'package>=1.0.0,<2.0.0'`	Version range
`pip install -e ./path/to/pkg`	Editable/development install
`pip install -r requirements.txt`	Install from requirements file

Advanced Installation

Index URLs

# Use alternative index
pip install --index-url https://pypi.example.com/simple/ package

# Add extra index (fallback)
pip install --extra-index-url https://pypi.example.com/simple/ package

Constraints Files

Lock versions without installing packages:

# constraints.txt
requests==2.25.1
numpy>=1.20.0

# Usage
pip install -c constraints.txt package

Hash Verification

Ensure package integrity:

# requirements.txt with hashes
package==1.0.0 \
    --hash=sha256:2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824

# Install with hash checking
pip install --require-hashes -r requirements.txt

3. Publishing Packages to PyPI

Package Structure Essentials

mypackage/
├── mypackage/ # Actual package
│ ├── __init__.py
│ └── module.py
├── tests/ # Tests (optional)
├── pyproject.toml # Build system config
├── README.md # Project description
└── LICENSE # Legal info

Key Configuration Files

pyproject.toml (Modern Standard)

[build-system]
requires = ["setuptools>=42", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "mypackage"
version = "1.0.0"
authors = [{name = "Your Name", email = "you@example.com"}]
description = "My awesome package"
readme = "README.md"
requires-python = ">=3.8"
license = {text = "MIT"}
classifiers = [
    "Programming Language :: Python :: 3",
    "Operating System :: OS Independent",
]
dependencies = [
    "requests>=2.25.0",
    "numpy>=1.20.0",
]

[project.optional-dependencies]
dev = ["pytest>=6.0.0", "black"]
gui = ["pyqt5>=5.15.0"]

Legacy setup.py (Deprecated but still used)

from setuptools import setup, find_packages

setup(
    name="mypackage",
    version="1.0.0",
    author="Your Name",
    author_email="you@example.com",
    description="My awesome package",
    long_description=open("README.md").read(),
    long_description_content_type="text/markdown",
    packages=find_packages(),
    install_requires=[
        "requests>=2.25.0",
        "numpy>=1.20.0",
    ],
    extras_require={
        "dev": ["pytest>=6.0.0", "black"],
        "gui": ["pyqt5>=5.15.0"],
    },
    python_requires=">=3.8",
    classifiers=[
        "Programming Language :: Python :: 3",
        "Operating System :: OS Independent",
    ],
)

Publishing Steps

1. Build Distributions

# Install build tools
pip install build

# Create distribution packages
python -m build

# Output:
# dist/
#   mypackage-1.0.0.tar.gz
#   mypackage-1.0.0-py3-none-any.whl

2. Upload to PyPI

# Install twine
pip install twine

# Upload to test PyPI first (recommended)
twine upload --repository testpypi dist/*

# Upload to production PyPI
twine upload dist/*

# First time setup:
# 1. Create account at pypi.org
# 2. Generate API token
# 3. Configure ~/.pypirc or use environment variables

Security Note: Always use API tokens instead of passwords for PyPI uploads. Enable 2FA for your account.

4. Package Maintenance Best Practices

Versioning Strategy

Semantic Versioning (SemVer)

Follow the MAJOR.MINOR.PATCH pattern:

MAJOR: Breaking changes
MINOR: Backwards-compatible features
PATCH: Backwards-compatible bug fixes

Version in Code

Define version in your package's __init__.py:

# mypackage/__init__.py
__version__ = "1.0.0"

Dependency Management

Required Dependencies

Specify in pyproject.toml or setup.py:

# pyproject.toml
[project]
dependencies = [
    "requests>=2.25.0",
    "numpy>=1.20.0",
]

Optional Dependencies

# pyproject.toml
[project.optional-dependencies]
dev = ["pytest>=6.0.0", "black"]
gui = ["pyqt5>=5.15.0"]

# Installation
pip install "mypackage[gui]"

Development Setup

# Install package in editable mode with dev dependencies
pip install -e ".[dev]"

Security Considerations

Package Auditing

# Install pip-audit
pip install pip-audit

# Check for vulnerabilities
pip-audit

PyPI Security

Enable 2FA for your PyPI account (mandatory for critical operations)
Use API tokens instead of passwords
Rotate tokens regularly
Review package maintainers and collaborators

5. Troubleshooting Common Issues

Issue	Solution
PackageNotFoundError	Check spelling, try `--index-url`, verify package exists
Version conflicts	Use virtual environments, `pip check`, review dependency tree
Upload fails (403)	Verify PyPI credentials/token, check package name availability
Legacy metadata	Migrate to `pyproject.toml`, update build tools
Build failures	Ensure build dependencies are installed, check Python version compatibility
Slow downloads	Use `--index-url` with mirror, upgrade pip

6. Advanced Topics

Private Package Hosting

Alternate Index Servers

DevPI: Self-hosted PyPI-compatible server
Artifactory/Nexus: Enterprise artifact management
AWS CodeArtifact: Managed service from AWS
Google Artifact Registry: Google Cloud's solution

.pypirc Configuration

# ~/.pypirc
[distutils]
index-servers =
  pypi
  private

[pypi]
username = __token__
password = pypi-YourApiTokenHere

[private]
repository = https://your-private-repo.com
username = your-username
password = your-token

Automated Publishing

GitHub Actions Workflow

# .github/workflows/publish.yml
name: Publish to PyPI

on:
  release:
    types: [published]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    
    - name: Set up Python
      uses: actions/setup-python@v2
      with:
        python-version: '3.x'
    
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install build twine
    
    - name: Build package
      run: python -m build
    
    - name: Publish to PyPI
      run: twine upload dist/*
      env:
        TWINE_USERNAME: __token__
        TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }}

Package Signing

GPG signatures for additional security:

# Generate GPG key if you don't have one
gpg --gen-key

# Upload with signature
twine upload --sign --identity you@example.com dist/*

PyPI | Package Management in Python