Introduction: PyPI and friends

Education objectives

• PyPI

• packages and projects

• sdist, wheels

• pip

• Virtual environments

• PyPA

• pyproject.toml

• isolated build

• Python packaging ecosystem (user perspective)

The Python ecosystem thrives on the sharing of open-source libraries and applications. Python software is commonly organized as projects that provide one or more packages. A package is a directory containing modules (Python files) and sub-packages.

Note

For example, the project scikit-learn (which can be installed with pip install scikit-learn) provides a single package named sklearn (which can be imported with import sklearn).

There is an official and centralized package index called PyPI (Python Package Index). Anyone can upload projects to PyPI. While there are popular and reputable Python projects that serve as pillars of the ecosystem (such as NumPy), being on PyPI is not a guarantee of quality, reliability, or even good intentions.

Historically, Python packaging and dependency handling was problematic. One major issue was the tendency to install everything into the same Python environment and abuse PYTHONPATH. In this training, we present modern installation methods and contemporary Python packaging practices.

Two acceptations of the word package

• In Python, a package is just a directory containing some Python files.

• In packaging, a package is a directory containing all the files needed for installation. Since there are many packaging systems, there are different kinds of packages, for example Debian packages, PyPI packages, Conda packages, …

There is a tendency to use the word package to denote a Python project.

Different situations for package users

We will consider different methods adapted to various use cases:

• Installing Python applications

• Installing Python libraries for direct Python usage (for example in IPython or notebooks)

• Installing Python libraries for a Python script

• Installing Python libraries for package development. In this case, we can distinguish between different types of dependencies:

  • Build dependencies

  • Runtime dependencies

  • Development dependencies

Before presenting practical methods for each of these cases, we need to establish the context.

PyPI and different types of Python projects

The website pypi.org (PyPI) is the official package index for Python. It is also known as the “Cheese Shop”, in reference to a Monty Python sketch.

PyPI hosts a vast number of projects and handles a huge volume of requests. This infrastructure is provided by the PSF (Python Software Foundation).

Note

The Python Software Foundation (PSF) is a non-profit organization that holds the intellectual property rights to Python and manages the development of the language. Beyond maintaining the Python trademark and supporting core development, the PSF provides critical infrastructure like PyPI, organizes and supports PyCon conferences worldwide, and funds grants to support Python communities, educational initiatives, and ecosystem projects.

Project maintainers typically upload two different types of distributions:

• The sdist (source distribution): an archive containing the project’s source code

• Wheels for different operating systems, architectures, and potentially Python versions. Wheels are archives containing compiled projects, potentially including binaries (Python extensions and executables) and Python code files. Wheels are platform-specific and ready to be installed.

The wheel format is quite versatile, allowing for different types of packages: applications or libraries.

Examples of wheel types

Pure Python library (e.g., requests):

requests-2.31.0-py3-none-any.whl

The py3-none-any tag indicates it works with any Python 3 version, any ABI, and any platform.

Pure Python application (e.g., pdm):

pdm-2.12.3-py3-none-any.whl

Similar naming convention, but contains console scripts/entry points for command-line usage.

Python wrapper of a native library (e.g., PySide6):

PySide6-6.6.1-cp39-abi3-manylinux_2_28_x86_64.whl
PySide6-6.6.1-cp39-abi3-macosx_11_0_universal2.whl
PySide6-6.6.1-cp39-abi3-win_amd64.whl

Note

The abi3 tag indicates that these wheels use Python’s stable ABI (Application Binary Interface), introduced in PEP 384. This means a single wheel can work across multiple Python versions (from 3.9 onwards in this example), reducing the number of wheels maintainers need to build and distribute.

Python library with extensions (e.g., numpy):

numpy-1.26.3-cp312-cp312-manylinux_2_17_x86_64.whl
numpy-1.26.3-cp312-cp312-macosx_11_0_arm64.whl
numpy-1.26.3-cp312-cp312-win_amd64.whl

These contain compiled C extensions, so separate wheels are needed for each Python version and platform.

Pip, the official Python package installer

Pip is the official Python package installer, maintained by the Python Packaging Authority (PyPA). It is not designed to install non Python dependencies. pip installs by default from the Python Package Index (PyPI).

# Install a package in the current environment
pip install numpy

# List the packages installed in the current environment
pip list

If a compatible wheel is available, pip uses it. Otherwise, a wheel must be built locally, which can involve compilation (a potentially lengthy and resource-intensive process that requires compilers and libraries).

Virtual environments

For different projects, you may need to use different versions of the same package. Or you may be facing a situation where a package you want to use needs a specific version of another package (an older version than the one you already have installed). To get around potentially messy install, you should use virtual environments.

Virtual environments are central to modern Python installation workflows. A virtual environment is an isolated Python environment with its own installation directories that doesn’t share packages with other virtual environments or the system Python installation. This isolation prevents dependency conflicts between different projects.

Definition: environment

An environment is a set of programs and libraries with particular versions. An environment is defined by the software installed on your computer and by environment variables, in particular the variable PATH, which contains all the paths where your shell looks for programs (you can read the output of echo $PATH).

One can create/use virtual environments. conda can creates conda virtual environments (not specific to Python) and there are also Python virtual environments which can be created with a command like python3 -m venv venv-for-something.

The standard library includes a module for creating virtual environments:

python -m venv name-of-the-venv

The virtual environment can then be “activated”. On Unix systems:

. name-of-the-venv/bin/activate

On Windows (PowerShell):

name-of-the-venv\Scripts\Activate.ps1

Other tools for creating virtual environments

virtualenv: A more feature-rich alternative to the standard library venv module, offering faster environment creation and additional configuration options.

Next generation tools: Project managers like PDM, UV, Poetry, and Hatch integrate virtual environment management with dependency resolution and package building, providing a more comprehensive development workflow.

There are many tools available. Let’s explore why this diversity exists and how it’s possible.

PyPA

The Python Packaging Authority (PyPA) is a working group responsible for the evolution of the packaging ecosystem.

It hosts the development of packaging tools (Pip, Build, setuptools-scm, and others).

The PyPA oversees PEPs related to packaging.

Note

PEPs (Python Enhancement Proposals) are design documents that describe new features or processes for Python. They provide technical specifications and rationale for changes to the language, standard library, or supporting infrastructure. PEPs are the primary mechanism for proposing major changes, gathering community feedback, and documenting design decisions in the Python ecosystem.

The PyPA maintains an official guide at https://packaging.python.org about installing and creating Python packages.

Recent developments related to PEPs 517/518

The current state of Python packaging reflects significant changes introduced by two important PEPs: PEP 518 (pyproject.toml) and PEP 517 (frontends/backends and isolated builds).

PEP 518: pyproject.toml

PEP 518 introduced pyproject.toml as the standard way to describe Python projects and their build requirements. This file uses the TOML format and eliminates the need for setup.py in many cases, providing a cleaner and more declarative approach to project configuration.

PEP 517: Build frontends and backends

PEP 517 introduced a clear separation between:

• Frontends: build and installation tools used by developers and users (Pip, Build, PDM, UV, and others)

• Backends: build systems that create sdists and wheels (setuptools, Flit, Meson, pdm-backend, Hatchling, and others)

This separation enables isolated builds, where build dependencies are installed in a temporary environment, ensuring reproducible builds without polluting the user’s environment.

A diverse ecosystem of tools, both established and emerging

The standardization introduced by PEPs 517/518 has enabled a diverse ecosystem with many choices for PEP 517 frontends (Pip, Build, PDM, UV, and others) and PEP 517 backends (setuptools, Flit, Meson, pdm-backend, and others).

A new generation of tools has emerged: Python project managers (PDM, UV, Hatch, Poetry, and others) inspired by tooling from the NodeJS and Rust ecosystems. These tools integrate multiple aspects of the development workflow: environment management, dependency resolution, building, and publishing.

These modern tools introduce concepts such as declared requirements, freeze files, and lock files (results of dependency resolution). PEP 751 standardizes lock files with the pyproject.lock format.