{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using and creating Python packages\n",
    "\n",
    "This tutorial will show how to use Python packages from Nixpkgs and create new ones. For reference material on Python in Nixpkgs, see the [Python section in the Nixpkgs manual](https://nixos.org/nixpkgs/manual#python)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python interpreters\n",
    "\n",
    "Nixpkgs contains recipes for various Python interpreters. Multiple versions are available of the official reference implementation also known as CPython. Interpreters can be installed into a profile, used as part of a build or used temporary in a `nix-shell` or `nix run` session. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python applications, libraries and environments\n",
    "\n",
    "In Nixpkgs a distinction is made between applications, libraries and environments. \n",
    "- Applications are programs for end-users such as Calibre the e-book manager. \n",
    "- Libraries are packages providing importable modules. Additionally, tools used for development of libraries are also considered libraries.\n",
    "- Environments are a Python interpreter along with libraries, typically used for development or as dependency for other programs.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python application\n",
    "\n",
    "Let's consider first a Python application, the version control system Mercurial. The following recipe shows how to build it. For build and run-time dependencies we use the recipes provided by Nixpkgs. We then create a builder function for building our package. Finally, we call it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%file default.nix\n",
    "\n",
    "let\n",
    "  nixpkgs = fetchTarball \"channel:nixos-20.03\";\n",
    "  pkgs = import nixpkgs {};\n",
    "\n",
    "  recipe = { python3, fetchurl }: \n",
    "    with python3.pkgs;\n",
    "        \n",
    "    buildPythonApplication rec {\n",
    "      pname = \"mercurial\";\n",
    "      version = \"5.2.2\";\n",
    "\n",
    "      src = fetchurl {\n",
    "        url = \"https://mercurial-scm.org/release/mercurial-${version}.tar.gz\";\n",
    "        sha256 = \"0fy00q0k4f0q64jjlnb7cl6m0sglivq9jgdddsp5sywc913zzigz\";\n",
    "      };\n",
    "    \n",
    "      # By default tests are executed, but they need to be invoked differently for this package\n",
    "      dontUseSetuptoolsCheck = true;\n",
    "    };\n",
    "\n",
    "in pkgs.callPackage recipe {}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's build it. Note we suppress the build-output because of the amount."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! nix-build --no-build-output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now execute `hg` directly from the store path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! result/bin/hg --version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or install it in our profile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! nix-env -if ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "bringing the executable onto `$PATH`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! hg --version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating a basic recipe for Mercurial was easy. The recipe used in Nixpkgs is however more elaborate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! nix run -f channel:nixos-20.03 curl --command curl --silent https://raw.githubusercontent.com/NixOS/nixpkgs/nixos-20.03/pkgs/applications/version-management/mercurial/default.nix | pygmentize"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python library\n",
    "\n",
    "Python uses packages to distribute libraries called modules that are importable. In Nixpkgs all Python 3 libraries are in the attribute set `python3.pkgs`. The following example shows a recipe for the Python package `toolz`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%file toolz.nix\n",
    "\n",
    "{ lib, buildPythonPackage, fetchPypi }:\n",
    "\n",
    "buildPythonPackage rec {\n",
    "  pname = \"toolz\";\n",
    "  version = \"0.10.0\";\n",
    "\n",
    "  src = fetchPypi {\n",
    "    inherit pname version;\n",
    "    sha256 = \"08fdd5ef7c96480ad11c12d472de21acd32359996f69a5259299b540feba4560\";\n",
    "  };\n",
    "\n",
    "  doCheck = false;\n",
    "\n",
    "  meta = with lib; {\n",
    "    homepage = https://github.com/pytoolz/toolz;\n",
    "    description = \"List processing tools and functional utilities\";\n",
    "    license = licenses.bsd3;\n",
    "    maintainers = with maintainers; [ fridh ];\n",
    "  };\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Applications or other libraries may want to use this package. In order to make that possible, we need to update the Python packages set to include this package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%file mypython.nix\n",
    "\n",
    "let\n",
    "  nixpkgs = fetchTarball \"channel:nixos-20.03\";\n",
    "  pkgs = import nixpkgs {};\n",
    "\n",
    "  mypython = pkgs.python3.override {\n",
    "    self = mypython;\n",
    "    packageOverrides = pself: psuper: {\n",
    "      toolz = pself.callPackage ./toolz.nix {};\n",
    "    };\n",
    "  };\n",
    "in {\n",
    "  inherit mypython;\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have a Python interpreter that has a package set that includes our package. Using this type of overriding we can also inject modifications to existing packages that will then be used by all dependents in the set.\n",
    "\n",
    "Let's build the package. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! nix-build -A mypython.pkgs.toolz mypython.nix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The log shows the execution of several hooks that are responsible for building, installing and testing the package."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Environments\n",
    "\n",
    "Now that we know how to build a Python library, we want to use it in an environment, for example a development environment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following expression creates a Python environment, quite similar to a `virtualenv` or `venv` that includes the interpreter, our package and another package, the test runner `pytest`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%file myenv.nix\n",
    "\n",
    "(import ./mypython.nix).mypython.withPackages(ps: with ps; [ toolz pytest ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! nix run -f myenv.nix --command python3 -c 'import toolz; print(toolz); import pytest; print(pytest)'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `toolz` module, just like the `pytest` module can now be imported."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How does an environment like this look like? If we build the expression we get the link to the store path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! find $(nix-build myenv.nix) | tail"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The environment is essentially a bunch of symbolic links to the packages we've included, as well as the interpreter. Executables are also wrapped so that all tools can find each other."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}