Quick tip: How I use pip-tools to wrangle dependencies

pip-tools is a Python development tool for helping you ensure you have deterministic and predictable builds. The best way I can think of what that means is by example. Let's say you clone a project to work on a new feature. On the first day you create your virtualenv and install all the dependencies (via requirements.txt). By day 3 you have finished the feature and are ready for release. However in the mean time one of the underlying packages has changed and this causes a break during the build. You never caught this because you had different dependencies locally. pip-tools solves this by compiling a requirements.txt file with all of the packages being used, including underlying depenencies, pinned in the file.

I'm not going to go too in-depth into the various features of pip-tools given I don't use all of them and they are pretty well outlined in the documentation. You can read about it more on their GitHub I want to show how I generally setup my Python application projects when I'm using pip-tools.

Applications have dependencies for running as well as dependencies for developing. I like to have these in separate requirements.in and requirements-dev.in. This keeps the compiled dependencies separate from each other. So for example I would have the following requirements.in

django<3.1
djangorestframework

And then the following in requirements-dev.in.

pip-tools
pytest
pytest-django

I then like to setup a Makefile just to speed up development.

# This is usually my first target so as soon as a developer clones
# a project and creates a virtualenv they can just run `make` to get things going.
install:
  @pip install \
  -r requirements.txt \
  -r requirements-dev.txt

compile:
  @rm -f requirements*.txt
  @pip-compile requirements.in
  @pip-compile requirements-dev.in
    
sync:
  @pip-sync requirements*.txt

Basically I just run make compile && make sync to pull in the newest packages. I'll then usually have a make test target just to verify everything is still working.

I do want to highlight one of my favorite features of pip-tools and that is that it prints where dependencies came from in the requirements.txt files.

django==2.1.15
    # via
    #   -c requirements.txt
    #   django-debug-toolbar

I've found this very useful in tracking down unnecessary dependencies in applications. We once found ourselves installing pandas in applications that never used it. This is a massive package so we were able to track down where it was coming from and fix it. Another time this came in handy was we use to have what effectively became a catch all package. This package was used directly in applications as well as a ton of various underlying packages. It reached throughout our organization. This package caused dependencies such as Django to be installed in applications that weren't even web applications. Most apps barely used any of the package. At one point due to a developer error in the package it took down 3 APIs. The dependency list allowed us to target it's usage and remove it.

I hope you found this useful. Let me know what you think. Also I'm considering doing a short writeup on useful features of Makefiles that I like. Let me know if that is something that interests you as well.