This post was originally published on http://snoozy.ninja/.

Python is the "most powerful language you can still read" - Paul Dubois

In previous post I discussed tools such as pip and virtualenv. Thanks to them, we can easily install the libraries we need in isolated environments. These core package management tools are in most cases sufficient for standard problems. We can easily install, remove or update the package thanks to them. Virtualenv allows us to easily bypass the problem of keeping different versions of packages in one system.

However, when we actively participate in many different projects, we may encounter further problems:

• How to manage different versions of interpreter in one system?
• How to stick dependencies in a more readable (and safe) way than requirements.txt?
• How to manage a large number of virtual environments in a simple way?

These problems can be solved in many ways, but I would like to introduce you two new tools - pyenv, pipenv. They differ completely in what they offer to developers, but together they will solve the above-mentioned problems.

Pyenv

Pyenv is an easy-to-use tool that is used to install, delete and manage different versions of the Python interpreter on your system. Using a simple interface, we can change the interpreter versions without any problems for our computer (the one who tried to remove python 2.7 from ubuntu and replace it with version 3, knows what I'm talking about). In addition, automatically pyenv makes sure that other commands related to our interpreter, such as pip, refer to their counterparts.

The pyenv installation process itself is not as simple as other tools listed here (especially as we are used to using pip every day ) This is mainly due to the fact that pyenv is a tool written primarily in bash. However, the entire installation process is not complicated, and apart from copying the files, it also requires small changes in our .bashrc (or the equivalent used by us). Let's take a look at how help looks like:

❯ pyenv help                                                                                             mmasztalerczuk.github.io/git/master 
Usage: pyenv <command> [<args>]

Some useful pyenv commands are:
   commands    List all available pyenv commands
   local       Set or show the local application-specific Python version
   global      Set or show the global Python version
   shell       Set or show the shell-specific Python version
   install     Install a Python version using python-build
   uninstall   Uninstall a specific Python version
   rehash      Rehash pyenv shims (run this after installing executables)
   version     Show the current Python version and its origin
   versions    List all Python versions available to pyenv
   which       Display the full path to an executable
   whence      List all Python versions that contain the given executable

Let's see what versions we have installed and use a different version of Python.

Using pyenv versions we can check which versions of the interpreter we currently have under control pyenv (all supported versions can be found usingpyenv install --list). The interpreter is changed using pyenv global x (where x is the version you want to use). The pyenv operating principle is quite simple. When we enter the command, for example python, our terminal starts looking for an executable file in places that were specified in the environment variablePATH. Pyenv simply modifies this variable by adding some folders to the very beginning of PATH variable, in which it holds so-called shims files.

PATH=/home/mariusz/.pyenv/shims:/home/mariusz/.pyenv/bin:/usr/local/bin:/usr/local/sbin

There are very small scripts in the folder named shims, which are called exactly the same as executable files (e.g python). Because the name is the same, they are treated by our shell as what we want to find. They parse arguments and forward them by running pyenv which, using environmental variables, decides on the interpreter version. The whole process is described in more detail in the documentation and I recommend that you read it, because it is quite an interesting solution to the problem.

Thanks to this method, we are able to easily use many python versions with minimal work.

Pipenv

Pipenv is the newest of the tools I have listed (which translates to the fact that development of this tool is active and new functionalities are coming all the time). Pipenv is a tool that combines pip and virtualenv into one. In addition, it supports Pipfile, helps manage environmental variables by automatically loading .env files (a very popular solution for holding values for environment variables). In addition, it has support for pyenv, which means that when creating the environment, we can automatically install the Python version that interests us.

The installation process is simple and you can install pipenv using pip (and then forget that something like pip exists). Let's see what help looks like and briefly discuss the functionalities provided to us (I cut to the most important fragment):

Commands:
  check      Checks for security vulnerabilities and against PEP 508 markers
             provided in Pipfile.
  clean      Uninstalls all packages not specified in Pipfile.lock.
  graph      Displays currently–installed dependency graph information.
  install    Installs provided packages and adds them to Pipfile, or (if none
             is given), installs all packages.
  lock       Generates Pipfile.lock.
  open       View a given module in your editor.
  run        Spawns a command installed into the virtualenv.
  shell      Spawns a shell within the virtualenv.
  sync       Installs all packages specified in Pipfile.lock.
  uninstall  Un-installs a provided package and removes it from Pipfile.
  update     Runs lock, then sync.

Virtual environments

To create a new work environment, we start by running eg the pipenv --python 3.6 command. If we have installed pyenv, then in case of problems with the lack of a specific version of the interpreter, pipenv will take care of downloading and installing it. The new virtual environment that will be created by default, however, will not be located in the project folder but in standard location (please check the documentation for more information about this topic). For me, this is the path .local/share/virtualenvs. This behavior can be changed by setting the 'PIPENV_VENV_IN_PROJECT' flag correctly, then the virtual environment will be created in the same folder as project. In general, I recommend reading the list of environment variables that pipenv uses, because it is quite large and gives us a lot of possibilities to configure the behavior of the application.

It is worth paying attention to the name of the folder with the pipenv_blog-ldLvq6-K virtual environment. The name has been created as combined the name of the folder and hash of the project path. The name was constructed in such a way to uniquely identify the virtual environment for our project. This is a solution which helps us in a situation when we have more projects with the same name. At the moment managing virtual environments is still a bit problematic. Especially when it comes to removing unused folders. We can delete the environment through the pipenv --rm command, but we must be in the folder of a project that is the owner of that virtual env. This is a standard solution and has one disadvantage, unfortunately.

You must remember about this before deleting the project from the disk. Sometimes, if we want to rebuild the environment, we will delete the folder with the project, and then create it again (eg by doing git clone). Unfortunately, in this scenario, the old environment will still be used because the path and name have not been changed. The second problem may also be that the potential change of the folder name or moving it to another location will not automatically result in the existing virtual environment being used. We will have to remember that it is necessary to remove them and create new ones.

The environments themselves can simply be removed manually from where they are held or we can use the pew tool. It is automatically installed with pipenvem and is used by it to manage packages. With the help of pew ls, we can check what currently installed environments we have, then usepew rm to remove those that do not seem to be necessary anymore.

Piplock

Package installation is just as easy. It boils down to the command pipenv install mypy. I will show an example in which we will install two packages, only one of them with the flag --dev.

To understand what really happened, we need to know what Pipfile is and what it is for. Pipfile is such a better alternative to requirements.txt. When using pip very often we have a problem with the development process. Usually, this is done by holding several files as the requirements_dev.txt orrequirements_tests.txt in the project. This is not an ideal solution.

Pipfile solves these problems through its structure. There are actually two files - pipfile and pipfile.lock. In the first of these, we have our packages divided into two sections, in which we can hold the packages needed for the production and development version. By installing requests with the flag --dev we just add this package to this section. The second one, pipfile.lock is a set of hashes that confirm the versions of a given package. Let's look at our pipfile file:

Gits

Of course, this is a very basic file that can be expanded with many things. The syntax for determining the package of interest to us is very complicated (in all it is a language, which is called TOML), however, can very accurately determine what you want to use in the project. Let's take a look at a slightly more complicated example:

Gits

The second file is pipfile.lock. It is a set of hashes that confirm the versions of a given package. Discussing pip I mentioned the --hash flag. Generating hashes from files makes sure that the file is exactly the same file that we used when we originally installed the library. In requirements.txt there was a possibility to store the password information, but it was very impractical, because the readability of such a file was significantly reduced. Pipfile breaks it into two files, so that the things that interest us on a daily basis (the content of pipfile) are very readable. It is also worth mentioning the graph command. It works the same as mentioned in the previous entry pipdeptree. In a more accessible way, it displays the packages installed in our environment, showing the dependencies between them.

In order to be able to run the program in a given environment, we can do it in two ways. We can enter to the given environment using the pipenv shell command. This is exactly the same as what we did using source with venv or virtualenv. The second solution looks very similar to the syntax of the Docker tool. With the help of pipenv run we can run something that will work in the context of our environment.

In addition, the check command is a very interesting functionality, which allows us to easily check the correctness of the markers defining the dependencies of the packages described in PEP 508 (i.e. the syntax used in the pipfile file) and what's more interesting to check if any security fixes to one of our packages appeared. Thanks to this we will know when we have to raise our package to the most current version.

Summary

Pipenv is a very interesting piece of code and by integrating with a few other solutions it is ideally suited as the main tool in the developer's daily python work. Thanks to the fact that it partially supports pyenv we do not have to worry about which interpreter should be used. In addition, the pipfile support, or the functionalities that completely covers the pip application, makes it the most attractive package management tool in python at the moment.

This post was originally published on http://snoozy.ninja/.