Writing a simple Pytest hook

Pytest is my go to Python testing framework due to it's flexibility. One of the great features it has is the ability to write hooks into various points of the test suite execution. These can all be references via the API docs. At work we can have hundreds of tests and sometimes changes can cause tens of tests to fail. Now let's not too far into the "When you make changes only a few tests should break. You shouldn't be seeing 30-40 test failures." Well sure that is great with an ideal test suite but sometimes you're just not going to have that. Or you'll be making a change to a middleware which affects all calls and you have a mass of failures. When you have 30-40 tests failing, if not just 10, you'll find yourself sifting through lots of pytest output to just find the tests that failed. Because of that let's just hook into pytests execution and write all of our test failures to failures.txt so we can see very clearly which of our tests have failed.

Create a new virtualenv and install pytest in it. I am assuming Python3.6 This should be all you need.

Write a simply test that we force to fail in a file tests.py

def test_failed():
    assert False
    
def test_passed():
    assert True

Create a conftest.py and add the following code.

import pytest

@pytest.hookimpl()
def pytest_sessionstart(session):
    print("hello")

Pytest will automatically pick up our hook from conftest.py much like it would with fixtures.

If we run $ pytest tests.py we can see the output as follows. (Deleted some = for brevity)

19:45 $ pytest tests.py 
hello
=======================...=================== test session starts 

Per the api docs we can see that this would be an appropriate hook for creating our failures.txt file so let's do that.

Change the hook as follows

# pathlib is great
from pathlib import Path
from _pytest.main import Session

# Let's define our failures.txt as a constant as we will need it later
FAILURES_FILE = Path() / "failures.txt"

@pytest.hookimpl()
def pytest_sessionstart(session: Session):
    if FAILURES_FILE.exists():
        # We want to delete the file if it already exists
        # so we don't carry over failures form last run
        FAILURES_FILE.unlink()
    FAILURES_FILE.touch()

I like to use pathlib when working with anything on the path just because it's so much more enjoyable than os.path. The logic change is fairly straightforward. If the file already exists delete it, then create a new one.

Next we need to write a hook to append test failures to our failures.txt file. According to the api docs pytest_runtest_makereport would be our desired hook. This hook is ran immediately after the test case is ran and recieves the test case itself (Item) as well as the result of the test case (CallInfo). Add the following to conftest.py

# ...other imports
from _pytest.nodes import Item
from _pytest.runner import CallInfo

# ...previous code

@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item: Item, call: CallInfo):
    # All code prior to yield statement would be ran prior
    # to any other of the same fixtures defined
    
    outcome = yield  # Run all other pytest_runtest_makereport non wrapped hooks
    result = outcome.get_result()
    if result.when == "call" and result.failed:
        try:  # Just to not crash py.test reporting
            with open(str(FAILURE_FILE), "a") as f:
                f.write(result.nodeid + "\n")
        except Exception as e:
            print("ERROR", e)
            pass

Because we don't care to modify the test report itself we will write ours as a wrapped hook via hookwrapper=True. The api docs give's a great example of hook call ordering that is more in-depth than my comments.

At this point if we run $ pytest tests.py we can see that our one failing test is being written to failures.txt

$ cat failures.txt 
tests.py::test_failed

Developers on your team might not even notice this file so let's give them a little direction. Let's add a hook to print directions at the end of test running.
Again looking through the api docs it appears pytest_terminal_summary is our desired hook.

# ... previous imports
from _pytest.runner import CallInfo
from _pytest.terminal import TerminalReporter

# ... previous hooks
@pytest.hookimpl(hookwrapper=True)
def pytest_terminal_summary(
    terminalreporter: TerminalReporter, exitstatus: int, config: Config
):
    yield
    print(f"Failures outputted to: {FAILURE_FILE}")
    print(f"to see run\ncat {FAILURE_FILE}")

In our case I think it makes sense to run all other hooks prior to ours being ran so we can ensure our directions are the last thing printed to the terminal. Again we'll use hookwrapper=True and put our code AFTER the yield statement. In our case we're not doing anything with the the result of the yield so we can simply invoke it and assign it to nothing. With this code change run $ pytest tests.py again and we can see the following.

tests.py:3: AssertionError
Failures outputted to: failures.txt
to see run
cat failures.txt
=====================...== 1 failed, 1 passed in 0.04s=====...===

I hope this provided a good real world example of using writing custom hooks in pytest. Feel free to reach out to me with any questions or feedback regarding this post.