Introduction to Python Decorators

Introduction

I'll start off by admitting, decorators are hard! Some of the code you will see in this tutorial will be necessarily complicated. Most people seem to struggle with decorators at least for a while so don't feel disheartened if this looks weird to you. But then most people can get over that struggle. In this tutorial, I'll walk you slowly through the process of understanding decorators. I will assume that you can write basic functions and basic classes. If you can't do these things I suggest you learn how to before coming back here (unless if you are lost, in which case you are excused).

A Use Case: Timing Function Execution

Let's assume we are executing a piece of code that is taking a bit longer to execute than we would like. The piece of code is made up of a bunch of function calls and we are convinced that at least one of those calls constitutes a bottleneck in our code. How do we find the bottleneck? One solution, the solution we will focus on now, is to time function execution.

Let's start with a simple example. We have just one function that we want to time, func_a

def func_a(stuff):
    do_important_things_1()
    do_important_things_2()
    do_important_things_3()

One way to proceed would be to put our timing code around each function call. So this:

func_a(current_stuff)

will look a little more like this:

before = datetime.datetime.now()
func_a(current_stuff)
after = datetime.datetime.now()
print "Elapsed Time = {0}".format(after-before)

That will work just fine. But what happens if we have multiple calls to func_a and we want to time all of them? We could surround every call to func_a with our timing code, but that has a bad smell to it. It would be ready to write the timing code only once. So instead of putting it outside the function, we put it inside the function definition.

def func_a(stuff):
    before = datetime.datetime.now()
    do_important_things_1()
    do_important_things_2()
    do_important_things_3()
    after = datetime.datetime.now()
    print "Elapsed Time = {0}".format(after-before)

The benefits of this approach are:

1. We have the code in one place so if we want to change it (for example if we want to store the elapsed time in a database or log) then we need to change it in only one place instead of at every single function call
2. We don't have to remember to write four lines of code instead of one every time we call func_a which is just an all round good thing

Alright, but needing to time just one function is not so realistic. If you need to time one thing there's a very good chance that you'll need to time at least two things. So we'll go for three

def func_a(stuff):
    before = datetime.datetime.now()
    do_important_things_1()
    do_important_things_2()
    do_important_things_3()
    after = datetime.datetime.now()
    print "Elapsed Time = {0}".format(after-before)

def func_b(stuff):
    before = datetime.datetime.now()
    do_important_things_4()
    do_important_things_5()
    do_important_things_6()
    after = datetime.datetime.now()
    print "Elapsed Time = {0}".format(after-before)
    
def func_c(stuff):
    before = datetime.datetime.now()
    do_important_things_7()
    do_important_things_8()
    do_important_things_9()
    after = datetime.datetime.now()
    print "Elapsed Time = {0}".format(after-before)

This is looking pretty nasty. What if we want to time 8 functions. Then we decide we want to store the timing information in a log file. Then we decide a database will be better. Yuck is the word. What we need here is a way to incorporate the same code into func_a, func_b and func_c in a way that doesn't have us copy pasting code all over the place.

A brief Detour: Functions that Return Functions

Python is a pretty special language in that functions are first class objects. What that means is once a function is defined in a scope it can be passed to functions, assigned to variables, even returned from functions. This simple fact is what makes python decorators possible. Look at the code below and see if you can guess what happens for the lines labelled A, B, C and D.

def get_function():
    print "inside get_function"                 
    def returned_function():                    
        print "inside returned_function"        
        return 1
    print "outside returned_function"
    return returned_function
   
returned_function()     # A                         
x = get_function()      # B                         
x                       # C                        
x()                     # D                     

A

This line gives us a NameError and states that returned_function does not exist. But we just defined it, right? What you need to know here is that it is defined in the scope of get_function. That is, inside get_function it is defined. Outside of get_function it is not. if this confuses you try playing with the locals() function a little bit and read up on Python scoping.

B

This prints the following:

inside get_function
outside returned_function

Python does not execute anything inside returned_function at this point.

C

This line outputs:

<function returned_function at 0x7fdc4463f5f0>

That is, x, the value returned from get_function(), is itself a function.

Try running lines B and C again. Notice that every time you repeat this process the address of the returned returned_function is different. Every time get_function is called it makes a new returned function.

D

Since x is a function, it can be called. Calling x is calling an instance of returned_function. What this outputs is:

inside returned_function
1

That is, it prints the string, and returns the value 1.

Back to the Timing Problem

Still with us? Aren't you cute. Ok, so armed with our new knowledge, how do we solve our old problem? I would suggest we make a function, let's call it time_this, that takes another function as a parameter and wraps the parameter function in some timing code. A little something like:

def time_this(original_function):                            # 1
    def new_function(*args,**kwargs):                        # 2
        before = datetime.datetime.now()                     # 3
        x = original_function(*args,**kwargs)                # 4
        after = datetime.datetime.now()                      # 5
        print "Elapsed Time = {0}".format(after-before)      # 6
        return x                                             # 7
    return new_function()                                    # 8

I admit it is kinda crazy looking, so we'll go through it line by line:

1 This is just the prototype of time_this. time_this is a function just like any other and has one parameter.
2 Inside time_this we are defining a function. Every time time_this executes it will create a new function.
3 Timing code, just like before.
4 We call the original function and keep the result for later.
5,6 The rest of the timing code.
7 The new_function must act just like the original function and so returns the stored result.
8 The function created in time_this is finally returned.

And now we want to make sure that our functions are timed:

def func_a(stuff):
    do_important_things_1()
    do_important_things_2()
    do_important_things_3()
func_a = time_this(func_a)        # <---------

def func_b(stuff):
    do_important_things_4()
    do_important_things_5()
    do_important_things_6()
func_b = time_this(func_b)        # <---------
    
def func_c(stuff):
    do_important_things_7()
    do_important_things_8()
    do_important_things_9()
func_c = time_this(func_c)        # <---------

Looking at func_a, when w execute func_a = time_this(func_a) we replace func_a with the function returned from time_this. So we replace func_A with a function that does some timing stuff (line 3 above), stores the result of func a in a variable called x (line 4), does a little more timing stuff (line 5 and 6) and then returns whatever func_a would have returned anyway. In other words func_a is still called in the same way and returns the same thing, it just gets timed as well. Neat eh?

Introducing Decorators

What we did works fine and is great and stuff, but it's ugly and hard to read. So the lovely authors of Python gave us a different and much prettier way of writing it:

@time_this
def func_a(stuff):
    do_important_things_1()
    do_important_things_2()
    do_important_things_3()

Is exactly equivalent to:

def func_a(stuff):
    do_important_things_1()
    do_important_things_2()
    do_important_things_3()
func_a = time_this(func_a)

That is commonly referred to as syntactic sugar. There is nothing magical about @. It is just a convention that has agreed on. Somewhere along the line it was decided.

Conclusion

So a decorator is just a function that returns a function. If this stuff all looks like crazy-talk to you then the make sure the following topics make sense to you then come back to this tutorial:

• Python functions
• Scope
• Python functions as first class objects (maybe even lookup lambda functions, it might make it easier to understand).

If, on the other hand, you are hungry for more then topics you might find interesting would be:

• Decorating classes eg:

  @add_class_functionality
  class MyClass:
      ...
  

• Decorators with more arguments
  eg:

  @requires_permission(name="edit")
  def save_changes(stuff):
     ...
  

I intend to cover advanced decorator topics in another tutorial. I'll put some links here for you guys once I've done that.
The End.