Python Decorators
Decorators
Use cases for decorators in Python
Core built-in decorators
- @classmethod
- @staticmethod.
- TODO @property
- TODO @getter, @setter, @deleter
Standard libraries
- TODO functools
- @cache
- @cached_property
- @lru_cache
- @total_ordering
- @singledispatch
- @wraps
- TODO dataclasses
- TODO https://docs.python.org/3/library/contextlib.html#contextlib.contextmanager
- TODO https://docs.python.org/3/library/abc.html#abc.abstractmethod
- TODO https://docs.python.org/3/library/enum.html#enum.unique
- TODO https://docs.python.org/3/library/atexit.html#atexit.register
Some Libraries
Other uses
- Logging calls with parameters.
- Logging elapsed time of calls.
- Access control in Django or other web frameworks. (e.g. login required)
- Memoization (caching)
- Retry
- Function timeout
- Locking for thread safety
- Decorator Library
Decorators: simple example
- A decorator is that
@somethingjust before the declaration of the function. - Decorators can modify the behavior of functions or can set some meta information about them.
- In this book first we’ll see a few examples of existing decorators of Python and 3rd party libraries.
- Then we’ll learn when and how to create our own decorators.
- Then we’ll take a look at the implementation of some of the well-known decorators.
@some_decorator
def some_function():
pass
Decorators - Pytest
- In Pytest we can use decorators to add special marks to test functions
- … or to mark them as fixtures.
import sys
import pytest
@pytest.mark.skipif(sys.platform != 'linux', reason="Linux tests")
def test_linux():
assert True
@pytest.mark.skip(reason="To show we can skip tests without any condition.")
def test_any():
assert True
@pytest.fixture(autouse = True, scope="module")
def module_demo():
print(f"Fixture")
$ pytest -v test_with_decorator.py
Decorators - Flask
In Flask we use decorators to map pathes to functions and make them “routes”.
from flask import Flask
app = Flask(__name__)
@app.get("/")
def main():
return "Hello World!"
@app.get("/login")
def login():
return "Showing the login page ..."
$ flask --app flask_app run
Testing Flask
Throughout this book we’ll have various examples. To make sure they work properly we’ll also have tests for these examples. This is the test for the example with Flask. It uses pytest and it has another example of a decorator. Here we decorate a function to become a fixture.
import pytest
import flask_app
@pytest.fixture()
def web():
return flask_app.app.test_client()
def test_main_page(web):
rv = web.get('/')
assert rv.status == '200 OK'
assert b'Hello World!' == rv.data
def test_main_page(web):
rv = web.get('/login')
assert rv.status == '200 OK'
assert b'Showing the login page ...' == rv.data
Core built-in decorators
OOP - classmethod - staticmethod
class Person(object):
def __init__(self, name):
print(f"init: '{self}' '{self.__class__.__name__}'")
self.name = name
def show_name(self):
print(f"instance method: '{self}' '{self.__class__.__name__}'")
@classmethod
def from_occupation(cls, occupation):
print(f"class method '{cls}' '{cls.__class__.__name__}'")
@staticmethod
def is_valid_occupation(param):
print(f"static method '{param}' '{param.__class__.__name__}'")
fb = Person('Foo Bar')
fb.show_name()
fb.from_occupation('Tailor')
Person.from_occupation('Tailor') # This is how we should call it.
fb.is_valid_occupation('Tailor')
Person.is_valid_occupation('Tailor')
init: '<__main__.Person object at 0x7fb008f3a640>' 'Person'
instance method: '<__main__.Person object at 0x7fb008f3a640>' 'Person'
class method '<class '__main__.Person'>' 'type'
class method '<class '__main__.Person'>' 'type'
static method 'Tailor' 'str'
static method 'Tailor' 'str'
Functools
functools provieds a number of decorators.
Decorators caching - no cache
If we have a function that for a given set of parameters always returns the same result (so no randomness, no time dependency, no persistent part) then we might be able to trade some memory to gain some speed. We could use a cache to remember the result the first time we call a functions and return the same result without doing the computation for every subsequent call.
First let’s see a case without cache. Each call will execute the function and do the (expensive) computation.
def compute(x, y):
print(f"Called with {x} and {y}")
# some long computation here
return x+y
if __name__ == '__main__':
print(compute(2, 3))
print(compute(3, 4))
print(compute(2, 3))
Called with 2 and 3
5
Called with 3 and 4
7
Called with 2 and 3
5
from no_cache import compute
def test_compute(capsys):
assert compute(2, 3) == 5
out, err = capsys.readouterr()
assert err == ''
assert out == 'Called with 2 and 3\n'
assert compute(3, 4) == 7
out, err = capsys.readouterr()
assert err == ''
assert out == 'Called with 3 and 4\n'
assert compute(2, 3) == 5
out, err = capsys.readouterr()
assert err == ''
assert out == 'Called with 2 and 3\n'
Decorators caching - with cache - lru_cache
- By adding the lru_cache decorator we can tell Python to cache the result and save on computation time.
import functools
@functools.lru_cache()
def compute(x, y):
print(f"Called with {x} and {y}")
# some long computation here
return x+y
if __name__ == "__main__":
print(compute(2, 3))
print(compute(3, 4))
print(compute(2, 3))
Called with 2 and 3
5
Called with 3 and 4
7
5
from with_lru_cache import compute
def test_compute(capsys):
assert compute(2, 3) == 5
out, err = capsys.readouterr()
assert err == ''
assert out == 'Called with 2 and 3\n'
assert compute(3, 4) == 7
out, err = capsys.readouterr()
assert err == ''
assert out == 'Called with 3 and 4\n'
assert compute(2, 3) == 5
out, err = capsys.readouterr()
assert err == ''
assert out == '' # compute is not called!
LRU - Least recently used cache
- LRU - Cache replacement policy.
- When we call the function with (1, 5) it removes the least recently used results of (1, 2).
- So next time it has to be computed again.
import functools
@functools.lru_cache(maxsize=3)
def compute(x, y):
print(f"Called with {x} and {y}")
# some long computation here
return x+y
if __name__ == "__main__":
compute(1, 2) # Called with 1 and 2
compute(1, 2)
compute(1, 2)
compute(1, 3) # Called with 1 and 3
compute(1, 3)
compute(1, 4) # Called with 1 and 4
compute(1, 4)
compute(1, 5) # Called with 1 and 5
compute(1, 2) # Called with 1 and 2
compute(1, 2)
from lru_cache_example_1 import compute
def test_compute(check_out):
compute.cache_clear()
compute(1, 2)
check_out("Called with 1 and 2\n")
compute(1, 2)
check_out("")
compute(1, 2)
check_out("")
compute(1, 3)
check_out("Called with 1 and 3\n")
compute(1, 3)
check_out("")
compute(1, 4)
check_out("Called with 1 and 4\n")
compute(1, 4)
check_out("")
compute(1, 5)
check_out("Called with 1 and 5\n")
# This is called again as the last addition pushed this out from the cache
compute(1, 2)
check_out("Called with 1 and 2\n")
compute(1, 2)
check_out("")
assert compute.cache_info().hits == 5
assert compute.cache_info().misses == 5
assert compute.cache_info().maxsize == 3
assert compute.cache_info().currsize == 3
LRU - Least recently used cache
- Here we called (1, 2) after (1, 4) when it was still in the cache
- When we called (1, 5) it removed the LRU pair, but it was NOT the (1, 2) pair
- So it was in the cache even after the (1, 5) call.
import functools
@functools.lru_cache(maxsize=3)
def compute(x, y):
print(f"Called with {x} and {y}")
# some long computation here
return x+y
if __name__ == "__main__":
compute(1, 2) # Called with 1 and 2
compute(1, 2)
compute(1, 2)
compute(1, 3) # Called with 1 and 3
compute(1, 3)
compute(1, 4) # Called with 1 and 4
compute(1, 4)
compute(1, 2)
compute(1, 5) # Called with 1 and 5
compute(1, 2)
from lru_cache_example_1 import compute
def test_compute(check_out):
compute.cache_clear()
compute(1, 2)
check_out("Called with 1 and 2\n")
compute(1, 2)
check_out("")
compute(1, 2)
check_out("")
compute(1, 3)
check_out("Called with 1 and 3\n")
compute(1, 3)
check_out("")
compute(1, 4)
check_out("Called with 1 and 4\n")
compute(1, 4)
check_out("")
compute(1, 2)
check_out("")
compute(1, 5)
check_out("Called with 1 and 5\n")
# This is now in the cache
compute(1, 2)
check_out("")
# This is called again as the last addition pushed this out from the cache
compute(1, 3)
check_out("Called with 1 and 3\n")
assert compute.cache_info().hits == 6
assert compute.cache_info().misses == 5
assert compute.cache_info().maxsize == 3
assert compute.cache_info().currsize == 3
Functions and closures
Before we learn how decorators work let’s remember a few things about functions in Python.
Function assignment
We can assign functions to variable and then use the new variable as the original function. We effectively create an alias.
Note, we did not call the hello function when we assigned it it greet.
def hello(name):
print(f"Hello {name}")
if __name__ == "__main__":
hello("Python")
print(hello)
greet = hello
greet("Rust")
print(greet)
Hello Python
<function hello at 0x7f8aee3401f0>
Hello Rust
<function hello at 0x7f8aee3401f0>
from function_assignment import hello
def test_assignment(check_out):
hello("Python")
check_out("Hello Python\n")
assert hello.__name__ == "hello"
greet = hello
greet("Rust")
check_out("Hello Rust\n")
assert greet.__name__ == "hello"
Function assignment - alias print to say
It looks more useful when we shorten the name of a function.
say = print
say("Hello World")
Function assignment - don’t do this
One can go crazy and assign a function to another existing function. Then the old exiting function is gone and it now does something completely different.
It is probably not a very good idea to do this.
numbers = [2, 4, 3, 1, 1, 1]
print(sum(numbers)) # 12
print(max(numbers)) # 4
sum = max
print(sum(numbers)) # 4
print(max(numbers)) # 4
sum = lambda values: len(values)
print(sum(numbers)) # 6
Passing functions as parameters
Maybe assigning one function to a variable just to shorten the name is not for you, but this capability allows us to pass a function as a parameter to another function.
def call(func):
return func(42)
def double(val):
return 2 * val
def square(val):
return val * val
if __name__ == "__main__":
print(call(double)) # 84
print(call(square)) # 1764
print(call(lambda x: x // 2)) # 21
from passing_function import call, double, square
def test_call():
assert double(3) == 6
assert call(double) == 84
assert square(2) == 4
assert call(square) == 1764
TODO: prepare a more useful example!
Traversing directory tree
Here we created our own directory tree-walker that gets a path to a folder and a function assigned to the todo variable. It then traverses the tree and calls the function on every item. This is another example wewhere one function accepts another function as a parameter.
import sys
import os
def walker(path, todo):
if os.path.isdir(path):
items = os.listdir(path)
for item in items:
walker(os.path.join(path, item), todo)
else:
todo(path)
def print_size(name):
print(f"{os.stat(name).st_size:6} {name} ")
if __name__ == '__main__':
if len(sys.argv) < 2:
exit(f"Usage: {sys.argv[0]} PATH")
walker(sys.argv[1], print)
#walker(sys.argv[1], print_size)
#walker(sys.argv[1], lambda name: print(f"{os.stat(name).st_size:6} {name[::-1]} "))
Declaring Functions inside other function
Let’s also remember that we can define a function inside another function and then the internally defined function only exists in the scope of the function where it was defined in. Not outside.
def f():
def g():
print("in g")
print("start f")
g()
print("end f")
f()
g()
start f
in g
end f
Traceback (most recent call last):
File "examples/decorators/function_in_function.py", line 9, in <module>
g()
NameError: name 'g' is not defined
Returning a new function from a function
As we can pass a function as a parameter to a function, we can also return a function from another one.
Combining it with the previouse example, iniside our create_function we define a new function. Normally it only exists inside the create_function, but we can return it to the caller and then it stays around.
def create_function():
print("creating a function")
def internal():
print("This is the generated function")
print("creation done")
return internal
func = create_function()
func()
creating a function
creation done
This is the generated function
Returning a closure
In this example the internally created function depends on a parameter the create_incrementer received.
This parameter will go out of scope at the end of the create_incrementer function, but because it is used inside the internal function which was returned the caller, inside it will stay alive.
This is called a closure and it can be extremly useful in certain cases.
def create_incrementer(num):
def inc(val):
return num + val
return inc
inc_5 = create_incrementer(5)
inc_7 = create_incrementer(7)
if __name__ == "__main__":
print(inc_5(10)) # 15
print(inc_5(0)) # 5
print(inc_7(10)) # 17
print(inc_7(0)) # 7
from incrementer import inc_5, inc_7
def test_inc_5():
assert inc_5(1) == 6
assert inc_5(-5) == 0
def test_inc_7():
assert inc_7(1) == 8
assert inc_7(-5) == 2
Decorator
- A function that changes the behaviour of other functions.
- The input of a decorator is a function.
- The returned value of a decorator is a modified version of the same function.
Normally a decorator is used with the @ prefix just above the declaration of a function:
from some_module import some_decorator
@some_decorator
def f(...):
...
However, that syntax is only to make it look nice. In reality it is basically the same as this code:
def f(...):
...
f = some_decorator(f)
A simple function - use as it is
This is just a simple function. We can call it. It is just an example.
import time
def myfunc():
print("myfunc started")
time.sleep(1)
print("myfunc ended")
myfunc()
The output looks simple:
myfunc started
myfunc ended
wrapper
We created a wrapper function called wrap that receives a function as a parameter.
Inside it creates a new funcion called new_function. (I am really not very creative with names.)
The return value of the wrap function is this `new_function.
The new_function first prints something on the screen and saves the current time.
Then it calls the function that was received as a parameter.
Then gets the current time again and prints the elapsed time.
That’s the new_function
On the next pages we’ll see how we can use this function.
import time
def wrap(func):
def new_function():
print(f"start new '{func.__name__}'")
start = time.time()
func()
end = time.time()
print(f"end new '{func.__name__}' {end-start}")
return new_function
Use wrapper as a function
We can take any arbirary fuction, for example the myfunc, pass it to the wrap function
and assign the returned value back to the myfunc name. This will replace the original myfunc by one returned by the wrap function.
So far myfunc was not called.
Then we call the new myfunc.
This will basically call the new_function that will call the original myfunc.
We did not use any “decoration” for this. Just plain function calls.
from wrapper import wrap
import time
def myfunc():
print("myfunc started")
time.sleep(1)
print("myfunc ended")
myfunc = wrap(myfunc)
myfunc()
$ python use_wrapper.py
start new 'myfunc'
myfunc started
myfunc ended
end new 'myfunc' 1.0002148151397705
Use wrapper as a decorator
We can get the same result by putting @wrap as a decorator above the definition of our function.
from wrapper import wrap
import time
@wrap
def myfunc():
print("myfunc started")
time.sleep(1)
print("myfunc ended")
myfunc()
Decorator to register function
- Pytest, Flask probably do this
functions = []
def register(func):
global functions
functions.append(func.__name__)
return func
@register
def f():
print("in f")
print(functions)
A recursive Fibonacci
def fibo(n):
if n in (1,2):
return 1
return fibo(n-1) + fibo(n-2)
print(fibo(5)) # 5
trace fibo
import decor
@decor.tron
def fibo(n):
if n in (1,2):
return 1
return fibo(n-1) + fibo(n-2)
print(fibo(5))
Calling fibo(5)
Calling fibo(4)
Calling fibo(3)
Calling fibo(2)
Calling fibo(1)
Calling fibo(2)
Calling fibo(3)
Calling fibo(2)
Calling fibo(1)
5
tron decorator
def tron(func):
def new_func(v):
print(f"Calling {func.__name__}({v})")
return func(v)
return new_func
Decorate with direct call
import decor
def fibo(n):
if n in (1,2):
return 1
return fibo(n-1) + fibo(n-2)
fibo = decor.tron(fibo)
print(fibo(5))
Decorate with parameter
import decor_param
@decor_param.tron('foo')
def fibo(n):
if n in (1,2):
return 1
return fibo(n-1) + fibo(n-2)
print(fibo(5))
foo Calling fibo(5)
foo Calling fibo(4)
foo Calling fibo(3)
foo Calling fibo(2)
foo Calling fibo(1)
foo Calling fibo(2)
foo Calling fibo(3)
foo Calling fibo(2)
foo Calling fibo(1)
5
Decorator accepting parameter
def tron(prefix):
def real_tron(func):
def new_func(v):
print("{} Calling {}({})".format(prefix, func.__name__, v))
return func(v)
return new_func
return real_tron
Decorate function with any signature
- How can we decorate a function that is flexible on the number of arguments?
- Accept
*argsand**kwargsand pass them on.
from decor_any import tron
@tron
def one(param):
print(f"one({param})")
@tron
def two(first, second = 42):
print(f"two({first}, {second})")
one("hello")
one(param = "world")
two("hi")
two(first = "Foo", second = "Bar")
Decorate function with any signature - implementation
def tron(func):
def new_func(*args, **kw):
params = list(map(lambda p: str(p), args))
for (k, v) in kw.items():
params.append(f"{k}={v}")
print("Calling {}({})".format(func.__name__, ', '.join(params)))
return func(*args, **kw)
return new_func
Calling one(hello)
one(hello)
Calling one(param=world)
one(world)
Calling two(hi)
two(hi, 42)
Calling two(first=Foo, second=Bar)
two(Foo, Bar)
Decorate function with any signature - skeleton
def decorator(func):
def wrapper(*args, **kw):
return func(*args, **kw)
return wrapper
@decorator
def zero():
print("zero")
@decorator
def one(x):
print(f"one({x})")
@decorator
def two(x, y):
print(f"two({x, y})")
zero()
one('hello')
two( y = 7, x = 8 )
print(zero)
print(one)
print(two)
print(zero.__name__)
print(one.__name__)
print(two.__name__)
zero
one(hello)
two((8, 7))
<function decorator.<locals>.wrapper at 0x7f1165258a60>
<function decorator.<locals>.wrapper at 0x7f1165258b80>
<function decorator.<locals>.wrapper at 0x7f1165258ca0>
Decorate function with any signature - skeleton with name
import functools
def decorator(func):
@functools.wraps(func)
def wrapper(*args, **kw):
return func(*args, **kw)
return wrapper
@decorator
def zero():
print("zero")
@decorator
def one(x):
print(f"one({x})")
@decorator
def two(x, y):
print(f"two({x, y})")
zero()
one('hello')
two( y = 7, x = 8 )
print(zero)
print(one)
print(two)
print(zero.__name__)
print(one.__name__)
print(two.__name__)
zero
one(hello)
two((8, 7))
<function zero at 0x7f9079bdca60>
<function one at 0x7f9079bdcb80>
<function two at 0x7f9079bdcca0>
Functool - partial
- partial
from functools import partial
val = '101010'
print(int(val, base=2))
basetwo = partial(int, base=2)
basetwo.__doc__ = 'Convert base 2 string to an int.'
print(basetwo(val))
# Based on example from https://docs.python.org/3/library/functools.html
Exercise: Logger decorator
- In the previous pages we created a decorator that can decorate arbitrary function logging the call and its parameters.
- Add time measurement to each call to see how long each function took.
Exercise: decorators decorator
Write a function that gets a functions as attribute and returns a new functions while memoizing (caching) the input/output pairs. Then write a unit test that checks it. You probably will need to create a subroutine to be decoratorsd.
- Write tests for the fibonacci functions.
- Implement the decorators decorator for a function with a single parameter.
- Apply the decorator.
- Run the tests again.
- Check the speed differences.
- or decorate with tron to see the calls…
Solution: Logger decorator
import time
def tron(func):
def new_func(*args, **kwargs):
start = time.time()
print("Calling {}({}, {})".format(func.__name__, args, kwargs))
out = func(*args, **kwargs)
end = time.time()
print("Finished {}({})".format(func.__name__, out))
print("Elapsed time: {}".format(end - start))
return out
return new_func
Solution: Logger decorator (testing)
from logger_decor import tron
@tron
def f(a, b=1, *args, **kwargs):
print('a: ', a)
print('b: ', b)
print('args: ', args)
print('kwargs:', kwargs)
return a + b
f(2, 3, 4, 5, c=6, d=7)
print()
f(2, c=5, d=6)
print()
f(10)
Calling f((2, 3, 4, 5), {'c': 6, 'd': 7})
a: 2
b: 3
args: (4, 5)
kwargs: {'c': 6, 'd': 7}
Finished f(5)
Elapsed time: 1.3589859008789062e-05
Calling f((2,), {'c': 5, 'd': 6})
a: 2
b: 1
args: ()
kwargs: {'c': 5, 'd': 6}
Finished f(3)
Elapsed time: 5.245208740234375e-06
Calling f((10,), {})
a: 10
b: 1
args: ()
kwargs: {}
Finished f(11)
Elapsed time: 4.291534423828125e-06
Solution decorators decorator
import sys
import memoize_attribute
import memoize_nonlocal
import decor_any
#@memoize_attribute.memoize
#@memoize_nonlocal.memoize
#@decor_any.tron
def fibonacci(n):
if n == 1:
return 1
if n == 2:
return 1
return fibonacci(n-1) + fibonacci(n-2)
if __name__ == '__main__':
if len(sys.argv) != 2:
sys.stderr.write("Usage: {} N\n".format(sys.argv[0]))
exit(1)
print(fibonacci(int(sys.argv[1])))
def memoize(f):
data = {}
def caching(n):
nonlocal data
key = n
if key not in data:
data[key] = f(n)
return data[key]
return caching
def memoize(f):
def caching(n):
key = n
#if 'data' not in caching.__dict__:
# caching.data = {}
if key not in caching.data:
caching.data[key] = f(n)
return caching.data[key]
caching.data = {}
return caching
Before
$ time python fibonacci.py 35
9227465
real 0m3.850s
user 0m3.832s
sys 0m0.015s
After
$ time python fibonacci.py 35
9227465
real 0m0.034s
user 0m0.019s
sys 0m0.014s
A list of functions
def hello(name):
print(f"Hello {name}")
def morning(name):
print(f"Good morning {name}")
hello("Jane")
morning("Jane")
print()
funcs = [hello, morning]
funcs[0]("Peter")
print()
for func in funcs:
func("Mary")
Hello Jane
Good morning Jane
Hello Peter
Hello Mary
Good morning Mary
Insert element in sorted list using insort
- insort
import bisect
solar_system = ['Earth', 'Jupiter', 'Mercury', 'Saturn', 'Venus']
name = 'Mars'
# Find the location where to insert the element to keep the list sorted and insert the element
bisect.insort(solar_system, name)
print(solar_system)
print(sorted(solar_system))
import sys
import os
def traverse(path):
if os.path.isfile(path):
print(path)
return
if os.path.isdir(path):
for item in os.listdir(path):
traverse(os.path.join(path, item))
return
# other unhandled things
if len(sys.argv) < 2:
exit(f"Usage: {sys.argv[0]} DIR|FILE")
traverse(sys.argv[1])
import sys
import os
def traverse(path, func):
response = {}
if os.path.isfile(path):
func(path)
return response
if os.path.isdir(path):
for item in os.listdir(path):
traverse(os.path.join(path, item), func)
return response
# other unhandled things
if len(sys.argv) < 2:
exit(f"Usage: {sys.argv[0]} DIR|FILE")
#traverse(sys.argv[1], print)
#traverse(sys.argv[1], lambda path: print(f"{os.path.getsize(path):>6} {path}"))
import sys
import os
def traverse(path, func):
if os.path.isfile(path):
func(path)
return
if os.path.isdir(path):
for item in os.listdir(path):
traverse(os.path.join(path, item), func)
return
# other unhandled things
if len(sys.argv) < 2:
exit(f"Usage: {sys.argv[0]} DIR|FILE")
#traverse(sys.argv[1], print)
#traverse(sys.argv[1], lambda path: print(f"{os.path.getsize(path):>6} {path}"))
#from inspect import getmembers, isfunction
import inspect
def change(sub):
def new(*args, **kw):
print("before")
res = sub(*args, **kw)
print("after")
return res
return new
def add(x, y):
return x+y
#print(add(2, 3))
fixed = change(add)
#print(fixed(3, 4))
def replace(subname):
def new(*args, **kw):
print("before")
res = locals()[subname](*args, **kw)
print("after")
return res
locals()[subname] = new
replace('add')
add(1, 7)
def say():
print("hello")
#print(dir())
#getattr('say')