makefun

Dynamically create python functions with a proper signature.

New prepend_args, append_args, and remove_args parameters in @wraps. See below for details !

makefun helps you create functions dynamically, with the signature of your choice. It was largely inspired by decorator and functools, and created mainly to cover some of their limitations.

The typical use cases are:

• creating signature-preserving function wrappers - just like functools.wraps but with accurate TypeError exception raising when user-provided arguments are wrong, and with a very convenient way to access argument values.

• creating function wrappers that have more or less arguments that the function they wrap. A bit like functools.partial but a lot more flexible and friendly for your users. For example, I use it in my pytest plugins to add a requests parameter to users' tests or fixtures when they do not already have it.

• more generally, creating functions with a signature derived from a reference signature,

• or even creating functions with a signature completely defined at runtime.

It currently supports three ways to define the signature of the created function

• from a given reference function, e.g. foo.
• from strings, e.g. 'foo(a, b=1)'
• from Signature objects, either manually created, or obtained by using the inspect.signature (or its backport funcsigs.signature) method.

creating signature-preserving decorators

Creating decorators and creating signature-preserving function wrappers are two independent problems. makefun is solely focused on the second problem. If you wish to solve the first problem you can look at decopatch. It provides a compact syntax, relying on makefun, if you wish to tackle both at once.

Installing

> pip install makefun

Usage

1- Ex-nihilo creation

Let's create a function foo(b, a=0) implemented by func_impl. The easiest way to provide the signature is as a str:

from makefun import create_function

# (1) define the signature. Warning: do not put 'def' keyword here!
func_sig = "foo(b, a=0)"

# (2) define the function implementation
def func_impl(*args, **kwargs):
    """This docstring will be used in the generated function by default"""
    print("func_impl called !")
    return args, kwargs

# (3) create the dynamic function
gen_func = create_function(func_sig, func_impl)

We can test it:

>>> args, kwargs = gen_func(2)
func_impl called !
>>> assert args == ()
>>> assert kwargs == {'a': 0, 'b': 2}

You can also:

• remove the name from the signature string (e.g. '(b, a=0)') to directly use the function name of func_impl.
• override the function name, docstring, qualname and module name if you pass a non-None func_name, doc, qualname and module_name argument
• add other attributes on the generated function if you pass additional keyword arguments

See API reference for details.

Arguments mapping

We can see above that args is empty, even if we called gen_func with a positional argument. This is completely normal: this is because the created function does not expose (*args, **kwargs) but exposes the desired signature (b, a=0). So as for usual python function calls, we lose the information about what was provided as positional and what was provided as keyword. You can try it yourself: write a function def foo(b, a=0) and now try to guess from the function body what was provided as positional, and what was provided as keyword...

This behaviour is actually a great feature because it makes it much easier to develop the func_impl! Indeed, except if your desired signature contains positional-only (not yet available as of python 3.7) or var-positional (e.g. *args) arguments, you will always find all named arguments in **kwargs.

More compact syntax

You can use the @with_signature decorator to perform exactly the same things than create_function, but in a more compact way:

from makefun import with_signature

@with_signature("foo(b, a=0)")
def gen_func(*args, **kwargs):
    """This docstring will be used in the generated function by default"""
    print("func_impl called !")
    return args, kwargs

It also has the capability to take None as a signature, if you just want to update the metadata (func_name, doc, qualname, module_name) without creating any function:

@with_signature(None, func_name='f')
def foo(a):
    return a

assert foo.__name__ == 'f'

See API reference for details.

PEP484 type hints in str

PEP484 type hints are supported in string function definitions:

func_sig = "foo(b: int, a: float = 0) -> str"

PEP484 type comments are also supported:

func_signature = """
foo(b,      # type: int
    a = 0,  # type: float
    ):
    # type: (...) -> str
"""

but unfortunately inspect.signature is not able to detect them so the generated function does not contain the annotations. See this example.

Using Signature objects

create_function and @with_signature are able to accept a Signature object as input, instead of a str. That might be more convenient than using strings to programmatically define signatures. For example we can rewrite the above script using Signature:

from makefun import with_signature
from inspect import Signature, Parameter

# (1) define the signature using objects.
parameters = [Parameter('b', kind=Parameter.POSITIONAL_OR_KEYWORD),
              Parameter('a', kind=Parameter.POSITIONAL_OR_KEYWORD, default=0), ]
func_sig = Signature(parameters)
func_name = 'foo'

# (2) define the function
@with_signature(func_sig, func_name=func_name)
def gen_func(*args, **kwargs):
    """This docstring will be used in the generated function by default"""
    print("func_impl called !")
    return args, kwargs

Note that Signature objects do not contain any function name information. You therefore have to provide an explicit func_name argument to @with_signature (or to create_function) as shown above.

Signature availability in python 2

In python 2 the inspect package does not provide any signature-related features, but a complete backport is available: funcsigs.

2- Deriving from existing signatures

In many real-world applications we want to reuse "as is", or slightly modify, an existing signature.

Copying a signature

If you just want to expose the same signature as a reference function (and not wrap it nor appear like it), the easiest way to copy the signature from another function f is to use signature(f) from inspect/funcsigs.

Signature-preserving function wrappers

@functools.wraps is a famous decorator to create "signature-preserving" function wrappers. However it does not actually preserve the signature, it just uses a trick (setting the __wrapped__ attribute) to trigger special dedicated behaviour in stdlib's help() and signature() methods. See here.

This has two major limitations:

1. the wrapper code will execute even when the provided arguments are invalid.
2. the wrapper code can not easily access an argument using its name, from the received *args, **kwargs. Indeed one would have to handle all cases (positional, keyword, default) and therefore to use something like Signature.bind().

makefun provides a convenient replacement for @wraps that fixes these two issues:

from makefun import wraps

# a dummy function
def foo(a, b=1):
    """ foo doc """
    return a + b

# our signature-preserving wrapper
@wraps(foo)
def enhanced_foo(*args, **kwargs):
    print('hello!')
    print('b=%s' % kwargs['b'])  # we can reliably access 'b'
    return foo(*args, **kwargs)

We can check that the wrapper behaves correctly whatever the call modes:

>>> assert enhanced_foo(1, 2) == 3  # positional 'b'
hello!
b=2
>>> assert enhanced_foo(b=0, a=1) == 1  # keyword 'b'
hello!
b=0
>>> assert enhanced_foo(1) == 2  # default 'b'
hello!
b=1

And let's pass wrong arguments to it: we see that the wrapper is not executed.

>>> enhanced_foo()
TypeError: foo() missing 1 required positional argument: 'a'

You can try to do the same experiment with functools.wraps to see the difference.

Finally note that a create_wrapper function is also provided for convenience ; it is the equivalent of @wraps but as a standard function - not a decorator.

creating signature-preserving decorators

Creating decorators and creating signature-preserving function wrappers are two independent problems. makefun is solely focused on the second problem. If you wish to solve the first problem you can look at decopatch. It provides a compact syntax, relying on makefun, if you wish to tackle both at once.

Editing a signature

Below we show how to add a parameter to a function. We first capture its Signature using inspect.signature(f), we modify it to add a parameter, and finally we use it in wraps to create our final function:

from makefun import wraps
from inspect import signature, Parameter

# (0) the reference function
def foo(b, a=0):
    print("foo called: b=%s, a=%s" % (b, a))
    return b, a

# (1a) capture the signature of reference function `foo`
foo_sig = signature(foo)
print("Original Signature: %s" % foo_sig)

# (1b) modify the signature to add a new parameter 'z' as first argument
params = list(foo_sig.parameters.values())
params.insert(0, Parameter('z', kind=Parameter.POSITIONAL_OR_KEYWORD))
new_sig = foo_sig.replace(parameters=params)
print("New Signature: %s" % new_sig)

# (2) define the wrapper implementation
@wraps(foo, new_sig=new_sig)
def foo_wrapper(z, *args, **kwargs):
    print("foo_wrapper called ! z=%s" % z)
    # call the foo function 
    output = foo(*args, **kwargs)
    # return augmented output
    return z, output

# call it
assert foo_wrapper(3, 2) == (3, (2, 0))

yields

Original Signature: (b, a=0)
New Signature: (z, b, a=0)

foo_wrapper called ! z=3
foo called: b=2, a=0

This way you can therefore easily create function wrappers with different signatures: not only adding, but also removing parameters, changing their kind (forcing keyword-only for example), etc. The possibilities are as endless as the capabilities of the Signature objects.

Two helper functions are provided in this toolbox to make it a bit easier for you to edit Signature objects:

• remove_signature_parameters creates a new signature from an existing one by removing all parameters corresponding to the names provided
• add_signature_parameters prepends the Parameters provided in its first= argument, and appends the ones provided in its last argument.

from makefun import add_signature_parameters, remove_signature_parameters

def foo(b, c, a=0):
    pass

# original signature
foo_sig = signature(foo)
print("original signature: %s" % foo_sig)

# let's modify it
new_sig = add_signature_parameters(foo_sig,
                first=Parameter('z', kind=Parameter.POSITIONAL_OR_KEYWORD),
                last=Parameter('o', kind=Parameter.POSITIONAL_OR_KEYWORD, 
                               default=True)
          )
new_sig = remove_signature_parameters(new_sig, 'b', 'a')
print("modified signature: %s" % new_sig)

yields

original signature: (b, c, a=0)
modified signature: (z, c, o=True)

They might save you a few lines of code if your use-case is not too specific.

Easier edits

Now @wraps supports three new parameters to easily add or remove parameters to a signature: append_args, prepend_args, and remove_args. The above example can therefore be simplified to

from makefun import wraps

def foo(b, a=0):
    print("foo called: b=%s, a=%s" % (b, a))
    return b, a

@wraps(foo, prepend_args='z')
def foo_wrapper(z, *args, **kwargs):
    print("foo_wrapper called ! z=%s" % z)
    # call the foo function 
    output = foo(*args, **kwargs)
    # return augmented output
    return z, output

# call it
assert foo_wrapper(3, 2) == (3, (2, 0))

See api documentation for details.

Removing parameters easily

To replace them with a hardcoded value

As goodies, makefun provides a partial function that are equivalent to functools.partial, except that it is fully signature-preserving and modifies the documentation with a nice helper message explaining that this is a partial view:

def foo(x, y):
    """
    a `foo` function

    :param x:
    :param y:
    :return:
    """
    return x + y

from makefun import partial
bar = partial(foo, x=12)

we can test it:

>>> assert bar(1) == 13
>>> help(bar)
Help on function bar in module makefun.tests.test_partial_and_macros:

bar(y)
    <This function is equivalent to 'foo(y, x=12)', see original 'foo' doc below.>

    a `foo` function

    :param x:
    :param y:
    :return:

A @with_partial decorator is also available to create partial views easily for quick tests:

@with_partial(x=12)
def foo(x, y):
    """
    a `foo` function

    :param x:
    :param y:
    :return:
    """
    return x + y

To inject a dynamically baked value

As mentioned previously, @wraps provides a remove_args parameter where you can pass one or several argument names.

def inject_random_a(f):
    """
    A decorator that injects a random number inside the `a` argument, 
    removing it from the exposed signature
    """
    @wraps(f, remove_args='a')
    def my_wrapper(*args, **kwargs):
        # generate a random value for a and inject it in the args for f
        kwargs['a'] = random()
        return f(*args, **kwargs)

    return my_wrapper

@inject_random_a
def summer(a, b):
    return a + b

assert 12 <= summer(b=12) <= 13

3- Advanced topics

Generators and Coroutines

create_function and @with_signature will automatically create a generator if your implementation is a generator:

# define the implementation
def my_generator_impl(b, a=0):
    for i in range(a, b):
        yield i * i

# create the dynamic function
gen_func = create_function("foo(a, b)", my_generator_impl)

# verify that the new function is a generator and behaves as such
assert isgeneratorfunction(gen_func)
assert list(gen_func(1, 4)) == [1, 4, 9]

The same goes for generator-based coroutines:

# define the impl that should be called
def my_gencoroutine_impl(first_msg):
    second_msg = (yield first_msg)
    yield second_msg

# create the dynamic function
gen_func = create_function("foo(first_msg='hello')", my_gencoroutine_impl)

# verify that the new func is a generator-based coroutine and behaves correctly
cor = gen_func('hi')
assert next(cor) == 'hi'
assert cor.send('chaps') == 'chaps'
cor.send('ola')  # raises StopIteration

and asyncio coroutines as well

# define the impl that should be called
async def my_native_coroutine_impl(sleep_time):
    await sleep(sleep_time)
    return sleep_time

# create the dynamic function
gen_func = create_function("foo(sleep_time=2)", my_native_coroutine_impl)

# verify that the new function is a native coroutine and behaves correctly
from asyncio import get_event_loop
out = get_event_loop().run_until_complete(gen_func(5))
assert out == 5

Generated source code

The generated source code is in the __source__ field of the generated function:

print(gen_func.__source__)

prints the following source code:

def foo(b, a=0):
    return _func_impl_(b=b, a=a)

The _func_impl_ symbol represents your implementation. As already mentioned, you see that the variables are passed to it as keyword arguments when possible (_func_impl_(b=b), not simply _func_impl_(b)). Of course if it is not possible it adapts:

gen_func = create_function("foo(a=0, *args, **kwargs)", func_impl)
print(gen_func.__source__)

prints the following source code:

def foo(a=0, *args, **kwargs):
    return _func_impl_(a=a, *args, **kwargs)

Function reference injection

In some scenarios you may wish to share the same implementation among several created functions, for example to expose slightly different signatures on top of the same core.

In that case you may wish your implementation to know from which dynamically generated function it is being called. For this, simply use inject_as_first_arg=True, and the called function will be injected as the first argument:

def core_impl(f, *args, **kwargs):
    print("This is generic core called by %s" % f.__name__)
    # here you could use f.__name__ in a if statement to determine what to do
    if f.__name__ == "func1":
        print("called from func1 !")
    return args, kwargs

# generate 2 functions
func1 = create_function("func1(a, b)", core_impl, inject_as_first_arg=True)
func2 = create_function("func2(a, d)", core_impl, inject_as_first_arg=True)

func1(1, 2)
func2(1, 2)

yields

This is generic core called by func1
called from func1 !
This is generic core called by func2

4. Other goodies

@compile_fun

A draft decorator to compile any existing function so that users cant debug through it. It can be handy to mask some code from your users for convenience (note that this does not provide any obfuscation, people can still reverse engineer your code easily. Actually the source code even gets copied in the function's __source__ attribute for convenience):

from makefun import compile_fun

@compile_fun
def foo(a, b):
    return a + b

assert foo(5, -5.0) == 0
print(foo.__source__)

yields

@compile_fun
def foo(a, b):
    return a + b

If the function closure includes functions, they are recursively replaced with compiled versions too (only for this closure, this does not modify them otherwise). You may disable this behaviour entirely with recurse=False, or exclude some symbols from this recursion with the except_names=(...) arg (a tuple of names to exclude).

IMPORTANT this decorator is a "goodie" in early stage and has not been extensively tested. Feel free to contribute !

Note that according to this post compiling does not make the code run any faster.

Known issues: NameError may appear if your function code depends on symbols that have not yet been defined. Make sure all symbols exist first ! See this issue.

Main features / benefits

• Generate functions with a dynamically defined signature: the signature can be provided as a string or as a Signature object, thus making it handy to derive from other functions.
• Implement them easily: the generated functions redirect their calls to the provided implementation function. As long as the signature is compliant, it will work as expected. For example the signature can be specific (a: int, b=None), and the implementation more generic (*args, **kwargs). Arguments will always be passed as keywords arguments when possible.
• Replace `@functools.wraps so that it correctly preserves signatures, and enable you to easily access named arguments.

See Also

• decorator, which largely inspired this code
• PEP362 - Function Signature Object
• A blog entry on dynamic function creation
• functools.wraps

Others

Do you like this library ? You might also like my other python libraries

Want to contribute ?

Details on the github page: https://github.com/smarie/python-makefun