Writing Plugins for the Pappy Proxy

Introduction

Are macros not powerful enough? Want to make something reusable? Want to add console commands?! Then you might want to write yourself a plugin. Some quick highlights about plugins:

• Python scripts stored in ~/.pappy/plugins
• Can add console commands
• For actions which aren’t specific to one project
• Harder to write than macros

Since macros can also use the plugin API, plugins aren’t any more powerful than macros (besides adding console commands). However, if you find yourself copying a useful macro to more than one project, it may be worth it to just bind it to some commands, put the script in one place, and stop worrying about copying it around. Plus then you can put it on GitHub for some sweet sweet nerd cred.

Should I Write a Plugin or a Macro?

A lot of the time, you can get away with writing a macro. However, you may consider writing a plugin if:

• You find yourself copying one macro to multiple projects
• You want to write a general tool that can be applied to any website
• You need to maintain state during the Pappy session

My guess is that if you need one quick thing for a project, you’re better off writing a macro first and seeing if you end up using it in future projects. Then if you find yourself needing it a lot, write a plugin for it. You may also consider keeping a mine.py plugin where you can write out commands that you use regularly but may not be worth creating a dedicated plugin for.

Plugins Get Merged

If you write a useful plugin, as long as it isn’t uber niche, I’ll try and merge it into the core project.

Creating a Plugin

Whenever you make a macro, you’ll have to bind some functions to some console commands. To do this, you’ll have to define a load_cmds function in your plugin. This function should take one argument. When the plugin is loaded, this function will be called and the console object will be passed to this function. You can then use set_cmds and add_aliases to bind functions to console commands.

Writing a Hello World Plugin

It’s probably easiest to explain how to write a plugin by writing one. Here is a simple plugin that defines a hello command and gives an alias hlo (we’ll go over all the parts in a second):

## hello.py

def hello_world(line):
    print "Hello, world!"

###############
## Plugin hooks

def load_cmds(cmd):
    cmd.set_cmds({
        'hello': (hello_world, None),
    })
    cmd.add_aliases([
        ('hello', 'hlo'),
    ])

Save this as ~/.pappy/plugins/hello.py and run Pappy. You should have a new hello command that prints your message:

$ cp hello.py ~/.pappy/plugins/
$ pappy -l
Temporary datafile is /tmp/tmp1Myw6q
Proxy is listening on port 8000
pappy> hello
Hello, world!
pappy> hlo
Hello, world!
pappy>

Awesome! So let’s go over the code. Here are the important parts of the code:

• We define a function that we want to call
• We define load_cmds(cmd) to be called when our plugin is loaded to bind our function to a command
• We use cmd.set_cmds to set all our commands
• We use cmd.add_aliases to add aliases for commands

Now let’s go over it in detail

Passing Arguments to Your Function

Each command gets bound to one function which takes one argument. That argument is all the text that was entered after the name of the command in the console. For example if we run hello foo bar, in our function line would be “foo bar”. I suggest using shlex.split(line) to parse multiple arguments. So let’s update our script to take some arguments:

## hello.py
import shlex

def hello_world(line):
    if line:
        args = shlex.split(line)
        print 'Hello, %s!' % (', '.join(args))
    else:
        print "Hello, world!"

###############
## Plugin hooks

def load_cmds(cmd):
    cmd.set_cmds({
        'hello': (hello_world, None),
    })
    cmd.add_aliases([
        ('hello', 'hlo'),
    ])

Save your changes and restart pappy to reload the plugin:

$ pappy -l
Temporary datafile is /tmp/tmpBOXyJ3
Proxy is listening on port 8000
pappy> hello
Hello, world!
pappy> hello foo bar baz
Hello, foo, bar, baz!
pappy> hello foo bar "baz lihtyur"
Hello, foo, bar, baz lihtyur!
pappy>

Adding More Aliases

So now let’s add some more aliases to our command. If we want to add a new alias, we just add another tuple to the list passed into cmd.add_aliases. The first element is the real name of the command (what you set with set_cmds) and the second value is the alias you want to type. So let’s make it so we can just type ho to say hello:

## hello.py
import shlex

def hello_world(line):
    if line:
        args = shlex.split(line)
        print 'Hello, %s!' % (', '.join(args))
    else:
        print "Hello, world!"

###############
## Plugin hooks

def load_cmds(cmd):
    cmd.set_cmds({
        'hello': (hello_world, None),
    })
    cmd.add_aliases([
        ('hello', 'hlo'),
        ('hello', 'ho'),
    ])

Note

You must use the actual name of the command that you used in set_cmds. You can’t “chain” alieases. As a result, in our example we couldn’t add the alias ('hlo', 'ho') to add ho as our alias.

Then reload the plugin:

$ pappy -l
Temporary datafile is /tmp/tmpBOXyJ3
Proxy is listening on port 8000
pappy> ho
Hello, world!
pappy> ho foo bar baz
Hello, foo, bar, baz!
pappy> ho foo bar "baz lihtyur"
Hello, foo, bar, baz lihtyur!
pappy>

Adding Another Command

So now let’s add a robe_and_wizard_hat command. To do this, we will define another function, then add another entry in the dict that is passed to set_cmds. The second value in the tuple is the autocomplete function, but we’ll get to that later. For now, just put in None to say we don’t have one. We will also add a wh alias to it:

$ pappy -l
Temporary datafile is /tmp/tmpyl2cEZ
Proxy is listening on port 8000
pappy> wh
I put on my robe and wizard hat
pappy>

Adding Autocompletion

You can also define a function to handle autocompletion for your command. In order to do this, you define a function that takes 4 arguments: text, line, begidx, and endidx. From the Cmd docs, this is what the arguments mean:

text is the string prefix we are attempting to match: all returned matches must begin with it. line is the current input line with leading whitespace removed, begidx and endidx are the beginning and ending indexes of the prefix text, which could be used to provide different completion depending upon which position the argument is in.

Let’s let the user to autocomplete some names in our plugin:

import shlex

_AUTOCOMPLETE_NAMES = ['alice', 'allie', 'sarah', 'mallory', 'slagathor']

def hello_world(line):
    if line:
        args = shlex.split(line)
        print 'Hello, %s!' % (', '.join(args))
    else:
        print "Hello, world!"

def put_on_rope_and_wizard_hat(line):
    if line:
        print '%s puts on their robe and wizard hat' % line
    else:
        print 'I put on my robe and wizard hat'

def complete_hello_world(text, line, begidx, endidx):
    return [n for n in _AUTOCOMPLETE_NAMES if n.startswith(text)]

###############
## Plugin hooks

def load_cmds(cmd):
    cmd.set_cmds({
        'hello': (hello_world, complete_hello_world),
        'wizard_hat': (put_on_rope_and_wizard_hat, None),
    })
    cmd.add_aliases([
        ('hello', 'hlo'),
        ('wizard_hat', 'wh'),
    ])

Then restart and run:

$ pappy -l
Temporary datafile is /tmp/tmp3J97rE
Proxy is listening on port 8000
pappy> hello
alice      allie      mallory    sarah      slagathor
pappy> hello allie
Hello, allie!
pappy>

You can’t see it, but I hit tab twice after typing hello to get the completions to appear.

Adding Help

Now let’s say we want to add some help to the command so that when the user runs help hello they get something useful. To do that, just add a docstring to your function:

import shlex

_AUTOCOMPLETE_NAMES = ['alice', 'allie', 'sarah', 'mallory', 'slagathor']

def hello_world(line):
    """
    Say hello to the world. Usage: hello [name]
    """

    if line:
        args = shlex.split(line)
        print 'Hello, %s!' % (', '.join(args))
    else:
        print "Hello, world!"

def put_on_rope_and_wizard_hat(line):
    if line:
        print '%s puts on their robe and wizard hat' % line
    else:
        print 'I put on my robe and wizard hat'

def complete_hello_world(text, line, begidx, endidx):
    return [n for n in _AUTOCOMPLETE_NAMES if n.startswith(text)]

###############
## Plugin hooks

def load_cmds(cmd):
    cmd.set_cmds({
        'hello': (hello_world, complete_hello_world),
        'wizard_hat': (put_on_rope_and_wizard_hat, None),
    })
    cmd.add_aliases([
        ('hello', 'hlo'),
        ('wizard_hat', 'wh'),
    ])

Using defer.inlineCallbacks With a Command

Note

If you are using inlineCallbacks, you can’t use any functions which are blocking versions of async functions. For example, you cannot use pappyproxy.http.Request.save() and must instead use pappyproxy.http.Request.async_deep_save().

Note

This tutorial won’t tell you how to use inlineCallbacks in general. Type “twisted inline callbacks” into google to figure out what they are. This is mainly just a reminder to use the crochet wrapper for console commands and warning you that some functions may return deferreds that you may have to deal with.

Since you’re writing a plugin, you’ll probably be using functions which return a deferred. And to keep things readable, you’ll want to use the defer.inlineCallbacks function wrapper. Unfortunately, you can’t bind async functions to commands. Luckily, there’s a library called crochet which lets you add another wrapper to the function that lets it be used like a blocking function. Rather than talking about it, let’s write a plugin to call pappyproxy.util.load_reqlist() to print out some requests’ hosts. Let’s start by pretending it’s a normal function:

import shlex
from pappyproxy.util import load_reqlist

def print_hosts(line):
    args = shlex.split(line)
    reqs = load_reqlist(args[0]) # It's supposed to return a list of requests, right?
    for r in reqs:
        print 'The host for request %s is: %s' % (r.reqid, r.host)

###############
## Plugin hooks

def load_cmds(cmd):
    cmd.set_cmds({
        'print_hosts': (print_hosts, None),
    })
    cmd.add_aliases([
    ])

And we run it:

pappy> print_hosts 1
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/cmd2.py", line 788, in onecmd_plus_hooks
    stop = self.onecmd(statement)
  File "/usr/local/lib/python2.7/dist-packages/cmd2.py", line 871, in onecmd
    stop = func(statement)
  File "/home/supahacker/pappy/pappyproxy/console.py", line 15, in catch
    func(*args, **kwargs)
  File "/home/supahacker/.pappy/plugins/hosts.py", line 7, in print_hosts
    for r in reqs:
TypeError: iteration over non-sequence
iteration over non-sequence
pappy>

Iteration over a non-sequence? what? Well, pappyproxy.util.load_reqlist() doesn’t actually return a list of requests. It returns a deferred which returns a list of requests. I’m not going into the details (look up some stuff on using inline callbacks with Twisted if you want more info), but the way to fix it is to slap an inlineCallbacks wrapper on the function and yield the result of the function. Now it looks like this:

import shlex
from pappyproxy.util import load_reqlist
from twisted.internet import defer

@defer.inlineCallbacks
def print_hosts(line):
    args = shlex.split(line)
    reqs = yield load_reqlist(args[0])
    for r in reqs:
        print 'The host for request %s is: %s' % (r.reqid, r.host)

###############
## Plugin hooks

def load_cmds(cmd):
    cmd.set_cmds({
        'print_hosts': (print_hosts, None),
    })
    cmd.add_aliases([
    ])

However, the console assumes that any functions it calls will be blocking. As a result, we need to add the crochet.wait_for wrapper:

import shlex
import crochet
from pappyproxy.util import load_reqlist
from twisted.internet import defer

@crochet.wait_for(timeout=None)
@defer.inlineCallbacks
def print_hosts(line):
    args = shlex.split(line)
    reqs = yield load_reqlist(args[0])
    for r in reqs:
        print 'The host for request %s is: %s' % (r.reqid, r.host)

###############
## Plugin hooks

def load_cmds(cmd):
    cmd.set_cmds({
        'print_hosts': (print_hosts, None),
    })
    cmd.add_aliases([
    ])

And now we’re good! If you run it without the crochet wrapper, it may still work. However, since the console assumes any functions it calls will be blocking, not having the wrapper could lead to weird errors.

Plugin API

There are also some useful functions that you can use to interact with the request history and the context. It’s somewhat limited for now, but for now you can at least look through history and create/send new requests. Hopefully the API will expand as people find themselves wanting to do new things. That means if you’re writing a plugin, let me know and I’ll add any APIs you need. For now at least, plugins will let you maintain state over the course of the session and let you define commands.

The best way to learn what you can do is to go through the pappyproxy-package and look at all the available functions.

API Functions

See pappyproxy.plugin for docs on all the functions you can use. You can also use any of the functions provided for writing macros (and vice-versa).

Storing Data on Disk

Unfortunately, you’re on your own if you want to store plugin specific stuff on disk. It’s also important that you store any data that is specific to a project in the same directory as the data file. This is to make sure that if you encrypt your project folder, you can be sure that no sensitive data about the test can be found anywhere else. The only time you should store anything outside of the current directory is to store global plugin settings, and even then it would probably be better to parse options from config.config_dict. Pappy doesn’t even store data outside of the project directory except for its CA certificates.

However, if your plugin is a special snowflake that needs to store unencrypted, global settings, you should create a directory for your plugin in {config.DATA_DIR}/plugindata and put your files there. But again, avoid this if you can.

Note

Any project-specific data (ie anything that contains info about requests) should be stored in the project directory unless you have a really really good reason. This is because it must be possible to secure any sensitive data by encrypting the project folder and storing data outside of the directory will add complications.

Warning

Do not modify the data file schema. There is a good chance the schema will break in future versions of Pappy.

Storing Custom Request Metadata

pappyproxy.http.Request objects have a plugin_data attribute. It is a dictionary that is intended to be used by plugins to give the request custom metadata. If you want to store metadata about a request, it is suggested that you add a key to this dictionary and store any metadata you want under that key. You can use pappyproxy.http.Request.get_plugin_dict() to get a dictionary for a specific name. It will create an entry for that name if it doesn’t exist. I also suggest defining a function plugin-wide for getting the plugin’s data dict from a specific request. Since dictionaries are always passed by reference, any modifications you make to the returned dict will be applied to the request as well.

Note

You will need to save the request using something like pappyproxy.http.Request.save() or pappyproxy.http.Request.async_deep_save() in order to store the changes in the data file.

Here is an example plugin for storing the user-agent (if it exists) in the plugin_data dict of a request under the key agent:

import crochet
import shlex
from twisted.internet import defer

from pappyproxy.util import load_reqlist
from pappyproxy.plugin import main_context
from pappyproxy.util import PappyException

DATA_KEY = 'agent'

def get_data(r):
    return r.get_plugin_dict(DATA_KEY)

@crochet.wait_for(timeout=None)
@defer.inlineCallbacks
def update_agent_metadata(line):
    for r in main_context().active_requests:
        if 'user-agent' in r.headers:
            get_data(r)['agent'] = r.headers['user-agent']
        yield r.async_deep_save()

@crochet.wait_for(timeout=None)
@defer.inlineCallbacks
def view_agent(line):
    args = shlex.split(line)
    reqs = yield load_reqlist(args[0])
    for r in reqs:
        if 'agent' in get_data(r):
            print 'The user agent for %s is "%s"' % (r.reqid, get_data(r)['agent'])
        else:
            print 'Request %s has no user agent data' % r.reqid

###############
## Plugin hooks

def load_cmds(cmd):
    cmd.set_cmds({
        'agent_update': (update_agent_metadata, None),
        'view_agent': (view_agent, None),
    })
    cmd.add_aliases([
    ])

Useful Functions

See pappyproxy.plugin and pappyproxy.util for useful functions

Built In Plugins As Examples

Built In Plugins

All the commands in Pappy are implemented as plugins. I have done what I could to avoid using internal functions as much as I could, but there are still some instances where I had to implement an internal function in order to get the functions I needed. However, you can still look them over to see how things are structured and see some examples of semi-complicated plugins.

Interceptor and Repeater

Pappy’s interceptor and repeater are fully implemented as a plugin. It defines an intercepting macro that handles saving then editing messages and commands that read those files and edit them. It relies on Twisted to switch between the macro handling the request and the command modifying it, so if you want to make something similar, you’ll have to learn how to use deferreds.