jab2870/PappyProxy
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity

Version 0.2.7

Browse source
This commit is contained in:
Rob Glew 2016-02-18 15:29:43 -06:00
parent fda0166e72
commit f4274e1e82
27 changed files with 2136 additions and 493 deletions
Download patch file Download diff file Expand all files Collapse all files

4
docs/source/conf.py
View file

@ -59,9 +59,9 @@ author = u'Rob Glew'
# built documents.
#
# The short X.Y version.
version = u'0.2.6'
version = u'0.2.7'
# The full version, including alpha/beta/rc tags.
release = u'0.2.6'
release = u'0.2.7'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.

647
docs/source/overview.rst
View file

@ -1,6 +1,9 @@
The Pappy Proxy
===============
`Documentation <https://roglew.github.io/pappy-proxy/>`__ -
`Tutorial <https://roglew.github.io/pappy-proxy/tutorial.html>`__
Introduction
------------
@ -10,8 +13,8 @@ testing. Its features are often similar, or straight up rippoffs from
`Burp Suite <https://portswigger.net/burp/>`__. However, Burp Suite is
neither open source nor a command line tool, thus making a proxy like
Pappy inevitable. The project is still in its early stages, so there are
bugs and only the bare minimum features, but it should be able to do
some cool stuff soon (I'm already using it for real work).
bugs and only the bare minimum features, but it can already do some cool
stuff.
Contributing
------------
@ -28,6 +31,15 @@ Right now the codebase is kind of rough and I have refactored it a few
times already, but I would be more than happy to find a stable part of
the codebase that you can contribute to.
Another option is to try writing a plugin. It might be a bit easier than
contributing code and plugins are extremely easy to integrate as a core
feature. So you can also contribute by writing a plugin and letting me
know about it. You can find out more by looking at `the official plugin
docs <https://roglew.github.io/pappy-proxy/pappyplugins.html>`__.
You can find ideas for features to add on `the contributing page in the
docs <https://roglew.github.io/pappy-proxy/contributing.html>`__.
How to Use It
=============
@ -49,9 +61,9 @@ Quickstart
----------
Pappy projects take up an entire directory. Any generated scripts,
exported responses, etc. will be placed in the current directory so it's
good to give your project a directory of its own. To start a project, do
something like:
exported responses, plugin data, etc. will be placed in the current
directory so it's good to give your project a directory of its own. To
start a project, do something like:
::
@ -164,6 +176,130 @@ The following tokens will also be replaced with values:
See the default ``config.json`` for examples.
General Console Techniques
--------------------------
There are a few tricks you can use in general when using Pappy's
console. Most of these are provided by the
`cmd <https://docs.python.org/2/library/cmd.html>`__ and
`cmd2 <https://pythonhosted.org/cmd2/index.html>`__.
Run a shell command
~~~~~~~~~~~~~~~~~~~
You can run a shell command with ``!``:
::
pappy> ls
ID Verb Host Path S-Code Req Len Rsp Len Time Mngl
5 GET vitaly.sexy /netscape.gif 304 Not Modified 0 0 0.08 --
4 GET vitaly.sexy /esr1.jpg 304 Not Modified 0 0 0.07 --
3 GET vitaly.sexy /construction.gif 304 Not Modified 0 0 0.07 --
2 GET vitaly.sexy /vitaly2.jpg 0 N/A -- --
1 GET vitaly.sexy / 304 Not Modified 0 0 0.07 --
pappy> !ls
cmdhistory config.json data.db
pappy>
Running Python Code
~~~~~~~~~~~~~~~~~~~
You can use the ``py`` command to either run python code or to drop down
to a Python shell.
::
pappy> py print ':D '*10
:D :D :D :D :D :D :D :D :D :D
pappy> py
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(ProxyCmd)
py <command>: Executes a Python command.
py: Enters interactive Python mode.
End with ``Ctrl-D`` (Unix) / ``Ctrl-Z`` (Windows), ``quit()``, '`exit()``.
Non-python commands can be issued with ``cmd("your command")``.
Run python code from external files with ``run("filename.py")``
>>> from pappyproxy import config
>>> config.CONFIG_DICT
{u'data_file': u'./data.db', u'history_size': 1000, u'cert_dir': u'{DATADIR}/certs', u'proxy_listeners': [{u'interface': u'127.0.0.1', u'port': 8000}]}
>>> exit()
pappy>
Redirect Output To File
~~~~~~~~~~~~~~~~~~~~~~~
You can use ``>`` to direct output to a file. However, a number of
commands use colored output. If you just redirect these to a file, there
will be additional bytes which represent the ANSI color codes. To get
around this, use the ``nocolor`` command to remove the color from the
command output.
::
pappy> ls > ls.txt
pappy> !xxd -c 32 -g 4 ls.txt
0000000: 1b5b316d 1b5b346d 49442020 56657262 2020486f 73742020 20202020 20202050 .[1m.[4mID Verb Host P
0000020: 61746820 20202020 20202020 20202020 2020532d 436f6465 20202020 20202020 ath S-Code
0000040: 20202020 52657120 4c656e20 20527370 204c656e 20205469 6d652020 20204d6e Req Len Rsp Len Time Mn
0000060: 676c2020 1b5b306d 0a352020 201b5b33 366d4745 541b5b30 6d202020 1b5b3931 gl .[0m.5 .[36mGET.[0m .[91
0000080: 6d766974 616c792e 73657879 1b5b306d 20201b5b 33366d1b 5b306d2f 1b5b3334 mvitaly.sexy.[0m .[36m.[0m/.[34
00000a0: 6d6e6574 73636170 652e6769 661b5b30 6d202020 2020201b 5b33356d 33303420 mnetscape.gif.[0m .[35m304
00000c0: 4e6f7420 4d6f6469 66696564 1b5b306d 20203020 20202020 20202030 20202020 Not Modified.[0m 0 0
00000e0: 20202020 302e3038 20202020 2d2d2020 20200a34 2020201b 5b33366d 4745541b 0.08 -- .4 .[36mGET.
0000100: 5b306d20 20201b5b 39316d76 6974616c 792e7365 78791b5b 306d2020 1b5b3336 [0m .[91mvitaly.sexy.[0m .[36
0000120: 6d1b5b30 6d2f1b5b 33346d65 7372312e 6a70671b 5b306d20 20202020 20202020 m.[0m/.[34mesr1.jpg.[0m
0000140: 201b5b33 356d3330 34204e6f 74204d6f 64696669 65641b5b 306d2020 30202020 .[35m304 Not Modified.[0m 0
0000160: 20202020 20302020 20202020 2020302e 30372020 20202d2d 20202020 0a332020 0 0.07 -- .3
0000180: 201b5b33 366d4745 541b5b30 6d202020 1b5b3931 6d766974 616c792e 73657879 .[36mGET.[0m .[91mvitaly.sexy
00001a0: 1b5b306d 20201b5b 33366d1b 5b306d2f 1b5b3334 6d636f6e 73747275 6374696f .[0m .[36m.[0m/.[34mconstructio
00001c0: 6e2e6769 661b5b30 6d20201b 5b33356d 33303420 4e6f7420 4d6f6469 66696564 n.gif.[0m .[35m304 Not Modified
00001e0: 1b5b306d 20203020 20202020 20202030 20202020 20202020 302e3037 20202020 .[0m 0 0 0.07
0000200: 2d2d2020 20200a32 2020201b 5b33366d 4745541b 5b306d20 20201b5b 39316d76 -- .2 .[36mGET.[0m .[91mv
0000220: 6974616c 792e7365 78791b5b 306d2020 1b5b3336 6d1b5b30 6d2f1b5b 33346d76 italy.sexy.[0m .[36m.[0m/.[34mv
0000240: 6974616c 79322e6a 70671b5b 306d2020 20202020 201b5b33 366d3230 30204f4b italy2.jpg.[0m .[36m200 OK
0000260: 1b5b306d 20202020 20202020 20202020 30202020 20202020 20323033 34303033 .[0m 0 2034003
0000280: 20203135 352e3131 20202d2d 20202020 0a312020 201b5b33 366d4745 541b5b30 155.11 -- .1 .[36mGET.[0
00002a0: 6d202020 1b5b3931 6d766974 616c792e 73657879 1b5b306d 20201b5b 33366d1b m .[91mvitaly.sexy.[0m .[36m.
00002c0: 5b306d2f 1b5b3334 6d1b5b30 6d202020 20202020 20202020 20202020 2020201b [0m/.[34m.[0m .
00002e0: 5b33356d 33303420 4e6f7420 4d6f6469 66696564 1b5b306d 20203020 20202020 [35m304 Not Modified.[0m 0
0000300: 20202030 20202020 20202020 302e3037 20202020 2d2d2020 20200a 0 0.07 -- .
pappy> nocolor ls > ls2.txt
pappy> !xxd -c 32 -g 4 ls2.txt
0000000: 49442020 56657262 2020486f 73742020 20202020 20202050 61746820 20202020 ID Verb Host Path
0000020: 20202020 20202020 2020532d 436f6465 20202020 20202020 20202020 52657120 S-Code Req
0000040: 4c656e20 20527370 204c656e 20205469 6d652020 20204d6e 676c2020 0a352020 Len Rsp Len Time Mngl .5
0000060: 20474554 20202076 6974616c 792e7365 78792020 2f6e6574 73636170 652e6769 GET vitaly.sexy /netscape.gi
0000080: 66202020 20202033 3034204e 6f74204d 6f646966 69656420 20302020 20202020 f 304 Not Modified 0
00000a0: 20203020 20202020 20202030 2e303820 2020202d 2d202020 200a3420 20204745 0 0.08 -- .4 GE
00000c0: 54202020 76697461 6c792e73 65787920 202f6573 72312e6a 70672020 20202020 T vitaly.sexy /esr1.jpg
00000e0: 20202020 33303420 4e6f7420 4d6f6469 66696564 20203020 20202020 20202030 304 Not Modified 0 0
0000100: 20202020 20202020 302e3037 20202020 2d2d2020 20200a33 20202047 45542020 0.07 -- .3 GET
0000120: 20766974 616c792e 73657879 20202f63 6f6e7374 72756374 696f6e2e 67696620 vitaly.sexy /construction.gif
0000140: 20333034 204e6f74 204d6f64 69666965 64202030 20202020 20202020 30202020 304 Not Modified 0 0
0000160: 20202020 20302e30 37202020 202d2d20 2020200a 32202020 47455420 20207669 0.07 -- .2 GET vi
0000180: 74616c79 2e736578 7920202f 76697461 6c79322e 6a706720 20202020 20203230 taly.sexy /vitaly2.jpg 20
00001a0: 30204f4b 20202020 20202020 20202020 30202020 20202020 20323033 34303033 0 OK 0 2034003
00001c0: 20203135 352e3131 20202d2d 20202020 0a312020 20474554 20202076 6974616c 155.11 -- .1 GET vital
00001e0: 792e7365 78792020 2f202020 20202020 20202020 20202020 20202033 3034204e y.sexy / 304 N
0000200: 6f74204d 6f646966 69656420 20302020 20202020 20203020 20202020 20202030 ot Modified 0 0 0
0000220: 2e303720 2020202d 2d202020 200a0a .07 -- ..
pappy>
If you want to write the contents of a request or response to a file,
don't use ``nocolor`` with ``vfq`` or ``vfs``. Use just the ``vbq`` or
``vbs`` commands.
+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Command | Description |
+===============+==============================================================================================================================================================================+
| ``nocolor`` | Run a command and print its output without ASCII escape codes. Intended for use when redirecting output to a file. Should only be used with text and not with binary data. |
+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Generating Pappy's CA Cert
--------------------------
@ -186,23 +322,45 @@ Browsing Recorded Requests/Responses
The following commands can be used to view requests and responses
+--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Command | Aliases | Description |
+====================+================================+====================================================================================================================================================================================================================================================================================================================================================================================================================================+
| ``ls [a|<num>``] | list, ls | List requests that are in the current context (see Context section). Has information like the host, target path, and status code. With no arguments, it will print the 25 most recent requests in the current context. If you pass 'a' or 'all' as an argument, it will print all the requests in the current context. If you pass a number "n" as an argument, it will print the n most recent requests in the current context. |
+--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``sm`` | sm, site\_map | Print a tree showing the site map. It will display all requests in the current context that did not have a 404 response. |
+--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``viq <id(s)>`` | view\_request\_info, viq | View additional information about requests. Includes the target port, if SSL was used, applied tags, and other information. |
+--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``vfq <id(s)>`` | view\_full\_request, vfq | [V]iew [F]ull Re[Q]uest, prints the full request including headers and data. |
+--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``vhq <id(s)>`` | view\_request\_headers, vhq | [V]iew [H]eaders of a Re[Q]uest. Prints just the headers of a request. |
+--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``vfs <id(s)>`` | view\_full\_response, vfs | [V]iew [F]ull Re[S]ponse, prints the full response associated with a request including headers and data. |
+--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``vhs <id(s)>`` | view\_response\_headers, vhs | [V]iew [H]eaders of a Re[S]ponse. Prints just the headers of a response associated with a request. |
+--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Command | Aliases | Description |
+==============================================================================================================================================================================================================================================================================================================+==================================+====================================================================================================================================================================================================================================================================================================================================================================================================================================+
| ``ls [a|<num>``] | list, ls | List requests that are in the current context (see Context section). Has information like the host, target path, and status code. With no arguments, it will print the 25 most recent requests in the current context. If you pass 'a' or 'all' as an argument, it will print all the requests in the current context. If you pass a number "n" as an argument, it will print the n most recent requests in the current context. |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``sm`` [p] | sm, site\_map | Print a tree showing the site map. It will display all requests in the current context that did not have a 404 response. This has to go through all of the requests in the current context so it may be slow. If the ``p`` option is given, it will print the paths as paths rather than as a tree. |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``viq <id(s)>`` | view\_request\_info, viq | View additional information about requests. Includes the target port, if SSL was used, applied tags, and other information. |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``vfq <id(s)>`` | view\_full\_request, vfq, kjq | [V]iew [F]ull Re[Q]uest, prints the full request including headers and data. |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``vbq <id(s)>`` | view\_request\_bytes, vbq | [V]iew [B]ytes of Re[Q]uest, prints the full request including headers and data without coloring or additional newlines. Use this if you want to write a request to a file. |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``ppq <format> <id(s)>`` | pretty\_print\_request, ppq | Pretty print a request with a specific format. See the table below for a list of formats. |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``vhq <id(s)>`` | view\_request\_headers, vhq | [V]iew [H]eaders of a Re[Q]uest. Prints just the headers of a request. |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``vfs <id(s)>`` | view\_full\_response, vfs, kjs | [V]iew [F]ull Re[S]ponse, prints the full response associated with a request including headers and data. |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``vhs <id(s)>`` | view\_response\_headers, vhs | [V]iew [H]eaders of a Re[S]ponse. Prints just the headers of a response associated with a request. |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``vbs <id(s)>`` | view\_response\_bytes, vbs | [V]iew [B]ytes of Re[S]ponse, prints the full response including headers and data without coloring or additional newlines. Use this if you want to write a response to a file. |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``pps <format> <id(s)>`` | pretty\_print\_response, pps | Pretty print a response with a specific format. See the table below for a list of formats. |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``pprm <id(s)>`` | print\_params, pprm | Print a summary of the parameters submitted with the request. It will include URL params, POST params, and/or cookies |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``pri [ct] [key(s)] | param_info, pri | Print a summary of the parameters and values submitted by in-context requests. You can pass in keys to limit which values will be shown. If you also provide``\ ct\ ``as the first argument, it will include any keys that are passed as arguments. | |``\ watch\` | watch | Print requests and responses in real time as they pass through the proxy. |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Available formats for ``ppq`` and ``pps`` commands:
+------------+------------------------------------------------------------+
| Format | Description |
+============+============================================================+
| ``form`` | Print POST data submitted from a form (normal post data) |
+------------+------------------------------------------------------------+
| ``json`` | Print as JSON |
+------------+------------------------------------------------------------+
The table shown by ``ls`` will have the following columns:
@ -399,12 +557,20 @@ List of fields
+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
| body | body, data, bd, dt | The body (data section) of either the request or the response | String |
+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
| reqbody | qbody, qdata, qbd, qdt | The body (data section) of th request | String |
+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
| rspbody | sbody, sdata, sbd, sdt | The body (data section) of th response | String |
+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
| verb | verb, vb | The HTTP verb of the request (ie GET, POST) | String |
+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
| param | param, pm | Either the get or post parameters | Key/Value |
+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
| header | header, hd | An HTTP header (ie User-Agent, Basic-Authorization) in the request or response | Key/Value |
+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
| reqheader | reqheader, qhd | An HTTP header in the request | Key/Value |
+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
| rspheader | rspheader, shd | An HTTP header in the response | Key/Value |
+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
| rawheaders | rawheaders, rh | The entire header section (as one string) of either the head or the response | String |
+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
| sentcookie | sentcookie, sck | A cookie sent in a request | Key/Value |
@ -456,6 +622,21 @@ can still negate these.
+-----------+------------------+---------------------------------------------------------------------------------------------------------+
| after | after, af | Filters out any request that is not before the given request. Filters out any request without a time. |
+-----------+------------------+---------------------------------------------------------------------------------------------------------+
| inv | inf | Inverts a filter string. Anything that matches the filter string will not pass the filter. |
+-----------+------------------+---------------------------------------------------------------------------------------------------------+
Examples:
::
Only show requests before request 1234
f b4 1234
Only show requests after request 1234
f af 1234
Show requests without a csrf parameter
f inv param ct csrf
Scope
-----
@ -505,6 +686,81 @@ The ``fbi`` command also supports tab completion.
| ``fbi <filter>`` | ``builtin_filter``, ``fbi`` | Apply a built-in filter to the current context |
+--------------------+-------------------------------+--------------------------------------------------+
Decoding Strings
----------------
These features try to fill a similar role to Burp's decoder. Each
command will automatically copy the results to the clipboard. In
addition, if no string is given, the commands will encode/decode
whatever is already in the clipboard. Here is an example of how to
base64 encode/decode a string.
::
pappy> b64e "Hello World!"
SGVsbG8gV29ybGQh
pappy> b64d
Hello World!
pappy>
And if the result contains non-printable characters, a hexdump will be
produced instead
::
pappy> b64d ImALittleTeapot=
0000 22 60 0b 8a db 65 79 37 9a a6 8b "`...ey7...
pappy>
The following commands can be used to encode/decode strings:
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Command | Aliases | Description |
+===========================+=====================================+=====================================================================================================================================================================+
| ``base64_decode`` | ``base64_decode``, ``b64d`` | Base64 decode a string |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``base64_encode`` | ``base64_encode``, ``b64e`` | Base64 encode a string |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``asciihex_decode`` | ``asciihex_decode``, ``ahd`` | Decode an ASCII hex string |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``asciihex_encode`` | ``asciihex_encode``, ``ahe`` | Encode an ASCII hex string |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``html_decode`` | ``html_decode``, ``htmld`` | Decode an html encoded string |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``html_encode`` | ``html_encode``, ``htmle`` | Encode a string to html encode all of the characters |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``url_decode`` | ``url_decode``, ``urld`` | Url decode a string |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``url_encode`` | ``url_encode``, ``urle`` | Url encode a string |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``gzip_decode`` | ``gzip_decode``, ``gzd`` | Gzip decompress a string. Probably won't work too well since there's not a great way to get binary data passed in as an argument. I'm working on this. |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``gzip_encode`` | ``gzip_encode``, ``gze`` | Gzip compress a string. Result doesn't get copied to the clipboard. |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``base64_decode_raw`` | ``base64_decode_raw``, ``b64dr`` | Same as ``base64_decode`` but will not print a hexdump if it contains non-printable characters. It is suggested you use ``>`` to redirect the output to a file. |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``base64_encode_raw`` | ``base64_encode_raw``, ``b64er`` | Same as ``base64_encode`` but will not print a hexdump if it contains non-printable characters. It is suggested you use ``>`` to redirect the output to a file. |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``asciihex_decode_raw`` | ``asciihex_decode_raw``, ``ahdr`` | Same as ``asciihex_decode`` but will not print a hexdump if it contains non-printable characters. It is suggested you use ``>`` to redirect the output to a file. |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``asciihex_encode_raw`` | ``asciihex_encode_raw``, ``aher`` | Same as ``asciihex_encode`` but will not print a hexdump if it contains non-printable characters. It is suggested you use ``>`` to redirect the output to a file. |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``html_decode_raw`` | ``html_decode_raw``, ``htmldr`` | Same as ``html_decode`` but will not print a hexdump if it contains non-printable characters. It is suggested you use ``>`` to redirect the output to a file. |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``html_encode_raw`` | ``html_encode_raw``, ``htmler`` | Same as ``html_encode`` but will not print a hexdump if it contains non-printable characters. It is suggested you use ``>`` to redirect the output to a file. |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``url_decode_raw`` | ``url_decode_raw``, ``urldr`` | Same as ``url_decode`` but will not print a hexdump if it contains non-printable characters. It is suggested you use ``>`` to redirect the output to a file. |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``url_encode_raw`` | ``url_encode_raw``, ``urler`` | Same as ``url_encode`` but will not print a hexdump if it contains non-printable characters. It is suggested you use ``>`` to redirect the output to a file. |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``gzip_decode_raw`` | ``gzip_decode_raw``, ``gzdr`` | Same as ``gzip_decode`` but will not print a hexdump if it contains non-printable characters. It is suggested you use ``>`` to redirect the output to a file. |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``gzip_encode_raw`` | ``gzip_encode_raw``, ``gzer`` | Same as ``gzip_encode`` but will not print a hexdump if it contains non-printable characters. It is suggested you use ``>`` to redirect the output to a file. |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``unixtime_decode`` | ``unixtime_decode``, ``uxtd`` | Take in a unix timestamp and print a human readable timestamp |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Interceptor
-----------
@ -604,7 +860,7 @@ a simple macro would be:
::
--- macro_print.py
### macro_print.py
MACRO_NAME = 'Print Macro'
@ -717,15 +973,15 @@ Request Objects
The main method of interacting with the proxy is through ``Request``
objects. You can submit a request with ``req.sumbit()`` and save it to
the data file with ``req.save()``. The objects also have attributes
which can be used to modify the request in a high-level way.
Unfortunately, I haven't gotten around to writing full docs on the API
and it's still changing every once in a while so I apologize if I pull
the carpet out from underneath you.
which can be used to modify the request in a high-level way. You can see
the `full
documentation <https://roglew.github.io/pappy-proxy/pappyproxy.html#module-pappyproxy.http>`__
for more details on using these objects.
Dict-like objects are represented with a custom class called a
``RepeatableDict``. I haven't gotten around to writing docs on it yet,
so just interact with it like a dict and don't be surprised if it's
missing some methods you would expect a dict to have.
``RepeatableDict``. Again, look at the docs for details. For the most
part, you can interact with it like a normal dictionary, but don't be
surprised if it's missing some methods you would expect.
Here is a quick list of attributes that you can use with ``Request``
objects:
@ -1024,6 +1280,8 @@ error checking.
+----------------------------------------+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``export <req|rsp> <reqid>`` | ``export`` | Writes either the full request or response to a file in the current directory. |
+----------------------------------------+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``merge <dbfile>`` | ``merge`` | Add all the requests from another datafile to the current datafile |
+----------------------------------------+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+
Response streaming
~~~~~~~~~~~~~~~~~~
@ -1033,9 +1291,241 @@ data to the browser as it gets it. However, if you're trying to mangle
messages/responses, Pappy will need to download the entire message
first.
Plugins
-------
Note that this section is a very quick overview of plugins. For a full
description of how to write them, please see `the official
docs <https://roglew.github.io/pappy-proxy/pappyplugins.html>`__.
It is also possible to write plugins which are reusable across projects.
Plugins are simply Python scripts located in ``~/.pappy/plugins``.
Plugins are able to create new console commands and maintain state
throughout a Pappy session. They can access the same API as macros, but
the plugin system is designed to allow you to create general purpose
commands as compared to macros which are meant to be project-specific
scripts. Still, it may not be a bad idea to try building a macro to do
something in a quick and dirty way before writing a plugin since plugins
are more complicated to write.
A simple hello world plugin could be something like:
::
## hello.py
import shlex
def hello_world(line):
if line:
args = shlex.split(line)
print 'Hello, %s!' % (', '.join(args))
else:
print "Hello, world!"
###############
## Plugin hooks
def load_cmds(cmd):
cmd.set_cmds({
'hello': (hello_world, None),
})
cmd.add_aliases([
('hello', 'hlo'),
('hello', 'ho'),
])
You can also create commands which support autocomplete:
::
import shlex
_AUTOCOMPLETE_NAMES = ['alice', 'allie', 'sarah', 'mallory', 'slagathor']
def hello_world(line):
if line:
args = shlex.split(line)
print 'Hello, %s!' % (', '.join(args))
else:
print "Hello, world!"
def complete_hello_world(text, line, begidx, endidx):
return [n for n in _AUTOCOMPLETE_NAMES if n.startswith(text)]
###############
## Plugin hooks
def load_cmds(cmd):
cmd.set_cmds({
'hello': (hello_world, complete_hello_world),
})
cmd.add_aliases([
('hello', 'hlo'),
])
Then when you run Pappy you can use the ``hello`` command:
::
$ pappy -l
Temporary datafile is /tmp/tmpBOXyJ3
Proxy is listening on port 8000
pappy> ho
Hello, world!
pappy> ho foo bar baz
Hello, foo, bar, baz!
pappy> ho foo bar "baz lihtyur"
Hello, foo, bar, baz lihtyur!
pappy>
Should I Write a Plugin or a Macro?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A lot of the time, you can get away with writing a macro. However, you
may consider writing a plugin if:
- You find yourself copying one macro to multiple projects
- You want to write a general tool that can be applied to any website
- You need to maintain state during the Pappy session
My guess is that if you need one quick thing for a project, you're
better off writing a macro first and seeing if you end up using it in
future projects. Then if you find yourself needing it a lot, write a
plugin for it. You may also consider keeping a ``mine.py`` plugin where
you can write out commands that you use regularly but may not be worth
creating a dedicated plugin for.
Global Settings
---------------
There are some settings that apply to Pappy as a whole and are stored in
``~/.pappy/global_config.json``. These settings are generally for tuning
performance or modifying behavior on a system-wide level. No information
about projects is put in here since it is world readable. You can
technically add settings in here for plugins that you write, but if it's
at all possible, please keep settings in the normal project config.
Settings included in ``~/.pappy/global_config.json``:
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Setting | Description |
+===============+===============================================================================================================================================================================+
| cache\_size | The number of requests from history that will be included in memory at any given time. Set to -1 to keep everything in memory. See the request cache section for more info. |
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Using a SOCKS Server
--------------------
Pappy allows you to use an upstream SOCKS server. You can do this by
adding a ``socks_proxy`` value to config.json. You can use the following
for anonymous access to the proxy:
::
"socks_proxy": {"host":"socks.proxy.host", "port":5555}
To use credentials you add a ``username`` and ``password`` value to the
dictionary:
::
"socks_proxy": {"host":"socks.proxy.host", "port":5555, "username": "mario", "password":"ilovemushrooms"}
Anything that passes through any of the active listeners will use the
proxy.
Transparent Host Redirection
----------------------------
Sometimes you get a frustrating thick client that doesn’t let you mess
with proxy settings to get it to go through a proxy. However, if you can
redirect where it sends its traffic to localhost, you can get Pappy to
take that traffic and redirect it to go where it should.
It takes root permissions to listen on low numbered ports. As a result,
we’ll need to do some root stuff to listen on ports 80 and 443 and get
the data to Pappy. There are two ways to get the traffic to Pappy. The
first is to set up port forwarding as root to send traffic from
localhost:80 to localhost:8080 and localhost:443 to localhost:8443
(since we can listen on 8080 and 8443 without root). Or you can YOLO,
run Pappy as root and just have it listen on 80 and 443.
According to Google you can use the following command to forward port 80
on localhost to 8080 on Linux:
::
iptables -t nat -A PREROUTING -i ppp0 -p tcp --dport 80 -j REDIRECT --to-ports 8080
Then to route 443 to 8443:
::
iptables -t nat -A PREROUTING -i ppp0 -p tcp --dport 443 -j REDIRECT --to-ports 8443
Of course, both of these need to be run as root.
Then on mac it’s
::
echo "
rdr pass inet proto tcp from any to any port 80 -> 127.0.0.1 port 8080
rdr pass inet proto tcp from any to any port 443 -> 127.0.0.1 port 8443
" | sudo pfctl -ef -
Then to turn it off on mac it’s
sudo pfctl -F all -f /etc/pf.conf
Then modify the listener settings in the project’s config.json to be:
::
"proxy_listeners": [
{"port": 8080, "interface": "127.0.0.1", "forward_host": "www.example.faketld"},
{"port": 8443, "interface": "127.0.0.1", "forward_host_ssl": "www.example.faketld"},
]
This configuration will cause Pappy to open a port on 8080 that will
accept connections normally and a port on 8443 which will accept SSL
connections. The forward\_host setting tells Pappy to redirect any
requests sent to the port to the given host. It will also update the
request’s host header. forward\_host\_ssl does the same thing, but it
listens for SSL connections and forces the connection to use SSL.
Or if you’re going to YOLO it do the same thing then listen on port
80/443 directly. I do not suggest you do this.
::
"proxy_listeners": [
{"port": 80, "interface": "127.0.0.1", "forward_host": "www.example.faketld"},
{"port": 443, "interface": "127.0.0.1", "forward_host_ssl": "www.example.faketld"},
]
Pappy will automatically use this host to make the connection and
forward the request to the new server.
FAQ
---
I still like Burp, but Pappy looks interesting, can I use both?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Yes! If you don't want to go completely over to Pappy yet, you can
configure Burp to use Pappy as an upstream proxy server. That way,
traffic will go through both Burp and Pappy and you can use whichever
you want to do your testing.
How to have Burp forward traffic through Pappy:
1. Open Burp
2. Go to ``Options -> Connections -> Upstream Proxy Servers``
3. Click ``Add``
4. Leave ``Destination Host`` blank, but put ``127.0.0.1`` in
``Proxy Host`` and ``8000`` into ``Port`` (assuming you're using the
default listener)
5. Configure your browser to use Burp as a proxy
Why does my request have an id of ``--``?!?!
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -1044,17 +1534,98 @@ saved to disk. In between the time when a request is decoded and when
it's saved to disk, it will have an ID of ``--``. So just wait a little
bit and it will get an ID you can use.
Boring, Technical Stuff
-----------------------
I do some stuff to try and keep speed and memory usage to reasonable
levels. Unfortunately, things might seem slow in some areas. This is
where I try and explain why those exist. Honestly, you probably don't
care about this, but I'd rather have it written down and have nobody
read it than just leave people in the dark.
Request Cache / Memory usage
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
For performance reasons, Pappy by default will not store every request
in memory. The cache will store a certain number of the most recently
accessed requests in memory. This means that if you go through all of
history, it could be slow (for example running ``ls a`` or ``sm``). If
you have enough RAM to keep everything in memory, you can set the
request cache size to -1 to just keep everything in memory. However,
even if the cache size is unlimited, it still won't load a request into
memory untill you access it. So if you want to load everything in
memory, run ``ls a``.
By default, Pappy will cache 2000 requests. This is kind of heavy, but
it's assumed you're doing testing on a reasonably specced laptop.
Personally, I live on the edge and use -1 until I run into memory
issues.
Changelog
---------
The boring part of the readme
- 0.1.2
- Refactor almost every part of proxy
- Basic framework for plugins
- Bugfixes probably
- Create changelog
- 0.1.1
- Start using sane versioning system
- No idea what I added
- 0.2.7
- boring unit tests
- should make future releases more stable I guess
- Support for upstream SOCKS servers
- ``print_params`` command
- ``inv`` filter
- ``param_info`` command
- Filters by request/response only headers/body
- Transparent host redirection
- Some easier to type aliases for common commands
- 0.2.6
- Fix pip being dumb
- ``watch`` command to watch requests/responses in real time
- Added ``pp[qs] form <id>`` to print POST data
- Bugfixes
- 0.2.5
- Requests sent with repeater now are given ``repeater`` tag
- Add ppq and pps commands
- Look at the pretty prompt
- Bugfixes
- 0.2.4
- Add command history saving between sessions
- Add html encoder/decoder
- All the bugs were fixed so I added some more for 0.2.5
- 0.2.3
- Decoder functions
- Add ``merge`` command
- Bugfixes
- 0.2.2
- COLORS
- Performance improvements
- Bugfixes (duh)
- 0.2.1
- Improve memory usage
- Tweaked plugin API
- 0.2.0
- Lots of refactoring
- Plugins
- Bugfixes probably
- Change prompt to make Pappy look more professional (but it will
always be pappy time in your heart, I promise)
- Create changelog
- Add response streaming if no intercepting macros are active
- 0.1.1
- Start using sane versioning system
- Did proxy things