Version 0.2.12

This commit is contained in:
Rob Glew 2016-05-02 13:17:25 -04:00
parent 992edab315
commit f28ab4fe96
32 changed files with 2324 additions and 754 deletions

View file

@ -14,6 +14,7 @@ Contents:
overview
tutorial
pappyplugins
contributing
Indices and tables

View file

@ -37,6 +37,9 @@ feature. So you can also contribute by writing a plugin and letting me
know about it. You can find out more by looking at `the official plugin
docs <https://roglew.github.io/pappy-proxy/pappyplugins.html>`__.
You can find ideas for features to add on `the contributing page in the
docs <https://roglew.github.io/pappy-proxy/contributing.html>`__.
How to Use It
=============
@ -173,6 +176,130 @@ The following tokens will also be replaced with values:
See the default ``config.json`` for examples.
General Console Techniques
--------------------------
There are a few tricks you can use in general when using Pappy's
console. Most of these are provided by the
`cmd <https://docs.python.org/2/library/cmd.html>`__ and
`cmd2 <https://pythonhosted.org/cmd2/index.html>`__.
Run a shell command
~~~~~~~~~~~~~~~~~~~
You can run a shell command with ``!``:
::
pappy> ls
ID Verb Host Path S-Code Req Len Rsp Len Time Mngl
5 GET vitaly.sexy /netscape.gif 304 Not Modified 0 0 0.08 --
4 GET vitaly.sexy /esr1.jpg 304 Not Modified 0 0 0.07 --
3 GET vitaly.sexy /construction.gif 304 Not Modified 0 0 0.07 --
2 GET vitaly.sexy /vitaly2.jpg 0 N/A -- --
1 GET vitaly.sexy / 304 Not Modified 0 0 0.07 --
pappy> !ls
cmdhistory config.json data.db
pappy>
Running Python Code
~~~~~~~~~~~~~~~~~~~
You can use the ``py`` command to either run python code or to drop down
to a Python shell.
::
pappy> py print ':D '*10
:D :D :D :D :D :D :D :D :D :D
pappy> py
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(ProxyCmd)
py <command>: Executes a Python command.
py: Enters interactive Python mode.
End with ``Ctrl-D`` (Unix) / ``Ctrl-Z`` (Windows), ``quit()``, '`exit()``.
Non-python commands can be issued with ``cmd("your command")``.
Run python code from external files with ``run("filename.py")``
>>> from pappyproxy import config
>>> config.CONFIG_DICT
{u'data_file': u'./data.db', u'history_size': 1000, u'cert_dir': u'{DATADIR}/certs', u'proxy_listeners': [{u'interface': u'127.0.0.1', u'port': 8000}]}
>>> exit()
pappy>
Redirect Output To File
~~~~~~~~~~~~~~~~~~~~~~~
You can use ``>`` to direct output to a file. However, a number of
commands use colored output. If you just redirect these to a file, there
will be additional bytes which represent the ANSI color codes. To get
around this, use the ``nocolor`` command to remove the color from the
command output.
::
pappy> ls > ls.txt
pappy> !xxd -c 32 -g 4 ls.txt
0000000: 1b5b316d 1b5b346d 49442020 56657262 2020486f 73742020 20202020 20202050 .[1m.[4mID Verb Host P
0000020: 61746820 20202020 20202020 20202020 2020532d 436f6465 20202020 20202020 ath S-Code
0000040: 20202020 52657120 4c656e20 20527370 204c656e 20205469 6d652020 20204d6e Req Len Rsp Len Time Mn
0000060: 676c2020 1b5b306d 0a352020 201b5b33 366d4745 541b5b30 6d202020 1b5b3931 gl .[0m.5 .[36mGET.[0m .[91
0000080: 6d766974 616c792e 73657879 1b5b306d 20201b5b 33366d1b 5b306d2f 1b5b3334 mvitaly.sexy.[0m .[36m.[0m/.[34
00000a0: 6d6e6574 73636170 652e6769 661b5b30 6d202020 2020201b 5b33356d 33303420 mnetscape.gif.[0m .[35m304
00000c0: 4e6f7420 4d6f6469 66696564 1b5b306d 20203020 20202020 20202030 20202020 Not Modified.[0m 0 0
00000e0: 20202020 302e3038 20202020 2d2d2020 20200a34 2020201b 5b33366d 4745541b 0.08 -- .4 .[36mGET.
0000100: 5b306d20 20201b5b 39316d76 6974616c 792e7365 78791b5b 306d2020 1b5b3336 [0m .[91mvitaly.sexy.[0m .[36
0000120: 6d1b5b30 6d2f1b5b 33346d65 7372312e 6a70671b 5b306d20 20202020 20202020 m.[0m/.[34mesr1.jpg.[0m
0000140: 201b5b33 356d3330 34204e6f 74204d6f 64696669 65641b5b 306d2020 30202020 .[35m304 Not Modified.[0m 0
0000160: 20202020 20302020 20202020 2020302e 30372020 20202d2d 20202020 0a332020 0 0.07 -- .3
0000180: 201b5b33 366d4745 541b5b30 6d202020 1b5b3931 6d766974 616c792e 73657879 .[36mGET.[0m .[91mvitaly.sexy
00001a0: 1b5b306d 20201b5b 33366d1b 5b306d2f 1b5b3334 6d636f6e 73747275 6374696f .[0m .[36m.[0m/.[34mconstructio
00001c0: 6e2e6769 661b5b30 6d20201b 5b33356d 33303420 4e6f7420 4d6f6469 66696564 n.gif.[0m .[35m304 Not Modified
00001e0: 1b5b306d 20203020 20202020 20202030 20202020 20202020 302e3037 20202020 .[0m 0 0 0.07
0000200: 2d2d2020 20200a32 2020201b 5b33366d 4745541b 5b306d20 20201b5b 39316d76 -- .2 .[36mGET.[0m .[91mv
0000220: 6974616c 792e7365 78791b5b 306d2020 1b5b3336 6d1b5b30 6d2f1b5b 33346d76 italy.sexy.[0m .[36m.[0m/.[34mv
0000240: 6974616c 79322e6a 70671b5b 306d2020 20202020 201b5b33 366d3230 30204f4b italy2.jpg.[0m .[36m200 OK
0000260: 1b5b306d 20202020 20202020 20202020 30202020 20202020 20323033 34303033 .[0m 0 2034003
0000280: 20203135 352e3131 20202d2d 20202020 0a312020 201b5b33 366d4745 541b5b30 155.11 -- .1 .[36mGET.[0
00002a0: 6d202020 1b5b3931 6d766974 616c792e 73657879 1b5b306d 20201b5b 33366d1b m .[91mvitaly.sexy.[0m .[36m.
00002c0: 5b306d2f 1b5b3334 6d1b5b30 6d202020 20202020 20202020 20202020 2020201b [0m/.[34m.[0m .
00002e0: 5b33356d 33303420 4e6f7420 4d6f6469 66696564 1b5b306d 20203020 20202020 [35m304 Not Modified.[0m 0
0000300: 20202030 20202020 20202020 302e3037 20202020 2d2d2020 20200a 0 0.07 -- .
pappy> nocolor ls > ls2.txt
pappy> !xxd -c 32 -g 4 ls2.txt
0000000: 49442020 56657262 2020486f 73742020 20202020 20202050 61746820 20202020 ID Verb Host Path
0000020: 20202020 20202020 2020532d 436f6465 20202020 20202020 20202020 52657120 S-Code Req
0000040: 4c656e20 20527370 204c656e 20205469 6d652020 20204d6e 676c2020 0a352020 Len Rsp Len Time Mngl .5
0000060: 20474554 20202076 6974616c 792e7365 78792020 2f6e6574 73636170 652e6769 GET vitaly.sexy /netscape.gi
0000080: 66202020 20202033 3034204e 6f74204d 6f646966 69656420 20302020 20202020 f 304 Not Modified 0
00000a0: 20203020 20202020 20202030 2e303820 2020202d 2d202020 200a3420 20204745 0 0.08 -- .4 GE
00000c0: 54202020 76697461 6c792e73 65787920 202f6573 72312e6a 70672020 20202020 T vitaly.sexy /esr1.jpg
00000e0: 20202020 33303420 4e6f7420 4d6f6469 66696564 20203020 20202020 20202030 304 Not Modified 0 0
0000100: 20202020 20202020 302e3037 20202020 2d2d2020 20200a33 20202047 45542020 0.07 -- .3 GET
0000120: 20766974 616c792e 73657879 20202f63 6f6e7374 72756374 696f6e2e 67696620 vitaly.sexy /construction.gif
0000140: 20333034 204e6f74 204d6f64 69666965 64202030 20202020 20202020 30202020 304 Not Modified 0 0
0000160: 20202020 20302e30 37202020 202d2d20 2020200a 32202020 47455420 20207669 0.07 -- .2 GET vi
0000180: 74616c79 2e736578 7920202f 76697461 6c79322e 6a706720 20202020 20203230 taly.sexy /vitaly2.jpg 20
00001a0: 30204f4b 20202020 20202020 20202020 30202020 20202020 20323033 34303033 0 OK 0 2034003
00001c0: 20203135 352e3131 20202d2d 20202020 0a312020 20474554 20202076 6974616c 155.11 -- .1 GET vital
00001e0: 792e7365 78792020 2f202020 20202020 20202020 20202020 20202033 3034204e y.sexy / 304 N
0000200: 6f74204d 6f646966 69656420 20302020 20202020 20203020 20202020 20202030 ot Modified 0 0 0
0000220: 2e303720 2020202d 2d202020 200a0a .07 -- ..
pappy>
If you want to write the contents of a request or response to a file,
don't use ``nocolor`` with ``vfq`` or ``vfs``. Use just the ``vbq`` or
``vbs`` commands.
+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Command | Description |
+===============+==============================================================================================================================================================================+
| ``nocolor`` | Run a command and print its output without ASCII escape codes. Intended for use when redirecting output to a file. Should only be used with text and not with binary data. |
+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Generating Pappy's CA Cert
--------------------------
@ -195,23 +322,45 @@ Browsing Recorded Requests/Responses
The following commands can be used to view requests and responses
+--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Command | Aliases | Description |
+====================+================================+====================================================================================================================================================================================================================================================================================================================================================================================================================================+
| ``ls [a|<num>``] | list, ls | List requests that are in the current context (see Context section). Has information like the host, target path, and status code. With no arguments, it will print the 25 most recent requests in the current context. If you pass 'a' or 'all' as an argument, it will print all the requests in the current context. If you pass a number "n" as an argument, it will print the n most recent requests in the current context. |
+--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``sm`` | sm, site\_map | Print a tree showing the site map. It will display all requests in the current context that did not have a 404 response. |
+--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``viq <id(s)>`` | view\_request\_info, viq | View additional information about requests. Includes the target port, if SSL was used, applied tags, and other information. |
+--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``vfq <id(s)>`` | view\_full\_request, vfq | [V]iew [F]ull Re[Q]uest, prints the full request including headers and data. |
+--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``vhq <id(s)>`` | view\_request\_headers, vhq | [V]iew [H]eaders of a Re[Q]uest. Prints just the headers of a request. |
+--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``vfs <id(s)>`` | view\_full\_response, vfs | [V]iew [F]ull Re[S]ponse, prints the full response associated with a request including headers and data. |
+--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``vhs <id(s)>`` | view\_response\_headers, vhs | [V]iew [H]eaders of a Re[S]ponse. Prints just the headers of a response associated with a request. |
+--------------------+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Command | Aliases | Description |
+==============================================================================================================================================================================================================================================================================================================+==================================+====================================================================================================================================================================================================================================================================================================================================================================================================================================+
| ``ls [a|<num>``] | list, ls | List requests that are in the current context (see Context section). Has information like the host, target path, and status code. With no arguments, it will print the 25 most recent requests in the current context. If you pass 'a' or 'all' as an argument, it will print all the requests in the current context. If you pass a number "n" as an argument, it will print the n most recent requests in the current context. |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``sm`` [p] | sm, site\_map | Print a tree showing the site map. It will display all requests in the current context that did not have a 404 response. This has to go through all of the requests in the current context so it may be slow. If the ``p`` option is given, it will print the paths as paths rather than as a tree. |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``viq <id(s)>`` | view\_request\_info, viq | View additional information about requests. Includes the target port, if SSL was used, applied tags, and other information. |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``vfq <id(s)>`` | view\_full\_request, vfq, kjq | [V]iew [F]ull Re[Q]uest, prints the full request including headers and data. |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``vbq <id(s)>`` | view\_request\_bytes, vbq | [V]iew [B]ytes of Re[Q]uest, prints the full request including headers and data without coloring or additional newlines. Use this if you want to write a request to a file. |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``ppq <format> <id(s)>`` | pretty\_print\_request, ppq | Pretty print a request with a specific format. See the table below for a list of formats. |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``vhq <id(s)>`` | view\_request\_headers, vhq | [V]iew [H]eaders of a Re[Q]uest. Prints just the headers of a request. |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``vfs <id(s)>`` | view\_full\_response, vfs, kjs | [V]iew [F]ull Re[S]ponse, prints the full response associated with a request including headers and data. |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``vhs <id(s)>`` | view\_response\_headers, vhs | [V]iew [H]eaders of a Re[S]ponse. Prints just the headers of a response associated with a request. |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``vbs <id(s)>`` | view\_response\_bytes, vbs | [V]iew [B]ytes of Re[S]ponse, prints the full response including headers and data without coloring or additional newlines. Use this if you want to write a response to a file. |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``pps <format> <id(s)>`` | pretty\_print\_response, pps | Pretty print a response with a specific format. See the table below for a list of formats. |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``pprm <id(s)>`` | print\_params, pprm | Print a summary of the parameters submitted with the request. It will include URL params, POST params, and/or cookies |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``pri [ct] [key(s)] | param_info, pri | Print a summary of the parameters and values submitted by in-context requests. You can pass in keys to limit which values will be shown. If you also provide``\ ct\ ``as the first argument, it will include any keys that are passed as arguments. | |``\ watch\` | watch | Print requests and responses in real time as they pass through the proxy. |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Available formats for ``ppq`` and ``pps`` commands:
+------------+------------------------------------------------------------+
| Format | Description |
+============+============================================================+
| ``form`` | Print POST data submitted from a form (normal post data) |
+------------+------------------------------------------------------------+
| ``json`` | Print as JSON |
+------------+------------------------------------------------------------+
The table shown by ``ls`` will have the following columns:
@ -408,12 +557,20 @@ List of fields
+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
| body | body, data, bd, dt | The body (data section) of either the request or the response | String |
+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
| reqbody | qbody, qdata, qbd, qdt | The body (data section) of th request | String |
+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
| rspbody | sbody, sdata, sbd, sdt | The body (data section) of th response | String |
+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
| verb | verb, vb | The HTTP verb of the request (ie GET, POST) | String |
+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
| param | param, pm | Either the get or post parameters | Key/Value |
+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
| header | header, hd | An HTTP header (ie User-Agent, Basic-Authorization) in the request or response | Key/Value |
+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
| reqheader | reqheader, qhd | An HTTP header in the request | Key/Value |
+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
| rspheader | rspheader, shd | An HTTP header in the response | Key/Value |
+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
| rawheaders | rawheaders, rh | The entire header section (as one string) of either the head or the response | String |
+--------------+--------------------------------+----------------------------------------------------------------------------------+-------------+
| sentcookie | sentcookie, sck | A cookie sent in a request | Key/Value |
@ -465,6 +622,21 @@ can still negate these.
+-----------+------------------+---------------------------------------------------------------------------------------------------------+
| after | after, af | Filters out any request that is not before the given request. Filters out any request without a time. |
+-----------+------------------+---------------------------------------------------------------------------------------------------------+
| inv | inf | Inverts a filter string. Anything that matches the filter string will not pass the filter. |
+-----------+------------------+---------------------------------------------------------------------------------------------------------+
Examples:
::
Only show requests before request 1234
f b4 1234
Only show requests after request 1234
f af 1234
Show requests without a csrf parameter
f inv param ct csrf
Scope
-----
@ -514,6 +686,81 @@ The ``fbi`` command also supports tab completion.
| ``fbi <filter>`` | ``builtin_filter``, ``fbi`` | Apply a built-in filter to the current context |
+--------------------+-------------------------------+--------------------------------------------------+
Decoding Strings
----------------
These features try to fill a similar role to Burp's decoder. Each
command will automatically copy the results to the clipboard. In
addition, if no string is given, the commands will encode/decode
whatever is already in the clipboard. Here is an example of how to
base64 encode/decode a string.
::
pappy> b64e "Hello World!"
SGVsbG8gV29ybGQh
pappy> b64d
Hello World!
pappy>
And if the result contains non-printable characters, a hexdump will be
produced instead
::
pappy> b64d ImALittleTeapot=
0000 22 60 0b 8a db 65 79 37 9a a6 8b "`...ey7...
pappy>
The following commands can be used to encode/decode strings:
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Command | Aliases | Description |
+===========================+=====================================+=====================================================================================================================================================================+
| ``base64_decode`` | ``base64_decode``, ``b64d`` | Base64 decode a string |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``base64_encode`` | ``base64_encode``, ``b64e`` | Base64 encode a string |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``asciihex_decode`` | ``asciihex_decode``, ``ahd`` | Decode an ASCII hex string |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``asciihex_encode`` | ``asciihex_encode``, ``ahe`` | Encode an ASCII hex string |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``html_decode`` | ``html_decode``, ``htmld`` | Decode an html encoded string |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``html_encode`` | ``html_encode``, ``htmle`` | Encode a string to html encode all of the characters |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``url_decode`` | ``url_decode``, ``urld`` | Url decode a string |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``url_encode`` | ``url_encode``, ``urle`` | Url encode a string |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``gzip_decode`` | ``gzip_decode``, ``gzd`` | Gzip decompress a string. Probably won't work too well since there's not a great way to get binary data passed in as an argument. I'm working on this. |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``gzip_encode`` | ``gzip_encode``, ``gze`` | Gzip compress a string. Result doesn't get copied to the clipboard. |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``base64_decode_raw`` | ``base64_decode_raw``, ``b64dr`` | Same as ``base64_decode`` but will not print a hexdump if it contains non-printable characters. It is suggested you use ``>`` to redirect the output to a file. |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``base64_encode_raw`` | ``base64_encode_raw``, ``b64er`` | Same as ``base64_encode`` but will not print a hexdump if it contains non-printable characters. It is suggested you use ``>`` to redirect the output to a file. |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``asciihex_decode_raw`` | ``asciihex_decode_raw``, ``ahdr`` | Same as ``asciihex_decode`` but will not print a hexdump if it contains non-printable characters. It is suggested you use ``>`` to redirect the output to a file. |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``asciihex_encode_raw`` | ``asciihex_encode_raw``, ``aher`` | Same as ``asciihex_encode`` but will not print a hexdump if it contains non-printable characters. It is suggested you use ``>`` to redirect the output to a file. |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``html_decode_raw`` | ``html_decode_raw``, ``htmldr`` | Same as ``html_decode`` but will not print a hexdump if it contains non-printable characters. It is suggested you use ``>`` to redirect the output to a file. |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``html_encode_raw`` | ``html_encode_raw``, ``htmler`` | Same as ``html_encode`` but will not print a hexdump if it contains non-printable characters. It is suggested you use ``>`` to redirect the output to a file. |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``url_decode_raw`` | ``url_decode_raw``, ``urldr`` | Same as ``url_decode`` but will not print a hexdump if it contains non-printable characters. It is suggested you use ``>`` to redirect the output to a file. |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``url_encode_raw`` | ``url_encode_raw``, ``urler`` | Same as ``url_encode`` but will not print a hexdump if it contains non-printable characters. It is suggested you use ``>`` to redirect the output to a file. |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``gzip_decode_raw`` | ``gzip_decode_raw``, ``gzdr`` | Same as ``gzip_decode`` but will not print a hexdump if it contains non-printable characters. It is suggested you use ``>`` to redirect the output to a file. |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``gzip_encode_raw`` | ``gzip_encode_raw``, ``gzer`` | Same as ``gzip_encode`` but will not print a hexdump if it contains non-printable characters. It is suggested you use ``>`` to redirect the output to a file. |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``unixtime_decode`` | ``unixtime_decode``, ``uxtd`` | Take in a unix timestamp and print a human readable timestamp |
+---------------------------+-------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Interceptor
-----------
@ -1033,6 +1280,8 @@ error checking.
+----------------------------------------+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``export <req|rsp> <reqid>`` | ``export`` | Writes either the full request or response to a file in the current directory. |
+----------------------------------------+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``merge <dbfile>`` | ``merge`` | Add all the requests from another datafile to the current datafile |
+----------------------------------------+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+
Response streaming
~~~~~~~~~~~~~~~~~~
@ -1146,6 +1395,116 @@ plugin for it. You may also consider keeping a ``mine.py`` plugin where
you can write out commands that you use regularly but may not be worth
creating a dedicated plugin for.
Global Settings
---------------
There are some settings that apply to Pappy as a whole and are stored in
``~/.pappy/global_config.json``. These settings are generally for tuning
performance or modifying behavior on a system-wide level. No information
about projects is put in here since it is world readable. You can
technically add settings in here for plugins that you write, but if it's
at all possible, please keep settings in the normal project config.
Settings included in ``~/.pappy/global_config.json``:
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Setting | Description |
+===============+===============================================================================================================================================================================+
| cache\_size | The number of requests from history that will be included in memory at any given time. Set to -1 to keep everything in memory. See the request cache section for more info. |
+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Using a SOCKS Server
--------------------
Pappy allows you to use an upstream SOCKS server. You can do this by
adding a ``socks_proxy`` value to config.json. You can use the following
for anonymous access to the proxy:
::
"socks_proxy": {"host":"socks.proxy.host", "port":5555}
To use credentials you add a ``username`` and ``password`` value to the
dictionary:
::
"socks_proxy": {"host":"socks.proxy.host", "port":5555, "username": "mario", "password":"ilovemushrooms"}
Anything that passes through any of the active listeners will use the
proxy.
Transparent Host Redirection
----------------------------
Sometimes you get a frustrating thick client that doesnt let you mess
with proxy settings to get it to go through a proxy. However, if you can
redirect where it sends its traffic to localhost, you can get Pappy to
take that traffic and redirect it to go where it should.
It takes root permissions to listen on low numbered ports. As a result,
well need to do some root stuff to listen on ports 80 and 443 and get
the data to Pappy. There are two ways to get the traffic to Pappy. The
first is to set up port forwarding as root to send traffic from
localhost:80 to localhost:8080 and localhost:443 to localhost:8443
(since we can listen on 8080 and 8443 without root). Or you can YOLO,
run Pappy as root and just have it listen on 80 and 443.
According to Google you can use the following command to forward port 80
on localhost to 8080 on Linux:
::
iptables -t nat -A PREROUTING -i ppp0 -p tcp --dport 80 -j REDIRECT --to-ports 8080
Then to route 443 to 8443:
::
iptables -t nat -A PREROUTING -i ppp0 -p tcp --dport 443 -j REDIRECT --to-ports 8443
Of course, both of these need to be run as root.
Then on mac its
::
echo "
rdr pass inet proto tcp from any to any port 80 -> 127.0.0.1 port 8080
rdr pass inet proto tcp from any to any port 443 -> 127.0.0.1 port 8443
" | sudo pfctl -ef -
Then to turn it off on mac its
sudo pfctl -F all -f /etc/pf.conf
Then modify the listener settings in the projects config.json to be:
::
"proxy_listeners": [
{"port": 8080, "interface": "127.0.0.1", "forward_host": "www.example.faketld"},
{"port": 8443, "interface": "127.0.0.1", "forward_host_ssl": "www.example.faketld"},
]
This configuration will cause Pappy to open a port on 8080 that will
accept connections normally and a port on 8443 which will accept SSL
connections. The forward\_host setting tells Pappy to redirect any
requests sent to the port to the given host. It will also update the
requests host header. forward\_host\_ssl does the same thing, but it
listens for SSL connections and forces the connection to use SSL.
Or if youre going to YOLO it do the same thing then listen on port
80/443 directly. I do not suggest you do this.
::
"proxy_listeners": [
{"port": 80, "interface": "127.0.0.1", "forward_host": "www.example.faketld"},
{"port": 443, "interface": "127.0.0.1", "forward_host_ssl": "www.example.faketld"},
]
Pappy will automatically use this host to make the connection and
forward the request to the new server.
FAQ
---
@ -1175,11 +1534,87 @@ saved to disk. In between the time when a request is decoded and when
it's saved to disk, it will have an ID of ``--``. So just wait a little
bit and it will get an ID you can use.
Boring, Technical Stuff
-----------------------
I do some stuff to try and keep speed and memory usage to reasonable
levels. Unfortunately, things might seem slow in some areas. This is
where I try and explain why those exist. Honestly, you probably don't
care about this, but I'd rather have it written down and have nobody
read it than just leave people in the dark.
Request Cache / Memory usage
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
For performance reasons, Pappy by default will not store every request
in memory. The cache will store a certain number of the most recently
accessed requests in memory. This means that if you go through all of
history, it could be slow (for example running ``ls a`` or ``sm``). If
you have enough RAM to keep everything in memory, you can set the
request cache size to -1 to just keep everything in memory. However,
even if the cache size is unlimited, it still won't load a request into
memory untill you access it. So if you want to load everything in
memory, run ``ls a``.
By default, Pappy will cache 2000 requests. This is kind of heavy, but
it's assumed you're doing testing on a reasonably specced laptop.
Personally, I live on the edge and use -1 until I run into memory
issues.
Changelog
---------
The boring part of the readme
- 0.2.7
- boring unit tests
- should make future releases more stable I guess
- Support for upstream SOCKS servers
- ``print_params`` command
- ``inv`` filter
- ``param_info`` command
- Filters by request/response only headers/body
- Transparent host redirection
- Some easier to type aliases for common commands
- 0.2.6
- Fix pip being dumb
- ``watch`` command to watch requests/responses in real time
- Added ``pp[qs] form <id>`` to print POST data
- Bugfixes
- 0.2.5
- Requests sent with repeater now are given ``repeater`` tag
- Add ppq and pps commands
- Look at the pretty prompt
- Bugfixes
- 0.2.4
- Add command history saving between sessions
- Add html encoder/decoder
- All the bugs were fixed so I added some more for 0.2.5
- 0.2.3
- Decoder functions
- Add ``merge`` command
- Bugfixes
- 0.2.2
- COLORS
- Performance improvements
- Bugfixes (duh)
- 0.2.1
- Improve memory usage
- Tweaked plugin API
- 0.2.0
- Lots of refactoring

View file

@ -272,10 +272,10 @@ Using defer.inlineCallbacks With a Command
.. note::
This tutorial won't tell you how to use inlineCallbacks in general. Type "twisted inline callbacks" into google to figure out what they are. This is mainly just a reminder to use the ``crochet`` wrapper for console commands and warning you that some functions may return deferreds that you may have to deal with.
Since you're writing a plugin, you'll probably be using functions which return a deferred. And to keep things readable, you'll want to use the ``defer.inlineCallbacks`` function wrapper. Unfortunately, you can't bind async functions to commands. Luckily, there's a library called `crochet <https://pypi.python.org/pypi/crochet>`_ which lets you add another wrapper to the function that lets it be used like a blocking function. Rather than talking about it, let's write a plugin to call :func:`pappyproxy.console.load_reqlist` to print out some requests' hosts. Let's start by pretending it's a normal function::
Since you're writing a plugin, you'll probably be using functions which return a deferred. And to keep things readable, you'll want to use the ``defer.inlineCallbacks`` function wrapper. Unfortunately, you can't bind async functions to commands. Luckily, there's a library called `crochet <https://pypi.python.org/pypi/crochet>`_ which lets you add another wrapper to the function that lets it be used like a blocking function. Rather than talking about it, let's write a plugin to call :func:`pappyproxy.util.load_reqlist` to print out some requests' hosts. Let's start by pretending it's a normal function::
import shlex
from pappyproxy.console import load_reqlist
from pappyproxy.util import load_reqlist
def print_hosts(line):
args = shlex.split(line)
@ -309,10 +309,10 @@ And we run it::
iteration over non-sequence
pappy>
Iteration over a non-sequence? what? Well, :func:`pappyproxy.console.load_reqlist` doesn't actually return a list of requests. It returns a deferred which returns a list of requests. I'm not going into the details (look up some stuff on using inline callbacks with Twisted if you want more info), but the way to fix it is to slap an ``inlineCallbacks`` wrapper on the function and ``yield`` the result of the function. Now it looks like this::
Iteration over a non-sequence? what? Well, :func:`pappyproxy.util.load_reqlist` doesn't actually return a list of requests. It returns a deferred which returns a list of requests. I'm not going into the details (look up some stuff on using inline callbacks with Twisted if you want more info), but the way to fix it is to slap an ``inlineCallbacks`` wrapper on the function and ``yield`` the result of the function. Now it looks like this::
import shlex
from pappyproxy.console import load_reqlist
from pappyproxy.util import load_reqlist
from twisted.internet import defer
@defer.inlineCallbacks
@ -336,7 +336,7 @@ However, the console assumes that any functions it calls will be blocking. As a
import shlex
import crochet
from pappyproxy.console import load_reqlist
from pappyproxy.util import load_reqlist
from twisted.internet import defer
@crochet.wait_for(timeout=None)
@ -394,7 +394,7 @@ Here is an example plugin for storing the user-agent (if it exists) in the ``plu
import shlex
from twisted.internet import defer
from pappyproxy.console import load_reqlist
from pappyproxy.util import load_reqlist
from pappyproxy.plugin import main_context
from pappyproxy.util import PappyException
@ -435,8 +435,7 @@ Here is an example plugin for storing the user-agent (if it exists) in the ``plu
Useful Functions
----------------
* Load a request by id: :func:`pappyproxy.http.Request.load_request`
* Create a filter from a filter string: :func:`pappyproxy.context.Filter.from_filter_string`
See :mod:`pappyproxy.plugin` and :mod:`pappyproxy.util` for useful functions
Built In Plugins As Examples
============================