Debugging HTTP services with mitmproxy
Posted on
I spend a lot of my time at work writing Go services that talk to other Go services over HTTP. Much of the time, everything works as expected, but every now and then a situation arises where I’m struggling to understand why my program is receiving a specific value. Is my request not being built correctly? Am I not properly deserializing the response? Logging can be helpful, but sometimes I really just want to look at the HTTP traffic between services.
One tool that I really love in these situations is mitmproxy: “a free and open source interactive HTTPS proxy,” according to its website.
There are no shortage of features and options for mitmproxy, and when I was first exploring it they were a little daunting. I’m sure there are still tons of things that it can do that I don’t know about, but the main thing I tend to use it for is reverse proxying.
Reverse proxying is a pretty simple concept; basically it means that you send your HTTP requests to a specific proxy endpoint, and the proxy repeats your request to some specific origin server. Then the same thing just happens backwards: the origin server sends its reply to the reverse proxy which passes it along back to you. This means that the proxy can see (and log!) the actual HTTP request you send, and the response sent by the origin server.
Earlier today, I was working on a program that sent requests to a HTTP server and my program’s output didn’t make sense. I wasn’t sure if my requests were being sent incorrectly, or maybe there was a bug in the server I was talking to. So I fired up mitmproxy to take a look. In my shell, I ran:
mitmproxy --mode reverse:https://service.dev.example.com --listen-port 8080
This opens up a log window where any requests handled by the reverse proxy will
be displayed. I quickly updated my program to make its requests to
http://localhost:8080 instead of https://service.dev.example.com and re-ran
it. The request and response were logged in the terminal window, and I was
quickly able to identify that a particular dependency needed to be updated.
Of course, you could also use Wireshark or tcpdump to inspect network traffic, and these are great options that I also use frequently! But the main reason I tend to turn first to mitmproxy is because it handles TLS like a honey badger -- it just doesn’t give a shit. Basically, you can throw whatever you want at it and it’ll just do the right thing:
Client --TLS--> mitmproxy --TLS--> Origin
Client --HTTP--> mitmproxy --TLS--> Origin
Client --TLS--> mitmproxy --HTTP--> Origin
Client --HTTP--> mitmproxy --HTTP--> Origin
How does this work? When you first run mitmproxy, it generates a certificate
authority that it uses to generate certificates on-the-fly. All you need to do
is add the CA certificate to your OS trust store (see their docs about
certificates here). For example, if I run mitmproxy --mode reverse:https://www.benburwell.com --listen-port 8080, and then connect over
SSL, I can see the certificate that mitmproxy generated:
$ openssl s_client -connect localhost:8080
CONNECTED(00000005)
depth=1 CN = mitmproxy, O = mitmproxy
verify error:num=19:self signed certificate in certificate chain
verify return:0
---
Certificate chain
0 s:/CN=www.benburwell.com
i:/CN=mitmproxy/O=mitmproxy
1 s:/CN=mitmproxy/O=mitmproxy
i:/CN=mitmproxy/O=mitmproxy
Here, mitmproxy CA has generated a certificate with CN=www.benburwell.com to
match the hostname I’m reverse proxying to!
Now, there are ways to snoop on TLS encrypted traffic with Wireshark as well using a TLS key log file, but this usually involves making somewhat non-trivial modifications to the program you’re working with. It’s not very complicated or difficult, and it’s a technique I’ve used a few times, but mitmproxy is usually quicker and easier for me. I plan to write a post about this topic in the future, so stay tuned! (Update: see my post about decrypting TLS in Wireshark)