In order to learn a bit more about how both the TCP and HTTP/2 protocols work, I recently created the tcp-h2-describe reverse proxy in Python. I was excited about some of the insights I was able to glean from the process, in particular a full example tracing a connection between a gRPC client and server over HTTP/2.

When a serialized protobuf comes across the wire as an HTTP/2 DATA frame, the tcp-h2-describe proxy is able to give a faithful description of the frame:

...
----------------------------------------
Frame Length = 21 (00 00 15)
Frame Type = DATA (00)
Flags = END_STREAM:0x1 (01)
Stream Identifier = 3 (00 00 00 03)
gRPC Compressed Flag = 0 (00)
Protobuf Length = 16 (00 00 00 10)
Protobuf Message (users.v1.User) =
   first_name: "Alice"
   last_name: "Redmond"
Hexdump (Protobuf Message) =
   0a 05 41 6c 69 63 65 12 07 52 65 64 6d 6f 6e 64
----------------------------------------
...

In addition to decoding the data frame and providing a "pretty" description of the deserialized users.v1.User coming across the wire, this also preserves the raw bytes being sent as part of the larger TCP packet data. For example in ... 6c 69 63 ... the 69 (0x69 in hexadecimal is 105 in decimal) corresponds to the ASCII encoding of the i in Alice.

Why?

When I excitedly told my CTO Eugene "look what I built", he quickly said "is this like Wireshark?" and he hit the nail on the head. So why did I build something much less capable than Wireshark? Well the first answer to that question is something like "Oops".

It gets even better: I originally went down the path of implementing this myself because of a silly mistake. I was working on a Kubernetes service running a gRPC server (more on that in a moment) and failed to get off-the-shelf reverse proxies working. I tried seven different (e.g.) existing proxies of varying quality (some were just in GitHub gists) but all of them failed to actually do anything at all within the container. The problem: I kept binding the proxy to loopback IP (i.e. localhost or 127.0.0.1) rather than the broadcast IP (0.0.0.0).

In the end though I'm better for not having been "able" to use off-the-shelf reverse proxies because I got to learn more about the guts of TCP (which I use all the time) and HTTP/2 (which is new to me).

Where?

As for where this code is running, I mentioned I'd say a bit more about the Kubernetes service running a gRPC server. I'm currently doing some discovery¹ on gRPC patterns; I'm trying to understand if gRPC can work for me and possibly my team or other groups at Blend.

Within our infrastructure, I had a Kubernetes pod handling raw TCP traffic on a port of my choosing. Unfortunately the AWS ELB² uses the proxy protocol to add a prefix to the packet data from the first TCP segment. This prefix gives information about the proxied IP for raw TCP connections that may not be able to convey³ such information, e.g.

PROXY TCP4 198.51.100.22 203.0.113.7 35646 80

Due to my loopback IP vs. broadcast IP mixup, I wasn't able to get our proxyprotocol package working with the google/tcpproxy package. Without a reverse proxy that could strip the prefix from the TCP packet data, my gRPC server could not handle the request. So I set off to understand TCP a little bit better and write my own reverse proxy that could strip the proxy protocol prefix.

What?

The tcp-h2-describe reverse proxy directs HTTP/2 traffic⁴ from a client to a server:

$ tcp-h2-describe --server-host website.invalid --server-port 80
Starting tcp-h2-describe proxy server on port 24909
  Proxying server located at website.invalid:80
...

Once a request is sent through, the TCP packet data is parsed as a series of HTTP/2 frames and each byte in the data is accounted for.

In the HTTP example provided in the library's documentation, proxied traffic is sent to a locally running HTTP/2 server. The packet data from the first TCP segment sent by the client contains 84 bytes:

00000000  50 52 49 20 2a 20 48 54  54 50 2f 32 2e 30 0d 0a  |PRI * HTTP/2.0..|
00000010  0d 0a 53 4d 0d 0a 0d 0a  00 00 24 04 00 00 00 00  |..SM......$.....|
00000020  00 00 01 00 00 10 00 00  02 00 00 00 01 00 04 00  |................|
00000030  00 ff ff 00 05 00 00 40  00 00 03 00 00 00 64 00  |.......@......d.|
00000040  06 00 01 00 00 00 00 06  04 00 00 00 00 00 00 02  |................|
00000050  00 00 00 00                                       |....|
00000054

We can see most of these bytes are not printable characters, however tcp-h2-describe breaks this TCP packet data down and explains how every single byte is used:

...
============================================================
client(127.0.0.1:59600)->proxy->server(localhost:8080)

Client Connection Preface = b'PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n'
Hexdump (Client Connection Preface) =
   50 52 49 20 2a 20 48 54 54 50 2f 32 2e 30 0d 0a
   0d 0a 53 4d 0d 0a 0d 0a
----------------------------------------
Frame Length = 36 (00 00 24)
Frame Type = SETTINGS (04)
Flags = UNSET (00)
Stream Identifier = 0 (00 00 00 00)
Settings =
   SETTINGS_HEADER_TABLE_SIZE:0x1 -> 4096 (00 01 | 00 00 10 00)
   SETTINGS_ENABLE_PUSH:0x2 -> 1 (00 02 | 00 00 00 01)
   SETTINGS_INITIAL_WINDOW_SIZE:0x4 -> 65535 (00 04 | 00 00 ff ff)
   SETTINGS_MAX_FRAME_SIZE:0x5 -> 16384 (00 05 | 00 00 40 00)
   SETTINGS_MAX_CONCURRENT_STREAMS:0x3 -> 100 (00 03 | 00 00 00 64)
   SETTINGS_MAX_HEADER_LIST_SIZE:0x6 -> 65536 (00 06 | 00 01 00 00)
----------------------------------------
Frame Length = 6 (00 00 06)
Frame Type = SETTINGS (04)
Flags = UNSET (00)
Stream Identifier = 0 (00 00 00 00)
Settings =
   SETTINGS_ENABLE_PUSH:0x2 -> 0 (00 02 | 00 00 00 00)
----------------------------------------
...

Go Deeper?

I was eventually able to handle the proxy protocol line that gets prepended by the AWS ELB (see an example). In order to handle proxy protocol and regular TCP connections the same way I had to use MSG_PEEK to avoid reading bytes I didn't want to use while handling the proxy protocol prefix⁵.

I did read a bit about going deep enough in the stack to actually construct a TCP packet, but luckily I was able to avoid having to deal with TCP segment structure

and was just able to focus on the TCP package data. This is because the socket module in Python's standard library provides a very nice interface intended to emulate the

Unix system call and library interface for sockets

In particular it allows focusing on the data within a TCP packet via recv() and send().

Once I got a handle on this process I was able to implement a reverse proxy that opens two sockets: one for the client making a TCP request to the proxy and one for the server being proxied. It simultaneously calls recv() and send() on each socket, making sure to act as a trampoline that sends data between the client and the server. For example, any data returned from a recv() and server socket immediately gets passed to send() on the client socket.

Since the data is "in the middle" while the proxy hands it from one socket to another, the proxy can decode the series of HTTP/2 frames contained in the data and print the findings to the console.

1. Many thanks to Alex Fish for being my sounding board during this process ↩
2. To my former Google Cloud colleagues, yes I use AWS every day ↩
3. This is in contrast to HTTP, where the X-Forwarded-For header is commonly used to indicate a proxy ↩
4. For now, this only supports unencrypted traffic, though it is possible to use TLS in such a proxy if the proxy is able to share the certs of the server being proxied ↩
5. Hat tip to Will Charczuk for the proxyprotocol package, which I used as a basis ↩