Decrypting ESP #2: Understanding the Differences Between the Two Modes of IPsec After Decryption

*This article is an English translation of the Engineer Notes article as of Jun. 20, 2023.
*Please note that the content may have been updated since then.

Original Article

Decrypting ESP #1 covered the mechanism of IPsec and the decryption process on Wireshark.
This time, we will actually input SA information, etc., and decrypt the packets on Wireshark to see the details of the protocol.

Environment Used for Decryption

The environment used this time is as follows. We quickly checked with pcap files from the official Wireshark website.
The official site offers various protocols besides IPsec, so if you're interested, check out the Wireshark Sample Captures page.

Comparing ESP Before and After Decryption - Transport Mode

Download and extract "ipsec_esp.tgz" from the Sample Capture page, and you will find four sample pcaps and SA information. Among them, "ipsec_esp_capture_1" is the ESP data in transport mode, so let's check with that file.

You can manually input the SA settings as described in Decoding ESP #1, or you can copy "esp_sa" from the same folder as the pcap to the Wireshark settings folder. The Wireshark settings folder is shown in the red frame on the SA registration screen.

ESP_SA Settings Screen

First, let's check the screen before decryption.

ESP Before Decryption

ESP in Transport Mode Before Decryption

The part in the red frame is encrypted.
Below is the screen after decryption.

ESP After Decryption

ESP in Transport Mode After Decryption

We can see that the upper layer protocol is ICMP after decryption.

By the way, the "SPI" item set in Wireshark can be checked even before decryption.
The SPI needs to be uniquely identified between the IPsec devices of the sender and recipient, and the values differ for communication from "A to B" and from "B to A".

IPsec SPI Value

Comparing ESP Before and After Decryption - Tunnel Mode

Next, let's check "ipsec_esp_capture_2". This is the ESP data in tunnel mode.

ESP Before Decryption

ESP in Tunnel Mode Before Decryption

ESP After Decryption

ESP in Tunnel Mode After Decryption

Unlike the transport mode, two ESP headers and two IP headers are decoded. The upper layer protocol is ICMP.

Transport Mode vs Tunnel Mode

Transport Mode

Transport mode operates at the transport layer of IPsec. In transport mode, the IP packet header remains unchanged, and only the payload of the communication is encrypted.
In other words, the source and destination IP addresses remain the same.

Tunnel Mode

Tunnel mode operates at the network layer of IPsec. In tunnel mode, both the IP packet header and payload are encrypted, and a new IP header is added.
Specifically, the source and destination IP addresses are replaced with the IP addresses of the IPsec gateways (or endpoints) for communication.

Difference Between the Two Modes

The difference between the two modes can be compared in the frame format as follows.

The Two Modes of IPsec

Comparing transport mode and tunnel mode, we can see that tunnel mode adds a new IP header. This means that the entire communication at the IP layer is encrypted, ensuring overall security from the source to the destination.
On the other hand, in transport mode, the original IP header is maintained, so only layers above the transport layer are encrypted.

Therefore, when using IPsec in typical VPNs, tunnel mode is used to secure the entire path. The advantage of transport mode is that it has fewer headers, making it beneficial for improving communication efficiency in special networks with multiple encapsulations (such as GRE over IPsec).
However, usually, encryption at the transport layer is more often done with TLS than IPsec.

SYNESIS Decode Screen with Decryption

Our product, SYNESIS, has a decryption function for encrypted packets. By pre-configuring the protocol settings, encrypted packets can be decrypted and displayed.

Supported Decryptable Protocols

  • SSL/TLS
  • ESP (IPsec)

SYNESIS Decoding Screen | TLS, ESP Decryption

SYNESIS Decode Screen with Decryption

If you are interested in SYNESIS with packet decryption, please contact us below.

Contact Us

Afterword

This time, I checked with pcap files on the official Wireshark website to test easily. The official site offers various protocol pcaps, so you might find hard-to-obtain files on this page.
This article explains using pcaps published on the official site. Therefore, global addresses are displayed as they are. Please understand this point.