Decrypting ESP #2: Understanding the Differences Between the Two Modes of IPsec After Decryption
*This article is an English translation of the Engineer Notes article as of Jun. 20, 2023.
*Please note that the content may have been updated since then.
Decrypting ESP #1 covered the mechanism of IPsec and the decryption process on Wireshark.
This time, we will actually input SA information, etc., and decrypt the packets on Wireshark to see the details of the protocol.
Table of Contents
Environment Used for Decryption
Comparing ESP Before and After Decryption - Transport Mode
Comparing ESP Before and After Decryption - Tunnel Mode
Environment Used for Decryption
The environment used this time is as follows. We quickly checked with pcap files from the official Wireshark website.
The official site offers various protocols besides IPsec, so if you're interested, check out the Wireshark Sample Captures page.
- PC Environment
- Windows 10 Pro
- Wireshark Version 4.0.6
- ESP Data
- Wireshark SampleCapture #IPsec pcaps and SA information
Comparing ESP Before and After Decryption - Transport Mode
Download and extract "ipsec_esp.tgz" from the Sample Capture page, and you will find four sample pcaps and SA information. Among them, "ipsec_esp_capture_1" is the ESP data in transport mode, so let's check with that file.
You can manually input the SA settings as described in Decoding ESP #1, or you can copy "esp_sa" from the same folder as the pcap to the Wireshark settings folder. The Wireshark settings folder is shown in the red frame on the SA registration screen.
First, let's check the screen before decryption.
ESP in Transport Mode Before Decryption
The part in the red frame is encrypted.
Below is the screen after decryption.
ESP in Transport Mode After Decryption
We can see that the upper layer protocol is ICMP after decryption.
By the way, the "SPI" item set in Wireshark can be checked even before decryption.
The SPI needs to be uniquely identified between the IPsec devices of the sender and recipient, and the values differ for communication from "A to B" and from "B to A".
Comparing ESP Before and After Decryption - Tunnel Mode
Next, let's check "ipsec_esp_capture_2". This is the ESP data in tunnel mode.
ESP in Tunnel Mode Before Decryption
ESP in Tunnel Mode After Decryption
Unlike the transport mode, two ESP headers and two IP headers are decoded. The upper layer protocol is ICMP.
Transport Mode vs Tunnel Mode
Transport Mode
Transport mode operates at the transport layer of IPsec. In transport mode, the IP packet header remains unchanged, and only the payload of the communication is encrypted.
In other words, the source and destination IP addresses remain the same.
Tunnel Mode
Tunnel mode operates at the network layer of IPsec. In tunnel mode, both the IP packet header and payload are encrypted, and a new IP header is added.
Specifically, the source and destination IP addresses are replaced with the IP addresses of the IPsec gateways (or endpoints) for communication.
Difference Between the Two Modes
The difference between the two modes can be compared in the frame format as follows.
Comparing transport mode and tunnel mode, we can see that tunnel mode adds a new IP header. This means that the entire communication at the IP layer is encrypted, ensuring overall security from the source to the destination.
On the other hand, in transport mode, the original IP header is maintained, so only layers above the transport layer are encrypted.
Therefore, when using IPsec in typical VPNs, tunnel mode is used to secure the entire path. The advantage of transport mode is that it has fewer headers, making it beneficial for improving communication efficiency in special networks with multiple encapsulations (such as GRE over IPsec).
However, usually, encryption at the transport layer is more often done with TLS than IPsec.
SYNESIS Decode Screen with Decryption
Our product, SYNESIS, has a decryption function for encrypted packets. By pre-configuring the protocol settings, encrypted packets can be decrypted and displayed.
Supported Decryptable Protocols
- SSL/TLS
- ESP (IPsec)
SYNESIS Decode Screen with Decryption
If you are interested in SYNESIS with packet decryption, please contact us below.
Afterword
This time, I checked with pcap files on the official Wireshark website to test easily. The official site offers various protocol pcaps, so you might find hard-to-obtain files on this page.
This article explains using pcaps published on the official site. Therefore, global addresses are displayed as they are. Please understand this point.