Decrypting ESP #1: Architecture of IPsec and Wireshark Settings

*This article is an English translation of the Engineer Notes article as of Jun. 16, 2023.
*Please note that the content may have been updated since then.

Original Article

ESP (Encapsulated Security Payload) is one of the protocols used in IPsec (Internet Protocol Security) and ensures the confidentiality and integrity of data. In other words, data on EPS is encrypted, making it normally impossible to check the contents of packets in Wireshark. This time, I will explain the mechanism of IPsec and the settings and procedures for decrypting ESP in Wireshark.

Previously, I wrote an article on decrypting SSL/TLS in Wireshark. Please take a look if you are interested.

Mechanism of IPsec

IPsec is a protocol designed to ensure secure communication over the Internet. It comprises multiple protocols that provide data confidentiality, integrity, and authentication of the sender. In practical terms, IPsec is used to establish VPN connections. Here is an image of an IPsec VPN setup. ("VPNs may use protocols other than IPsec, so I use the term 'IPsec VPN' here..)

IPsec Diagram

IPsec uses multiple protocols to achieve secure communication. RFC6071 lists and summarizes the main RFC documents related to IPsec and IKE, as well as their relevance and dependencies. It is a good starting point to understand the overview.

As shown in this diagram, IPsec is implemented through combinations of protocols such as "ESP+IKE", "AH+IKE", or "AH+ESP+IKE". The overview of each protocol is as follows:

Protocol Role Type
ESP Encryption and authentication of IP packets IP Protocol Number: 50
AH Authentication of IP packets IP Protocol Number: 51
IKE Establishing Security Association (SA) and key exchange UDP Port: 500

In IPsec, the encryption function is handled by ESP. Typically, since the data above the IP layer in ESP packets is encrypted, it is not possible to know what protocol data is flowing through. In Wireshark, by setting the SA parameters used during encryption, it is possible to decrypt and display the encrypted part of the data.

Both AH and ESP provide authentication functionality, while ESP additionally offers data encryption. Therefore, if ESP alone is sufficient, implementing AH is unnecessary. However, it is important to note that AH includes the IP header in its authentication scope, whereas ESP only targets the payload. In some environments, data authentication alone is required, in which case AH is used. According to my research, the implementation of IPsec in Japan predominantly uses "ESP+IKE", with AH being used only in limited cases.

IKE plays a crucial role in establishing and maintaining secure communication. By establishing an SA between communications and exchanging appropriate keys and security parameters, it ensures the security functions of IPsec.

Steps to Decrypt ESP in Wireshark

With an understanding of IPsec, let's move on to the steps for decrypting ESP in Wireshark.

  1. Launch Wireshark, go to [Edit] menu > Preferences > Protocols > ESP, and check "Attempt to detect/decode encrypted ESP payloads".
  2. Click the [Edit] button next to "ESP SAs". Add the following items using the + button:

Wireshark ESP Settings Screen

Item Description
Protocol Select "IPv4" or "IPv6"
Src IP Source IP address
Dest IP Destination IP address
SPI Security Parameter Index, used to identify the SA
Encryption Select the encryption method
Encryption Key Enter the encryption key
Authentication Select the authentication method
Authentication Key Enter the authentication key

For more details, refer to the Wireshark Wiki ESP.

The encryption method and keys, as well as the authentication method and keys, cannot be discerned from the packet. (Obviously, if anyone could understand them, encryption would be meaningless...) This information is stored in the Security Association Database (SAD), which contains the parameters for the SA. The method of checking the SAD varies depending on the IPsec implementation, but it is often done using command lines or tools. Please consult the network administrator managing IPsec for more details.

Afterword

This time, I explained the architecture of IPsec and the steps to decrypt ESP, which handles the encryption function in IPsec. Next, I will explain how ESP packets look before and after decryption in Wireshark.

The packet capture product "SYNESIS" supports decryption of SSL/TLS and ESP in the decode screen from version 8.0. The basic input parameters are the same as in Wireshark. Please give it a try. For any questions about SYNESIS, please contact us below.

Contact Us