Decrypting ESP #1: Architecture of IPsec and Wireshark Settings
*This article is an English translation of the Engineer Notes article as of Jun. 16, 2023.
*Please note that the content may have been updated since then.
ESP (Encapsulated Security Payload) is one of the protocols used in IPsec (Internet Protocol Security) and ensures the confidentiality and integrity of data. In other words, data on EPS is encrypted, making it normally impossible to check the contents of packets in Wireshark. This time, I will explain the mechanism of IPsec and the settings and procedures for decrypting ESP in Wireshark.
Previously, I wrote an article on decrypting SSL/TLS in Wireshark. Please take a look if you are interested.
Mechanism of IPsec
IPsec is a protocol designed to ensure secure communication over the Internet. It comprises multiple protocols that provide data confidentiality, integrity, and authentication of the sender. In practical terms, IPsec is used to establish VPN connections. Here is an image of an IPsec VPN setup. ("VPNs may use protocols other than IPsec, so I use the term 'IPsec VPN' here..)
IPsec uses multiple protocols to achieve secure communication. RFC6071 lists and summarizes the main RFC documents related to IPsec and IKE, as well as their relevance and dependencies. It is a good starting point to understand the overview.
As shown in this diagram, IPsec is implemented through combinations of protocols such as "ESP+IKE", "AH+IKE", or "AH+ESP+IKE". The overview of each protocol is as follows:
| Protocol | Role | Type |
|---|---|---|
| ESP | Encryption and authentication of IP packets | IP Protocol Number: 50 |
| AH | Authentication of IP packets | IP Protocol Number: 51 |
| IKE | Establishing Security Association (SA) and key exchange | UDP Port: 500 |
In IPsec, the encryption function is handled by ESP. Typically, since the data above the IP layer in ESP packets is encrypted, it is not possible to know what protocol data is flowing through. In Wireshark, by setting the SA parameters used during encryption, it is possible to decrypt and display the encrypted part of the data.
Both AH and ESP provide authentication functionality, while ESP additionally offers data encryption. Therefore, if ESP alone is sufficient, implementing AH is unnecessary. However, it is important to note that AH includes the IP header in its authentication scope, whereas ESP only targets the payload. In some environments, data authentication alone is required, in which case AH is used. According to my research, the implementation of IPsec in Japan predominantly uses "ESP+IKE", with AH being used only in limited cases.
IKE plays a crucial role in establishing and maintaining secure communication. By establishing an SA between communications and exchanging appropriate keys and security parameters, it ensures the security functions of IPsec.
Steps to Decrypt ESP in Wireshark
With an understanding of IPsec, let's move on to the steps for decrypting ESP in Wireshark.
- Launch Wireshark, go to [Edit] menu > Preferences > Protocols > ESP, and check "Attempt to detect/decode encrypted ESP payloads".
- Click the [Edit] button next to "ESP SAs". Add the following items using the + button:
| Item | Description |
|---|---|
| Protocol | Select "IPv4" or "IPv6" |
| Src IP | Source IP address |
| Dest IP | Destination IP address |
| SPI | Security Parameter Index, used to identify the SA |
| Encryption | Select the encryption method |
| Encryption Key | Enter the encryption key |
| Authentication | Select the authentication method |
| Authentication Key | Enter the authentication key |
For more details, refer to the Wireshark Wiki ESP.
The encryption method and keys, as well as the authentication method and keys, cannot be discerned from the packet. (Obviously, if anyone could understand them, encryption would be meaningless...) This information is stored in the Security Association Database (SAD), which contains the parameters for the SA. The method of checking the SAD varies depending on the IPsec implementation, but it is often done using command lines or tools. Please consult the network administrator managing IPsec for more details.
Afterword
This time, I explained the architecture of IPsec and the steps to decrypt ESP, which handles the encryption function in IPsec. Next, I will explain how ESP packets look before and after decryption in Wireshark.
The packet capture product "SYNESIS" supports decryption of SSL/TLS and ESP in the decode screen from version 8.0. The basic input parameters are the same as in Wireshark. Please give it a try. For any questions about SYNESIS, please contact us below.
