Implementing Kerberos RPC encryption over SSPI
Pavlo Myroniuk April 05, 2025 #kerberos #rpc #sspi #rust #cryptographyIntro
Today's world has many unknown and magical things. Proprietary protocols and libraries are among them ๐คช. In this blog post, I will explain how RPC encryption works under the hood and how to implement it over the SSPI interface.
Goals
- Provide a detailed explanation of how RPC PDUs are encrypted.
- Explain how it is related to SSPI.
- Implement RPC PDUs encryption/decryption and test it against real RPC traffic ๐คญ.
Non-goals
- I'm not explaining RPC use cases or how to set it up.
- Do not expect RPC internals explanations.
This blog post is only about encrypting and decrypting RPC PDUs.
Getting started
I assume the reader has enough knowledge and experience with RPC and SSPI to read this article. If not, I highly recommend reading the following articles (they should give enough context to understand what I am going to do):
Microsoft frequently uses RPC for local and remote calls. Of course, many remote RPC calls are encrypted, and the caller needs to pass the authentication to communicate with the server.
Let's take, for example, [MS-GKDI]: Group Key Distribution Protocol,
...which enables clients to obtain cryptographic keys associated with Active Directory security principals.
It specifies only one GetKey
RPC method. I hope it's obvious that the key is encrypted and cannot be sent over the network as plaintext.
(If you want to open and analyze this packet with me, then here is a Wireshark recording: rpc.pcapng.)
Captured RPC communication shows us all authentication steps and encrypted GetKey
RPC method call. And we will decrypt it at the end of this article ๐.
RPC PDU structure
(Alternatively, you can read the RPC Payload section from the RPC Encryption - An Exercise in Frustration article).
RPC PDU is usually split into three parts (https://pubs.opengroup.org/onlinepubs/9629399/chap12.htm):
name | purpose |
---|---|
header | contains protocol control information |
body | the body of a request or response PDU contains data representing the input or output parameters for an operation |
security trailer | contains data specific to an authentication protocol. For example, an authentication protocol may ensure the integrity of a packet via inclusion of an encrypted checksum in the authentication verifier |
But for now, we need to dig a bit deeply. PDU body consists of its header and data. In turn, the PDU security trailer also has its header and auth value. Look at the screenshot below:
I think you've got the idea. In our case, the PDU body data and security trailer auth value are encrypted.
RPC encryption over SSPI
Why SSPI? There are several reasons why:
- This is how it is originally done in Windows.
- Some aspects of the encryption process make sense only with SSPI in mind.
- RPC and SSPI are interconnected in many places, so why not explain the SSPI part?
RPC PDUs are encrypted by calling the SSPI::EncryptMessage function (and, correspondingly, decryption is done by calling the SSPI::DecryptMessage function). Let's see its interface.
// https://learn.microsoft.com/en-us/windows/win32/api/sspi/nf-sspi-encryptmessage
SECURITY_STATUS SEC_ENTRY ;
// https://learn.microsoft.com/en-us/windows/win32/api/sspi/nf-sspi-decryptmessage
SECURITY_STATUS SEC_ENTRY ;
// https://learn.microsoft.com/en-us/windows/win32/api/sspi/ns-sspi-secbufferdesc
typedef struct _SecBufferDesc SecBufferDesc, *PSecBufferDesc;
// https://learn.microsoft.com/en-us/windows/win32/api/sspi/ns-sspi-secbuffer
typedef struct _SecBuffer SecBuffer, *PSecBuffer;
There are a lot of security buffer types and two security buffer flags defined in SSPI (SecBuffer structure (sspi.h
): BufferType
). But for now, we need only a few of them:
buffer type | meaning/purpose |
---|---|
SECBUFFER_TOKEN | in the general case, it contains the security token portion of the message. In our case, it contains a signature generated by the security package. |
SECBUFFER_DATA | contains input data to be processed by the security package. In our case, it's just data we want to encrypt or decrypt. |
Also, we have two possible buffer flags:
buffer flag | meaning |
---|---|
SECBUFFER_READONLY | the data in this buffer is read-only and is never can be overwritten by the security package. Usually, this flag is used to inform the security package about something. |
SECBUFFER_READONLY_WITH_CHECKSUM | The data in this buffer is included in the checksum calculation but not in the encryption/decryption process. |
As you can see, the message we want to encrypt is an array of buffers. The whole PDU is split into SSPI security buffers and passed to the SSPI::EncryptMessage
function. This process is described in the "Message Protection" section of the RPC Encryption - An Exercise in Frustration article. In short, it happens as follows:
security buffer type + flags | security buffer value | made-up name | |
---|---|---|---|
1 | SECBUFFER_DATA + SECBUFFER_READONLY_WITH_CHECKSUM | contains PDU header + PDU body header | Sign1 |
2 | SECBUFFER_DATA | contains PDU body data to be encrypted (in-place) | Enc |
3 | SECBUFFER_DATA + SECBUFFER_READONLY_WITH_CHECKSUM | contains PDU security trailer header | Sign2 |
4 | SECBUFFER_TOKEN | will contain PDU security trailer auth value | Token |
We will refer to these buffers a lot, so I added made-up names to each of them. This can be illustrated in the previous RPC request that we used as the example:
All encryption is always in-place. The security package overwrites the original data with encrypted data. This also means that the caller must allocate the last security buffer (4th which has the SECBUFFER_TOKEN
type), and the security package will overwrite it. The caller can get the token buffer length by calling the QueryContextAttributes
function with the SECPKG_ATTR_SIZES
parameter. The cbSecurityTrailer
field of the SecPkgContext_Sizes
structure contains a desired value.
SSPI::EncryptMessage
Before implementing it, I want to clarify some general function behavior. As I wrote above, it encrypts the input message in-place. Some security packages, besides encryption, also calculate checksum over the provided data. Even more, some security packages can only calculate a checksum over the provided data. Here is a citation from the MSDN:
Some packages do not have messages to be encrypted or decrypted but rather provide an integrity hash that can be checked.
In our case, both encryption and integrity hash (hmac) calculations happen. We have 4 buffers as an input message. The general encryption process looks like this:
// pseudo code
// Sign1: message[0] is PDU header + PDU body header. SECBUFFER_DATA + SECBUFFER_READONLY_WITH_CHECKSUM.
// Enc: message[1] is PDU body data. we need to encrypt it. SECBUFFER_DATA.
// Sign2: message[2] is PDU security trailer header. SECBUFFER_DATA + SECBUFFER_READONLY_WITH_CHECKSUM.
// Token: message[3] is PDU security trailer auth value. Currently, this buffer is blank (memory is allocated). SECBUFFER_TOKEN.
message = encrypt;
message = checksum;
Kerberos::EncryptMessage
Now that we have enough knowledge about the encryption process, we can talk about concrete encryption algorithms. The client can authenticate using one of the available protocols (security packages) like NTLM or Kerberos. The most secure these days is, of course, Kerberos. Our task is to understand how it works and implement it.
First, I should mention that the Kerberos::EncryptMessage
function behaves as the GSS_WrapEx
function. It encrypts the data, constructs a Wrap Token, and writes it in the input message. The wrap token construction and encryption process are defined in RPC 4121. Let's recall the token structure:
Octet no Name Description
--------------------------------------------------------------
0..1 TOK_ID Identification field. Tokens emitted by
GSS_Wrap() contain the hex value 05 04
expressed in big-endian order in this
field.
2 Flags Attributes field, as described in section
4.2.2.
3 Filler Contains the hex value FF.
4..5 EC Contains the "extra count" field, in big-
endian order as described in section 4.2.3.
6..7 RRC Contains the "right rotation count" in big-
endian order, as described in section
4.2.5.
8..15 SND_SEQ Sequence number field in clear text,
expressed in big-endian order.
16..last Data Encrypted data for Wrap tokens with
confidentiality, or plaintext data followed
by the checksum for Wrap tokens without
confidentiality, as described in section
4.2.4.
This can look familiar to you because we already saw this structure in the Wireshark:
If we want to decrypt the RPC request, then we need to know the meaning of each field:
name | meaning |
---|---|
TOK_ID | 0x0504 |
Flags | They depend on the authentication process. |
Filler | 0xff |
EC | RFC 4121: EC Field. It represents how many filler bytes we must insert after the plaintext data before encryption. In our case, it is equal to 16 (The sender should set extra count (EC) to 1 block - 16 bytes.) |
RRC | In our case, it is equal to 28. The RRC field is 28 if encryption is requested. |
SND_SEQ | This value is set by the Kerberos implementation. |
It might not make much sense to you so far, but soon enough, you will have one picture.
RFC 4121 also defines the encryption process:
The resulting Wrap token is
{"header" | encrypt(plaintext-data | filler | "header")}
, whereencrypt()
is the encryption operation.
It looks simple, and I think you got the idea. But there is a tricky moment. The negotiated Kerberos encryption algorithm is AES256-CTS-HMAC-SHA1-96
. This algorithm has two specific features:
- It appends a checksum (hmac) over the plaintext to the ciphertext.
- It prepends one block of random bytes (named confounder) to the plaintext data before encryption (it is a non-deterministic encryption).
So, the actual encryption scheme is a bit more complex and looks something like this:
// pseudocode
// I hope you didn't forget the `message` array from the previous pseudo code block.
let ec = 16;
let filler = ec * 0x00;
let confounder = ;
let data_to_encrypt = message | filler | wrap_token_header;
let ciphertext = encrypt;
let data_to_hmac = message | message | message | filler | wrap_token_header;
let checksum = hmac;
let wrap_token = wrap_token_header | ciphertext | checksum;
Cool. Can we finally write the resulting Wrap Token to input buffers?
Not yet ๐คญ. The RFC also defined the right rotation operation after the encryption. RFC 4121: RRC Field:
The "RRC" (Right Rotation Count) field in Wrap tokens is added to allow the data to be encrypted in-place by existing SSPI applications.
Excluding the first 16 octets of the token header, the resulting Wrap token in the previous section is rotated to the right by "RRC" octets. The net result is that "RRC" octets of trailing octets are moved toward the header.
Consider the following as an example of this rotation operation: Assume that the RRC value is 3 and the token before the rotation is
{"header" | aa | bb | cc | dd | ee | ff | gg | hh}
. The token after rotation would be{"header" | ff | gg | hh | aa | bb | cc | dd | ee }
.
It means, that we need to do the right-rotation operation on ciphertext | checksum
. But pay attention here: we must rotate it by RRC + RC bytes. The final Wrap Token looks like this:
// pseudocode
let rrc = 28;
let ec = 16;
let wrap_token = wrap_token_header | rotate_right;
The very last step is to split the resulting Wrap Token into buffers and write them into corresponding input buffers (input message). Let's recall the task. We have 4 input buffers:
- Sign1:
Data
buffer withSECBUFFER_READONLY_WITH_CHECKSUM
. - Enc:
Data
buffer. - Sign2:
Data
buffer withSECBUFFER_READONLY_WITH_CHECKSUM
. - Token:
Token
buffer.
And now we need to write the Wrap Token buffer into the second and fourth buffers. To do it correctly, we write first cbSecurityTrailer
bytes of the Wrap Token into the Token buffer, and the rest of the Wrap Token should be written into the Enc buffer.
// pseudocode
let cb_security_trailer = 76;
let = wrap_token.split_at;
// Write into the Enc buffer
message.copy_from_slice;
// Write into the Token buffer
message.copy_from_slice;
Some curious readers may ask me:
Why can't we fill the Enc buffer first and then just write the rest of the Wrap Token into the Token buffer? Most likely because the RPC server expects the same buffers, as you just explained.
But the question is why it works this way and not another.
Some things are done on purpose, while others are simply historical heritage (legacy). If you write down all these data manipulations, you can notice that, in the end, the Enc buffer ciphertext matches the original data. The checksum, encrypted confounder, and Wrap Token header will be in the Token buffer. Look at the scheme below:
I tried to show the whole encryption process in the diagram above. As you can see, the encrypted Enc buffer data is written directly into the Enc buffer. But the Wrap Token Header, encrypted confounder, filler, etc., are written exclusively in the Token buffer.
Assumptions
Now we know all the details of the encryption process. Of course, we can't know all the reasons behind Microsoft's decisions regarding their protocols and implementations. But for now, I can make a few assumptions:
- Why do we need to rotate
ciphertext | checksum
?. To make the encrypted Enc buffer data match the unencrypted one. This way, the encryption will be in-place. - Why is the EC value equal to 16?. Just to extend the ciphertext (to make it longer on purpose). Why do we need to make it longer? Kerberos AES256-CTS-HMAC-SHA1-96 encryption algorithm uses AES256 in CTS mode. In short, it changes the last two blocks of ciphertext, so the padding is unnecessary. To ensure that the encrypted Enc buffer data is not affected during CTS (cipher text stealing) (otherwise, it would be impossible for the encrypted Enc buffer data to match the unencrypted one), we need to be sure that the last two blocks of the ciphertext do not contain the Enc buffer data. Thus, we have two additional blocks: the first block is the filler bytes (
0x00 * 16
) and the second block is the Wrap Token header (16 bytes long). - Why is the
cbSecurityTrailer
value equal to 76? wrap token header len + confounder len + filler len + wrap token header (encrypted) len + checksum len = 16 + 16 + 16 + 16 + 12 = 76.
Alternatively, you can read Microsoft's example of the message encryption. [MS-KILE]: GSS_WrapEx
with AES128-CTS-HMAC-SHA1-96
:
The picture above shows the same process but with two buffers to encrypt instead of one. clearhdr
- Wrap Token header. padding
= ec * 0x00
.
Phew ๐ฎโ๐จ I hope you are not tired because we are going to implement it ๐ค.
Code: Encryptor
Note 1. The full code is available here: RPC decryptor github/TheBestTvarynka/trash-code/rpc-decryptor.
Note 2. The code in this article is not production-ready! The only purpose of the code below is to show the algorithm's correctness.
Let's start with simple things. Here is a small SecBuffer
implementation. We don't need anything more complex.
pub const DATA: u32 = 1;
pub const TOKEN: u32 = 2;
pub const READONLY_WITH_CHECKSUM_FLAG: u32 = 0x10000000;
Now we need a Wrap Token implementation. I don't want to overengineer it. I wrote a simple Wrap Token header encoding/decoding. Again, we don't need anything more complex.
Great! Now the fun part: encryption implementation. First of all, we should define the constants we are going to use.
// "Extra count"
const EC: u16 = 16;
// "Right Rotation Count"
const RRC: u16 = 28;
// https://www.rfc-editor.org/rfc/rfc4121.html#section-2
const CLIENT_ENCRYPTION_KEY_USAGE: i32 = 24;
const CLIENT_DECRYPTION_KEY_USAGE: i32 = 22;
const SERVER_ENCRYPTION_KEY_USAGE: i32 = CLIENT_DECRYPTION_KEY_USAGE;
const SERVER_DECRYPTION_KEY_USAGE: i32 = CLIENT_ENCRYPTION_KEY_USAGE;
const CB_SECURITY_TRAILER: usize = 16 /* wrap token header len */
+ 16 /* confounder len */
+ EC as usize
+ 16 /* wrap token header (encrypted) len */
+ 12 /* checksum (hmac) len */;
You may be surprised by key usage numbers. Don't worry.
The encryption and checksum specifications in RFC3961 require as input a "key usage number", to alter the encryption key used in any specific message to make certain types of cryptographic attack more difficult.
In other words, a key usage number is a public known value used to derive the encryption key from a base key (session key). You should not care about it. We always use key usage numbers specified in the RFC/specification.
Now we can start implementing RPC encryption. Finally! We start with the Wrap Token and filler generation.
Then, we encrypt the data. Let's do it in two steps:
- Collect the data to encrypt in one buffer: DATA sec buffer + filler + wrap token header.
- Encrypt it with the correct key usage number. There are not many Kerberos encryption algorithm implementations. The only actively used and well-maintained one is
picky-krb
: docs.rs/picky-krb/crypto/aes/.
I hope you didn't forget about the confounder. In the code above, it is autogenerated and returned from the encrypt_no_checksum
function. We need it to correctly calculate HMAC. What is next? Right, checksum calculation.
As you can see, we calculate checksum separately from the encryption because we may have READONLY_WITH_CHECKSUM_FLAG
buffers which we don't need to encrypt.
๐ฎโ๐จ What's left? Right rotation, final Wrap Token construction, copying the data to the input message
.
That's all! We just implemented RPC Kerberos encryption! ๐ ๐ฅณ
Code: Decryptor
Note. The full code is available here: RPC decryptor github/TheBestTvarynka/trash-code/rpc-decryptor.
I didn't explain the decryption process in detail, but it is completely the reverse of encryption. Look at the scheme below (you may need to zoom in or open it in a new tab):
As you can see, the decryption process is a reversed encryption process (obviously, makes sense). Time to start implementing it. We start with the Wrap Token construction and split it into the header and encrypted parts.
Then, we do the left rotation and data decryption:
Nice! And, of course, we need to calculate the checksum and compare it with the provided one.
๐ฎโ๐จ And the last one: copying the plaintext to the input message
.
๐ ๐ฅณ That's all! Now we have encryptor and decryptor implemented.
Code: testing
Let's add a simple test to ensure that it works at least against itself (encrypt-decrypt test). I wrote small wrappers for the encryption/decryption functions to make testing easier:
Pretty straightforward and understandable. And most importantly, we don't need more. Here is a simple test:
Good results so far! But do you remember our main goal? We wanted to decrypt the captured encrypted RPC request. Let's do it! Now nothing can stop us ๐ ๐.
All these buffers were extracted from the RPC Request PDU, which you already saw at the start of this article. And yes, I already expect the question from you:
Wait! Wait! Wait! Where the hell did you get the correct (I suppose correct) session key? ๐คจ
๐ I know the user's password, so I can decrypt the previous Kerberos messages and extract the session key ๐คญ ๐คซ. Maybe I'll write an article about it someday. Ping me if you need to know how to do it ๐.
Enough talking. Let's run this test. We've been waiting for this for so long.
Yeeeeeey. We have successfully decrypted the RPC request! It proves that the implementation is correct! The plaintext buffer on the screenshot is decrypted GetKey
RPC request.
Hope you find this article informative and useful. Cheers! ๐
Unaddressed topics
If you want to get any of the following topics explained, feel free to contact me.
- NTLM RPC encryption. But if you are really interested, you can read how it is implemented in
sspi-rs
: github.com/Devolutions/sspi-rs/193f92a0fd73e43d672e3f9ac5775c894c02b200/src/ntlm/mod.rs#L419-L455. - Kerberos auth process. Three-Leg DCE-Style Mutual Authentication.
- How I decrypted Kerberos traffic and extracted the session key.