Talos Vulnerability Report

Summary

A stack overflow vulnerability present in the PDF filter of KeyView as used by Domino can lead to process crash and possible arbitrary code execution.

Tested Versions

• KeyView 10.16 as used by IBM Domino 9.0.1

Product URLs

http://www-03.ibm.com/software/products/en/ibmdomino

Details

While parsing a specially crafted PDF file, a user controlled length field is used in a write loop with fixed destination size leading to a stack based buffer overflow. The vulnerability is triggered while parsing the PDF file that specifies an encrypted stream. As per the PDF specification, the Length value specifies the key length and is at most 16 bytes long. In the vulnerable function a stack buffer 16 bytes in length is reserved, but unchecked Length value is used during the copy operation which allows adjacent stack data to be overwritten, including the return address.

The minimized test case that triggers the vulnerability is as follows:

`
%PDF-1.4 
trailer
<</Size 9 
/Root 1 0 R
/Encrypt 8 0 R                    
8 0 obj<<
	/Length 768
	/Filter/Standard
	/Type/Catalog
	/O (41414141414141414141414141414141)
    /U (42424242424242424242424242424242) 
    /P 0 /R 3                           
	>> 
>>
obj<< >>
endobj
%%EOF
`

In the above test case, the PDF trailer specifies that object 8 is encrypted. Further, object 8 specifies that it is using a standard filter for encryption (/Filter/Standard) and is using a revision 3 (/R 3) of the algorithm. Owner password (/O) and user password (‘/U’), as well as object type don’t play a significant role in this test case.

While parsing the supplied test case, the CPDFConvertToUserPassword function in pdfsr.so will be called. This function implements the algorithm for deriving the decryption key. The overflow happens in the following code (image base being 0xB79BA00):

.text:B79E97A5 loc_B79E97A5:
.text:B79E97A5 movzx   eax, [ebp+edx+var_40] 	[1]
.text:B79E97AA xor     eax, esi 				[2]
.text:B79E97AC mov     [ebp+edx+var_20], al 	[3]
.text:B79E97B0 add     edx, 1
.text:B79E97B3
.text:B79E97B3 loc_B79E97B3:
.text:B79E97B3 mov     ecx, [ebp+var_3E0]
.text:B79E97B9 mov     eax, [ecx+13CCh]
.text:B79E97BF cmp     edx, eax 				[4]
.text:B79E97C1 jl      short loc_B79E97A5

In the above code, edx serves as a counter. At [1], a byte is zero extended from a stack based buffer, is xored with esi at [2] and written to a stack buffer at [3]. The value of esi comes from an outer loop counter, starts at 19 and is decreased untill 0. At [4], the counter in edx is compared against a maximum value in eax which comes straight from the Length value divided by 8. To reiterate, the PDF specification states that Length will be at most 128 bits, so the maximum value in eax should be 16. Appropriate ly, 16 bytes are allocated for var_20 buffer. If the value of Length is more, a buffer overflow will occur, overwriting the adjacent stack memory.

The supplied test case triggers the vulnerability and leads to a crash as the buffer overflow overwrites the return address as well as the stack cookie present on the stack.

There are two mitigating factors that lower the chance of successful exploitation of this vulnerability. First, the function is protected with a stack cookie making a straight forward return address overwrite difficult. And second, the bytes that end up overflowing the buffer are constant.

To elaborate on the second point, a shortened pseudo code of the algorithm follows:

if Revision == 3:
	if len(UserPassword) > 0:
		if len(UserPassword) < 32:
			#add padding to UserPassword
	else:
		#UserPassword = padding
UserPassword = md5(UserPassword)
if Revision == 3:
	for i in range(50):
		UserPassword = md5(UserPassword)
	for esi in range(13):
		for edx in range(Length/8):
			key[edx] = UserPassword[edx] ^ esi #here is the overflow
		initialize_arc4_key(key)

As can be seen from the pseudocode above, algorithm revision must be set to 3. Also, in examined use cases of this function, the UserPassword will always be blank, length 0, meaning that the UserPassword will be initialized to the fixed value of padding which is equal to magic value “28bf4e5e4e758a4164004e56fffa01082e2e00b6d0683e802f0ca9fe6453697a” that comes from PDF specification. This means that the attacker has limited control over overflowing bytes as it always depends on this fixed string (51 iterations of md5 of it, to be precise) and past contents of the stack.

By controlling the size of the overwrite, data past the stack cookie and return address can be overwritten potentially leading to further abuse in certain circumstances.

Detection of PDF files specifically crafted to trigger this vulnerability can be based on the presence of objects encrypted with revision 3 of the encryption algorithm (the exact algorithm is specified in PDF specification version 1.4) with abnormally, illegally, large Length value.

Exploit Proof-of-Concept

The vulnerability can be triggered with the supplied test case in the filter standalone KeyView binary shipped with IBM Domino, or by sending it as an attachment with an email to a Domino mail server.

Timeline

2016-02-09 - Vendor Notification
2016-06-08 – Public Disclosure