Talos Vulnerability Report
IBM Domino KeyView PDF Filter Encrypted Stream Code Execution Vulnerability
June 8, 2016
A stack overflow vulnerability present in the PDF filter of KeyView as used by Domino can lead to process crash and possible arbitrary code execution.
- KeyView 10.16 as used by IBM Domino 9.0.1
While parsing a specially crafted PDF file, a user controlled length field is used in a
write loop with fixed destination size leading to a stack based buffer overflow. The vulnerability
is triggered while parsing the PDF file that specifies an encrypted stream. As per the PDF specification, the
Length value specifies the key length and is at most 16 bytes long. In the vulnerable function a stack buffer 16 bytes in length is reserved, but unchecked
Length value is used during the copy operation which allows adjacent stack data to be overwritten, including the return address.
The minimized test case that triggers the vulnerability is as follows:
` %PDF-1.4 trailer <</Size 9 /Root 1 0 R /Encrypt 8 0 R 8 0 obj<< /Length 768 /Filter/Standard /Type/Catalog /O (41414141414141414141414141414141) /U (42424242424242424242424242424242) /P 0 /R 3 >> >> obj<< >> endobj %%EOF `
In the above test case, the PDF
trailer specifies that object 8 is encrypted. Further, object 8 specifies that it is using a standard filter for encryption (
/Filter/Standard) and is using a revision 3 (
/R 3) of the algorithm. Owner password (
/O) and user password (‘/U’), as well as object type don’t play a significant role in this test case.
While parsing the supplied test case, the
CPDFConvertToUserPassword function in
pdfsr.so will be called.
This function implements the algorithm for deriving the decryption key. The overflow happens in the following
code (image base being 0xB79BA00):
.text:B79E97A5 loc_B79E97A5: .text:B79E97A5 movzx eax, [ebp+edx+var_40]  .text:B79E97AA xor eax, esi  .text:B79E97AC mov [ebp+edx+var_20], al  .text:B79E97B0 add edx, 1 .text:B79E97B3 .text:B79E97B3 loc_B79E97B3: .text:B79E97B3 mov ecx, [ebp+var_3E0] .text:B79E97B9 mov eax, [ecx+13CCh] .text:B79E97BF cmp edx, eax  .text:B79E97C1 jl short loc_B79E97A5
In the above code,
edx serves as a counter. At , a byte is zero extended from a stack based
buffer, is xored with
esi at  and written to a stack buffer at . The value of
from an outer loop counter, starts at 19 and is decreased untill 0. At , the counter in
compared against a maximum value in
eax which comes straight from the
Length value divided by 8.
To reiterate, the PDF specification states that
Length will be at most 128 bits, so the maximum
eax should be 16. Appropriate ly, 16 bytes are allocated for
var_20 buffer. If the
Length is more, a buffer overflow will occur, overwriting the adjacent stack memory.
The supplied test case triggers the vulnerability and leads to a crash as the buffer overflow overwrites the return address as well as the stack cookie present on the stack.
There are two mitigating factors that lower the chance of successful exploitation of this vulnerability. First, the function is protected with a stack cookie making a straight forward return address overwrite difficult. And second, the bytes that end up overflowing the buffer are constant.
To elaborate on the second point, a shortened pseudo code of the algorithm follows:
if Revision == 3: if len(UserPassword) > 0: if len(UserPassword) < 32: #add padding to UserPassword else: #UserPassword = padding UserPassword = md5(UserPassword) if Revision == 3: for i in range(50): UserPassword = md5(UserPassword) for esi in range(13): for edx in range(Length/8): key[edx] = UserPassword[edx] ^ esi #here is the overflow initialize_arc4_key(key)
As can be seen from the pseudocode above, algorithm revision must be set to 3. Also,
in examined use cases of this function, the UserPassword will always be blank, length 0,
meaning that the
UserPassword will be initialized to the fixed value of padding which is
equal to magic value “28bf4e5e4e758a4164004e56fffa01082e2e00b6d0683e802f0ca9fe6453697a” that
comes from PDF specification. This means that the attacker has limited control over overflowing
bytes as it always depends on this fixed string (51 iterations of md5 of it, to be precise) and
past contents of the stack.
By controlling the size of the overwrite, data past the stack cookie and return address can be overwritten potentially leading to further abuse in certain circumstances.
Detection of PDF files specifically crafted to trigger this vulnerability can be based on the
presence of objects encrypted with revision 3 of the encryption algorithm (the exact algorithm is
specified in PDF specification version 1.4) with abnormally, illegally, large
The vulnerability can be triggered with the supplied test case in the
filter standalone KeyView
binary shipped with IBM Domino, or by sending it as an attachment with an email to a Domino
Discovered by Aleksandar Nikolic of Cisco Talos.
2016-02-09 - Vendor Notification
2016-06-08 – Public Disclosure