Talos Vulnerability Report
Oracle OIT IX SDK libvs_pdf Size Integer Overflow Vulnerability
July 19, 2016
An integer overflow leading to two distinct issues can be triggered by a specially crafted PDF file.
Oracle Outside In IX sdk 8.5.1
While parsing a PDF file with specific /Size element, a memory allocation operation
can fail, returning a NULL pointer due to integer overflow, which is unchecked and leads to a crash
memset() call. A carefully selected size value can also lead to further memory
The supplied testcase can be abbreviated to the following:
` %PDF-1.3 xref trailer <</Size 444444444444444444499999999999>> startxref 4 %%EOF `
/Size value leads to failed memory allocation in the following basic block:
` .text:B74ECE59 mov edi, eax  .text:B74ECE5B shl edi, 4  .text:B74ECE5E mov [esp+6BCh+s], edi .text:B74ECE61 call _SYSNativeAlloc  .text:B74ECE66 mov edx, [esp+6BCh+arg_10] .text:B74ECE6D mov [edx+1D6Ch], eax  .text:B74ECE73 test eax, eax .text:B74ECE75 jz loc_B7 `
At , the value in
eax comes straight from the 32bit rounded value from the
/Size element. At , it is multiplied by four therefore invalidating the
integer overflow check that was done previously. A
malloc wrapper is called
at  and the returned pointer (NULL in this case) is saved at .
Even though the pointer is checked against NULL at the end, in a subsequent
basic block it is still used as a destination for
` .text:B74ECE7B loc_B74ECE7B: .text:B74ECE7B mov ecx, [esp+6BCh+arg_10] .text:B74ECE82 mov eax, [ecx+1D6Ch] .text:B74ECE88 mov [esp+6BCh+n], edi ; n  .text:B74ECE8C mov [esp+6BCh+c], 0 ; c .text:B74ECE94 mov [esp+6BCh+s], eax ; s  .text:B74ECE97 call _memset  `
The same size derived in the previous basic block is used at  as a size
memset. At , saved pointer is retrieved and is NULL in this
case. The application crashes at  due to invalid pointer.
If a size value is chosen carefully, it can lead to an integer
overflow at  in the first basic block such that a small value
is passed to SYSNativeAlloc at . In this case, the subsequent
memset call would pass without issue. The problem arises
when, due to rounding, heap allocator returns a pointer
to a bigger heap chunk than requested. In this case, the
call will initialize only the originally requested size, leaving the
rest of the buffer uninitialized to zero. Later on in the code,
this buffer is treated as a pointer array with checks for NULL pointers,
but the uninitialized portion of the buffer may have non-NULL values
leading to further issues.
As an example, if the size value is specified to be 0x10000001 it will
pass the check before allocation in the first basic block above, but when shifted by 4,
it becomes 0x10, making a small allocation. Depending on an underlying allocator,
the actual size of the allocated chunk would be bigger. In case of Linux, in
this case, the returned chunk will be 24 bytes long and subsequent
only initialize the first 16 bytes.
Afterwards, the code reaches the following loop inside VwStreamClose:
` .text:B74D17BB mov esi, eax  .text:B74D17BD xor edi, edi  .text:B74D17BF jmp short loc_BB74D17C4 .text:B74D17C1 .text:B74D17C1 loc_B74D17C1: .text:B74D17C1 add esi, 10h  .text:B74D17C4 .text:B74D17C4 loc_B74D17C4: .text:B74D17C4 mov eax, [esi]  .text:B74D17C6 test eax, eax .text:B74D17C8 jz short loc_B74D17D3  .text:B74D17CA mov edx, [esp+4Ch+arg_4] .text:B74D17CE call sub_B74D14C0  .text:B74D17D3 .text:B74D17D3 loc_B74D17D3: .text:B74D17D3 add edi, 1  .text:B74D17D6 mov eax, [esp+4Ch+arg_4] .text:B74D17DA cmp [eax+1D70h], edi  .text:B74D17E0 ja short loc_B74D17C1 `
At , a pointer to the previously allocated buffer is moved into
esi and used as a starting
position of a loop. At , a counter is initialized to 0. At , a pointer stored in the
buffer is copied into
eax and tested against being NULL at . If it’s not NULL, a usercall
function who’s first argument is
eax is called at . After the function call, or if the pointer
was NULL, a counter in
edi is advanced by one and then compared to the upper bound which is
Size value as specified in the file (before the overflow) at . Finally, the code
jumps back to , where the pointer into the buffer is increased by 16.
It is now clear that if only first 16 bytes of the buffer are initialized, when the code executes the loop for the second time, at  it will be accessing memory that is uninitialized to zero effectively turning this into a sort of use-after-free vulnerability. Function called at  deals with heap structures and, if sufficient heap control is achieved, leftover data present in uninitialized part can cause further memory corruption, potentially leading to code execution.
Discovered by Aleksandar Nikolic of Cisco Talos.
2016-04-12 - Initial Vendor Communication
2016-07-19 – Public Disclosure