A vulnerability in PDF parser of the IX SDK exists that results in out of bounds heap memory access following an unchecked memory allocation operation under specific conditions.
Oracle Outside In IX sdk 8.5.1
In a PDF file an xref table contains multiple rows each containing three values (
except for the first row which specifies the first object being referenced and the
number of objects). First value represents the 10 digit offset into the file where
object is to be found. In a specially crafted PDF file, OID SDK PDF parser
uses the specified value as a parameter in a call to
realloc() which can fail.
The return value is checked for errors but is subsequently ignored. The original
numerical value is then used as an upper bound in a loop where out of bounds
read happens during process cleanup.
In a function at 0xB74C603A in libvs_pdf.so (base address being 0xB74BF000)
either a call to
realloc is being made indirectly. Final size is calculated
` .text:B74C608E mov [edi+8], ecx .text:B74C6091 imul ecx, ebp .text:B74C6094 mov [esp+2Ch+var_14], ecx .text:B74C6098 mov [esp+2Ch+c], ecx .text:B74C609C mov eax, [edi] .text:B74C609E mov [esp+2Ch+s], eax .text:B74C60A1 call _SYSNativeReAlloc `
ecx holds the value from the file. In this case
it is multiplied by 0x10 in
ebp. The upper bound for the value in the
file is 0x7ffffff, so the maximum size that can be passed to
is 0x7ffffff0. In limited memory conditions, of about less than 2 gigabytes of virtual
memory available, this reallocation will fail returning zero. Although
there are checks for this condition, the same buffer is iterated over
during process cleanup. Specifically, in the following code in function
` .text:B74D17C1 add esi, 10h  .text:B74D17C4 mov eax, [esi]  .text:B74D17C6 test eax, eax .text:B74D17C8 jz short loc_B74D17D3 .text:B74D17CA mov edx, [esp+4Ch+arg_4] .text:B74D17CE call sub_B74D14C0 .text:B74D17D3 .text:B74D17D3 loc_B74D17D3: ; CODE XREF: VwStreamClose+19Ej .text:B74D17D3 add edi, 1  .text:B74D17D6 mov eax, [esp+4Ch+arg_4] .text:B74D17DA cmp [eax+1D70h], edi  .text:B74D17E0 ja short loc_B74D17C1 `
esi points to the buffer that was previously subject to failed
edi serves as a counter, is increased by 1 each turn at  and compared to
the initial value from the file at . At , pointer in
esi is increased by 0x10.
At this point, the parser expects that the memory reallocation was successful which
leads to an out of bounds memory access.
The supplied minimized testcase triggers the out of bounds access in
application supplied with the SDK. In order to trigger it, a virtual memory
limit must be set by executing:
` # ulimit -Sv 1000000 `
2016-04-12 - Vendor Notification
2016-07-19 – Public Disclosure
Discovered by Aleksandar Nikolic of Cisco Talos.