Talos Vulnerability Report


Oracle OIT libim_gem2 Gem_Text Code Execution Vulnerability

July 19, 2016

Report ID



An integer overflow vulnerability exists in file parsing code of Oracle Outside In Technology libim_gem2 library. A specially crafted Gem file can trigger an integer overflow leading to multiple heap based buffer overflows which can be abused to achieve remote code execution.

Tested Versions

  • Oracle Outside In Technology Content Access SDK 8.5.1.

Product URLs



While parsing a Gem metafile data an unchecked memory allocation is performed. In function Gem_Text in libim_gem library, a 16 bit value read from the file is used in arithmetic operations without checks leading to an integer overflow.

As an example, the vulnerability is triggered in the ixsample demo application supplied in the SDK. The supplied minimized testcase crashes due to overwritten heap structures resulting from a buffer overflow enabled by unchecked memory allocation.

Vulnerability is present in function Gem_Text in the libim_gem2 library. Significant code excerpt:

.text:00006323                 push    ebp
.text:00006324                 push    edi
.text:00006325                 push    esi
.text:00006326                 push    ebx
.text:00006327                 sub     esp, 3Ch
.text:0000632A                 call    sub_3D35
.text:0000632F                 add     ebx, 0C321h
.text:00006335                 mov     ebp, [esp+4Ch+arg_4]
.text:00006339                 mov     edx, [esp+4Ch+arg_0]
.text:0000633D                 mov     eax, [edx+4]
.text:00006340                 mov     eax, [eax+4758h]
.text:00006346                 mov     edx, [eax+8]
.text:00006349                 mov     [esp+4Ch+var_48], 1
.text:00006351                 mov     eax, [eax+4]
.text:00006354                 mov     [esp+4Ch+var_4C], eax
.text:00006357                 call    dword ptr [edx+134h]
.text:0000635D                 movsx   edi, bp 					; [1]
.text:00006360                 lea     esi, [edi+edi]			; [2]
.text:00006363                 lea     eax, [esi+5]				; [3]
.text:00006366                 mov     [esp+4Ch+var_4C], eax	; [4]
.text:00006369                 call    _memalloc
.text:0000636E                 mov     [esp+4Ch+var_14], eax
.text:00006372                 lea     eax, [edi+1]				; [5]
.text:00006375                 mov     [esp+4Ch+var_4C], eax
.text:00006378                 call    _memalloc
.text:0000637D                 mov     [esp+4Ch+var_18], eax	; [6]

The 16 bit value read from a file is sign extended into edi at [1]. At [2] and [3] a size argument for memalloc is calculated leading to the first integer overflow. If the value in bp was 0xffff, sign extending if makes edx 0xffffffff, then [2] and [3] overflow this value which ends up in eax and as a parameter to memalloc. No check is performed for sane values of size parameter. In case of initial value being 0xffff, the size of allocated memory would be small (3 bytes requested).

Second integer overflow can occur at [5] and, again, an unchecked result is used as a size parameter to memalloc. If the initial value was 0xffff, the requested memory size would be 0. Pointer to the allocated memory is saved at [6] and is subsequently used.

Pointer from the second allocation is used as a parameter to the gem_char_translate function which,in essence, is tasked with translating one codepage to another. Original size value is also supplied to this function and serves as a counter.

Function gem_char_translate can be simply explained with the following pseudo-C:

gem_char_translate(char *cp_string,uint16 size){

	for(int i; i < size; i++){
		if(cp_string[i] < 0x1f){
			cp_string[i] = 0x20;
		}else if(cp_string[i] > 0x7e){
			//change byte according to the codepage


If an integer overflow did occur during memory allocation with initial size value of 0xffff, gem_char_translate will happily write up to 0xffff bytes starting at the allocated buffer leading to a heap buffer overflow.

By manipulating the heap to place specially crafted values in memory after the wrongfully allocated buffer, arbitrary content can be placed on the heap.

Later on in the code, the gem_char_translate translated buffer is used as a source string in a strcpy() call which leads to another heap buffer overflow which can be abused to achieve remote code execution. Second heap overflow happens during a strcpy call in imsStrCpy function. Since the gem_char_translate function will, in case of integer overflow, always generate a string of length 0xffff, this buffer overflow can be abused to overwrite different control structures that are present on the heap.


Discovered by Aleksandar Nikolic of Cisco Talos.


2015-10-19 - Discovery
2016-04-20 - Initial Vendor Communication
2016-07-19 - Public Release