Talos Vulnerability Report


GTKWave FST FST_BL_GEOM parsing maxhandle integer overflow vulnerability

January 8, 2024
CVE Number



An integer overflow vulnerability exists in the FST_BL_GEOM parsing maxhandle functionality of GTKWave 3.3.115, when compiled as a 32-bit binary. A specially crafted .fst file can lead to memory corruption. A victim would need to open a malicious file to trigger this vulnerability.


The versions below were either tested or verified to be vulnerable by Talos or confirmed to be vulnerable by the vendor.

GTKWave 3.3.115


GTKWave - https://gtkwave.sourceforge.net


7.0 - CVSS:3.1/AV:L/AC:H/PR:N/UI:R/S:U/C:H/I:H/A:H


CWE-190 - Integer Overflow or Wraparound


GTKWave is a wave viewer, often used to analyze FPGA simulations and logic analyzer captures. It includes a GUI to view and analyze traces, as well as convert across several file formats (.lxt, .lxt2, .vzt, .fst, .ghw, .vcd, .evcd) either by using the UI or its command line tools. GTKWave is available for Linux, Windows and MacOS. Trace files can be shared within teams or organizations, for example to compare results of simulation runs across different design implementations, to analyze protocols captured with logic analyzers or just as a reference when porting design implementations.

GTKWave sets up mime types for its supported extensions. So, for example, it’s enough for a victim to double-click on a wave file received by e-mail to cause the gtkwave program to be executed and load a potentially malicious file.

The function fstReaderOpen is used to parse .fst files.

     void *fstReaderOpen(const char *nam)
[1]  struct fstReaderContext *xc = (struct fstReaderContext *)calloc(1, sizeof(struct fstReaderContext));
[2]  if((!nam)||(!(xc->f=fopen(nam, "rb"))))
             int flen = strlen(nam);
             char *hf = (char *)calloc(1, flen + 6);
             int rc;
     #if defined(FST_UNBUFFERED_IO)
             setvbuf(xc->f, (char *)NULL, _IONBF, 0);   /* keeps gzip from acting weird in tandem with fopen */
             memcpy(hf, nam, flen);
             strcpy(hf + flen, ".hier");
             xc->fh = fopen(hf, "rb");
             xc->filename = strdup(nam);
[3]          rc = fstReaderInit(xc);

At [1] the xc structure is created, a big (about 65KB) structure used to hold information regarding the .fst file.
At [2] the xc->f field is set to the file pointer returned by fopen after opening the file being parsed.

Then, the function fstReaderInit is called [3] to parse the various .fst sections from the input file.

     int fstReaderInit(struct fstReaderContext *xc)
     fst_off_t blkpos = 0;
     fst_off_t endfile;
     uint64_t seclen;
     int sectype;
     uint64_t vc_section_count_actual = 0;
     int hdr_incomplete = 0;
     int hdr_seen = 0;
     int gzread_pass_status = 1;

     sectype = fgetc(xc->f);
             fstReaderFseeko(xc, xc->f, 0, SEEK_END);
             endfile = ftello(xc->f);

             while(blkpos < endfile)
                     fstReaderFseeko(xc, xc->f, blkpos, SEEK_SET);

[4]                  sectype = fgetc(xc->f);
                     seclen = fstReaderUint64(xc->f);

                     if(sectype == EOF)

[5]                  if((hdr_incomplete) && (!seclen))

[6]                  if(!hdr_seen && (sectype != FST_BL_HDR))


At [4], the sectype (1 byte) is read off the file. This variable will tell us which kind of sector we’re parsing. The checks at [5] and [6] are important because we don’t want to trigger them in order to reach the vulnerable code. We need to make sure hdr_incomplete stays 0, and that we start with a FST_BL_HDR sector, which will set hdr_seen to 1.

So, if the input file starts with a 0x00, we’ll land inside the following condition:

     if(sectype == FST_BL_HDR)
                     int ch;
                     double dcheck;

                     xc->start_time = fstReaderUint64(xc->f);
                     xc->end_time = fstReaderUint64(xc->f);

[7]                  hdr_incomplete = (xc->start_time == 0) && (xc->end_time == 0);

[8]                  hdr_seen = 1;

[9]                  xc->mem_used_by_writer = fstReaderUint64(xc->f);
                     xc->scope_count = fstReaderUint64(xc->f);
                     xc->var_count = fstReaderUint64(xc->f);

fstReaderUint64 is used to read a 64-bit unsigned integer (big endian) off the file, incrementing the current file offset.
We need to make sure that either start_time or end_time are not 0, to not set hdr_incomplete [7].
At [8] hdr_seen is set, so that we can pass the check at [6] when parsing the next sections.
Finally at [9], other values are read off the file and saved into the xc structure.

Next, if the file contains the FST_BL_GEOM (0x03) section, we’ll land inside the following condition:

     else if(sectype == FST_BL_GEOM)
[10]                 uint64_t clen = seclen - 24;
                     uint64_t uclen = fstReaderUint64(xc->f);
[11]                 unsigned char *ucdata = (unsigned char *)malloc(uclen);
[14]                 unsigned char *pnt = ucdata;
                     unsigned int i;

                     xc->contains_geom_section = 1;
[15]                 xc->maxhandle = fstReaderUint64(xc->f);
                     xc->longest_signal_value_len = 32; /* arbitrarily set at 32...this is much longer than an expanded double */

                     xc->process_mask = (unsigned char *)calloc(1, (xc->maxhandle+7)/8);

[12]                 if(clen != uclen)
[13]                         fstFread(ucdata, uclen, 1, xc->f);

At [10] clen and uclen are read; these represent the section data size and the uncompressed section data size respectively.
In fact if clen and uclen don’t match [12], the data portion of this section is uncompressed. Otherwise it is simply read into the ucdata buffer. The allocation for ucdata happens at [11] and is also assigned to pnt [14], which is a pointer moving within the ucdata buffer.

At [15] xc->maxhandle is read using the same fstReaderUint64 already described. Note that while this function returns a uint64_t, xc->maxhandle is of type uint32_t. For this reason, this issue only affects code compiled as 32-bit.

Continuing on the same path:

[16]                 xc->signal_lens = (uint32_t *)malloc(sizeof(uint32_t) * xc->maxhandle);
                     xc->signal_typs = (unsigned char *)malloc(sizeof(unsigned char) * xc->maxhandle);

[17]                 for(i=0;i<xc->maxhandle;i++)
                             int skiplen;
[18]                         uint64_t val = fstGetVarint32(pnt, &skiplen);

                             pnt += skiplen;

[19]                                 xc->signal_lens[i] = (val != 0xFFFFFFFF) ? val : 0;
                                     xc->signal_typs[i] = FST_VT_VCD_WIRE;
                                     if(xc->signal_lens[i] > xc->longest_signal_value_len)
                                             xc->longest_signal_value_len = xc->signal_lens[i];
                                     xc->signal_lens[i] = 8; /* backpatch in real */
                                     xc->signal_typs[i] = FST_VT_VCD_REAL;
                                     /* xc->longest_signal_value_len handled above by overly large init size */

At [16] xc->maxhandle is multiplied by 4, in order to allocate the xc->signal_lens buffer. In 32-bit mode, this multiplication will wrap around. For example, with an xc->maxhandle of 0x40000000 malloc(0) will be called, allocating 0 or just a few bytes depending on the malloc implementation.
For each xc->maxhandle [17], a variable 32-bit integer is read from ucdata at [18] using fstGetVarint32, which is then assigned to xc->signal_lens[i] at [19]. Since the xc->signal_lens buffer is not big enough for 0x40000000 elements, the write at [19] will eventually write out-of-bounds on the heap, leading to memory corruption. Because of the multi-threaded nature of GTKWave, an attacker may be able to exploit this issue to execute arbitrary code.

Finally, note that the value written out-of-bounds is almost fully controlled by an attacker. The fstGetVarint32 function returns a 32-bit “variable” (in length) integer using LEB128 encoding. In short, an integer is encoded as a sequence of 7-bit values using a stop bit on the MSB:

static uint32_t fstGetVarint32(unsigned char *mem, int *skiplen)
unsigned char *mem_orig = mem;
uint32_t rc = 0;
while(*mem & 0x80)

*skiplen = mem - mem_orig + 1;
        rc <<= 7;
        rc |= (uint32_t)(*mem & 0x7f);
        if(mem == mem_orig)


For example, 0xaabbccdd would be encoded as DD99EFD50A.

Crash Information

==1102==ERROR: AddressSanitizer: heap-buffer-overflow on address 0xf5f006b0 at pc 0x5659ccb9 bp 0xffff5598 sp 0xffff558c
WRITE of size 4 at 0xf5f006b0 thread T0
    #0 0x5659ccb8 in fstReaderInit fst/fstapi.c:4804
    #1 0x5659dde7 in fstReaderOpen fst/fstapi.c:4917
    #2 0x56558e33 in main src/helpers/fst2vcd.c:153
    #3 0xf765c294 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #4 0xf765c357 in __libc_start_main_impl ../csu/libc-start.c:381
    #5 0x56558686 in _start (fst2vcd+0x3686)

0xf5f006b1 is located 0 bytes to the right of 1-byte region [0xf5f006b0,0xf5f006b1)
allocated by thread T0 here:
    #0 0xf7a58ffb in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0x5659ca24 in fstReaderInit fst/fstapi.c:4791
    #2 0x5659dde7 in fstReaderOpen fst/fstapi.c:4917
    #3 0x56558e33 in main src/helpers/fst2vcd.c:153
    #4 0xf765c294 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58

SUMMARY: AddressSanitizer: heap-buffer-overflow fst/fstapi.c:4804 in fstReaderInit
Shadow bytes around the buggy address:
  0x3ebe0080: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x3ebe0090: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x3ebe00a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x3ebe00b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x3ebe00c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x3ebe00d0: fa fa fa fa fa fa[01]fa fa fa 05 fa fa fa 00 04
  0x3ebe00e0: fa fa 00 03 fa fa 00 04 fa fa 00 05 fa fa 00 04
  0x3ebe00f0: fa fa 00 05 fa fa 00 00 fa fa fa fa fa fa fa fa
  0x3ebe0100: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x3ebe0110: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x3ebe0120: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb

Fixed in version 3.3.118, available from https://sourceforge.net/projects/gtkwave/files/gtkwave-3.3.118/


2023-07-10 - Initial Vendor Contact
2023-07-18 - Vendor Disclosure
2023-12-31 - Vendor Patch Release
2024-01-08 - Public Release


Discovered by Claudio Bozzato of Cisco Talos.