My Simple C Program Demonstrating Heap Buffer Overflow

Note

This is the twelth assignment from my Masters Advanced Network Security Course which has never been published anywhere and I, as the author and copyright holder, license this assignment customized CC-BY-SA where anyone can share, copy, republish, and sell on condition to state my name as the author and notify that the original and open version available here.

1. Introduction

The C variables can be categorized into 2 which are the statically assigned variables or local variables resides in the stack area and dynamically assigned variables (using malloc function) resides in the heap area [1]. A stack is a data structure equivalent to last in first out. It mainly has push and pop instruction where push puts new element at the top of the stack and pop removes the top elements first. For instance, when a function calls another function, which in turn calls a third function, it’s important that the third function return back to the second function rather than the first [2] [3]. A stack buffer overflow occurs when the input data is larger than the assigned size of local variables, which corrupts the stack area, worst case a malicious user can insert a malicious code on the return address after overflowing the previous stack area [4].

On the other hand the heap is a tree based data structure where the value of the parent is always greater or equal then the children’s node. In C the heap area is an area that is not manage automatically but usually larger than stack. On the stack area the user only defines the size and variable, using last in first out it pushes and pops elements automatically, while on the heap area we have to manually free() the memory after it is used. To allocate memory on the heap area usually uses function calloc() or malloc(). Unlike stack it doesn’t have size restriction, but it’s slower because we have to use pointers to access [5]. A heap buffer overflow can occur on the heap area when the input size (using gets) of a dynamic allocated variable is larger then the allocate size which replace the next dynamic allocated variable [6], or cause by the statically assigned variable next to the heap area (using memset() e.g) which will overwrite starting from the beginning address of the heap area [7].