About the Reverse Engineering of Windows x64 Executables

Summary

The main focus of this blog is to give a push start to the beginners to get in the field of reverse engineering. Since this is the age of x64, I have skipped x86 architecture and will solely be focusing on x64 assembly. However, all the examples that will be written in C++ can be compiled to x86 ones as well, but I will leave that out for you guys. If you however have no experience in assembly, it won’t matter. The only thing that you would require throughout my blogs are logic and a bit of basic understanding of programming languages. This is however a reposting of my own blog from here.

As to get started, we will be writing a simple C++ program which will prompt for a password. It will check if the password matches, if it does, it will prompt its correct, else will prompt its incorrect. The main reason I took up this example is because this will give you an idea of how the jump, if else and other similar conditions work in assembly language. Another reason for this is that most programs which have hardcoded keys in them can be cracked in a similar manner except with a bit of more mathematics, and this is how most piracy distributors crack the legit softwares and spread the keys.

The code is pretty simple here. Our program takes one argument as an input which is basically the password. If I don’t enter any password, it will print the help command. If I specify a password, it gets stored as a char with 10 bytes and will send the password to the check_pass() function. Our hardcoded password is PASSWORD1 in the check_pass() function. Out here, our password get’s compared with the actual password variable mypass with the strcmp() function. If the password matches, it returns Zero, else it will return One. Back to our main function, if we receive One, it prints incorrect password, else it prints correct password.

Now, let’s get this code in our GDB debugger. We will execute the binary with GDB and we will first setup a breakpoint on main before we send the argument. Secondly, we will enable time travelling on our GDB, so that if we somehow go one step ahead by mistake, we can reverse that and come one step back again. This can be done with the following command: target record-full and reverse-stepi/nexti

Dont’ be scared if you don’t understand any of this. Just focus on the gdb$ part and as you can see above, I have given an incorrect password as pass123 after giving the breakpoint with break main. My compiled code should print an incorrect password as seen previously, but as we proceed, we will find two ways to bypass the code; one is by getting out the actual password from memory and second is by modifying the jump value and printing that the password is correct.

Disassembly

The next step is to disassemble the entire code and try to understand what actually is happening:

Our main point of intereset in the whole disassembled code would be the below few things:

1. je – je means jump to an address if its equal to something. If unequal, continue with the flow.

2. call – calls a new function. Remember that after this is loaded, the disassembled code will change from the main disassembly function to the new function’s disassembly code.

3. test – check if two values are equal

4. cmp – compare two values with each other

5. jne – jne means jump to and address if its not equal to something. Else, continue with the flow.

Some people might question why do we have test if we have cmp which does the same thing. The answer can be found here which is explained beautifully.
So, if we see the disassembly code above, we know that if we run the binary without a password or argument, it will print help, else will proceed to check the password. So this cmp should be the part where it checks whether we have an arguement. If an arguement doesn’t exist it will continue with the printing of help, else it will jump to <main+70>. If you see that numbers next to the addresses on the left hand side, we can see that at <+70>, we are moving something into the rax register. So, what we will do is we will setup a breakpoint at je, by specifying its address 0x0000000000400972 and then will see if it jumps to <+70> by asking it to continue with c. GDB command c will continue running the binary till it hits another breakpoint.

And now if you do a stepi which is step iteration, it will step one iteration of execution and it should take you to <+70> where it moves a Quad Word into the rax register.

So, since our logic is correct till now, let’s move on to the next interesting thing that we see, which is the call part. And if you see next to it, it says something like <_Z10check_passPc> which is nothing but our check_pass() function. Let’s jump to that using stepi and see what’s inside that function.

Once, you jump into the check_pass() function and disassemble it, you will see a new set of disassembled code which is the code of just the check_pass() function itself. And here, there are four interesting lines of assembly code here:

The first part is where the value of rdx register is moved to rsi and rax is moved to rdi. The next part is strcmp() function is called which is a string compare function of C++. Next, we have the test which compares the two values, and if the values are equal, we jump (je) to <_Z10check_passPc+77> which will move the value Zero in the eax register. If the values are not equal, the function will continue to proceed at <+70> and move the value One in the eax register. Now, these are nothing but just the return values that we specified in the check_pass() function previously. Since we have entered an invalid password, the return value which will be sent would be One. But if we can modify the return value to Zero, it would print out as “Correct Password”.

Also, we can go ahead and check what is being moved into the rsi and the rdi register. So, let’s put a breakpoint there and jump straight right to it.

Epilogue

As you can see above, the function moved 0x1 to eax in the binary, but before it can do a je, we modified the value to 0x0 in eax using set $eax = 0x0 and then continued the function with c as below, and Voila!!! We have a value returned as Correct Password!
Learning assembly isn’t really something as a rocket science. But given a proper amount of time, it does become understandable and easy with experience.

This was just a simple example to get you started in assembly and reverse engineering. Now as we go deeper, we will see socket functions, runtime encryption, encoded hidden domain names and so on. This whole process can be done using x64dbg in Windows as well which I will show in my next blogpost.

As for now, this is it. Do comment below or tweet to me @NinjaParanoid if you have any doubt.