Quartz 4

05 - Reverse Engineering

18 min read

Reverse Engineering

Objectives: By the end of this topic, you will be able to…

  • Identify key functions and strings within a binary
  • Use tools such as objdump, gdb, and Cutter to inspect machine code
  • Understand the basic execution flow without access to source code
  • Recognize limitations, risks, and ethical aspects of reverse engineering

What is reverse engineering and what is it used for?

Reverse engineering is the process of analyzing the internal workings of a program or system without access to its original source code, in order to understand its behavior, architecture, or potential weaknesses.

In cybersecurity, reverse engineering is used to analyze malware and determine its purpose, identify security vulnerabilities, and validate software integrity through auditing. It also plays a role in forensic investigations after incidents, in studying protection mechanisms such as anti-debugging and obfuscation, and in ensuring compatibility with legacy or proprietary software. It is a common skill in malware analysis, exploit development, and security auditing.

Differences between static and dynamic analysis

Static analysis

Static analysis is performed without running the program: the binary file is examined directly with a disassembler, which lets the analyst review assembly code, functions, strings, and data structures. It is safer than running unknown code, though raw assembly can be harder to interpret without context.

Before opening a disassembler, a quick pass with file tells you what kind of binary you are dealing with. file reports whether the binary is statically or dynamically linked and whether its symbol table has been stripped. A dynamically linked binary imports functions such as strcmp or printf from libc at runtime, so a disassembler can label those calls by name even before you do anything else; a stripped binary has had its internal symbol names removed beyond those imports, so functions you locate yourself, including main, will not carry a label and you have to recognize them by structure (a prolog, a sequence of library calls, a return) instead. This distinction changes how you work with a debugger later: on a binary with symbols you can set a breakpoint by function name, while on a stripped one you often have to disassemble first and break at a raw address instead.

strings output mixes genuine program text, such as prompts and error messages, with compiler-generated artifacts and leftover data from alignment padding. Treat every match as a candidate, not a conclusion: a short sequence of printable bytes can appear in a binary by coincidence, and confirming which string actually matters is the job of the static and dynamic steps that follow, not of strings itself.

Common tools: strings, objdump, Ghidra, IDA Free

Dynamic analysis

Dynamic analysis executes the binary in a controlled environment, such as a sandbox or virtual machine, and observes its real behavior: system calls made, network connections initiated, files read or written. This is especially useful for detecting hidden behavior or dependencies that are only resolved at runtime.

Common tools: gdb, strace, ltrace, radare2, Cutter

Executable formats

ELF (Executable and Linkable Format)

ELF is the standard executable format on Linux systems. It organizes code and data into named sections such as .text (instructions), .data (initialized variables), .bss (uninitialized variables), .plt, and .got, and it may include debugging symbols and exported function names.

PE (Portable Executable)

PE is the executable format on Windows (.exe, .dll). It follows a similar conceptual structure to ELF, with a DOS header, section table, and import/export tables, and is widely used by malware developers targeting Windows environments.

Key difference: ELF is native to Unix/Linux, PE is native to Windows. Analysis tools differ between platforms, though concepts like sections, imports, and headers are similar.

Disassemblers and debuggers

Disassemblers

Translate machine code into assembly language to understand what a binary does at a low level.

objdump -d binary

A raw objdump -d listing shows every instruction with no narrative structure, so the practical skill is knowing what to search for rather than reading top to bottom. Locate main first (its label appears in the left margin on a binary with symbols, or by recognizing prolog and epilog patterns on a stripped one), then scan forward for call instructions targeting library functions such as strcmp, printf, puts, or scanf (their PLT stub names appear next to the call target, often written as strcmp@plt). Seeing a call to an input function followed by a call to a comparison function is usually enough to infer, before touching a decompiler, that the program reads input and checks it against something. strcmp itself follows a convention shared with C’s other comparison functions: it returns 0 when the two strings it compares are equal and a nonzero value otherwise, so code that branches on strcmp(...) == 0 is taking its success path exactly there.

InvocationEffect
radare2 binaryopen read-only
radare2 -A binaryopen and auto-analyze immediately (equivalent to running aaa by hand)
radare2 -w binaryopen in write mode, required before patching any instruction

Once inside the interactive prompt, a handful of commands cover most of what you need:

CommandPurpose
aaaanalyze all functions and cross-references
s <address|symbol>seek the current position (s main jumps straight to main)
pdfprint the disassembly of the function at the current position
wa <instruction>write (assemble) a new instruction at the current address

Replacing a conditional jump with a NOP (no-operation) instruction removes a branch entirely, so execution always falls through to whichever path followed it; flipping je to jne, or the reverse, inverts which path is taken instead of removing the choice.

Ghidra provides both disassembly and pseudocode decompilation.

Debuggers

Allow running a binary step by step, using breakpoints to pause execution at critical points, and inspecting memory, registers, and call stack.

gdb binary

Basic debugging commands:

CommandPurpose
break <function|address>set a breakpoint, by symbol name or by raw address (break *0xADDR)
run / continuestart execution, or resume it after a breakpoint
next / stepstep one source line at a time. Requires debug/line information; on a stripped binary, use stepi and address-based breakpoints instead
stepistep a single machine instruction
disassemble <function>show the disassembly of the current function
x/s <address|register>examine memory (or a register holding a pointer) as a string
info registers / info registers <reg>inspect all registers, or one specific register
finishrun until the current function returns, then show its return value

Function analysis and conditional logic

Function identification

Functions can be located via symbol tables, prolog/epilog patterns, or cross-references. Tools like Ghidra detect them automatically.

Every function begins with a prolog that sets up the stack frame (push ebp, mov ebp, esp) and ends with an epilog that restores it and returns (pop ebp, ret). Between them, calls (call) transfer control to subroutines, while jumps (jmp, je, jne) implement conditional branching, often preceded by a comparison (cmp).

Argument passing and stack canaries

Function calls exchange data through more than the stack. On x86-64 Linux, the System V calling convention passes the first several integer or pointer arguments in specific registers rather than pushing them: rdi holds the first argument, rsi the second, rdx the third, and so on. When disassembly shows a call to a two-argument function like strcmp(a, b), a sits in rdi and b in rsi at the moment of the call, and a debugger lets you read both directly without decompiling anything. This is what makes it possible to confirm, at the exact instruction where a comparison happens, what a program is actually checking user input against.

When stack protection is enabled, the default on most current Linux toolchains including Kali’s, the compiler inserts a check called a stack canary around functions that declare a vulnerable local buffer, as a defense against stack buffer overflows. A known value is placed on the stack just before the buffer when the function starts, and compared against its original value just before the function returns; if the two differ, something overwrote memory it should not have, and the program calls __stack_chk_fail to abort rather than continue with a potentially corrupted return address. This is a protection mechanism, not part of a program’s own decision logic, so when a canary check appears in disassembly or in a decompiler’s pseudocode next to the comparison you are actually analyzing, you can recognize it as unrelated scaffolding and move past it.

Conditional analysis

Program decisions are represented as blocks with comparison instructions (cmp) and conditional jumps (je, jne, jg, jl). Understanding this logic helps identify critical branches: password validation, flow decisions, anti-analysis tricks.

Common techniques include following the execution flow with gdb or a graphical tool, focusing analysis on the code around main, printf, system, and network or system calls, and tracing how variables and buffers move on the stack.

Decompilation with Ghidra

A decompiler goes a step further than a disassembler: instead of assembly mnemonics, it reconstructs approximate C-like pseudocode from the binary, trading some precision for much faster readability. Ghidra’s workflow starts with a project, a container for the binaries you import, and an auto-analysis pass that walks the binary once to identify functions, data types, and cross-references before you look at anything by hand; on a non-trivial binary this pass can take anywhere from a few seconds to a few minutes, during which the tool builds the Symbol Tree you navigate from afterward.

Ghidra’s Listing panel shows synchronized disassembly, and its Decompile panel shows the reconstructed pseudocode for whichever function is selected in the Listing or the Symbol Tree; the two stay in step as you scroll one or the other. Because a decompiler cannot recover names the compiler discarded, it invents its own: local stack variables appear as local_<offset>, named after their position on the stack, and temporary values as iVar1, iVar2, and so on. Renaming these to something meaningful, such as calling a local buffer user_input, does not change the binary; it only changes what you see, and is worth doing early, since you will refer back to the same variable many times as you read the surrounding logic.

Ghidra’s own comparison syntax follows directly from strcmp’s return convention covered earlier: a branch the decompiler renders as iVar1 == 0 is the same check as !strcmp(a, b) in another decompiler’s convention, taken when the two compared strings are equal.

Ethics and legality of binary analysis

Reverse engineering carries the same legal and ethical obligations as any other security-testing activity in this course (see Class 01’s discussion of authorization and scope), with two additions specific to working with someone else’s binary: copyright and end-user license agreements can restrict what you are allowed to do with software you did not write even when you have legitimate access to it, and modifying or redistributing a binary without permission can create liability independent of whatever you found through analysis. In practice, this means analyzing only software from legitimate sources or binaries built specifically for educational reverse-engineering practice, not redistributing anything you modify without permission, and keeping authorization and scope explicit before you start, the same discipline you already apply to any other test.

Hands-on lab

Requirements: Kali Linux, gdb, Ghidra or Cutter

In this lab you will analyze a binary that asks the user for a password. Your goal is to find the correct password without access to the source code, using only reverse engineering tools and techniques.

This lab is done in pairs, with each partner taking a different role. Both of you complete Part 0 independently, so each of you has a working copy of the binary. From Part 1 on, one of you becomes the Static Analyst, working through Part 1 (recon with file, strings, and objdump) and Part 2 (Ghidra decompilation, variable renaming, and password extraction) alone; your job is done once you have the password and an annotated screenshot of the decompiled logic. The other becomes the Dynamic Analyst, who does not open Ghidra or search for the password independently, and instead works through Part 3 (gdb confirmation) using only what the Static Analyst hands over. From Part 4 onward you work together: the Dynamic Analyst drives Cutter or radare2, and the Static Analyst annotates the graph and cross-checks it against the Part 2 pseudocode. Write the Submission jointly.

Part 0: Download the binary

  1. Download the crackme from crackmes.one. You will get a .zip file.

  2. Extract it. The zip file is password-protected — the password is crackmes.one:

unzip crackme.zip
  1. Make the binary executable:
chmod +x passguess

You should now have the passguess binary ready to analyze.

Part 1: Initial reconnaissance

Before opening any advanced tool, gather basic information about the binary.

  1. Determine the file type. Run the following command and note whether the binary is 32-bit or 64-bit, statically or dynamically linked, and whether symbols have been stripped:
file passguess

Write down the architecture (e.g. ELF 64-bit LSB executable, x86-64) — you will need this to choose the correct analysis mode later.

  1. Extract readable strings. Many binaries contain plaintext strings (prompts, error messages, hardcoded values). Search for them:
strings passguess | less

Look for anything that resembles a user-facing message (e.g. "Guess The Pass", "OK", "ERROR"). Also look for any short, suspicious strings that could be a hardcoded password or key. Write down every string that seems relevant. Note that strings will also show internal compiler artifacts and memory alignment data — not every short string is meaningful. You will confirm which one is the actual password in later parts.

  1. Inspect the disassembly for key functions. Get a raw disassembly listing and search for well-known function names:
objdump -d passguess | less

Inside less, press / and type main to jump to the main function. Also search for calls to library functions like strcmp, printf, puts, and scanf. These tell you what the program does: reads input, compares it, and prints a result.

Checkpoint: At this point you should have a rough idea of what the program does — it reads a password from the user and checks it against something. You may have already spotted the answer in the strings output. The next parts will confirm your hypothesis.

Part 2: Decompilation with Ghidra

Ghidra can reconstruct approximate C source code (pseudocode) from a binary, which is much easier to read than raw assembly.

  1. Open Ghidra and create a new project (File → New Project → Non-Shared Project). Give it any name.

  2. Import the binary. Go to File → Import File and select passguess. Ghidra will auto-detect the format and architecture. Click OK.

  3. Run the auto-analysis. When prompted with “program has not been analyzed. Would you like to analyze it now?”, click Yes and accept the default analyzers. Wait for the analysis to complete (progress bar at the bottom right).

  4. Navigate to main. In the Symbol Tree panel on the left, expand Functions and click on main. The Listing panel will show the disassembly and the Decompile panel will show the pseudocode.

  5. Read the pseudocode carefully. You should see something similar to this structure:

    • A local buffer (array) is declared on the stack (e.g. local_118)
    • printf prints a prompt asking for input
    • scanf reads user input into the buffer
    • strcmp compares the buffer against a hardcoded string
    • An if/else prints either a success or failure message
    • You may also see a __stack_chk_fail() call near the end — this is an automatic stack canary check inserted by the compiler to detect buffer overflows. You can ignore it for this exercise.

  6. Rename variables for clarity. Ghidra generates placeholder names like local_118 or iVar1. You can rename them to make the pseudocode easier to read: right-click a variable name and select Rename Variable (or press L). For example:

    • Rename the buffer (e.g. local_118) to user_input
    • Rename the strcmp return value (e.g. iVar1) to password_match

  7. Identify the comparison. Look at the strcmp call. It takes two arguments:

    • The buffer containing user input
    • A string literal — the hardcoded password

The strcmp function returns 0 when both strings are equal. Depending on the decompiler, you may see the check written as iVar1 == 0 (Ghidra style) or !strcmp(...) (IDA style) — both mean the same thing: the success branch executes when the strings match.

  1. Extract the password. The second argument to strcmp is the correct password. Write it down.

  2. Verify your finding. Run the binary and enter the password you found:

./passguess

You should see the success message. If you do, you have successfully reverse engineered the binary.

Handoff: Static Analyst, send your partner the password you extracted and your annotated Ghidra screenshot before continuing. Dynamic Analyst, don’t open Ghidra or search for the password yourself: Part 3 exists to confirm what your partner found, by reading it directly out of the registers at the strcmp call, not to rediscover it independently. If what you see at the breakpoint doesn’t match the password you were handed, that’s a signal to go back and check Part 2 together before moving on, not to keep going as if nothing happened.

Part 3: Dynamic analysis with gdb

Even though you already know the password, use gdb to confirm the finding dynamically and practice debugging skills.

  1. Start the debugger:
gdb ./passguess
  1. Break at main and run:
(gdb) break main
(gdb) run
  1. Disassemble main to find the strcmp call. Since the binary has no debug symbols, we cannot step line-by-line with next. Instead, we will find the exact address of the strcmp call and set a precise breakpoint:
(gdb) disassemble main

Scroll through the output and look for a line containing call and strcmp (it may appear as strcmp@plt). Note the address on the left side of that line (e.g. 0x00005555555551a8).

  1. Set a breakpoint at the strcmp call address and continue. Replace the address below with the one you found. This ensures you only break on the password comparison, not on internal library calls to strcmp:
(gdb) break *0x00005555555551a8
(gdb) continue

The program will print the prompt and wait for input. Type a test password (e.g. hello) and press Enter. Execution will pause right before the strcmp call that checks your password.

  1. Inspect the arguments. On x86-64, function arguments are passed in registers: rdi holds the first argument and rsi holds the second. Examine them as strings:
(gdb) x/s $rdi
(gdb) x/s $rsi

One of these will show the password you typed, and the other will show the hardcoded password from the binary. This confirms what you found in Ghidra.

  1. Inspect the return value. Step into strcmp and then use finish to let it complete. The return value will be in rax:
(gdb) stepi
(gdb) finish
(gdb) info registers rax

If rax is 0, the strings matched. Any other value means they differ.

Part 4: Visual flow analysis with Cutter

Cutter (or the Ghidra graph view) provides a visual representation of the program’s control flow, making it easy to see decision points.

  1. Open the binary in Cutter (or use Ghidra’s Function Graph view: Window → Function Graph).

  2. Navigate to main using the functions list or the search bar.

  3. Switch to the graph view. You should see a flowchart with blocks connected by arrows:

    • A block that calls scanf to read input
    • A block that calls strcmp to compare strings
    • A conditional branch that splits into two paths:
      • Success path → prints the “OK” message
      • Failure path → prints the “ERROR” message

  4. Take a screenshot of the graph. Annotate it to show:

    • Where user input is read
    • Where the comparison happens
    • Which branch leads to success and which leads to failure

The graph makes it visually obvious that there is exactly one condition that determines the outcome — matching the hardcoded password.

Part 5 (Optional): Patching the binary with radare2

Instead of finding the password, you can modify the binary so that any password is accepted.

  1. Open the binary in write mode:
cp passguess passguess_patched
radare2 -w ./passguess_patched
  1. Analyze and navigate to main:
[0x00...]> aaa
[0x00...]> s main
[0x00...]> pdf

  1. Find the conditional jump. Look for a je (jump if equal) or jne (jump if not equal) instruction near the strcmp call. This is the instruction that decides which branch to take.

  2. Patch the instruction. There are two common approaches:

    Option A — Flip the condition. If the instruction says je, change it to jne, or vice versa. This inverts the logic so wrong passwords are accepted and the correct one is rejected:

    [0x00...]> s <address of the jump instruction>
[0x00...]> wa jne

    Option B — NOP the jump. Replace the conditional jump with a NOP (no-operation) instruction. This removes the branch entirely, so execution always falls through to the success path:

    [0x00...]> s <address of the jump instruction>
[0x00...]> wa nop

    Both techniques achieve the goal of bypassing the password check. Option A inverts the logic; Option B eliminates the decision entirely.

  3. Save and exit:

[0x00...]> quit
  1. Test the patched binary. Run it and enter any random password:
./passguess_patched

The program should now accept any input as correct (or reject the real password). This demonstrates how a single byte change can completely alter program behavior.

Submission

Write a short report (PDF or Markdown) containing:

  1. Reconnaissance results: output of file and strings, with the relevant strings highlighted
  2. Pseudocode analysis: a screenshot of the Ghidra decompiler output for main, with annotations explaining each line
  3. The password you found and a screenshot of the program accepting it
  4. gdb session: show the output of x/s $rdi and x/s $rsi at the strcmp breakpoint, confirming the hardcoded password
  5. Flow graph: an annotated screenshot from Cutter or Ghidra showing the two branches
  6. (Optional) If you completed Part 5, show the patched instruction and demonstrate the modified behavior
  7. Reflection: In your own words, explain why hardcoding passwords in a binary is insecure, and suggest a more secure alternative

Key concepts

TermDefinition
Static analysisExamination of code or binaries without executing them
Dynamic analysisAnalysis of behavior during execution
ELFStandard executable format on Linux systems
PEExecutable format on Windows systems
GDBEssential command-line debugger for binary analysis

Navigation: ← Previous | Home | Next →

Graph View

Backlinks