Quartz 4

05 - Reverse Engineering

5 min read

Reverse Engineering

Objectives: By the end of this topic, you will be able to…

  • Identify key functions and strings within a binary
  • Use tools such as objdump, gdb, and Cutter to inspect machine code
  • Understand the basic execution flow without access to source code
  • Recognize limitations, risks, and ethical aspects of reverse engineering

What is reverse engineering and what is it used for?

Reverse engineering is the process of analyzing the internal workings of a program or system without access to its original source code, in order to understand its behavior, architecture, or potential weaknesses.

In cybersecurity, it is used to:

  • Analyze malware and determine its purpose
  • Identify security vulnerabilities
  • Validate software integrity (auditing)
  • Perform forensic analysis during incidents
  • Study protection mechanisms (anti-debugging, obfuscation)
  • Ensure compatibility with legacy or proprietary software

It is a common skill in malware analysis, exploit development, and security auditing.

Differences between static and dynamic analysis

Static analysis

  • Performed without running the program
  • The binary file is examined directly using a disassembler
  • Allows reviewing assembly code, functions, strings, and structures
  • Safer, but can be harder to interpret

Common tools: strings, objdump, Ghidra, IDA Free

Dynamic analysis

  • The binary is executed in a controlled environment (sandbox or VM)
  • Allows observing real behavior: system calls, network access, file modifications
  • Useful for detecting hidden behavior or dynamic dependencies

Common tools: gdb, strace, ltrace, radare2, Cutter

Executable formats

ELF (Executable and Linkable Format)

  • Standard executable format on Linux systems
  • Contains sections like .text, .data, .bss, .plt, .got
  • May include debugging symbols and exported functions

PE (Portable Executable)

  • Executable format on Windows (.exe, .dll)
  • Similar structure to ELF with DOS header and import/export tables
  • Widely used by malware developers

Key difference: ELF is native to Unix/Linux, PE is native to Windows. Analysis tools differ between platforms, though concepts like sections, imports, and headers are similar.

Disassemblers and debuggers

Disassemblers

Translate machine code into assembly language to understand what a binary does at a low level.

objdump -d binary
radare2 -A binary

Ghidra provides both disassembly and pseudocode decompilation.

Debuggers

Allow running a binary step by step, using breakpoints to pause execution at critical points, and inspecting memory, registers, and call stack.

gdb binary

Basic debugging commands:

  • break <function|address> — set a breakpoint
  • run — start execution
  • next / step — step through execution
  • info registers / x / print — inspect state

Function analysis and conditional logic

Function identification

Functions can be located via symbol tables, prolog/epilog patterns, or cross-references. Tools like Ghidra detect them automatically.

Key concepts:

  • Function prolog: stack setup (push ebp, mov ebp, esp)
  • Epilog: stack restoration and return (pop ebp, ret)
  • Calls (call) and jumps (jmp, je, jne, cmp)

Conditional analysis

Program decisions are represented as blocks with comparison instructions (cmp) and conditional jumps (je, jne, jg, jl). Understanding this logic helps identify critical branches: password validation, flow decisions, anti-analysis tricks.

Common techniques:

  • Follow execution flow with gdb or graphical tools
  • Analyze code near main, printf, system, or network/system calls
  • Trace variables and buffers on the stack

Ethics and legality of binary analysis

Reverse engineering can have legitimate uses (auditing, research, learning), but it can also violate:

  • Copyright laws (restrictive licenses)
  • Terms of use (EULA)
  • Cybercrime laws if done without consent

Best practices:

  • Only analyze software from legitimate sources or created for educational purposes
  • Avoid distributing modified code without permission
  • Document analysis for academic or security improvement purposes

All technical practices must be backed by clear ethical principles and explicit authorization.

Hands-on lab

Requirements: Kali Linux, provided ELF binary, gdb, Cutter

Part 1: Basic static inspection

  1. Run initial analysis on the provided binary:
file program
strings program | less
objdump -d program | less
  1. Identify:
    • Whether it is 32-bit or 64-bit
    • Potential strings containing passwords or function names (strcmp, printf)
    • Key functions (main, check_pass) and their addresses
    • Draw a basic flowchart of the program

Part 2: Step-by-step debugging with gdb

  1. Debug the binary:
gdb ./program
break main
run
next / step
info registers
x/s $rsp
  1. Activities:
    • Set a breakpoint at main and step through execution
    • Inspect registers (eax, rax, rsp)
    • Observe how the password comparison is performed
    • Identify the exact decision point

Part 3: Visual analysis with Cutter

  1. Open the binary in Cutter
  2. Use the flowchart view to see conditionals and jumps
  3. Navigate to the main function and locate the conditional tied to the success message
  4. Take screenshots and correlate findings with gdb and objdump

Part 4 (Optional): Binary manipulation with radare2

  1. Locate a conditional jump instruction (je, jne):
radare2 -w ./program
aaa
s main
pdf
  1. Modify the instruction:
s <instruction address>
wa jne   ; or wa je as needed
  1. Save and rerun to verify the behavior changed

Submission

Short report containing:

  • Tools used and key commands
  • Description of the logical flow discovered
  • Evidence: screenshots, assembly code snippets
  • Explanation of how the key was found or behavior was forced

Key concepts

TermDefinition
Static analysisExamination of code or binaries without executing them
Dynamic analysisAnalysis of behavior during execution
ELFStandard executable format on Linux systems
PEExecutable format on Windows systems
GDBEssential command-line debugger for binary analysis

Test yourself

  1. Conceptual: What is the main difference between a disassembler and a debugger? In what situation would you use each one?

  2. Practice: Given an unknown binary, describe your analysis methodology step by step, starting with static tools and progressing to dynamic ones.

  3. Ethics: Under what circumstances is it legal to reverse engineer commercial software? What precautions should you take?

Navigation: ← Previous | Home | Next →

Graph View

Backlinks