A comprehensive study guide covering endpoint security platforms, sandboxing and malware analysis, reverse engineering, malware exploitation techniques, Windows process behavioural analysis, EDR configuration, and block listing vs allow listing.
Objective 1.1 — System & network architecture in security operationsObjective 1.3 — Tools & techniques to determine malicious activity
An endpoint is any device connected to the network — desktops, laptops, smartphones, tablets. Endpoint monitoring requires identifying behavioural anomalies and detecting the techniques malware uses to achieve privilege escalation and persistence on the host.
AV / Anti-Malware
Antivirus / Anti-malware
Detects and removes viruses, worms, Trojans, rootkits, adware, spyware, and password crackers. Primarily signature-based. Foundation of endpoint protection but insufficient alone.
HIDS / HIPS
Host-based IDS/IPS
Monitors a single host for unexpected behaviour and drastic state changes — file system integrity monitoring, log monitoring, driver/application change detection. Sees what network-based IDS cannot.
EPP
Endpoint Protection Platform
The Swiss Army Knife — a single agent combining AV, HIDS/HIPS, host firewall, DLP, and file encryption. Mostly signature-driven. Leaders: Microsoft, CrowdStrike, Symantec (Gartner Magic Quadrant).
EDR
Endpoint Detection & Response
Behaviour and anomaly focused — not just signature-based. Collects system data and logs for continuous monitoring. Goal: runtime and historical visibility into a compromise, plus facilitating remediation once detected.
UEBA
User & Entity Behaviour Analytics
Automated identification of suspicious activity by user accounts and computer hosts. Establishes a behavioural baseline, then flags deviations. Heavily dependent on AI and machine learning. Examples: Microsoft Advanced Threat Analytics, Splunk UBA.
Market convergence: These five categories are increasingly merging into hybrid products marketed as ATP (Advanced Threat Protection), AEP (Advanced Endpoint Protection), and NGAV (Next-Generation AV). These are combinations of EPP + EDR + UEBA capabilities under a single product umbrella.
2
Sandboxing — Controlled Malware Analysis
Because signature-based tools are increasingly defeated by obfuscated and polymorphic malware, analysts must run malware in a controlled environment to observe its effects. A sandbox is an isolated computing environment — typically a virtual machine — where malware can execute without touching production systems or spreading to the network.
Static analysis — examine without running
🔬Inspect file structure, strings, imports, and metadata without execution.
🔑Use file signatures (magic numbers) to identify true file type regardless of extension.
📝Run strings tool to extract all ASCII/Unicode sequences — find URLs, function names, passwords.
⚠️Blocked by encrypted or packed malware — must unpack first.
🛠Tools: FLOSS, Strings, IDA Free/Pro, disassemblers.
Dynamic analysis — run and observe
▶️Execute malware inside the sandbox and monitor all system changes in real time.
📊Monitor registry changes, file creations/deletions, new processes, network connections, API calls.
📸Use VM snapshots — take a clean snapshot before running, roll back in 30–60 seconds vs. hours for a full restore.
💾Memory dump the running process to capture malware in its decrypted/unpacked state.
🛠Tools: FLARE VM, Cuckoo, Joe Sandbox, Process Monitor, Autoruns, Wireshark.
Key sandboxing tools
FLARE VM
Mandiant's Windows analysis VM
Free toolset installable on Windows 10. Provides a pre-configured Windows environment for running and analyzing Windows binaries. Can run in any VM platform (VMware, VirtualBox, Parallels).
Cuckoo
Automated multi-OS sandbox
Open-source tool that automatically executes malware samples and reports results. Supports Windows, Linux, and macOS environments. Reduces manual analysis burden through automation.
Joe Sandbox
Multi-platform dynamic analysis
Commercial tool that analyses malware across Windows, macOS, Linux, and Android. Classifies malware by family, tracks evolutionary relationships, and provides a clean dashboard interface. Automates the analysis pipeline end-to-end.
Sandbox isolation rule: The sandbox VM must never be used for any other purpose. Once infected, it must remain isolated. Only the sandbox host VM should be used for malware analysis — and it must have no path to your production network.
Exam scope: Know the concept of sandboxing, the static vs. dynamic analysis distinction, the snapshot feature, and the names of the three tools (FLARE VM, Cuckoo, Joe Sandbox). You do not need to know how to operate them — they are not tested in depth on the CySA+ exam.
Reverse engineering is the process of analyzing compiled software to understand its function without access to the original source code. In malware analysis, it reveals what the malware does, identifies the author's coding style (enabling attribution), and discovers strings for signature-based detection rules.
The code abstraction ladder
Machine code
Binary / hex — processor-native
Ones and zeros executed directly by the CPU. Represented as two hex digits per byte. Unreadable by humans without tools. File signatures (magic numbers) live at this level — the first 2–4 bytes identify the file type.
Assembly code
Via disassembler
Human-readable representation of machine instructions: PUSH, MOV, CALL, RET. Still technical but meaningful to experienced analysts. The 1970s/80s programming language. Produced by a disassembler.
High-level code
Via decompiler
Pseudo-code resembling C, Java, or Python — the most human-readable form. Not identical to the original source, but functionally equivalent. Produced by a decompiler (e.g. IDA Pro). Java can often be decompiled back to readable source.
File signatures (magic numbers)
The first 2–4 bytes of any file identify its true type — regardless of the file extension. Attackers routinely rename malware with innocent extensions (e.g. .jpg, .docx). Always check the magic bytes, never rely on the extension.
magic numbers — key file signatures to know
# File signature lookup: filesignatures.net
Windows PE executable(EXE, DLL, SYS, DRV, COM) Hex: 4D 5A ASCII: MZ Base64: TVo → ANY of these three = Windows executable, regardless of file extension
PDF document Hex: 25 50 44 46ASCII: %PDF
PNG image Hex: 89 50 4E 47ASCII: .PNG
ZIP archive Hex: 50 4B 03 04ASCII: PK..
Security Onion lab context (Ch.7): When Sguil alerted on a file with Content-Type: text/html but magic bytes MZ — that was a Windows executable disguised as an HTML file. File signature analysis explained the discrepancy.
Strings analysis
The strings tool (and its more capable version FLOSS) dumps all ASCII and Unicode character sequences over 3 characters from a binary file. Useful finds include: hardcoded URLs (C2 addresses), function names (InternetOpenUrl = downloads something), usernames, file paths, and registry keys. Limitation: packed/encrypted malware produces garbled output — the malware must be unpacked or memory-dumped first.
Program packers
A program packer compresses and/or encrypts an executable so the content is hidden until runtime. The file contains a small decompression stub + the compressed payload. Malware authors use packers to: (1) shrink file size for easier delivery, and (2) defeat static analysis tools. Packed files are not automatically malicious — legitimate software uses packing to protect intellectual property — but they warrant investigation. Always analyze both the packed and unpacked versions.
4
Malware Exploitation Techniques
Modern malware — especially APT implants — uses fileless techniques that execute directly in system memory, leaving minimal or no artifacts on disk. Understanding the attack chain helps analysts identify which stage they're seeing and what to hunt for next.
The modern APT attack chain
Stage 1
Dropper
Lightweight shellcode tricks user into execution. Installs or runs embedded payload.
Stage 2
Downloader
Fetches full implant (RAT/backdoor) from C2 server over the internet.
Stage 3
C2 & persistence
Remote access Trojan establishes command & control channel. Maintains access.
Stage 4
Lateral movement
Spreads to high-value targets (servers, DCs). Elevates privileges across the domain.
Stage 5
Actions on objectives
Steals, encrypts, or destroys data. Executes the original mission objective.
A specialized malware type designed to install or run other types of malware embedded in its payload. The stage-one component — once executed, it fetches the full implant.
Downloader
Code that connects to the internet after initial dropper infection to retrieve additional tools. The stage-two component — turns the dropper's access into a fully armed implant.
Shell code
Lightweight code designed to run an exploit on a target. In CySA+ context: any lightweight exploit code, not necessarily a command shell. (PenTest+ uses a narrower definition — shell access specifically.)
Code injection
Runs malicious code under the process ID of a legitimate process — hides malware inside a trusted process like explorer.exe. Sub-types: masquerading, DLL injection, DLL sideloading, process hollowing.
Living off the Land
Uses native system tools (PowerShell, WMI, Bash) for malicious purposes — no new executables to detect via signatures. The hardest APT technique to identify because the tools are legitimate and already trusted.
Code injection sub-types
Masquerading
Replaces a genuine executable with a malicious one using the same name. Analysts see a known process name but it's a swap.
DLL injection
Forces a running process to load a malicious DLL alongside its legitimate ones. The malicious code runs in the context of the trusted process.
DLL sideloading
Exploits a vulnerability in a legitimate program to load a malicious DLL at runtime by placing it where the program looks first.
Process hollowing
Creates a legitimate process in a suspended state, then overwrites its memory with malicious code before resuming execution. The process name in Task Manager looks legitimate.
5
Behavioural Analysis — Windows Process Forensics
Because living-off-the-land and fileless techniques evade signature detection, analysts must use behavioural analysis to identify infections. The Microsoft Sysinternals suite (Process Explorer, Process Monitor, Autoruns, etc.) enables this analysis. The foundation is establishing a known-good baseline, then flagging deviations.
Critical Windows processes — legitimacy reference (signature element)
Every entry below shows what is expected for a healthy system. Anything deviating from these patterns is suspicious and requires investigation.
Process (image)
PID / instances
Expected parent
Legitimate path
Red flags
System Idle
Always PID 0
None (kernel)
Kernel
If not PID 0, definitely suspicious
System
Always PID 4
None (kernel)
Kernel
If not PID 4, suspicious
smss.exe
One instance
System (PID 4)
%SystemRoot%\System32
Multiple instances or wrong parent
csrss.exe
Multiple (per session)
smss.exe (no parent shown)
%SystemRoot%\System32
Has a visible parent process (should appear parentless)
wininit.exe
Single instance
smss.exe
%SystemRoot%\System32
More than one instance
services.exe
Single instance
wininit.exe
%SystemRoot%\System32
High-frequency masquerade target. Wrong parent, multiple instances, or started by a username (not SYSTEM)
lsass.exe
Single instance
wininit.exe
%SystemRoot%\System32
Multiple instances or wrong parent
winlogon.exe
One per user session
smss.exe
%SystemRoot%\System32
Wrong parent or extra instances outside active sessions
userinit.exe
Transient (logon only)
winlogon.exe
%SystemRoot%\System32
Still running 30+ minutes after logon — should terminate quickly after launching explorer.exe
explorer.exe
One per logged-on user
userinit.exe
%SystemRoot%
Parent of all user-launched processes. Any process not descending from explorer (or services/system) for a user is suspicious
services.exe masquerade watch: Malware frequently impersonates services.exe. Check: (1) there is only one instance, (2) it is a child of wininit.exe, (3) it was started by the SYSTEM account — not a username. Any violation of these three rules is a strong malware indicator.
Eight signals that make a process suspicious
1
Unrecognised name
Google it on Microsoft's official site before concluding it's malicious — but always investigate the unknown.
2
Name similar to a legitimate system process
scvhost.exe vs svchost.exe, lssas.exe vs lsass.exe — typosquatted names exploiting analyst inattention.
3
No icon, no version info, no description, no company name
All legitimate Microsoft binaries carry full metadata. Missing metadata indicates either poor coding or deliberate concealment.
4
Unsigned binary from a well-known publisher
Microsoft signs all its binaries. A process claiming to be from Microsoft Corporation but without a valid signature is almost certainly malicious.
5
Digital signature doesn't match the claimed publisher
If a stolen developer private key was used to sign malware, the company name field and the actual digital certificate won't match.
6
Wrong parent-child relationship
User-launched processes should descend from explorer.exe. Background services from services.exe or svchost.exe. Anything launched from an unexpected parent warrants investigation.
7
Running from a temp or AppData directory
Legitimate system processes run from %SystemRoot%\System32 or %ProgramFiles%. Malware frequently installs itself to %TEMP% or %AppData%\Roaming.
8
Packed or compressed (shown in purple in Process Explorer)
Purple highlighting in Process Explorer indicates a packed/compressed binary. Not conclusive evidence of malice (legitimate software packs too), but always warrants deeper analysis.
Investigation questions for a suspicious process
process investigation checklist
# Seven questions to answer for every suspicious process:
1.Registry & file system changes— Is it modifying the registry or creating files? 2.Launch method— User, service, scheduled task, or something else? 3.Image file location— System32 (trusted) or TEMP/AppData (suspicious)? 4.File manipulation— What is the process reading and writing? 5.Persistence on reboot— Delete it; does it come back after reboot? 6.Privilege impact— Does deleting it break system privileges or services? 7.Network activity— Is it communicating outbound? What IP/domain?
Sysinternals tools used in endpoint forensics
Process Explorer
Visualises the full process tree with parent-child relationships. Shows process metadata, digital signatures, and highlights packed files in purple. The primary tool for identifying suspicious processes.
Process Monitor
Real-time log of all registry, file system, network, and process activity. Captures exactly what a process touches — essential for dynamic analysis.
Autoruns
Lists every program configured to run at startup — registry Run keys, scheduled tasks, services, drivers. Compare a pre-infection baseline save against a post-infection scan to identify exactly what malware added.
6
EDR Configuration — VirusTotal, MAEC & YARA
EDR systems require continuous tuning to reduce false positives. Community sharing accelerates signature development and benefits everyone. Custom rules handle organisation-specific threats that commercial feeds don't cover.
VirusTotal
VirusTotal scans submitted files and URLs against 70+ antivirus engines and blacklisting services simultaneously. It's free and widely used for rapid triage. Important caveat: before submitting a malware sample, check your organisation's policy. Attackers monitor VirusTotal — if they see their malware detected there, they'll update their techniques to evade it. The intelligence value of your sample may outweigh the benefit of sharing it publicly.
MAEC — Malware Attribute Enumeration and Characterization
MAEC is a standardised language for sharing structured malware information. It is complementary to STIX and TAXII (Ch. 4), enabling automated cross-organisation malware intelligence sharing. Use MAEC when your organisation needs to share custom malware findings with partners or vendors in a machine-readable format.
YARA rules
YARA is a multi-platform (Windows/Linux/macOS) pattern-matching tool that identifies, classifies, and describes malware samples. A YARA rule defines string combinations to match against binary files, log files, packet captures, or emails — searching for the needle in the haystack.
strings:← sequences found during analysis to use as detection indicators $s1 = "InternetOpenUrl" $s2 = "CreateRemoteThread" $s3 = "\\AppData\\Roaming\\"
condition:← when to fire: any, all, 2 of them, etc. 2 of ($s*)
}
In practice, YARA rules are run against PCAP files on network sensors (Security Onion) or against file system paths during threat hunting. They're the primary mechanism for searching large datasets for a specific piece of malware once you know what strings to look for. Exam: know what YARA is and what it does — you do not need to write rules.
7
Block Listing & Allow Listing
Two complementary access control philosophies govern what is permitted to run on or communicate with your systems. The nightclub bouncer analogy is useful: block list = name on the banned list means no entry; allow list = name must be on the approved list or no entry.
Block listing — deny known bad
✅ Everything is permitted unless it appears on the block list.
✅ Lower operational overhead — most traffic flows normally.
✅ Good for day-to-day operations and IP/URL blocking.
⚠️ You can only block what you know is bad — unknown threats pass through.
⚠️ False positives can block legitimate traffic (e.g. shared IP with former spammer).
🔒 Nothing is permitted unless it appears on the allow list.
🔒 Maximum security — only explicitly approved items can run/connect.
🔒 Excellent as a fallback posture during incident response.
⚠️ Highly restrictive — breaks access to any new or changed destination.
⚠️ High maintenance: every new supplier, site, or IP must be added.
⚠️ Load-balanced services (multiple IPs) require all IPs to be listed.
Application allow listing — execution control
Execution control determines what additional software may be installed or run on a client/server beyond its baseline. The most effective place to apply allow listing is at the application level — preventing unauthorised executables from running at all.
Windows SRP
Software Restriction Policies
Available for most Windows versions via Group Policy Objects. Creates allow/block rules based on file system locations where executables can run. Can restrict execution to System32 and Program Files only — blocking anything running from %TEMP% or %AppData%.
AppLocker
Enhanced SRP (Enterprise/Ultimate)
Improves on SRP by allowing policies to apply per-user or per-group (not just per-machine). Supports rules based on file hash, publisher signature, or file path. Requires Windows Enterprise or Ultimate edition.
WDAC
Windows Defender Application Control
Code integrity policy engine — can be used standalone or with AppLocker. Supports version-aware rules and digital signature enforcement. Can prevent even administrative accounts from disabling execution controls.
SELinux / AppArmor
Linux Security Modules
Mandatory Access Control (MAC) for Linux systems. SELinux and AppArmor are the most widely deployed Linux Security Module implementations for enterprise environments. Enforce access control at kernel level.
Allow list drift: Allow lists are only as good as their maintenance. A new supplier website, a load-balanced IP change, or a software update with a new hash will all break access unless the list is updated proactively. Use configuration management processes to handle updates — risk assessment and business impact analysis before large changes.
Exam tip: Block listing = known-bad denied, all others permitted. Allow listing = known-good permitted, all others denied. Application allow listing is one of the strongest endpoint controls available. Know the Windows tools (SRP → AppLocker → WDAC) and the Linux equivalent (SELinux/AppArmor via MAC/LSM). Configuration management governs updates to both types of lists.
Isolated VM for malware execution. Static = no execution (strings, file sig). Dynamic = run and observe (registry, network, processes). Snapshots = fast rollback. Memory dump = captures decrypted malware. Tools: FLARE VM, Cuckoo, Joe Sandbox.
File signatures
First 2–4 bytes reveal true file type. Windows PE = 4D5A (hex) / MZ (ASCII) / TVo (Base64). Never trust the extension — check the magic bytes. Look up at filesignatures.net.
Reverse engineering tools
Disassembler: machine code → assembly. Decompiler: assembly → high-level pseudo-code. IDA Pro = gold standard decompiler. Strings / FLOSS = dump readable strings. Packers encrypt/compress — unpack before analyzing.
Uses native tools (PowerShell, WMI, Bash) maliciously. No new executables = no signatures to detect. Hardest APT technique to identify. Requires behavioural baseline comparison to spot.
Windows process indicators
System=PID 4, Idle=PID 0. services.exe = single instance, child of wininit, started by SYSTEM (not a username). userinit.exe = transient (logon only). explorer.exe = parent of all user processes. Wrong parent = red flag.
Process suspicion signals
Unrecognised name, typosquatted name, no metadata/signature, wrong parent, runs from TEMP/AppData, packed file (purple in Process Explorer), network activity to unknown IPs, restores itself after deletion.
EDR tools
VirusTotal = 70+ AV engines, free, but attackers monitor it. MAEC = standardised malware sharing language (complements STIX/TAXII). YARA = pattern-matching rules against files/PCAPs/logs to hunt specific malware strings.
Block vs. allow listing
Block = deny known bad, all else permitted (day-to-day default). Allow = permit known good only (fallback during IR). Allow listing is harder to maintain but stronger. False positives can block legitimate traffic on both.
Execution control (Windows)
SRP = basic, via GPO, location-based rules. AppLocker = user/group-based, Enterprise+ only. WDAC = code integrity policy, strongest, admin-proof. All can use hash-based, publisher-based, or path-based rules.
Execution control (Linux)
MAC = Mandatory Access Control (general concept). LSM = Linux Security Modules (implementation framework). SELinux and AppArmor = the two most common enterprise LSM implementations.