Log Data

Firewall, Application, and Endpoint Logs

Security monitoring depends on visibility into what is happening across the environment. Different log sources provide different perspectives. No single log type provides complete visibility; effective security operations require correlating multiple log sources through a SIEM (Security Information and Event Management) platform.

Firewall Logs

Firewall logs record every connection attempt that passes through the firewall, whether permitted or denied. Standard firewall log fields:

• Source IP and port: Where the connection originated
• Destination IP and port: Where the connection was directed
• Protocol: TCP, UDP, ICMP
• Disposition (action): Allow or deny — the firewall decision
• Timestamp: When the connection was attempted

Next-generation firewall (NGFW) additions: Traditional firewalls operate at Layer 3/4 (IP and port). NGFWs add:

• Application identification: Identifies the application generating traffic regardless of port (e.g., BitTorrent using port 80, Dropbox using HTTPS)
• URL categories: Classifies destinations by content type (social media, malware, streaming)
• Anomaly detection: Flags unusual traffic patterns that port/IP rules alone would miss
• User identity: Associates traffic with authenticated user accounts rather than just IP addresses

Application Logs

Application logs record events generated by software: successful and failed logins, data access, transactions, errors, and configuration changes. Location varies by platform:

• Windows: Event Viewer — System, Security, and Application log channels. Each event has an Event ID that identifies the event type (e.g., Event ID 4625 = failed logon, 4648 = explicit credential use).
• Linux: /var/log directory — syslog, auth.log, apache/access.log, and application-specific log files. Format is text-based and parseable.
• Web servers: Access logs (every HTTP request with source IP, URL, user agent, HTTP method, response code) and error logs.

Application logs are critical for detecting account-based attacks: repeated failed logins, privilege escalation, access to sensitive data outside normal patterns.

Endpoint Logs

Endpoint logs capture activity on individual workstations and servers: authentication events, policy changes, process execution, file system changes, and network connections. When forwarded to a SIEM, endpoint logs from thousands of systems can be correlated to identify:

• Accounts logging in at unusual times or from unusual locations
• Policy changes made outside authorized change windows
• New processes executing on many endpoints simultaneously (malware propagation indicator)
• Lateral movement: a single account authenticating to an unusual number of systems in a short time

OS Security Logs

Operating system security logs provide authentication details beyond what application logs capture: specific credential types used, logon session IDs, account lockout events, file permission changes, and audit policy changes. OS security logs are especially valuable for detecting brute force attacks (repeated 4625 events on Windows), privilege escalation (local group membership changes), and covering of tracks (audit log clearing — which itself generates a log event). Because OS security logs are extremely verbose, filtering rules are required before forwarding to SIEM to avoid overwhelming storage and correlation capacity.

IPS, Network, and Metadata Logs

IPS/IDS Logs

Intrusion Prevention System (IPS) and Intrusion Detection System (IDS) logs record detected attack signatures and anomalies. Standard IPS/IDS log fields:

• Timestamp: Exact time of detection
• Attack classification: Category of attack (buffer overflow, SQL injection, port scan, malware communication)
• Source IP and port: Origin of the detected attack
• Destination IP and port: Target of the attack
• Severity: Risk level of the detected signature
• Action taken: For IPS, whether the traffic was dropped or permitted with alert

Modern NGFWs typically integrate IPS functionality. Standalone IDS sensors (passive, detection only) are also deployed at network taps for deep traffic analysis. IPS logs must be correlated with firewall logs: an IPS alert on traffic that the firewall should have blocked indicates a firewall misconfiguration.

Network Logs

Network infrastructure devices generate logs that provide visibility into traffic patterns and connectivity:

Metadata

Metadata is data that describes other data. It is distinct from content: metadata tells you who, when, where, and how — without requiring analysis of the content itself.

• Email metadata (headers): Sender, recipient, all relay servers the message passed through with timestamps and IP addresses, message ID, authentication results (SPF, DKIM, DMARC). Email headers are forensically valuable for tracing the origin of phishing messages even when the display name is spoofed.
• Mobile device metadata: GPS coordinates embedded in photos (EXIF data), cell tower triangulation records, call detail records (CDR) showing who called whom and from where.
• Web browser metadata: Browser type and version (user agent string), operating system, screen resolution, installed plugins, timezone, language — used in browser fingerprinting. Also: cookies, cached credentials, download history.
• File metadata: Author name, creation date, last modified date, last accessed date, document revision history. Often persists in documents even after attempts to sanitize identifying information.
• Network metadata (NetFlow/IPFIX): Source/destination IP and port, bytes transferred, packet count, flow duration — without packet content. Useful for identifying large data transfers and unusual connection patterns without requiring full packet capture storage.

Vulnerability Scans, SIEM, Dashboards, and Packet Captures

Vulnerability Scan Data

Vulnerability scanners (Nessus, Qualys, OpenVAS) probe systems for known vulnerabilities and produce detailed reports. Vulnerability scan data provides:

• Current attack surface: Which systems have which vulnerabilities, ranked by severity (CVSS score)
• Patch status: Missing patches across the environment
• Configuration weaknesses: Default credentials, open ports, weak cipher suites
• Trending: When vulnerability scan data is correlated over time in SIEM, it shows whether the patch management program is reducing the attack surface or whether vulnerabilities are accumulating

Vulnerability scan data combined with IPS logs enables prioritization: an IPS alert for an exploit of CVE-XXXX is critical if vulnerability scans confirm that CVE-XXXX is present on the targeted system, and lower priority if the system is already patched.

SIEM: The Central Correlation Platform

A SIEM (Security Information and Event Management) platform ingests logs from all sources, normalizes them into a common format, correlates events across sources using rules, and generates alerts. SIEM capabilities:

• Log aggregation: Collects logs from firewalls, endpoints, servers, applications, IPS, network devices, and cloud platforms into a central repository
• Normalization: Converts disparate log formats into a common data model for consistent querying and correlation
• Correlation rules: Matches patterns across multiple log sources to identify incidents that no single source would reveal. Example: a failed VPN authentication from China followed within 5 minutes by a successful authentication from the same account from the US is suspicious — geographically impossible travel.
• Alerting: Generates alerts when correlation rules fire, with severity ratings based on the rule and involved assets
• Automated reports: Scheduled reports for compliance (failed logins per week, admin account changes, firewall policy changes) and trend analysis
• Retention: Stores historical log data for forensic investigation and compliance requirements

Dashboards: Real-Time Visibility (Not Long-Term)

Security dashboards provide real-time or near-real-time visualization of current security status: active alerts, traffic volumes, authentication activity, top blocked IPs. Dashboards are designed for current situational awareness — they are not historical analysis tools. Key distinction:

• Dashboards show what is happening right now (or in the last hour)
• SIEM reports analyze what happened over a longer period (last week, last quarter)

Packet Captures

Packet captures (pcap files, analyzed with Wireshark) record the complete content of network traffic at the deepest possible level of detail. Packet captures capture everything: source and destination IPs, ports, protocols, and the full payload of each packet. Uses in security:

• Incident investigation: Reconstruct exactly what data was transmitted during a compromise; identify malware command-and-control communication protocols; extract files transferred over the network
• Protocol analysis: Verify that encrypted communications are using correct cipher suites; detect cleartext credentials being transmitted
• Malware analysis: Capture malware network behavior in a sandbox environment

Packet captures are the deepest level of network visibility but have significant limitations at scale: storage requirements are enormous (a 10 Gbps link generates terabytes per day), and legal/privacy constraints limit where packet capture can be deployed (especially for encrypted traffic and in jurisdictions with strict privacy laws). Full packet capture is typically restricted to specific segments (DMZ, critical servers) rather than deployed pervasively.