Investigation Workflow

The Difference Between Reacting and Investigating

A new analyst sees an alert and asks: "Is this bad?" A senior analyst sees the same alert and asks: "What story does this alert begin to tell, and what other evidence would confirm or deny it?"

That shift — from reacting to investigating — is the single most important skill transition in a SOC career. Lessons 4.1 and 4.2 gave you the classification framework and the context dimensions. This lesson gives you the workflow — the repeatable, step-by-step process that turns a raw alert into a confident verdict.

Every experienced analyst follows this workflow, whether they realize it or not. Some do it in 90 seconds on a routine false positive. Others spend 45 minutes on a complex chain of suspicious events. But the steps are always the same.

Click to enlarge

The 5-Step Investigation Workflow

Total target: 5-10 minutes for a Tier 1 analyst on a standard alert. Complex investigations that exceed this window get escalated — not abandoned.

Step 1: READ — What Actually Fired?

Before you do anything, read the raw alert. Not the summary. Not the title. The actual event data. Summaries lie — they truncate, they simplify, they strip context. The raw JSON never lies.

Here's a real alert from Wazuh. Take 30 seconds and read every field:

{
  "rule": {
    "id": "5551",
    "level": 10,
    "description": "Multiple SSH authentication failures",
    "groups": ["syslog", "sshd", "authentication_failures"],
    "mitre": {
      "id": ["T1110.001"],
      "tactic": ["Credential Access"],
      "technique": ["Brute Force: Password Guessing"]
    }
  },
  "agent": {
    "id": "001",
    "name": "linux-web-01",
    "ip": "10.0.0.10"
  },
  "data": {
    "srcip": "185.220.101.42",
    "srcport": "44892",
    "dstuser": "root"
  },
  "timestamp": "2026-02-23T02:47:33.000+0000",
  "location": "/var/log/auth.log"
}

In 30 seconds, you should extract these facts:

What to Extract on Every Alert

Build the habit of extracting these seven fields from every alert, regardless of type:

1. What rule fired — ID, description, severity level
2. What host was involved — agent name, IP, role in the environment
3. Who was involved — username, account type, privilege level
4. Where did it come from — source IP, source port, geographic origin
5. When did it happen — exact timestamp, relative to business hours
6. What log source generated it — auth.log, sysmon, firewall, web server
7. What ATT&CK technique maps to it — tactic and technique ID

Step 2: ENRICH — Add Context

Raw data tells you what happened. Context tells you whether it matters. In Lesson 4.2, you learned the five context dimensions. Now apply them systematically to the SSH brute-force alert:

GeoIP and Threat Intel Lookups

For the source IP 185.220.101.42, a quick lookup reveals:

AbuseIPDB:    Confidence Score 100% — reported 2,847 times
GreyNoise:    Classification: malicious — known scanner
Tor Project:  Listed as exit node
GeoIP:        Germany (DE), AS205100 — F3 Netze e.V.

This enrichment immediately shifts your assessment. This isn't an employee mistyping their password. This is a known malicious IP associated with thousands of abuse reports, running through the Tor network.

Step 3: PIVOT — Search for Related Events

A single alert is a data point. Multiple related alerts are a story. Pivoting means searching for other events connected to your alert — expanding your view from a single dot to the full picture.

There are four pivot directions. Run all four on every investigation.

Pivot 1: Same Host (±30-Minute Window)

Question: What else happened on linux-web-01 in the 30 minutes before and after this alert?

In Wazuh, filter by agent name and time range:

agent.name: "linux-web-01" AND timestamp:[2026-02-23T02:17:00 TO 2026-02-23T03:17:00]

What you're looking for:

• Other authentication failures (escalating brute force?)
• Successful authentication after failures (brute force succeeded?)
• New process execution (post-compromise activity?)
• File integrity changes (persistence mechanism installed?)
• Sudo or privilege escalation events
• Outbound connections to unusual IPs (C2 callback?)

Pivot 2: Same User

Question: Has the root account been targeted or used anywhere else recently?

data.dstuser: "root" AND timestamp:[2026-02-23T00:00:00 TO 2026-02-23T03:17:00]

What you're looking for:

• Is root being targeted on multiple hosts? (Coordinated campaign)
• Any successful root logons from unexpected sources?
• Sudo-to-root escalations from compromised user accounts?

Pivot 3: Same Source IP

Question: Is 185.220.101.42 hitting other systems in our environment?

data.srcip: "185.220.101.42"

What you're looking for:

• Same IP targeting multiple hosts → automated scanning campaign
• Same IP targeting multiple services (SSH, HTTP, RDP) → reconnaissance
• Same IP appearing in Suricata alerts → network-level detection correlation
• Same IP in firewall deny logs → how many attempts were blocked?

Pivot 4: Same Technique

Question: Are other IPs also brute-forcing SSH across our environment right now?

rule.id: "5551" AND timestamp:[2026-02-23T02:00:00 TO 2026-02-23T03:00:00]

What you're looking for:

• Multiple source IPs hitting the same target → distributed brute force
• Same rule firing across many hosts → worm-like propagation or mass scanning
• Increase in frequency compared to baseline → active campaign vs background noise

Step 4: CORRELATE — Build the Story

Pivoting gives you related events. Correlation turns those events into a narrative. The question shifts from "what happened?" to "is this an isolated event or part of an attack chain?"

Building a Timeline

Arrange your pivot findings chronologically:

Kill Chain Mapping

The timeline above maps cleanly to MITRE ATT&CK:

This is no longer a "brute-force alert." It's a confirmed compromise with persistence and C2 — a completely different severity and response.

Isolated Event vs. Attack Chain

Ask yourself three questions:

1. Is there progression? Do events follow a logical attack sequence? (Failures → success → post-exploitation = YES)
2. Is there a common thread? Do events share the same source IP, user, or host? (185.220.101.42 appears in all = YES)
3. Do later events depend on earlier ones? Could the curl download happen without the SSH access? (NO — they're causally linked)

If all three answers are "yes," you have an attack chain. If only one event exists with no follow-on activity, it's likely isolated.

Step 5: DECIDE — Verdict and Action

After reading, enriching, pivoting, and correlating, you arrive at the decision point. There are exactly three outcomes:

True Positive → Escalate and Respond

The evidence confirms malicious activity. Your responsibilities:

1. Escalate immediately to Tier 2/3 or incident response, per your escalation procedures
2. Preserve evidence — do NOT remediate yet (don't kill processes, don't block IPs) unless the attack is actively causing damage. Evidence preservation comes first.
3. Document everything you found — timeline, queries, indicators

False Positive → Document and Close

The evidence confirms benign activity. Your responsibilities:

1. Document your reasoning — why is this a false positive?
2. Close the alert with a clear explanation another analyst can understand
3. Consider tuning — if this FP recurs, submit a tuning request to reduce noise

Ambiguous → Collect More Data (With a Time Limit)

You can't tell. The evidence is inconclusive. Your responsibilities:

1. Expand your pivots — wider time window, additional log sources
2. Set a time limit — give yourself 5 more minutes, not infinity
3. If still ambiguous after the time limit, escalate — "I found X but couldn't confirm Y, here's what I checked" is a perfectly valid escalation

Time Management: The 10-Minute Rule

Tier 1 analysts process volume. Your queue has 50-100 alerts per shift. If you spend 45 minutes on each one, you process 10 alerts and the other 90 sit untouched — including the one that's a real breach.

The 10-Minute Rule:

This doesn't mean complex investigations can't exceed 10 minutes. It means that Tier 1 should make a triage decision within 10 minutes. If the investigation needs more time, the decision is to escalate — which is itself a valid and complete triage outcome.

Documentation: What Goes in Your Ticket

Every alert you touch gets documented, even if it takes only 90 seconds to close as a false positive. Your ticket is the audit trail, the knowledge base for other analysts, and your proof of work.

At Each Step, Record:

Example Ticket — SSH Brute Force on linux-web-01

ALERT:      Rule 5551 — Multiple SSH authentication failures
SEVERITY:   High (Level 10)
TARGET:     linux-web-01 (10.0.0.10) — Tier 2 production web server
SOURCE:     185.220.101.42 (Tor exit node, AbuseIPDB 100%, GreyNoise: malicious)
TIME:       2026-02-23 02:47 UTC (outside business hours)

INVESTIGATION:
- Same Host pivot: Found 847 failed SSH attempts (02:31-02:47),
  followed by successful SSH logon at 02:48 from SAME source IP.
  Post-logon: new user created (02:49), curl download (02:51),
  crontab modified (02:55).
- Same Source IP: 185.220.101.42 also hit dns-server-01 (SSH scan,
  no success).
- Same User: root account — no other unusual activity prior to
  this event.
- Same Technique: Rule 5551 fired only for this source IP in
  the past 24 hours.

TIMELINE: Brute force (16 min) → success → user creation →
C2 download → persistence

VERDICT:    TRUE POSITIVE — Confirmed compromise
ACTION:     Escalated to Tier 2 / IR team. Host requires
            immediate containment.
CONFIDENCE: High — attack chain confirmed across 6 correlated events

Practical Walkthrough: From Alert to Verdict

Let's trace the complete workflow on the SSH brute-force alert from Step 1, simulating what you'd do in the Wazuh dashboard during Lab 4.1.

Minute 0:00 — READ. Open the alert. Rule 5551, level 10, SSH brute force against root on linux-web-01 from 185.220.101.42 at 02:47 UTC. Source log: /var/log/auth.log. ATT&CK mapping: T1110.001 (Brute Force: Password Guessing).

Minute 0:30 — ENRICH. linux-web-01 is a Tier 2 production web server (asset context). Root is the highest-privilege account (user context). 02:47 UTC is outside business hours — no maintenance scheduled (temporal context). 185.220.101.42 is a known Tor exit node with 100% abuse confidence (geographic + TI context). SSH failures on this host baseline is ~5/day — 847 in 16 minutes is a 169x deviation (behavioral context).

Minute 2:00 — PIVOT. Same Host (±30 min): discover 847 failures, then successful logon, then new user creation, curl download, crontab modification. Same Source IP: this IP also probed dns-server-01. Same User: root has no other unusual activity. Same Technique: rule 5551 only triggered for this IP.

Minute 6:00 — CORRELATE. Build the timeline (shown in Step 4 above). Map to kill chain: reconnaissance → initial access → persistence → C2. Six correlated events. Attack chain confirmed.

Minute 8:00 — DECIDE. This is a true positive. The brute force succeeded, the attacker established persistence (new user + crontab), and initiated C2 communications. Escalate immediately. Document the full timeline, queries run, and indicators.

Total time: 8 minutes. Well within the 10-minute guideline. The key: no wasted motion, systematic pivoting, and the confidence to make a call once the evidence is clear.

Common Investigation Mistakes

What's Next

You now have the investigation workflow — the repeatable process that turns any alert into a confident verdict. In Lesson 4.4: Decoding & Deobfuscation, you'll learn what to do when attackers hide their payload behind layers of Base64, URL encoding, and PowerShell obfuscation. Because sometimes Step 1 (Read) reveals an event full of encoded strings — and you can't investigate what you can't read. CyberChef becomes your decoder ring.