Protocol Analysis: HTTP, TLS & SMB

Why Protocol Analysis Matters

Signature-based rules catch known-bad patterns — a specific exploit payload, a blacklisted IP, a malicious file hash. But what about threats that don't match any signature? What about the attacker who uses a legitimate tool with default settings, or the malware that communicates over standard HTTPS?

This is where protocol analysis becomes essential. Instead of asking "does this match a known-bad pattern?", you're asking "does this behavior look normal for this protocol?"

In Lessons 3.1 and 3.2, you learned how Suricata monitors traffic and fires alerts when rules match. Now we go deeper. Rules tell you something happened. Protocol analysis tells you what it means. A rule fires on a suspicious User-Agent — but is it really suspicious? A TLS connection goes to an unusual domain — but is the certificate legitimate? An SMB session opens between two workstations — but should those machines be talking?

A skilled analyst reads protocol headers the way a detective reads body language. The words might say one thing, but the posture, timing, and context reveal the truth.

Click to enlarge

---

HTTP — The Most Analyzed Protocol

HTTP is the most common protocol on any network and the most frequently abused. Web applications, APIs, file downloads, C2 communications, data exfiltration — all of it can ride on HTTP. Because HTTP headers are transmitted in cleartext (or visible before TLS encryption in the case of HTTPS), they provide a wealth of metadata for analysis.

User-Agent Analysis

The User-Agent header identifies the client software making the request. Legitimate browsers announce themselves with long, specific strings. Attack tools often use distinctive or lazy User-Agents.

URI Analysis

The URI (path and query string) reveals what the client is requesting. Normal URIs follow predictable patterns. Attack URIs contain payloads, probing patterns, or access to files that shouldn't exist.

HTTP Methods

Standard web traffic is overwhelmingly GET and POST. Unusual methods to unexpected paths warrant investigation.

Response Code Analysis

The response code tells you whether the server fulfilled the request. A 200 OK to a web shell path is far more alarming than a 404.

---

TLS — What You Can Still See Through Encryption

A common misconception is that HTTPS makes traffic invisible to network monitoring. Encryption protects the payload — the actual data being exchanged. But the TLS handshake that establishes encryption happens in cleartext, leaking valuable metadata.

The TLS Handshake (Simplified)

1. Client Hello — Client announces supported cipher suites, extensions, and the target domain (SNI)
2. Server Hello — Server selects cipher suite and sends its certificate
3. Key Exchange — Both sides agree on encryption keys
4. Encrypted Session — All subsequent data is encrypted

Steps 1 and 2 happen in cleartext. This is where NIDS extracts metadata.

Click to enlarge

SNI — Server Name Indication

SNI is part of the Client Hello message. It tells the server which domain the client wants to connect to (necessary because multiple domains can share one IP via virtual hosting). NIDS sees this and logs it.

Why it matters: Even though the payload is encrypted, you can see that a workstation connected to update.malware-c2.com. The domain name alone can be an IOC.

JA3 / JA3S Fingerprints

JA3 is a method of fingerprinting TLS clients by hashing specific parameters from the Client Hello message: TLS version, cipher suites, extensions, elliptic curves, and EC point formats. JA3S does the same for the Server Hello.

The result is an MD5 hash that is unique to the application, not the network or user. The same application produces the same JA3 hash regardless of what domain it connects to or what IP it uses.

Why it matters for detection:

Certificate Analysis

Suricata logs certificate details from the Server Hello: issuer, subject, validity period, and whether the certificate is self-signed. Suspicious indicators include:

• Self-signed certificates on public-facing connections (legitimate for internal services, suspicious for external)
• Recently issued certificates (< 7 days) on newly registered domains — common with phishing and C2
• Mismatched subject/SNI — certificate says *.legit-company.com but SNI is malware-c2.evil
• Unusual issuers — free CAs like Let's Encrypt are legitimate but also heavily used by attackers for short-lived C2 domains

What NIDS Cannot See

After the handshake completes and the session is encrypted:

• HTTP request/response bodies (form data, file uploads, API payloads)
• Specific URLs and paths (only the domain from SNI is visible)
• Cookies and authentication tokens
• Any application-layer data

This is why endpoint monitoring (Wazuh agent, EDR) complements network monitoring — the endpoint sees the decrypted data that the network sensor cannot.

---

SMB — Lateral Movement on the Wire

SMB (Server Message Block) is the Windows file sharing protocol. It's used for file access, printer sharing, remote administration, and Group Policy delivery. Because SMB is fundamental to Windows networks, it's always present — which makes it perfect cover for lateral movement.

Normal vs Suspicious SMB Traffic

Attack Patterns Over SMB

PsExec-style execution: Creates a service on the target via the ADMIN$ or C$ share, uploads an executable, starts the service remotely. Suricata sees: SMB TREE_CONNECT to ADMIN$ or C$, followed by CREATE (file upload), followed by IOCTL (service start).

Pass-the-Hash / Pass-the-Ticket: Attacker uses stolen credentials to authenticate to SMB without knowing the plaintext password. The SMB session looks legitimate from a protocol perspective — detection relies on behavioral analysis (unusual source, unusual time, unusual share access pattern).

Lateral file copy: Attacker copies tools or exfiltrates data through SMB file shares. Large file transfers between workstations that don't normally communicate are a strong indicator.

---

Reading Suricata Protocol Logs

Suricata's eve.json output contains far more than just alerts. It logs protocol-level metadata for every connection it processes, creating a rich dataset for investigation.

Event Types Beyond Alerts

How Protocol Logs Support Alert Investigation

When a Suricata alert fires, the alert record alone tells you what rule matched. The protocol logs tell you everything else about that connection:

1. Alert: "ET TROJAN Possible Metasploit Payload" on 10.0.1.50 → 203.0.113.66
2. TLS log: SNI = update.legit-looking.com, JA3 = 5d65ea3fb1d4aa7d826733d2f2cbf7df (known Metasploit hash)
3. DNS log: update.legit-looking.com resolved to 203.0.113.66, queried 30 seconds before the connection
4. Flow log: Connection duration 3600 seconds (1 hour), 50KB transferred — consistent with a C2 beacon

Each protocol log adds a layer of evidence. Together, they build a complete picture of the activity surrounding the alert.

---

What's Next

You've learned how Suricata monitors traffic (Lesson 3.1), how rules detect specific threats (Lesson 3.2), and now how to analyze the protocols themselves to understand what's really happening on the wire. But there's one protocol we haven't covered in depth — the one that nearly every attack touches at some point.

In Lesson 3.4 — DNS as a Weapon, you'll learn why DNS is a goldmine for both attackers and defenders. DNS tunneling, Domain Generation Algorithms, suspicious TLDs, and the visibility challenge posed by DNS over HTTPS. If network traffic is the circulatory system, DNS is the nervous system — and reading it reveals the attacker's intent.