Splunk-Driven Threat Feed Correlation Engine

Description

A real-time threat intelligence correlation engine that ingests threat feeds from sources like AlienVault OTX and AbuseIPDB, correlates them with internal VPC Flow Logs using Splunk, and generates automated alerts and dashboards to visualize and respond to suspicious activity.

Importance

This project solves the critical security operations problem of siloed external threat data and scattered internal telemetry. It demonstrates the ability to integrate threat intelligence with network visibility and enables timely incident response, aligning well with Security Operations Analyst and IT Automation Engineer roles.

Objectives

Automate ingestion of external threat intelligence from OTX

Correlate malicious IPs with internal VPC Flow Logs

Trigger real-time alerts for suspicious connections

Visualize threat patterns by IP, time, and action type

Tech Stack

AWS: VPC Flow Logs, CloudWatch Logs, IAM

Splunk: HEC endpoint, dashboards, correlation searches, alerts

Threat Intelligence APIs: AlienVault OTX

Architecture Overview

Python scripts periodically pull malicious IPs from OTX and AbuseIPDB

Parsed JSON is sent to Splunk via HEC into the threat_intel index

AWS VPC Flow Logs are streamed into CloudWatch, then into the vpc_flow_logs index in Splunk

Correlation SPL logic matches flow log srcaddr and dstaddr fields against the threat IPs

Alerts trigger when a match is found, and dashboards visualize threat activity

Implantation

1. Selected AlienVault OTX as primary external threat intelligence sources.

2. Created dedicated Splunk indexes (threat_intel for external feeds and vpc_flow_logs for internal traffic).

3. Configured AWS VPC Flow Logs to send data to CloudWatch Logs using a custom IAM role with VPCFlowLogsToCWLogs permissions.

4. Added a CloudWatch Logs input in Splunk to pull VPC Flow Logs and verified ingestion with SPL searches (index=vpc_flow_logs).

5. Developed Python scripts to query OTX APIs, normalize the indicators, and post them to Splunk’s HTTP Event Collector (HEC).

6. Validated external threat indicators in Splunk (index=threat_intel) and confirmed proper field extraction and format (indicator, threat_type, etc.).

7. Wrote correlation SPL to match threat indicators against internal traffic using dynamic subsearch and format to avoid literal IN errors.

8. Created test traffic using curl and ping to known threat IPs to simulate alerts and confirm correlation accuracy.

9. Built interactive Splunk dashboards showing recent threat matches, top indicators, traffic sources, and alert timelines.

Challenges and Resolutions

Challenge: Inconsistent threat intel formats

Resolution: Normalized to single indicator field in Python

Challenge: No results for early joins

Resolution: Validated IP format compatibility and log ingestion using field extraction verification

Challenge: Splunk CloudWatch Logs input missing

Resolution: Manually added CloudWatch log group in Splunk UI with correct IAM role and region