EndisProjects
Description
Designed and deployed a centralized security monitoring and event correlation solution by integrating AWS CloudTrail with Splunk using Amazon S3 and SQS. The system enables real-time ingestion, parsing, and visualization of API activity and user behavior across an AWS account. The pipeline is fully automated and security-hardened with IAM best practices and TLS encryption.
Importance
This project enhances AWS security posture and incident response by making AWS API activity fully observable in Splunk. It enables proactive threat detection, policy compliance audits, and forensic investigations.
Objectives
- Centralize AWS CloudTrail logs into Splunk in near real-time.
- Automate event forwarding using SQS and serverless compute.
- Secure data-in-transit with TLS/SSL encryption using Let’s Encrypt.
- Monitor IAM policy changes and suspicious activity.
- Enable dashboard-based analysis of cloud activity.
Tech Stack
AWS: CloudTrail, S3, SQS, IAM, EC2
Splunk: Splunk Enterprise, Splunk Add-on for AWS
Security: Let’s Encrypt, TLS 1.2+, IAM least privilege,
OS: Amazon Linux 2
CLI & Tools: AWS CLI, OpenSSL, Certbot (Let’s Encrypt)
Architecture Overview
CloudTrail delivers logs to an encrypted S3 bucket.
S3 triggers an SQS notification on new object arrival.
Splunk's AWS add-on polls the SQS queue.
Events are downloaded and parsed by Splunk via scripted input.
IAM Role (SplunkEC2S3AccessRole) assumed by EC2 Splunk instance with fine-grained permissions.
SSL/TLS encryption for S3 and Splunk Web secured using Let's Encrypt certificates.
Implantation
1. Enabled CloudTrail across all regions with logs delivered to a versioned S3 bucket.
2. Created an SQS queue and configured the S3 bucket to send event notifications.
3. Created and attached IAM role SplunkEC2S3AccessRole with access to CloudTrail S3 and SQS.
4. Deployed Splunk Enterprise on EC2 and configured input using the Splunk AWS Add-on.
5. Secured S3 bucket and Splunk Web UI using Let’s Encrypt and Certbot for SSL certificate provisioning.
6. Set up and tested dashboards for IAM changes, S3 access patterns, and API anomalies.
7. Created and triggered test events to verify pipeline behavior and dashboard ingestion.
Challenges and Resolutions
Access Denied Errors: Initial IAM role lacked proper S3:PutObject and SQS:ListQueue permissions. Resolved by iteratively refining IAM policies with least-privilege principle.
TLS Certificate Issues: Let’s Encrypt integration required configuration of firewall rules and domain name bindings. Resolved using Certbot on EC2 and proper DNS entries for validation.
Splunk Event Delay: Configured interval settings and ensured SQS queue visibility timeout aligned with Splunk input polling schedule.
Pictures
