- Streaming CloudWatch Logs to an S3 bucket using Amazon Kinesis Data Firehose for real-time log archival, analysis, and long-term storage.
Architecture flow would be
CloudWatch logs (subscription filter) — Kinesis Firehose — S3
- Step1: I have created an S3 bucket to store the logs
Bash
aws s3 mb s3://cloudwatch-logs-bucket –region us-east-1
- Step2: I have created IAM roles for CloudWatch logs and Kinesis firehose with required permissions
# Trust policy for CloudWatch logs:
{
“Version”: “2012-10-17”,
“Statement”: [
{
“Effect”: “Allow”,
“Principal”: {
“Service”: “logs.amazonaws.com”
},
“Action”: “sts:AssumeRole”
}
]
}
# IAM policy for Firehose access:
{
“Version”: “2012-10-17”,
“Statement”: [
{
“Effect”: “Allow”,
“Action”: [
“firehose:PutRecord”,
“firehose:PutRecordBatch”
],
“Resource”: [
“arn:aws:firehose:us-east-1:ACCOUNT-ID:deliverystream/cloudwatch-logs-to-s3”
]
}
]
}
# Create the role
aws iam create-role \
–role-name CloudWatchLogsToFirehoseRole \
–assume-role-policy-document file://cloudwatch-trust-policy.json
# Attach the policy
aws iam put-role-policy \
–role-name CloudWatchLogsToFirehoseRole \
–policy-name CloudWatchToFirehosePolicy \
–policy-document file://firehose-policy.json
# Trust policy for Kinesis firehose
{
“Version”: “2012-10-17”,
“Statement”: [
{
“Effect”: “Allow”,
“Principal”: {
“Service”: “firehose.amazonaws.com”
},
“Action”: “sts:AssumeRole”
}
]
}
# IAM policy for S3 access
{
“Version”: “2012-10-17”,
“Statement”: [
{
“Effect”: “Allow”,
“Action”: [
“s3:AbortMultipartUpload”,
“s3:GetBucketLocation”,
“s3:GetObject”,
“s3:ListBucket”,
“s3:ListBucketMultipartUploads”,
“s3:PutObject”
],
“Resource”: [
“arn:aws:s3:::my-cloudwatch-logs-bucket”,
“arn:aws:s3:::my-cloudwatch-logs-bucket/*”
]
},
{
“Effect”: “Allow”,
“Action”: [
“logs:PutLogEvents”
],
“Resource”: [
“arn:aws:logs:us-east-1:ACCOUNT-ID:log-group:/aws/kinesisfirehose/*:log-stream:*”
]
}
]
}
# Create the role
aws iam create-role \
–role-name FirehoseToS3Role \
–assume-role-policy-document file://trust-policy.json
# Attach the policy
aws iam put-role-policy \
–role-name FirehoseToS3Role \
–policy-name FirehoseS3Policy \
–policy-document file://s3-policy.json
- Step3: Created Kinesis firehose delivery stream by passing required values
Source: Direct PUT
Destination: Amazon S3
Delivery strem name: cloudwatch-logs-to-s3
Destination settings:
S3 bucket: cloudwatch-logs-bucket
Select IAM role (firehosetoS3Role)
- Step4: Created cloudwatch logs subscription filter using console
Navigate to cloudwatch and select the log group and click on Actions à create subscription filter à create amazon data subscription filter
In configure section:
Choose destination as Firehose delivery stream
Grant permission (cloudwatchLogsToFirehoseRole)
Configure log format and filters (csv)
Gave subscription name as cloudwatch-to-firehose-filter
Subscription pattern: leave empty to capture all logs or use a pattern
Then click start streaming
Leave a Reply