Effective logging is critical for monitoring and troubleshooting applications deployed on AWS. However, with high log volumes, identifying critical issues in CloudWatch can be challenging. Integrating logs with team communication tools like Slack and storing them for audits can streamline incident response and compliance. This article outlines how to build a custom logging system using AWS Lambda to filter logs with specific markers, send real-time Slack notifications, and store logs in DynamoDB. This approach, born from a need to quickly detect and act on critical errors, provides a scalable solution for AWS-based applications.


The Need for Custom Logging

AWS CloudWatch efficiently collects logs, but its broad scope can obscure critical events. Teams often rely on Slack for collaboration, making it an ideal destination for urgent alerts. A custom logging system addresses this by:



This system was developed after a critical database error went unnoticed in CloudWatch, prompting the need for faster detection and team notification.


System Architecture


The architecture leverages AWS serverless services and Slack’s webhook API:


Workflow:

  1. Applications send logs to CloudWatch with markers (e.g., [CRITICAL]).
  2. CloudWatch triggers a Lambda function via a subscription filter.
  3. Lambda filters logs, sends notifications to Slack, and stores data in DynamoDB.


The architecture can be visualised as:

[Application] → [CloudWatch Logs] → [Lambda] → [Slack]
                                     ↓
                                 [DynamoDB]


Implementation Steps

The following steps assume an AWS account, a Slack workspace, and familiarity with AWS services. The Lambda function uses Python, though it can be adapted to other runtimes.


Step 1: Configure CloudWatch Logs

Ensure applications send logs to CloudWatch using:

Create a log group (e.g., /aws/app/myapp) and include markers in logs, such as:

2025-06-29T10:00:00Z [INFO] User login successful
2025-06-29T10:01:00Z [CRITICAL] Database connection failed


Step 2: Create the Lambda Function

Set up a Lambda function in the AWS Console:


Below is the Lambda code:

import json
import base64
import zlib
import boto3
import requests
from datetime import datetime

def lambda_handler(event, context):
    # Decode CloudWatch log data
    log_data = base64.b64decode(event['awslogs']['data'])
    log_json = json.loads(zlib.decompress(log_data, 16+zlib.MAX_WBITS))
    
    # Initialize clients
    slack_webhook = "https://hooks.slack.com/services/xxx/yyy/zzz"  # Use Secrets Manager
    dynamodb = boto3.resource('dynamodb')
    table = dynamodb.Table('AppLogs')
    
    # Process log events
    for event in log_json['logEvents']:
        message = event['message']
        if '[CRITICAL]' in message or '[ALERT]' in message:
            # Send Slack notification
            marker = 'CRITICAL' if '[CRITICAL]' in message else 'ALERT'
            slack_payload = {
                "text": f"{marker} Log: {message}",
                "channel": "#app-alerts",
                "username": "AWS Logger",
                "icon_emoji": ":aws:"
            }
            requests.post(slack_webhook, json=slack_payload)
            
            # Store in DynamoDB
            table.put_item(
                Item={
                    'log_id': str(event['id']),
                    'timestamp': datetime.utcfromtimestamp(event['timestamp']/1000).isoformat(),
                    'marker': marker,
                    'message': message,
                    'app_name': log_json['logStream']
                }
            )
    
    return {
        'statusCode': 200,
        'body': json.dumps('Logs processed')
    }


Details:


Step 3: Set Up Slack Webhook

To enable Slack notifications:

  1. Create a Slack app at api.slack.com/apps and enable incoming webhooks.
  2. Generate a webhook URL for the target channel (e.g., #app-alerts).
  3. Store the URL in AWS Secrets Manager for security.


Test the webhook:

curl -X POST -H 'Content-type: application/json' \
--data '{"text":"Test alert from AWS"}' \
https://hooks.slack.com/services/xxx/yyy/zzz


Step 4: Configure DynamoDB

Create a DynamoDB table named AppLogs with:

Use on-demand capacity for scalability. Alternatively, use Amazon RDS for relational storage.


Step 5: Connect Components

Add a CloudWatch Logs subscription filter:

  1. Navigate to the log group in the CloudWatch console.
  2. Create a filter with a pattern like [CRITICAL] or [ALERT].
  3. Set the Lambda function as the destination.
  4. Test by sending logs with markers and verifying Slack notifications and DynamoDB entries.


Security Considerations

To secure the system:

secrets_client = boto3.client('secretsmanager')
slack_webhook = secrets_client.get_secret_value(SecretId='SlackWebhook')['SecretString']


Enhancements

The system may be extended by:


Conclusion

This custom logging system, built with AWS Lambda, CloudWatch, DynamoDB, and Slack, enhances monitoring by filtering critical logs, notifying teams in real-time, and storing data for audits. It’s a practical, scalable solution for AWS applications.