Send EC2 logs to Cloudwatch

Amazon cloudwatch is one of the most commonly used service for collecting metrics, monitoring logs & creating alerts. Although the name suggests, it has nothing to do with weather forecasting. ūüėČ Anyways, so this post covers step by step instructions to configure cloudwatch group & forward logs from Debian/Linux based EC2 instance.

1. Create IAM policy¬†– First of all, ec2 instance will need cloudwatch write access to put events to a specific cloudwatch group. So create a new policy with the json file as below –

{
    "Version": "2012-10-17",
    "Statement": [
        {
        "Effect": "Allow",
        "Action": [
            "logs:CreateLogGroup",
            "logs:CreateLogStream",
            "logs:PutLogEvents",
            "logs:DescribeLogStreams"
        ],
        "Resource": ["*"]
        }
    ]
}

2. Create IAM role – Create an IAM role & attach policy created in step 1. If you create IAM role from AWS console, it will also create an instance profile with the same name, however if you are using terraform to create role then you will have to create instance profile explicitly based on the role.

3. Create cloudwatch group – ¬†A cloudwatch¬†group will be required in order to collect logs. In a cloudwatch group, you may create different log streams to¬†segregate different types of logs from each other for e.g you may create system_log¬†stream to collect /var/log/syslog & jvm_log¬†stream to collect application logs. We will see how to configure agent to forward these logs to separate streams in a while. Creating log group from console just requires log group name. Terraform script can be used to create group as shown below –

resource "aws_cloudwatch_log_group" "my_ec2_logs" {
    name = "EC2_Logs"
    retention_in_days = 14
}

4. Attaching IAM instance profile РThis is straightforward, you can attach instance profile even if instance is already running. Before configuring, cloudwatch agent on the EC2 instance attach the instance profile created in step 2.

5. Create cloudwatch configuration РCloudwatch agent will look for config file to forward log files to appropriate log group/log stream. You can refer to the sample file below and change accordingly. Lets save this to /etc/cloudwatch.conf

[general]
state_file = /var/awslogs/state/agent-state

[/var/log/syslog]
file = /var/log/syslog
log_group_name = /aws/ec2/my_ec2_logs
log_stream_name = "system_logs"
datetime_format = %b %d %H:%M:%S

[/var/log/jvmlog]
file = /var/log/jvm.log
log_group_name = /aws/ec2/my_ec2_logs
log_stream_name = "jvm_logs"
datetime_format = %b %d %H:%M:%S

This will forward logs to 2 separate log streams under same log group. However, log stream is optional parameter.

6. Install cloudwatch agent – The easiest way to install cloudwatch agent on linux ec2 host is by using aws provided install script. Log in to ec2 instance & run below commands as root user and lets use the configuration file created in the last step.

Make sure you are using appropriate version of python installed on your system. If there isn’t ¬†any python interpreter installed on host then run then first install python –

sudo apt-get update
sudo apt-get install python

Then install the Cloudwatch agent as below –

curl https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O
chmod +x ./awslogs-agent-setup.py
python ./awslogs-agent-setup.py --region <REGION_NAME> -c /etc/cloudwatch.conf

Now go to AWS console, navigate to cloudwatch Service -> Logs & select the log group created in the step 3; by now you will be able to see events from running ec2 instance.

Leave a Reply

Your email address will not be published. Required fields are marked *