Stream AWS EC2 Logs To CloudWatch Using CloudWatch Agent Wizard

If you need one place to store and manage your AWS logs, look no further than Amazon CloudWatch. CloudWatch is a handy feature that helps with event correlation and is critical in maintaining visibility within your technical infrastructure.

Since Elastic Compute Cloud (EC2) Instances commonly run critical workloads, log visibility is vital, and it makes complete sense to integrate EC2 with CloudWatch.

In this blog, you’re going to learn how to set up one component of CloudWatch on your AWS EC2 instances, the CloudWatch Agent. Once configured, the agent will then send selected logs to AWS CloudWatch for further investigation.

Creating an IAM Role

Before CloudWatch will work with an EC2 instance, it must have an IAM role. An IAM role, if properly configured for least privilege, allows CloudWatch to work correctly but without unnecessary permissions.

To create an IAM role that will allow your EC2 Instance to communicate with CloudWatch:

1. Open a web browser.

2. Navigate to the AWS Management Console and sign in to your AWS account by supplying your AWS (root) or IAM account credentials.

3. Click Services on the upper left-hand corner of your screen.

4. Next, click IAM from the Services drop-down menu located under the Security, Identity, & Compliance category. This option will take you to the IAM console.

5. Now, click Roles from the menu on the screen’s left-hand side located under the Access Management category.

6. From the Roles screen, select Create Role, located at the top of your screen. You may see a message at the top of your screen that describes what an IAM role is. If that is the case, the Create Role option is located underneath the message.

7. On the Create Role page, designate the Type of Trusted Entity as AWS Service. AWS Service roles allow AWS services to interact with other resources (e.g. CloudWatch) on your behalf.

8. Select the EC2 option from the list of use cases since the CloudWatch Agent will be installed on an EC2 Instance and will communicate with CloudWatch.

9. From the list of use cases, select the EC2 option and then select Next: Permissions.

10. On the permissions page, enter “CloudWatchAgentServerPolicy” into the search bar and then check the box to the left of the CloudWatchAgentServerPolicy’s Policy Name. After checking the box, select Next: Tags. The CloudWatchAgentServerPolicy contains a set of list, read, and write permissions that allow your EC2 Instance to collect and send metrics and logs to CloudWatch. The policy’s JavaScript Object Notation (JSON) is shown below. For more information on JSON policy elements, reference AWS documentation.

11. Assign your role a unique name and select Create Role. In the following screenshot, this tutorial’s Role name is EC2CloudWatchAgentWizardRole

Attaching an IAM Role while creating new EC2 instance.

1. Assuming you’re still in the AWS Management Console, click Services in the upper left-hand corner of your screen.

2. Now click EC2 under All services. This option will take you to the EC2 console. Select Instances from the menu on the left-hand side of the screen under the Instances category.

3. Within the Instances window, click on launch instance and follow along

4. In Step 3 in IAM role field make sure to select our role “EC2CloudWatchAgentWizardRole” and follow along for next steps

5. On Review instance screen click on launch.

Attaching an IAM Role For existing EC2 instance

1. Assuming you’re still in the AWS Management Console, click Services in the upper left-hand corner of your screen.

2. Now click EC2 under All services. This option will take you to the EC2 console. Select Instances from the menu on the left-hand side of the screen under the Instances category.

3. Select Actions —> Security —> Modify IAM Role.

4. The Modify IAM Role option brings you to a menu that allows you to select and attach the IAM role created earlier. Now, select the role you created earlier “EC2CloudWatchAgentWizardRole” from the drop-down menu and then click Save.

5. Verify the role is attached to your EC2 Instance by viewing the success message on the top of your screen. The success message contains the IAM role name and the EC2 instance’s Instance ID.

Installing the CloudWatch Agent and configuring & Activating the CloudWatch Agent.

Note:- rpm repository path sometimes changes so for cloudwatch installation latest commands click here

1. ssh to Linux server using putty/gitbash. After login run below command to download the rpm package. After running below command you will see below output
   COPY

   COPY

   wget https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm
   

   
2. Install cloudwatch agent now using below command. After running below command you will see below output
   COPY

   COPY

   sudo rpm -U ./amazon-cloudwatch-agent.rpm
   

   

3. After successfull installation run below command and it will list down all files present in bin directory. Here you can see “amazon-cloudwatch-agent-config-wizard” executable file.

   COPY

   COPY

   ls /opt/aws/amazon-cloudwatch-agent/bin/
   

   

4. Run this executable file using below command and the cloudwatch wizard console will launch

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

Step 1:- On which OS are you planning to use the agent? > 1. Linux

Step 2 Are you using EC2 or On-Premises hosts? 1. EC2

Step 3 Which user are you planning to run the agent? 1. root [you can change as per your preference]

Step 4 Do you want to turn on StatsD daemon? 2. No

Step 5 Do you want to monitor metrics from CollectD? 2. No

Step 6 Do you want to monitor any host metrics? 1. Yes

Step 7 Do you want to monitor cpu metrics per core? 2. No

Step 8 Do you want to add ec2 dimensions? 1. Yes

Step 9 Would you like to collect your metrics at high resolution? 4. 60s

Step 10 Which default metrics config do you want? 1. Basic

Step 11 Are you satisfied with the above config? 1. Yes

Step 12 Do you have any existing CloudWatch Log Agent? 2. No

Step 13 Do you want to monitor any log files? 2. Yes

Step 14 Do you want to specify any additional log to monitor? 2. No

Step 15 Do you want to store the config in the SSM parameter store? 2. No

Output of the cloudwatch agent json file will be stored at “/opt/aws/amazon-cloudwatch-agent/bin/config.json” . Code snippet of json file with above selected parameters in cloudwatchagent wizard is as follows.

        "agent": {
                "metrics_collection_interval": 60,
                "run_as_user": "root"
        },
        "logs": {
                "logs_collected": {
                        "files": {
                                "collect_list": [
                                        {
                                                "file_path": "/var/log/boot.log",
                                                "log_group_name": "EC2Logs",
                                                "log_stream_name": "{local_hostname}"
                                        }
                                ]
                        }
                }
        },
        "metrics": {
                "append_dimensions": {
                        "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
                        "ImageId": "${aws:ImageId}",
                        "InstanceId": "${aws:InstanceId}",
                        "InstanceType": "${aws:InstanceType}"
                },
                "metrics_collected": {
                        "disk": {
                                "measurement": [
                                        "used_percent"
                                ],
                                "metrics_collection_interval": 60,
                                "resources": [
                                        "*"
                                ]
                        },
                        "mem": {
                                "measurement": [
                                        "mem_used_percent"
                                ],
                                "metrics_collection_interval": 60
                        }
                }
        }

It will now start streaming EC2 “boot.log” from path “/var/log/boot.log” to cloudwatch along with EC2 metrices that we have enabled. We can add multiple log files which needs to be streamed into this json file.

You can find practical implementation of above steps in below video.