Custom Metrics and Alerting with CloudWatch

In the first blogpost of this two part series, we talked about setting up our Elastic Beanstalk application with CloudFormation. Additionally we configured our application to send logs to CloudWatch Logs.

Elastic Beanstalk exposes many different metrics through CloudWatch, but in addition to ELB or usage metrics, we also want to be able to add our own custom metrics. A great way to do this is through your log statement and configuring CloudWatch to parse your log and turn it into metrics.

There are many use cases for custom metrics. At Codeship, for example, an important metric is all of the incoming webhooks that trigger builds. We want to know as early as possible if there is an issue either in our system or with a partner that results in builds not coming in correctly.

Writing a log message every time a hook comes in and then turning those log messages into a metric allows us to get an alarm whenever the number of hooks that are coming in gets too low.

CloudWatch even allows you to turn numbers in your log line into graphs, so they actually parse your log and don’t just count when a log statement occurs. You can find more about their very flexible system in their examples, specifically in their Bytes Transferred Example

In this post, we’re going to set up a MetricFilter that turns “Hello World” from our logs into a metric. We’ll then set up an alert that sends an email through SNS whenever the Hello World log messages happens more than five times in a minute.

The example builds on top of the flomotlik/cloudformation-elastic-beanstalk-blog-demo repository mentioned in the last blogpost.

To begin, we need to filter the Hello World log message that's written to the log by the Sinatra application whenever we open up the website. Following is a snippet from the CloudFormation template that transforms “Hello World” into a metric value of 1. If it gets called several times, it will be summed up accordingly.

The metric gets a name and namespace to be easily identified. You can take a look at the metric filter documentation for all the details on a metric filter.

Logs_MetricFilter('HelloWordMetric') do
    DependsOn %w(ElasticBeanstalkMainLogGroup)
    LogGroupName Ref('ElasticBeanstalkMainLogGroup')
    FilterPattern 'Hello World'
    MetricTransformations [
      {
        "MetricValue": 1,
        "MetricName": 'HelloWorldOccurences',
        "MetricNamespace":
          FnJoin('/', ['ElasticBeanstalk', Ref('DemoEnvironment')])
      }
    ]
  end

This will result in the following metric:

Now that we have all the data we need available, we can set up an alarm that will send us an email whenever we have too many Hello World log messages.

At first, we set up an SNS topic that lets us send an email. We hardcode the email in this example, but you could also set it up through a parameter.

SNS_Topic('AlarmNotifications') do
    DisplayName 'Email Alarm Notification'
    Subscription [
      {
        "Endpoint": 'dev@exampleteam.com',
        "Protocol": 'email'
      }
    ]
  end

Of course with SNS, you can set up many different ways to deliver this notification. You can check the Pagerduty documentation or Victor Ops documentation for examples on how to integrate SNS with either of those services.

Or if you want to create your own logic to fix issues, you could trigger Lambda functions that will react on a specific problem in your infrastructure. Take a look at the Lambda through SNS documentation to get an idea about how to set this up.

Now that we have the SNS topic, we can create the alarm.

CloudWatch_Alarm('HelloWorld') do
    DependsOn %w(HelloWordMetric AlarmNotifications)
    AlarmDescription 'Alert if HelloWorld is sent'
    MetricName 'HelloWorldOccurences'
    Namespace FnJoin('/', ['ElasticBeanstalk', Ref('DemoEnvironment')])
    Statistic 'Sum'
    Period '60'
    EvaluationPeriods '1'
    Threshold '5'
    ComparisonOperator 'GreaterThanThreshold'
    AlarmActions [Ref('AlarmNotifications')]
  end

It uses the same Namespace and Metric Name we’ve set before. Whenever the sum of all Hello World log statements is over five in one minute, the alarm will be triggered and, as an Action, sends the message to the SNS topic. Here's a sample of the email you will receive once this alarm happens:

To get all the details of the alarm, take a look at the CloudFormation CloudWatch Alarm documentation.

CloudFormation, Elastic Beanstalk, and CloudWatch are an incredibly powerful combination. You can build complex infrastructure, completely automate it, and make sure it handles outages and issues automatically while notifying you easily.

Let us know in the comments what other tools and services you’re using to manage your AWS infrastructure.