In last week’s post on Splunking CloudWatch Metric Streams we walked you through setting up CloudWatch Metric Streams and ingesting them into Splunk via Kinesis Data Firehose. The scability and performance benefits mean that if you are using the Splunk Add-on for Amazon Web Services to poll your CloudWatch metrics then you should look to switch over to this new approach. You can read that post here.

The Problem

The issue we left you with last time is that the CloudWatch data, once it arrives in Splunk, is well formed but does not match the JSON format supplied when using the Splunk Add-on for Amazon Web Services. If you want to simply swap from the old polling method of gathering your CloudWatch metrics to the new streaming method then your existing dashboards and searches might need a lot of re-work. Also, if you prefer to use Splunk metrics for your CloudWatch metrics, we left you hanging.

streamed cloudwatch metric event format
CloudWatch Metric Event without transformation

The Solution

The Splunk github repository is a busy place with many repositories actively being developed. You can take a look here: Splunk Github.

splunk github screenshot
Splunk Github: github.com/splunk

Within Splunk’s public github site you will find a repo called “splunk-aws-cloudwatch-streaming-metrics-processor”. This is where we can find a lambda function that will provide the necesary transformation to the JSON CloudWatch events, either into the event format or metrics format.

splunk github cloudwatch metric streaming screenshot
Splunk github repo – CloudWatch Streaming Metrics Processor

Steps To Implement

Essentially the steps are as per part 1 but instead of skipping the option to use a lambda function to transform the events we choose the aforementioned function. Before you can do this you must setup the lambda function in your AWS account. Note also the section below on setting the correct HEC endpoint type.

Add the Lamda function

You need to setup the lambda function before you can use it in your firehose:

  • From the AWS Console, search for Lambda
  • Click on “Create Function”
  • As per the screenshot below, select the “Browse serverless app repository” option
  • Search for the function using it’s name: splunk-aws-cloudwatch-streaming
  • Once selected you need to configure two settings:
    • MetricsOutputFormat = json (otel is not transformed)
    • SplunkCloudWatchSourcetype =
      • aws:cloudwatch (for the event based format)
      • aws:cloudwatch:metric (for the splunk metrics format)
AWS lamba function setup screenshot
Import the lambda function from the serverless app repo

Splunk HEC Endpoint Type

In your firehose configuration, make sure you use the “Event endpoint” now, the lambda function will format the data properly for HEC:

Kinesis Data Firehose Setup

Setup is the same as shown in part 1 but this time we choose to use the Transform Records option.

  • Hit the browse option and select the lambda function you setup in the previous section
  • or, paste the ARN in directly
AWS firehose transform records screenshot
Setup Kinesis Firehose to use the lambda function and transform the incoming events

The Result

Using the lambda function supplied by Splunk the cloudwatch events now come through in the aws:cloudwatch format. Job done!

correctly formatted splunk cloudwatch event screenshot

Further Reading

Read the announcement post on splunk.com – Stream Your AWS Services Metrics To Splunk

Posted by:Stuart Robertson

Stuart Robertson is the Consulting Director at iDelta. He is one of the initial founders of iDelta and has worked there since formation in 2001. Stuart holds various certifications in Core Splunk and ITSI. Stuart also holds a Bsc(Hons) in Computing Science from the University of Glasgow.