In last week’s post on Splunking CloudWatch Metric Streams we walked you through setting up CloudWatch Metric Streams and ingesting them into Splunk via Kinesis Data Firehose. The scability and performance benefits mean that if you are using the Splunk Add-on for Amazon Web Services to poll your CloudWatch metrics then you should look to switch over to this new approach. You can read that post here.
The Problem
The issue we left you with last time is that the CloudWatch data, once it arrives in Splunk, is well formed but does not match the JSON format supplied when using the Splunk Add-on for Amazon Web Services. If you want to simply swap from the old polling method of gathering your CloudWatch metrics to the new streaming method then your existing dashboards and searches might need a lot of re-work. Also, if you prefer to use Splunk metrics for your CloudWatch metrics, we left you hanging.
The Solution
The Splunk github repository is a busy place with many repositories actively being developed. You can take a look here: Splunk Github.
Within Splunk’s public github site you will find a repo called “splunk-aws-cloudwatch-streaming-metrics-processor”. This is where we can find a lambda function that will provide the necesary transformation to the JSON CloudWatch events, either into the event format or metrics format.
Steps To Implement
Essentially the steps are as per part 1 but instead of skipping the option to use a lambda function to transform the events we choose the aforementioned function. Before you can do this you must setup the lambda function in your AWS account. Note also the section below on setting the correct HEC endpoint type.
Add the Lamda function
You need to setup the lambda function before you can use it in your firehose:
- From the AWS Console, search for Lambda
- Click on “Create Function”
- As per the screenshot below, select the “Browse serverless app repository” option
- Search for the function using it’s name: splunk-aws-cloudwatch-streaming
- Once selected you need to configure two settings:
- MetricsOutputFormat = json (otel is not transformed)
- SplunkCloudWatchSourcetype =
- aws:cloudwatch (for the event based format)
- aws:cloudwatch:metric (for the splunk metrics format)
Splunk HEC Endpoint Type
In your firehose configuration, make sure you use the “Event endpoint” now, the lambda function will format the data properly for HEC:
Kinesis Data Firehose Setup
Setup is the same as shown in part 1 but this time we choose to use the Transform Records option.
- Hit the browse option and select the lambda function you setup in the previous section
- or, paste the ARN in directly
The Result
Using the lambda function supplied by Splunk the cloudwatch events now come through in the aws:cloudwatch format. Job done!
Further Reading
Read the announcement post on splunk.com – Stream Your AWS Services Metrics To Splunk