Unlocking Insights: Integrating Splunk with Amazon Bedrock for Advanced Data Analysis

By:

Alan Peaty, Senior Partner Solutions Architect – AWS
Greg Ainslie-Malik, Field CTO – Splunk
Varun Rajan, Senior Solutions Architect – AWS

Introduction: The Growing Challenge of Data Volume

Revolutionizing Data Insights with Large Language Models

Seamless Integration: Splunk Machine Learning Toolkit and Amazon Bedrock

Real-World Applications: Enhancing Security Investigations

Getting Started: Implementation Guidance

Cost Management Considerations

Security Best Practices

Conclusion: Driving Operational Excellence with AI Integration

Splunk – AWS Partner Spotlight

Unlocking the Power of Data with Splunk and Amazon Bedrock

By: Alan Peaty, Senior Partner Solutions Architect – AWS
By: Greg Ainslie-Malik, Field CTO – Splunk
By: Varun Rajan, Senior Solutions Architect – AWS

The Growing Need for Data Intelligence

In a world where global data volumes are projected to soar to 181 zettabytes by 2025—nearly tripling the amount created in 2020—the challenge of managing machine-generated data has never been greater. Organizations face a flood of security logs and system metrics, all of which must be sifted through to identify potential risks and drive actionable insights.

Enter Splunk, a partner of Amazon Web Services (AWS), which provides a robust data ecosystem for searching, monitoring, and analyzing machine data at scale. But even with this powerful tool, organizations still require resources for contextual analysis and skills for multi-source correlation. The global cybersecurity talent shortage, which exceeds 4 million professionals according to the World Economic Forum, further complicates the landscape, leaving organizations vulnerable to emerging threats.

Leveraging Large Language Models (LLMs)

One of the most promising solutions to this challenge is the use of Large Language Models (LLMs). These models excel at transforming raw, unstructured data into actionable insights. With Amazon Bedrock, AWS users can easily access high-performing foundation models (FMs) via a unified API, making it simpler to build and scale generative AI applications.

The integration of Splunk’s Machine Learning Toolkit (MLTK) with Amazon Bedrock can empower organizations to gain context from their data without requiring extensive expertise in AI.

Integrate Splunk MLTK with Amazon Bedrock

The new synergy between Splunk MLTK and Amazon Bedrock brings FMs into Splunk workflows, allowing security analysts to augment their traditional searches with natural language processing capabilities. This integration facilitates:

• Automatic Extraction of Contextual Insights: Quickly pull actionable information from raw events.
• Correlation of Diverse Data Sources: Analyze data from various origins to discover comprehensive insights.
• Simplified Communication: Translate complex technical findings into easy-to-understand next steps for stakeholders.

Splunk users can now utilize the latest version of MLTK (5.6.0) to directly integrate FMs from Amazon Bedrock into their workflows.

Key Benefits of Integration:

• Transform raw machine data into natural language insights.
• Accelerate security investigations by contextualizing alerts and logs automatically.
• Generate narrative explanations of technical events for non-technical stakeholders.
• Enhance SPL queries through advanced natural language processing capabilities.

Solution Overview

The integration reliably exchanges prompts and responses through secure methodologies between Splunk MLTK and Amazon Bedrock. Initial authentication is facilitated using AWS Identity and Access Management (IAM), ensuring secure access to foundation models.

Key Steps in the Integration Process:

1. IAM Role Assumption: Splunk MLTK uses IAM credentials to request a temporary role through AWS Security Token Service (STS).
2. Temporary Credentials: STS validates the request, issuing temporary credentials allowing specific permissions.
3. Model Invocation: Using these credentials, Splunk MLTK sends prompts to the selected foundation model in Bedrock.
4. Response Integration: The model’s generated response is incorporated back into Splunk’s search results.

Real-World Application: Investigating HTTP 400 Errors

Consider a situation where security analysts must investigate HTTP 400 errors to identify malicious activity. Typically a tedious process, integrating Amazon Bedrock into Splunk workflows significantly expedites this task.

For instance, after filtering the HTTP 400 error logs, analysts can extend their query with the new ai command, prompting the model to analyze the log entries. The model can identify key findings along with associated risk metrics, offering recommendations that would typically require extensive expertise to unearth.

Getting Started

For those looking to implement this powerful integration, our GitHub repository offers a comprehensive guide, including an AWS CloudFormation template for automated deployment, step-by-step instructions, and troubleshooting assistance.

Once the setup is complete, analysts can start leveraging the new ai command in their SPL queries, enabling the use of natural language processing capabilities directly in their data analysis workflows.

Cost Management Considerations

As with any evolving technology, cost management is important. Using the ai command will incur usage costs per API call. Organizations can implement best practices to ensure efficiency, such as:

• Testing SPL queries before executing them.
• Using commands like stats, dedup, or head to minimize result sets.
• Scheduling queries for controlled usage.

Security Considerations

Security is paramount, especially when handling sensitive data. The integration adheres to several security best practices:

• IAM Least-Privilege Permissions: Tailored to give minimal access required.
• Temporary Credentials: Facilitated via AWS STS for secure interactions.
• Encrypted Communications: Utilizing TLS 1.2+ for all data transfers.

Before deploying, organizations should evaluate their compliance needs and ensure that the integration aligns with their existing data governance frameworks.

Conclusion

By merging Splunk’s advanced data processing capabilities with Amazon Bedrock’s foundation models, organizations are now presented with a powerful way to unlock deeper insights from machine data—without the need for extensive expertise in AI.

This integration not only enhances security investigations but also enables operational teams to improve root cause analyses and businesses to translate technical findings into actionable insights—all using natural language within existing Splunk workflows.

Ready to Dive In?

Visit our GitHub repository for detailed deployment instructions and to explore how to effectively leverage this powerful integration.

Splunk – AWS Partner Spotlight

Splunk is proud to be an AWS Specialization Partner with expertise in Cloud Operations, Data and Analytics, DevOps, and much more. Leading organizations rely on Splunk’s unified security and observability platform to safeguard their digital infrastructure.

For more information, contact Splunk or explore the AWS Marketplace.