Cross-Account Integration of Amazon Bedrock Agents with Amazon Redshift: A Practical Solution

The Challenge

Solution Overview

Prerequisites

Assumption

Implementation Walkthrough

Clean Up Resources

Summary

Conclusion

About the Authors

Seamless Data Access for AI Agents: Connecting Amazon Bedrock with Cross-Account Redshift

In today’s data-driven world, organizations require seamless access to their structured data repositories to power intelligent AI agents. However, when these data pools span multiple AWS accounts, integration challenges often arise. In this blog post, we’ll explore a practical solution for connecting Amazon Bedrock agents to knowledge bases in Amazon Redshift clusters residing in different AWS accounts.

The Challenge

Organizations that leverage Amazon Bedrock for AI agent development often store their structured data in Amazon Redshift clusters. When these data repositories exist in separate AWS accounts, they encounter a significant limitation: Amazon Bedrock Knowledge Bases doesn’t natively support cross-account Redshift integration.

Key Constraints

This restriction presents challenges for enterprises with multi-account architectures eager to:

• Leverage existing structured data in Redshift for their AI agents.
• Maintain separation of concerns across different AWS accounts.
• Avoid duplicating data across accounts.
• Ensure proper security and access controls.

Solution Overview

To tackle these limitations, our solution facilitates cross-account knowledge base integration using a secure, serverless architecture, allowing AI agents to query structured data while maintaining robust access controls. The architecture utilizes AWS Lambda as an intermediary for safe cross-account data access.

Action Flow

1. Users submit their natural language questions via Amazon Bedrock Agents configured in the agent account.
2. The agents invoke a Lambda function through action groups to access the knowledge base in the agent-kb account.
3. The Lambda function, running in the agent account, assumes an IAM role created in the agent-kb account to connect to its knowledge base.
4. The Amazon Bedrock Knowledge Base in the agent-kb account employs an IAM role to access Amazon Redshift and query the data warehouse.

Key Components

• Amazon Bedrock agent in the agent account for user interactions.
• Amazon Redshift serverless workgroup in a private subnet of the agent-kb account, housing structured data.
• Amazon Bedrock knowledge base using the Redshift serverless workgroup as its structured data source.
• A Lambda function within the agent account.
• Action group configuration linking the agent to the Lambda function.
• IAM roles and policies to facilitate secure cross-account access.

Prerequisites

Before implementing this solution, ensure you have:

• Two AWS accounts: Create an AWS account if you do not have one.
• Specific permissions in both accounts, which will be set up in subsequent steps.
• AWS CLI (version 2.24.22 or above) installed.
• IAM user credentials configured for AWS CLI access to each account.
• jq tool installed, a lightweight command-line JSON processor.

Model Access

Ensure you enable access to the necessary models in the Amazon Bedrock console for both accounts in the us-west-2 (Oregon) AWS Region.

Assumptions

For clarity, we’ll refer to the AWS account with the Amazon Bedrock agent as agent and the account with the Amazon Bedrock knowledge base as agent-kb. We’ll be using the meta.llama3-1-70b-instruct-v1:0 model for the agent-kb, while the agent account will utilize an Amazon Bedrock agent optimized model such as us.amazon.nova-pro-v1:0.

Implementation Walkthrough

This section provides a step-by-step guide for deploying an end-to-end solution from scratch. If you’re already running some components, you can skip certain steps.

Step 1: Account Setup

Make a note of the following AWS account numbers:

• Agent Account: 111122223333
• Agent-KB Account: 999999999999

Step 2: Create Amazon Redshift Serverless Workgroup

• Log in to the agent-kb account.
• Follow instructions from a workshop link to create the Amazon Redshift Serverless workgroup.
• Document the namespace, workgroup, and other critical details.

Step 3: Set Up Data Warehouse

• In the agent-kb account, establish your data warehouse and create the AI knowledge base. Keep a record of the knowledge base ID for later use.

Step 4: Role and Policy Creation

Run the script create_bedrock_agent_kb_roles_policies.sh, passing in the necessary parameters, including the account IDs and role names.

Step 5: Download CloudFormation Script

Download the cloudformation_bedrock_agent_kb_query_cross_account.yaml script, log into the agent account, and set up resources through the CloudFormation console.

Step 6: Verify Model Access

Confirm that both accounts have access to the respective models. Navigate to the Amazon Bedrock console and check model access configurations.

Step 7: Test the Agent

Run the agent in the agent account and test various natural language queries.

- Who are the top 5 customers in Saudi Arabia?
- What is the total revenue by region for the year 1998?

Best Practices

• Phrase questions specifically.
• Use terminology that aligns with your dataset.
• Verify queries against existing data in the TPCH dataset.

Clean Up Resources

Upon completion, navigate to the CloudFormation console in both accounts to delete the stacks. For removing remaining IAM roles and policies, utilize the script delete-bedrock-agent-kb-roles-policies.sh.

Summary

This solution showcases how an Amazon Bedrock agent in one AWS account can query a knowledge base in another account, effectively overcoming constraints tied to multi-account architectures.

Conclusion

By employing an architecture that utilizes Amazon Bedrock Knowledge Bases in conjunction with Amazon Redshift Serverless, organizations can achieve a more integrated and secure approach to cross-account data access. This setup not only enhances data retrieval processes for AI agents but also upholds security and separation across AWS accounts.

As Amazon Bedrock evolves, this architecture positions you to leverage future enhancements while maintaining your multi-account strategy.

About the Authors

Kunal Ghosh, Arghya Banerjee, Indranil Banerjee, and Vinayak Datar—experts in AWS technologies—provide insights into utilizing generative AI, analytics, and cloud solutions to drive innovation and efficiency across industries.

This guide should serve as a solid foundation for connecting your Amazon Bedrock agents to knowledge bases in different AWS accounts. Happy querying!