Introducing an Open Source Chatbot: Leveraging Automated Reasoning for Enhanced Accuracy and Transparency

Improve Accuracy and Transparency with Automated Reasoning Checks

Chatbot Reference Implementation

How the Iterative Rewriting Loop Works

Initial Response and Validation

The Rewriting Loop: The Heart of the System

Getting Started with the Automated Reasoning Checks Rewriting Chatbot

Backend Implementation Details

About the Authors

Enhancing Chatbot Functionality with Automated Reasoning: An Open Source Sample Chatbot

Today, we are thrilled to announce the release of a new open-source sample chatbot designed to illustrate how to leverage Automated Reasoning checks. This innovative system iterates on generated content, asks clarifying questions, and rigorously proves the correctness of its answers.

Why Automated Reasoning?

In a landscape increasingly dominated by large language models (LLMs), maintaining accuracy and transparency is paramount. While LLMs can generate responses that seem plausible, they can sometimes produce factual inaccuracies—a phenomenon known as "hallucination." Here’s where Automated Reasoning checks come in. These checks validate user questions and LLM-generated answers by offering rewriting feedback that highlights ambiguities, broad assertions, and incorrect claims.

The chatbot, utilizing these checks, enhances accuracy by providing explicit answers to yes/no questions and improves transparency through mathematically verifiable proof of statement validity. As a result, our generative AI applications become auditable and compliant, even in highly regulated environments.

The Chatbot Reference Implementation

The sample chatbot is created using Flask, offering APIs to submit questions and check answer statuses. Moreover, its design allows developers to peek behind the curtain and understand how it arrives at its responses—thanks to detailed logs of every iteration and feedback from Automated Reasoning checks.

Frontend Configuration

A NodeJS front end links to an LLM sourced from Amazon Bedrock, allowing users to generate responses, choose an Automated Reasoning policy for validation, and set a maximum iteration limit. The interface displays a debug panel that includes a step-by-step overview of each content iteration and validation output.

Once Automated Reasoning checks deem a response valid, a verifiable explanation of the validity is presented.

How the Iterative Rewriting Loop Functions

The core of this chatbot is its iterative rewriting loop, which continuously refines answers based on Automated Reasoning feedback. Each time the chatbot processes a question and answer (Q&A), the checks return a list of findings. These findings analyze statements in the input Q&A, identifying premises and claims to validate and inform the rewriting process.

For instance, if the question is regarding Amazon S3 storage costs, the validation might return findings that confirm the costs for different regions—allowing the chatbot to clarify and reiterate the answer accurately.

The Backend Architecture

If you’re keen on adapting this implementation for production use, the primary components in the backend include:

1. ThreadManager: This component manages conversation lifecycles, ensuring state consistency throughout the rewrite process. It uses a locking mechanism to prevent race conditions.

2. ThreadProcessor: Employing a state machine pattern, it governs the rewriting loop, transitioning between phases like generating the initial response, validating it, and handling results.

3. ValidationService: Integrating with Amazon Bedrock Guardrails, it validates LLM responses by submitting them to the ApplyGuardrail API and parses structured findings.

4. LLMResponseParser: This interprets LLM intentions during the rewriting process, helping to extract structured information while handling markdown formatting and limiting questions.

5. AuditLogger: It maintains a dedicated log of events, preserving all necessary context as the chatbot processes user inquiries.

Getting Started

Ready to dive in? Follow these steps to set up your own Automated Reasoning checks rewriting chatbot:

1. Create Your Automated Reasoning Policy:

   • Navigate to Amazon Bedrock in your AWS Management Console.
   • Open the Automated Reasoning page and create a sample policy.

2. Download and Run the Reference Implementation:

   • Clone the Amazon Bedrock Samples repository.
   • Follow the README instructions to install dependencies and start the application.
   • Access it via your browser at http://localhost:8080.

Conclusion

Embedding Automated Reasoning checks in chatbot implementations allows for a new standard of accuracy and transparency in AI communication. With the complexities of today’s regulatory landscapes, building trustworthy AI applications is not just beneficial—it’s essential.

You’re now equipped with the knowledge to implement and customize this innovative chatbot.

About the Authors

Stefano Buliani
Stefano is a Product Manager in the Automated Reasoning team at AWS. With over a decade at AWS, he has been pivotal in developing serverless technologies and open-source projects that have empowered many customers to deploy successful applications.

Dive into the code, explore the capabilities, and let’s reshape the future of chatbot technology together!