Unlocking Generosity: How Pushpay’s AI-Powered Search Transforms Church Engagement

Navigating Insights: Co-Creating with Pushpay

Revolutionizing Ministry Support with Generative AI

Building a Smart Solution: AI Search Architecture Explained

Overcoming Initial Hurdles in AI Development

Enhancing Evaluation with a Custom AI Framework

Strategic Insights: Tackling Performance Challenges

Business Impact: Transforming User Experience and Development Velocity

Key Takeaways for Your AI Journey

Conclusion: The Road to Effective AI Agents

Meet the Authors: Experts Behind the Innovation

Empowering Ministries with AI: Pushpay’s Journey to Innovative Community Insights

This post was co-written with Saurabh Gupta and Todd Colby from Pushpay.

Pushpay is a market-leading digital giving and engagement platform designed to help churches and faith-based organizations enhance community engagement, manage donations, and improve generosity fundraising processes. Its comprehensive church management system equips church administrators and ministry leaders with insightful reporting, donor development dashboards, and automation of financial workflows to streamline operations.

Unleashing Generative AI for Ministry Needs

Recognizing the unique challenges ministry leaders face, Pushpay harnessed the power of generative AI to develop an innovative agentic AI search feature tailored for ministries. By leveraging natural language processing, this feature allows ministry staff to pose questions in plain English, enabling real-time, actionable insights derived from their community data.

For instance, a ministry leader can run queries such as “Show me members in a group who haven’t given this year” or “Identify individuals not engaged with my church.” This immediate access to critical information empowers leaders to make informed decisions that enhance support for their community. This tool is particularly valuable for community leaders who often have limited time and technical experience; they can now access meaningful data in seconds.

Accelerating Insights: The Architectural Overview

The AI search feature is constructed around several core components that enhance the user search experience:

1. User Interface Layer: Users submit natural language queries through the existing Pushpay application interface, ensuring accessibility without the need for technical training.

2. AI Search Agent: Central to the system is the AI search agent, comprising:

   • System Prompt: A set of role definitions, instructions, and application descriptions.
   • Dynamic Prompt Constructor (DPC): This component constructs customized system prompts based on user-specific information, improving both accuracy and user experience.

3. Amazon Bedrock Features: Pushpay employs advanced Amazon Bedrock services for:

   • Prompt Caching: Reducing latency and costs.
   • LLM Processing: Utilizing Claude Sonnet 4.5 to process prompts and deliver insights seamlessly.

4. Evaluation System: A closed-loop improvement mechanism captures user interactions and evaluates them offline, feeding insights into dashboards for ongoing enhancements.

Overcoming Initial Challenges

The initial iteration of the AI search feature faced hurdles, achieving only a 60-70% success rate with basic queries. Despite the implementation of prompt caching to reduce costs and latency, reaching further accuracy proved difficult due to the diverse range of user queries and extensive configurability.

Implementing a Generative AI Evaluation Framework

To enhance the agent’s accuracy, Pushpay introduced a custom generative AI evaluation framework, including:

• Golden Dataset: A curated repository of over 300 representative queries paired with expected outputs to enhance automated evaluations.
• Evaluator: This component compares agent-generated outputs against the golden dataset, producing core accuracy metrics and detailed performance data.
• Domain Category: Categorizing user queries allows nuanced performance analysis.
• Generative AI Evaluation Dashboard: Visualizes performance metrics for informed decision-making.

Business Impact of the AI Search Feature

The implementation of the evaluation framework yielded significant business benefits:

1. User Experience: The AI search feature reduced insight generation time from 120 seconds to under 4 seconds, democratizing access to valuable data insights.

2. Development Velocity: The scientific approach transformed optimization cycles, allowing rapid validation and enhancement based on genuine user feedback.

3. Production Readiness: Accuracy improved from 60-70% to over 95%, instilling confidence in the feature’s deployment among clients.

Key Takeaways for Your AI Journey

1. Plan for Production Early: Adopt a scaling mindset from the outset to identify and address accuracy issues before they impede progress.

2. Leverage Amazon Bedrock’s Features: Utilize prompt caching to optimize performance and cost effectively.

3. Evaluate Beyond Aggregate Metrics: Domain-level analysis uncovers hidden performance insights, enabling targeted optimizations.

4. Maintain Data Security: Prioritize information protection and ethical AI integration from the beginning, ensuring sensitive data is safeguarded.

Conclusion: Your Path to Effective AI Agents

Pushpay’s evolution from a basic accuracy prototype to a highly effective AI agent highlights the importance of a scientific, data-driven approach to evaluation and optimization. By incorporating a robust evaluation framework, Pushpay ensures rapid iteration, informed decisions, and sustained improvements across diverse domains.

Ready to embark on your own AI journey? Explore Amazon Bedrock and leverage learnings from Pushpay to create your production-ready AI solutions.