Revolutionizing Library Access: The Power of AI-Driven Search at Northwestern University Libraries

Enhancing User Experience Through Generative AI

Selecting AWS: A Flexible, Scalable Solution for Innovation

Crafting a Conversational, Multilingual Search Tool

Early Success and Insights for Other Libraries

The Future of AI in Academic Research: Ensuring Accessibility and Engagement

Libraries Reimagined: How Northwestern University is Transforming the Search Experience with AI

Libraries have long been considered treasure troves of knowledge, but navigating their vast digital collections can often feel like searching for a needle in a haystack. At Northwestern University Libraries (NUL), this challenge led to an innovative solution: a new, multilingual generative AI-powered search tool designed to enhance accessibility, intuitive use, and inclusivity for all users.

The Challenge of Traditional Search

The motivation for this project emerged from the need to create a better search experience for users struggling with traditional search methods on NUL’s Digital Collections site. This site, which serves over 800,000 page views annually, hosts hundreds of thousands of digitized works, spanning cultures and eras. For casual users and multilingual researchers, traditional keyword searches and Boolean logic fell short in efficiently sifting through this treasure trove.

Reimagining Search for a Multimedia World

Recognizing an opportunity with the rise of generative AI, the NUL team decided to overhaul the user search experience. Brendan Quinn, NUL’s lead AI engineer, pointed out that the Digital Collections were prime candidates for generative AI integration due to their high traffic and diverse content.

The collections encompass an astonishing array of materials—campus publications, maps, films, art, and audio recordings, among others. Acknowledging the barriers traditional search methods imposed, the team began exploring innovative solutions.

In early 2023, Quinn developed a prototype that utilized large language models (LLMs) in conjunction with vector databases to enhance semantic search capabilities. This initial pilot employed metadata from the library’s collections to illustrate how retrieval-augmented generation (RAG) could dramatically shift the search paradigm from keyword-aligned retrieval to meaning-based discovery.

Choosing AWS for Scalability and Flexibility

Given NUL’s early adoption of Amazon Web Services (AWS), it was only natural to turn to this platform for scaling the new AI search tool. After testing various generative AI platforms, AWS stood out due to its flexibility, cost transparency, and robust support for rapid iteration.

The early prototype utilized cloud-based services with a basic chatbot interface. As the project progressed, NUL incorporated advanced AWS services such as Amazon Bedrock for experimenting with foundation models and Amazon OpenSearch Service, which enabled semantic search through the use of vector embeddings—the mathematical representations that capture the meanings and relationships within the data.

Designing a Conversational, Multilingual Search Experience

In just a few months, the Northwestern team developed and launched the AI-powered search tool, featuring a user-friendly chat-based interface. Users can input natural language queries in various languages, receiving relevant results accompanied by contextual explanations and links that promote deep exploration of underrepresented topics.

Behind the scenes, the integration of AWS services supports the sophisticated semantic search process. When a user submits a query, Amazon OpenSearch embeds it as a vector, comparing it with stored vectors to yield the closest matches. This concept-based retrieval allows the system to surface results that may not share exact keywords—a significant leap in enhancing user experience.

Early Impact and Future Implications

Since its launch, the tool has attracted attention from academic institutions internationally. Northwestern’s pioneering efforts even earned them a National Leadership grant from the Institute of Museum and Library Services to further this innovative work. The tool’s success is evident through:

• Improved Accessibility: Users can search by concept, fostering a more approachable collection for non-experts.
• Multilingual Support: The AI can respond in multiple languages, breaking down barriers for diverse research communities.
• Enhanced Discovery: The system highlights underrepresented topics, aiding research on themes that might lack standard metadata.
• Flexible Formatting: Users can format results in various ways for faster analysis.

NUL’s team is also developing an open-source tool called Treetop Discovery, applying the valuable lessons learned to assist other institutions in undertaking similar AI projects.

Looking Ahead: The Future of AI in Academic Research

Northwestern University’s initiative exemplifies how academic libraries can drive responsible AI innovation. By making digital collections more accessible and intuitive, generative AI is poised to reshape how researchers interact with historical documents, rare manuscripts, and cultural artifacts.

As the team continues to refine their search tool, they emphasize the importance of adapting to evolving user expectations. According to Schober, “We need to be ready for what our users expect—and how they expect to search.”

If you’re intrigued and want to learn more about how AWS can empower your institution to develop innovative AI solutions, connect with an AWS representative today. Together, we can usher in a new era of accessible, intuitive digital discovery for libraries and research institutions worldwide.