Using AI Chatbots to Sniff Out Errors and Untruths: Researchers Find Potential Solution

AI chatbots have become increasingly sophisticated in mimicking human conversation, but along with that progress comes a concerning trend: they are prone to giving inaccurate or nonsensical answers, known as “hallucinations.” This raises serious concerns, especially in fields like medicine and law where inaccuracies could have severe consequences.

In a recent study published in the journal Nature, researchers proposed a unique solution to this problem: using chatbots to evaluate the responses of other chatbots. Sebastian Farquhar, a computer scientist at the University of Oxford, and his colleagues suggest that chatbots like ChatGPT or Google’s Gemini could be deployed to detect errors made by other AI chatbots.

Chatbots rely on large language models (LLMs) that analyze vast amounts of text to generate responses. However, these models lack human-like understanding, leading to errors and inconsistencies in their responses. By deploying one chatbot to review the responses of another, researchers aim to identify and eliminate these inaccuracies.

To test this approach, Farquhar and his team asked a chatbot a series of trivia questions and math problems, then used another chatbot to cross-check the responses for consistency. Surprisingly, the chatbots agreed with human raters 93% of the time, highlighting the potential effectiveness of this method.

Despite the promising results, not everyone is convinced of the efficacy of using chatbots to evaluate other chatbots. Karin Verspoor, a computing technologies professor at RMIT University, cautions against the circular nature of this approach, suggesting it may inadvertently reinforce errors rather than eliminate them.

Farquhar, on the other hand, sees this approach as a necessary step towards improving the reliability of AI chatbots. He likens it to building a wooden house with crossbeams for support, emphasizing the importance of reinforcing components to enhance overall stability.

In conclusion, the use of chatbots to evaluate the responses of other chatbots represents a novel approach to tackling the issue of AI hallucinations. While concerns remain about the potential biases and limitations of this method, it opens up new possibilities for enhancing the accuracy and reliability of AI chatbots in various industries.