Results of Turing Test: Human vs ChatGPT in Conversations

In a recent study conducted by researchers at UC San Diego, the ability of people to distinguish between human agents and AI models, particularly the GPT-4 model, was put to the test using a Turing test methodology. The results of this study suggest that individuals struggle to differentiate between GPT-4 and human agents during 2-person conversations.

The researchers designed a two-player computer game in which a human interrogator interacted with a “witness” who could be either human or an AI agent. Through a series of questions and interactions, the interrogator had to determine whether the witness was a human or a machine. The study included three different LLMs as potential witnesses: GPT-4, GPT 3.5, and ELIZA.

The findings revealed that while participants could often identify ELIZA and GPT-3.5 models as machines, they struggled to determine whether GPT-4 was a human or a machine, performing no better than random chance. This suggests that the GPT-4 model is highly convincing in mimicking human conversation during brief interactions.

The implications of these results are significant, as they suggest that AI models like GPT-4 are becoming increasingly difficult to distinguish from humans in online interactions. This has implications for areas such as automating client-facing jobs, fraud, misinformation, and more. As AI systems become more sophisticated at mimicking human behavior, people may become more uncertain about who they are interacting with online.

The researchers plan to further explore this topic by running additional experiments, including a three-person version of the game and testing AI’s persuasive capabilities in various scenarios. Their future work aims to provide more insight into the extent to which people can differentiate between humans and AI models in conversational settings.

Overall, the study highlights the remarkable capabilities of LLMs like GPT-4 in simulating human-like conversation and raises important questions about the implications of this technology for society. As AI continues to advance, distinguishing between human and machine interactions may become increasingly challenging, with potential consequences for trust, deception, and communication in the digital age.