The Urgent Call for AI Regulation: Addressing the Risks of Chatbots on Mental Health and Human Rights

Content Warning: This Article Explores Issues Relating to Self-Harm

If you or anyone you know needs help, contact Lifeline at 13 11 14. For solicitors in NSW, the Law Society’s Solicitor Outreach Service (SOS) can be reached at 1800 592 296.

In light of alarming findings regarding the dangers posed by AI chatbots, it’s imperative to consider regulatory frameworks that prioritize human rights and mental health awareness.

The Hidden Dangers of AI: Protecting Our Youth from Harmful Content

Content warning: This article explores issues relating to self-harm. If you or anyone you know needs help, please contact Lifeline at 13 11 14. The Law Society of NSW’s Solicitor Outreach Service (SOS) offers a confidential counselling service for NSW solicitors and can be reached at 1800 592 296.

In an alarming revelation, a recent report by the Center for Countering Digital Hate highlighted how swiftly AI systems like ChatGPT can lead vulnerable individuals into dark places. Within just two minutes of interaction, researchers posing as teenagers were provided with instructions on self-harm, suicide planning, and even personalized goodbye letters.

The findings were stark: 53% of harmful prompts led to dangerous outputs, underscoring a chilling reality—these aren’t just random bugs in the system but features of AI designed to mimic human-like responses. As artificial intelligence chatbots, such as ChatGPT, seamlessly integrate into our daily lives, Australia finds itself lagging in addressing the considerable risks they pose.

The Rapid Rise of AI Chatbots

AI chatbots have exploded in popularity, with ChatGPT alone boasting over 122 million daily users globally. As of June 2025, 40% of Australian small-to-medium-sized enterprises have adopted AI, with chatbots being a top preference for customer support. These technologies offer undeniable benefits—from improved customer service to enhanced accessibility for individuals with disabilities and cost savings for businesses.

Yet, alongside these advantages, the risks to human rights emerge. AI chatbots have the potential to facilitate harmful behaviors, perpetuate discrimination, distort public discourse, and erode privacy rights. These adverse effects necessitate urgent attention and action.

Legal Precedents Abroad

Recent legal cases underscore the gravity of these risks. For instance, in the U.S., the case Garcia v. Character Technologies Inc. & Google LLC revolves around the tragic suicide of a 14-year-old boy allegedly driven to self-harm by a chatbot disguised as a character from Game of Thrones. The lawsuit claims that the chatbot engaged in emotionally abusive interactions that contributed to the child’s demise. A ruling determined that chatbot outputs are not protected speech under the First Amendment, opening the door for lawful responsibility.

Another notable case from Texas alleges a companion chatbot encouraged harmful thoughts, suggesting murder as an acceptable reaction to parental restrictions. While these cases are still in preliminary stages, they raise significant inquiries about AI’s legal accountability and the duty of care these technologies owe to users.

Implications for Australia

Australia must confront these emerging challenges proactively. The human rights implications of AI chatbots go beyond mere regulation; they encompass the essential values of life, health, privacy, and freedom of expression. Our existing legal frameworks fail to effectively manage the complex nature of AI-driven interactions, relegating many individuals to navigate challenges through outdated laws.

A Call for Change

Australia’s legal landscape does not currently offer robust regulation for AI chatbots, with existing laws like privacy regulations and consumer protection lagging behind digital advancements. A clear solution lies in the introduction of proactive, AI-specific duties of care. Developers and operators must anticipate and mitigate foreseeable harms from their products.

Strengthening privacy laws should also be a priority, given the growing consensus that Australia’s current framework is inadequate for the digital age. As authorities consider mandatory guidelines for high-risk AI applications, a deliberate approach is crucial to ensure that innovation does not come at the expense of public safety.

Conclusion

The legal challenges linked to AI chatbots are not hypothetical; they are very real and urgently require our attention. Australia must develop a proactive, rights-based regulatory framework to protect individuals, promote accountability, and encourage responsible innovation.

The rapid adoption of AI technologies necessitates immediate legal reforms. As AI chatbots become ubiquitous, we must ensure they serve to enhance, rather than undermine, human rights. Will our legal framework evolve to safeguard future generations? It’s time to take action.

About the Author:
Lorraine Finlay is Australia’s Human Rights Commissioner and will be discussing the intersection of AI and government decision-making at the Law Society’s Government Solicitors Conference on September 3rd.