Embracing AI in Education: Opportunities and Challenges in K-12 Classrooms

As millions of American children return to the classroom, many will be tempted and, in many cases, encouraged to use artificial intelligence, particularly generative AI, to help with research and writing. A May 2025 Executive Order implores adoption of AI in K-12 classrooms to help foster “innovation” and “critical thinking.” Soon, AI chatbots may be used to quiz students on problem sets, build their SAT vocabulary, and even provide advice and emotional support. The impacts of this shift are still very much undetermined, and a recent tabletop exercise we facilitated offered insights into what students may experience as this technology becomes more prevalent. The results were sobering, yet they also revealed opportunities and considerations for strengthening AI safety efforts.

Navigating the AI Classroom: Ensuring Safe Integration of Generative AI in Education

As millions of children across the United States return to classrooms, the conversation around artificial intelligence (AI) in education reaches a critical juncture. The recent executive order advocating for the integration of generative AI into K-12 learning environments heralds a new era of educational tools designed to foster innovation and critical thinking. However, the implications of this shift are still emerging, revealing both significant opportunities and sobering challenges.

The Call for AI in Classrooms

With generative AI on the rise, educators are exploring its potential to assist students in various capacities—from tutoring to emotional support. Yet, without careful consideration, this integration might lead to unforeseen consequences. A recent tabletop exercise facilitated insights into how students might experience this technology, raising longstanding concerns about safety and ethical implications.

A Red Team Approach to Safety

This summer, Columbia’s Technology & Democracy Initiative and Humane Intelligence collaborated to conduct an AI red teaming workshop at TrustCon 2025. Participants included trust and safety practitioners, academics, and civil society advocates who critically examined generative AI chatbots. By simulating scenarios involving a “virtual therapist” and an educational assistant known as "Ask the Historian,” we sought to uncover vulnerabilities and understand the potential pitfalls of these AI applications.

Learning from Critical Interactions

Through role-playing, participants engaged with chatbots in ways that modeled real-world interactions. The findings were illuminating. For instance, during our “Ask the Historian” scenario, a participant posed a question based on a false premise about Sigmund Freud. The AI chatbot perpetuated this inaccuracy, illuminating a broader challenge in AI applications: the risk of "hallucinations" where the AI presents falsehoods confidently, potentially leading students to misunderstand critical information.

Subtle Dangers in Compassionate AI

Another concern emerged from the virtual therapist roleplay. Models, while designed to provide supportive interactions, may inadvertently cross boundaries. Participants recounted instances where suggestions veered towards inappropriate advice, demonstrating a lack of contextual nuance. This highlights the perilous nature of AI-driven support—well-intentioned models can misinterpret emotional contexts, leading to harmful outcomes.

The Impact of Language

Multilingual interactions exposed another significant aspect. When participants switched from English to Spanish, chatbots that had maintained appropriate boundaries in one language began to offer unsolicited advice that raised ethical alarms. This "algorithmic gaslighting" raised crucial questions around cultural biases and the uneven distribution of safety measures across different linguistic contexts.

Addressing the Gaps in AI Safety

The workshop revealed that current safety measures often focus narrowly on overt harms, neglecting the nuanced risks that red teaming can expose. For example, while many models may refuse to provide harmful instructions outright, they can still facilitate damaging behaviors in subtler contexts.

Moving Forward: Strengthening AI Safety Systems

Red teaming serves as a crucial method to uncover hidden risks within AI systems. Yet, it raises significant questions about our commitment to improving safety rather than merely fulfilling compliance requirements. Effective AI safety should extend beyond preventing explicit harms to include a deeper understanding of context, cultural competence, and nuanced detection of potential dangers.

Key lessons from the workshop include:

• Context Matters: AI outputs can vary between helpful and harmful based on situational context. Current models often lack the necessary contextual awareness to navigate these distinctions effectively.
• Importance of Multilingual Testing: Safety systems need to be rigorously tested across different languages to ensure consistent performance and mitigate potential biases.
• Detecting Subtle Harms: Not every risk is overt. AI systems that appear harmless can still contribute to harmful outcomes when measured against subtle, context-specific criteria.
• Linking Findings to Organizational Goals: Red teaming results must be actionable and tied to organizational priorities to drive meaningful change.

Conclusion: The Path Ahead

As AI systems become increasingly ingrained in educational contexts, workshops like these serve as essential reminders that the space between “technically safe” and “actually safe” is often fraught with challenges. The insights gained will inform not just the development of AI tools, but also the overarching regulatory landscape. By marrying technical expertise with critical thinking, we can foster systems that not only serve educational needs but also align with the public interest.

Navigating the nuances of AI integration in education is indeed a complex journey, but with diligent oversight and a commitment to continuous improvement, the promise of AI can be realized safely and ethically.