AI Chatbots Raise Financial Education Scores and Engagement

•

 61 min video

•

 11 min read

YouTube video ID: eyyHz7SU-7A

Source: YouTube video by Stanford Graduate School of Business — Watch original video

PDF

This article summarizes a panel discussion on the use of AI in financial education, focusing on a research project that tested AI-powered educational chatbots. The discussion highlights the effectiveness of these tools, their design principles, and practical implications for educators.

Introduction to the Project

The session began with an introduction to a collaborative project measuring the effectiveness of teaching with AI, specifically in financial education. Angela Amarillas, Program Director of the Stanford Financial Wellness Program (Mind Over Money), moderated the panel. The project involves collaborators from Stanford's Initiative for Financial Decision-Making and other universities.

The panel aimed to address three key questions: 1. Does AI in the classroom work? (Spoiler: Yes, it does.) 2. How does it work in practice? 3. How can this research be translated into one's own teaching?

The discussion was structured into three parts: an overview of the research, a conversation about findings and implementation, and a Q&A session.

Research Overview: AI in Personal Finance Education

Dr. Hakan Ozas, Experimental Research Director for Stanford's Initiative for Financial Decision-Making, presented initial results from an ongoing experiment on integrating AI into personal finance education. This project is a collaboration with Dr. Daniel Chi, Professor and Chair of Finance at the University of Nevada, Las Vegas, and Jodi Leovage, Professor at Cal State Northridge, along with Tim Kaiser, Annamaria Lusardi, Andrea Sestito, and Caroline Zhang.

The core motivation for the project is to improve the effectiveness of financial education, particularly by addressing the challenge of uniformity. Students come from diverse financial backgrounds, making personalized and relevant experiences crucial. Generative AI offers a promising solution for adaptive and personalized learning.

However, a significant concern with AI is the potential for over-reliance and reduced critical thinking. Educational psychology emphasizes "productive struggle" for long-term learning, where students learn from mistakes. General-purpose AI tools often provide direct answers, bypassing this critical thinking process. The project aims to leverage AI's benefits while mitigating the risk of over-reliance.

What the Project Doesn't Do:

  • It does not randomly provide students with access to general-purpose AI like ChatGPT, assuming students already have such access.
  • It does not replace instructors with AI bots; traditional instruction remains central.

Methodology:

The project developed 10 educational chatbots, one for each main personal finance area. These chatbots were tested at two large public universities: Cal State University Northridge and the University of Nevada, Las Vegas.

The experiment design involved instructors teaching two sections of personal finance: * Control Group: Traditional teaching methods. * Treatment Group: Traditional teaching augmented with educational chatbots as a complement, providing opportunities for students to practice what they learned.

Chatbot Design Principles (SPARK):

The chatbots were designed with a "risk-averse" approach to prevent over-reliance and promote critical thinking, following the SPARK principles: * S - Structured: Each chatbot is tied to a clear learning goal and follows a step-by-step path. For example, a chatbot might guide students through comparing two insurance plans. * P - Personalized: The chatbots gather information about students' financial situations to make interactions relevant. For instance, a time-value-of-money bot might ask about realistic monthly savings. * A - Adaptive: While AI is inherently adaptive, the chatbots include additional scaffolding. If a student gives an incorrect answer, the bot nudges them towards the correct understanding without directly providing the answer. * R - Reflective: The bot drives the interaction by asking questions about what students know or don't know, applying a Socratic method. Students provide answers, and the bots are explicitly instructed not to provide answers during these interactions. This limits over-reliance. * K - Knowledge-focused: (Implicitly, the focus is on building knowledge through these structured, personalized, adaptive, and reflective interactions).

Preliminary Findings:

  • Financial Knowledge: Students in AI-augmented sections showed significantly higher financial knowledge. The increase in test scores for the control group was around 6 percentage points, while AI sections essentially doubled this baseline effect.
  • Effect Size: The observed effect size on knowledge is comparable to the impact of receiving personal finance education versus not receiving it (typically around 20% of a standard deviation). Here, a similar gain was found among those already receiving education.
  • AI Usage: Students showed a greater understanding of when to rely on AI and when to use their own judgment. They used AI more as a learning tool, and no increase in over-reliance was detected.
  • Competence: Results were mixed regarding the ability to apply knowledge. Improvements were seen in areas like index funds and mortgages (hypothetical valuation decisions), but not in more complex domains like insurance. This suggests that the complexity of the financial decision itself influences competence gains.
  • Behavioral Change: Over a 3-4 month period, no significant increase in financial behaviors was found, though there was suggestive evidence of reduced costly borrowing. The study will collect long-term data to observe potential behavioral changes.

Practical Implementation and Insights

Demonstrating a Chatbot: Car Buying Scenario

Jodi Leovage demonstrated one of the chatbots designed for car buying. The chatbot guides students through the process, starting with their desired car and budget, then probing their thinking about factors like reliability and affordability. It also introduces realistic considerations beyond the car payment, such as insurance, parking, and maintenance costs, tailored to the student's location. The chatbot's introductory video emphasizes that it's not a sales tool but helps students understand trade-offs and make informed decisions without pressure.

Scaling Financial Education at UNLV

Daniel Chi discussed the scale of financial education at UNLV. They teach approximately 3,000 students annually across two course sequences. Despite the large numbers, classes are capped at 60 students, requiring 25 sections per semester. This necessitates a large pool of instructors, including full-time faculty and adjunct professors who are senior bankers, state treasurers, and CFA professionals.

The diverse student body at UNLV (ages 16 to 82, over 50% first-generation, many working full-time) makes personalized attention challenging. AI, with its adaptive and personalizable aspects, helps bridge this gap by delivering tailored messages and knowledge.

Why AI Increases Engagement

Hakan Ozas explained that the increased engagement observed in the study, rather than students offloading work to AI, stems from the structured and thought-provoking design of the chatbots. Unlike general-purpose AI, these tools are built to make students think and engage in "productive struggle," which is crucial for learning. While this can sometimes lead to frustration, it ultimately fosters deeper understanding.

Competence vs. Knowledge Gains

The mixed results on competence (the ability to apply knowledge) are attributed to the inherent complexity of financial decisions. While students gain knowledge, applying it in real-world scenarios, especially for complex products like insurance, requires more than just knowing the rules. The chatbots aim to foster reflection on decision-making processes.

Classroom Examples of Increased Engagement

Jodi Leovage shared an example from her class where students had to create spending plans and "smart goals." Students in the chatbot-augmented section produced significantly better smart goals, demonstrating a stronger grasp of the concept, likely because the chatbot helped them develop their first goals hands-on.

Daniel Chi noted that student feedback highlighted the value of the personalized aspect of the chatbots. Students, especially those intimidated to ask questions in class, appreciated that the chatbot met them at their knowledge level without judgment, increasing their confidence to engage with the topic.

Addressing Challenges in Financial Education with AI

1. Privacy

Jodi Leovage emphasized that financial information is sensitive, and students are often socialized not to discuss money. Chatbots offer a neutral, non-judgmental third party for students to explore their financial situations, slowly building confidence. While students are advised not to share highly sensitive data (like social security numbers), the chatbots provide a safe space for personal financial exploration.

2. Personalization

AI helps customize content in ways impossible in traditional classrooms, especially given diverse student backgrounds and financial situations. The housing chatbot, for instance, was designed to account for varying housing markets (e.g., Los Angeles vs. Iowa) and allowed students to explore renting vs. buying based on their specific location and circumstances, moving beyond generic advice.

3. Relevance and Timeliness

AI helps keep courses relevant in a rapidly changing financial landscape (evolving products, tax rules, fintech). Jodi Leovage argued that AI itself is the new landscape, and students are already using it. The chatbots teach students how to use AI constructively, responsibly, and to check sources, preparing them for a world where they will inevitably seek financial advice from AI tools.

AI's Role in a Diverse Student Body

Daniel Chi reiterated the critical role of personalization for UNLV's diverse student body. The structured nature of the AI tools provides a framework for learning, which is invaluable for students who might not know where to start with complex topics. This structured approach differentiates it from simply asking ChatGPT for answers, reinforcing the value of education in the AI age.

Heterogeneous Treatment Effects

Hakan Ozas noted that while preliminary, the study doesn't show significant differences in knowledge gains across demographics like gender. However, students who reported higher prior AI usage at baseline showed more concentrated effects, possibly due to greater comfort or perceived value in interacting with the tools.

Broader Challenges and Opportunities for AI in Financial Education

Daniel Chi reflected on the broader implications of AI for education. He highlighted that in the AI age, assessment (especially for online classes) becomes less meaningful, while learning becomes paramount. AI can facilitate better and faster learning. The focus should shift to how AI can enhance learning, with assessment strategies to follow.

Implementation Obstacles and Solutions

Jodi Leovage discussed practical challenges: * Technical Glitches: Expected with new technology (e.g., chatbots "forgetting" memory). These were addressed as they arose. * Student Skepticism: Some students viewed it as "just another assignment." Framing the chatbots as powerful, helpful tools for their personal benefit alleviated this. * Feedback: Mixed but mostly positive. Students appreciated access to the tools. * Instructor Advice: Frame potential glitches to students, and encourage them to report issues.

Actionable Advice for Educators (Rapid Fire)

  • Jodi Leovage: Go slowly. Don't implement all 10 chatbots at once. Integrate a few, get feedback, and build from there. Get comfortable with the tools yourself first.
  • Daniel Chi: Don't be intimidated by AI. Embrace it and think about how to use it for your mission. The goal is to facilitate and reinforce learning, not just find answers. Structured, well-designed chatbots require resources, and hopefully, these will become more widely available.
  • Hakan Ozas: General-purpose AI can make students think they know more than they do. The best use of AI is to make them realize they don't know as much, fostering deeper learning.

Q&A Session

Comparison to Other AI Studies

A question was raised about the uniqueness of these findings, given other studies (like PISA tests) suggesting negative impacts of AI overuse. Hakan Ozas clarified that most negative findings are based on providing access to general-purpose AI. This project's approach, which forces students to think rather than just receive answers, is distinct, and future studies with similar designs might yield comparable positive results. Daniel Chi added that while assessment is challenging, especially online, AI can reinforce learning, and students will eventually be assessed in real-world contexts.

Chatbot Access and Competence Measurement

A participant asked if students had access to the chatbots from the beginning or throughout the course. Hakan Ozas explained that chatbot interactions were sequential, tied to specific topics. For example, after covering budgeting, students would interact with the budgeting bot. Competence was measured shortly after each interaction.

Instructor Variation

The impact of instructor variation on results is a consideration, but the study currently lacks enough data to make definitive statements.

Quality and Quantity of Interaction

A question about assessing interaction quality and quantity was posed. Hakan Ozas noted that the study collects extensive chatbot interaction data. The chatbots incorporate a "badge system" with specific competence questions. While all students eventually earn a badge by providing correct answers, the initial responses help gauge their understanding.

Sharing the Chatbots

A practical question about sharing the developed chatbots was raised. The team confirmed their goal is to make the chatbots publicly available once the study concludes at the end of the academic year, though the infrastructure for broad accessibility is still being figured out.

Student Motivation and Gamification

A participant referenced Khan Academy's math AI tutor, where students often disengage if the tutor doesn't provide direct answers. Hakan Ozas speculated that the incentive structure (20% of the grade coming from chatbot interactions) plays a crucial role in student completion. Jodi Leovage added that the chatbots were designed to be engaging, with gamification elements like badges, and that they focus on self-discovery rather than providing direct answers.

Workload and Material Coverage

A question was asked about whether AI allowed covering more material or increased workload. The panelists clarified that the chatbots are embedded within existing modules, providing an additional tool for deeper learning without necessarily increasing the amount of material covered or the instructor's workload. The treatment and control groups covered the same material.

Student Resistance to AI

A participant asked about students who resist using AI due to pride, privacy concerns, or general boycotting. Jodi Leovage explained that the chatbots are presented as expert-built tools designed for personalized learning, not general AI. While guardrails are emphasized (e.g., not sharing sensitive data), the aim is to teach responsible AI use, acknowledging that AI is an inevitable part of the future.

  Takeaways

  • The AI‑augmented sections showed roughly double the knowledge gains compared with traditional teaching, increasing test scores by about 12 percentage points versus 6 for the control group.
  • The chatbots were built using the SPARK design (Structured, Personalized, Adaptive, Reflective, Knowledge‑focused) to encourage productive struggle and limit over‑reliance on AI.
  • Students used the bots as learning tools, demonstrating better judgment about when to rely on AI, and no increase in over‑reliance was observed.
  • Gains in competence were mixed; improvements appeared in simpler decisions like index funds and mortgages, while complex areas such as insurance showed little change.
  • Educators are advised to roll out a few chatbots gradually, become comfortable with the tools themselves, and present them as supportive, personalized aids rather than replacements.

Frequently Asked Questions

How do the SPARK principles prevent student over‑reliance on AI?

The SPARK principles embed scaffolding that requires students to work through steps, receive personalized prompts, and answer reflective questions without the bot giving direct answers, thereby encouraging productive struggle and limiting the temptation to accept AI‑generated answers outright.

Why did the study observe higher knowledge scores but no significant change in students' financial behaviors?

The AI chatbots improved conceptual understanding by guiding students through structured exercises, which raised test scores, but translating that knowledge into real‑world financial actions requires longer time horizons and external incentives that the short‑term study did not capture. Consequently, behavior changes were minimal within the 3‑4 month period.

Who is Stanford Graduate School of Business on YouTube?

Stanford Graduate School of Business is a YouTube channel that publishes videos on a range of topics. Browse more summaries from this channel below.

Does this page include the full transcript of the video?

Yes, the full transcript for this video is available on this page. Click 'Show transcript' in the sidebar to read it.

was raised about the uniqueness of these findings, given other studies (like PIS

tests) suggesting negative impacts of AI overuse. Hakan Ozas clarified that most negative findings are based on providing access to general-purpose AI. This project's approach, which forces students to *think* rather than just receive answers, is distinct, and future studies with similar designs might yield comparable positive results. Daniel Chi added that while assessment is challenging, especially online, AI can reinforce learning, and students will eventually be assessed in real-world contex

Helpful resources related to this video

If you want to practice or explore the concepts discussed in the video, these commonly used tools may help.

Links may be affiliate links. We only include resources that are genuinely relevant to the topic.

Full transcript is not shown on this page

This page focuses on the summary and original notes. For full verification, refer to the original YouTube video.

PDF