How Students Use ChatGPT in AI Courses: Insights From Over 1,000 Conversations

Introduction

Artificial intelligence (AI) is revolutionizing education, and tools like ChatGPT are becoming virtual study buddies for students worldwide. But how are students actually using these AI assistants? Are they leveraging them to deepen their learning, or are they relying on them to do the heavy lifting?

To answer these questions, researchers from the University of Massachusetts Amherst created StudyChat, a dataset capturing over 1,000 real conversations between university students and a ChatGPT-powered tutor during an artificial intelligence course. Their goal? To understand how students interact with AI chatbots, whether those interactions correlate with better learning outcomes, and whether some behaviors indicate over-reliance on AI tools.

Let’s dive into the key findings from this fascinating research and explore what they reveal about the future of AI-assisted learning.

How the Study Was Conducted

To collect real-world data, the researchers built a web-based chatbot, powered by ChatGPT (GPT-4o-mini), that replicated the experience of using the real ChatGPT. Students were free to use this chatbot for homework and programming assignments throughout the semester.

• Participants: 84 students enrolled in a university-level AI course.
• Data Collected: 1,197 chat sessions, 7,856 student messages, and 455 completed assignments.
• Analysis Method: Each student’s chat messages were categorized using dialogue act labeling, a technique that identifies the function of each message (e.g., asking a question, debugging code, or seeking explanations).

The dataset provides a rare, large-scale glimpse into how students actually engage with AI tutors in a real academic setting.

What Were Students Using ChatGPT For?

1. Asking Questions (Most Common Use Case)

The majority of student interactions with ChatGPT involved asking questions related to their assignments. These questions typically fell into two categories:
- Assignment Clarification: Students asked ChatGPT to explain the instructions or what was being asked of them.
- Code Explanation: Many students wanted ChatGPT to break down programming concepts or debug their code.

This suggests that most students weren’t simply using ChatGPT to copy answers, but rather to seek understanding and guidance—a promising sign for AI’s role as an educational tool.

2. Writing Code Together (“Co-Programming”)

Another surprising trend was that students collaborated with ChatGPT on coding tasks rather than just asking it to generate complete solutions. They fed it snippets of code, asked for improvements, and used its suggestions to refine their programs.

This interactive approach could be a new form of AI-assisted learning, where students and AI work together much like a tutor-student relationship.

3. Debugging and Fixing Errors

Some students heavily relied on ChatGPT to identify and fix errors in their programming assignments. While this might seem helpful, the study found that students who frequently used ChatGPT in this way performed worse on exams, possibly due to over-reliance on the AI rather than independently problem-solving.

Does Using ChatGPT Improve Student Performance?

The study examined whether certain types of interactions with ChatGPT correlated with better course performance. Here’s what they found:

✅ Positive Impact:
- Students who asked conceptual questions (e.g., theory, problem-solving strategies) scored higher on assignments and exams.
- Those who actively discussed assignment requirements with ChatGPT (instead of just pasting instructions) performed better overall.

💡 What This Means: Students who used ChatGPT as a tool to deepen their understanding rather than as a shortcut tended to learn more effectively and achieve higher grades.

❌ Negative Impact:
- Students who relied on ChatGPT primarily for debugging and fixing error messages performed worse on exams.
- Those who frequently used ChatGPT for tool-related assistance (e.g., using Python libraries) struggled in tests where ChatGPT was not allowed.

💡 What This Means: Some students may have become too dependent on AI, leading to weaker problem-solving skills when working independently.

What Can This Teach Us About AI in Education?

1. AI Can Be a Powerful Study Tool—If Used Wisely

This research suggests that ChatGPT (and similar AI tutors) can be invaluable for learning, especially when used to explore concepts, clarify assignments, and foster critical thinking. However, it’s essential that students don’t become overly reliant on AI for answers.

2. Educators Can Use AI Insights to Design Better Courses

With datasets like StudyChat, instructors can identify students at risk of AI over-reliance and intervene when necessary. Future AI tutors could even warn students if they seem to be depending too much on AI-generated solutions and encourage more interactive learning approaches.

3. More Research Is Needed on AI’s Long-Term Learning Effects

While this study provides valuable insights, we still don’t fully understand how AI-assisted learning impacts students in the long run. Future studies could look at:
- Whether AI-assisted students retain knowledge longer than those who don’t use AI.
- How different chatbot designs affect learning outcomes and behavior.
- Whether AI usage patterns change as students progress in their studies.

Key Takeaways

🔹 Most students use ChatGPT for asking questions and clarifying assignments, rather than just copying answers.
🔹 Students who actively engage in discussions and explore conceptual questions tend to perform better.
🔹 Over-reliance on AI for debugging and coding assistance can lead to weaker problem-solving skills, especially in tests.
🔹 AI tutors have great potential, but educators need to guide students on healthy AI usage to maximize learning benefits.

How Can You Improve Your Own AI-Powered Learning?

🏆 Use ChatGPT for learning, not just getting answers. Try asking “Why?” and “How?” more often!
💡 Explain concepts in your own words. Instead of just pasting an assignment, describe it to ChatGPT in your own way—it forces you to think critically.
🧠 Test yourself without AI. Before checking an AI-generated solution, see if you can solve it on your own first.

Final Thoughts

The StudyChat research highlights both the opportunities and pitfalls of AI-assisted learning. AI-powered tutors like ChatGPT can enhance education when used appropriately, but they also come with challenges—especially when students lean too much on AI instead of developing their own problem-solving abilities.

As AI in education continues to evolve, research like this will help us fine-tune how we integrate AI while keeping the focus on real learning. Whether you're a student, educator, or AI enthusiast, these insights are a powerful reminder that technology is a tool—not a replacement for thinking.

What do you think? Have you ever used ChatGPT for studying? Share your thoughts in the comments! 🚀