Student-Reported Benefits and Tensions about Generative AI in Academics: Part 1

My ongoing exploration of how generative AI might be used in assessment practices has revealed some benefits to teaching and learning, some drawbacks, and some tensions. Sharing this exploration through these blog posts and having numerous discussions about higher education and generative AI has expanded my appreciation for the complicated conversation taking place around this technology use.

In an effort to meet the challenge of how to address student use of generative AI in higher education, I decided to talk with students as partners (e.g., Manor et al. 2010). At the end of last semester, students from my Calculus I class volunteered to participate in a focus group about their use of generative AI. At the time of the focus group, students in that class had recently completed the artificial-intelligence supported assessment (AI-SA), Using ChatGPT to Explore Calculus Applications, and responded to a short survey about their experience. Seven students participated in the focus group, which I structured under two parts with preplanned questions.

During the first part of the focus group interview, students spoke to their perceptions of that AI-SA in reference to five questions I pulled from the Artificial Intelligence-Supported Assessment Framework.

To what degree does the AI-SA encourage students to achieve the teacher’s purpose and/or learning goals in efficient and powerful ways, which may not be feasible without the use of AI?
To what degree does the AI-SA appropriately assign assessment components the designation of AI-active or AI-inactive?
To what degree does the AI-SA encourage students to evaluate the accuracy and usability of AI output?
To what degree does the AI-SA promote critical thinking about the benefits and drawbacks of using AI?
To what degree does the AI-SA prepare students for using AI outside of academia?

Each question was displayed on an overhead screen and read aloud before students gave their responses. In the second part, I asked students to offer their thoughts and explanations on the summarized class survey results we received related to that AI-SA. I audio-recorded the focus group interview and had the audio transcribed. I then undertook a qualitative analysis of the transcript to identify themes. This analysis was guided by recommendations from Creswell (2013) to manage data, add memos on the transcripts, and classify the memos under larger themes.

In what follows, I share results of this thematic analysis for part one of the focus group interview. Results for this part of the focus group are organized under five questions I extracted from the AI-SA Framework. I have also provided the initial ranking I gave on the framework’s response scale (1–very low; 2–low; 3–middle; 4–high; 5–very high) for each question. Quotes from the focus group are slightly trimmed and edited for grammar. You are encouraged to consider the alignment between my ranking and themes that emerged from students’ responses.

To what degree does the AI-SA encourage students to achieve the teacher’s purpose and/or learning goals in efficient and powerful ways, which may not be feasible without the use of AI? [my initial framework ranking was 4–high]

Emergent Themes	Supporting Focus Group Quotes
Generative AI provides an efficient way to access information beyond manual internet searches.	• I definitely think that it helped us to be efficient. • Without ChatGPT, we’d probably be looking for a bit on the internet. • It helped us brainstorm ideas that we might not have thought of without AI. • It definitely helped me learn more about the topic in a short amount of time [rather] than having to sit down beforehand.
Generative AI is used to support main points students intend to make when providing evidence of their learning.	• I was able to add more information to support what I was saying, which was really helpful and couldn’t really be done without it [ChatGPT]. • I know what I’m saying is not just gibberish or just random thoughts. It actually made sense and it came together a lot better [after using ChatGPT].

Students’ responses aligned with my initial ranking of 4–high for this framework question. They reported that ChatGPT helped with efficiently brainstorming ideas and supporting those ideas in ways that would have taken more time and effort without the use of generative AI.

To what degree does the AI-SA appropriately assign assessment components the designation of AI-Active or AI-Inactive? [my initial framework ranking was 4–high]

Emergent Themes	Supporting Focus Group Quotes
Generative AI increases substance of student writing.	• When I actually sat down and wrote by myself, I already had a general idea of what I was going to write [due to work during AI-Active component].
AI-Inactive assessment task component promotes critical thinking.	• I feel like the inactive part really made me think about what I was talking about more. Whereas with the AI-Active, although it did give more examples and everything, the AI-Inactive part really just made me think about the whole problem.
The ordering of AI-Active and AI-Inactive assessment task components affects the way students provide evidence of their learning.	• I feel like if the inactive part was first, I would’ve done more thinking to apply things. Then, [with] whatever questions I did have, that’s what I would answer in the active part. So [I prefer] doing inactive first and then active, because I feel like when I started with active then going to inactive, I kind of had a skewed perception of it [the content to be learned]. I felt like I was trying not to say exactly what ChatGPT has already said. • I feel like reversing it [order of AI-Active and AI-Inactive] would help for a better understanding and just help generating that sort of curiosity.

My initial ranking for this question was 4–high, but after considering these focus group themes, I feel that 3–middle is the accurate ranking. Students appreciated the AI-Inactive portion of the task as it encouraged them to pause and be more purposeful when demonstrating their understanding of the learning goal. However, having students active in ChatGPT before they provided evidence of their learning had some drawbacks. For example, a student expressed their concern of trying to not directly quote ChatGPT in the inactive portion. A possible better assessment task structure may be to have an initial AI-Inactive brainstorming and outlining task component before the AI-Active portion. After engaging with the technology, the student could complete another AI-Inactive component where they demonstrate their understanding in documenting evidence of learning.

To what degree does the AI-SA encourage students to evaluate the accuracy and usability of AI output? [my initial framework ranking was 3–middle]

Emergent Themes	Supporting Focus Group Quotes
Generative AI is not always accurate, and students should be encouraged to evaluate the accuracy and usability of AI output.	• This assignment helped me realize that ChatGPT still has a way to go in terms of accuracy because when I did the practice problem [from ChatGPT], I looked through it and some of the work was wrong. • I think that people using AI for things that aren’t their assignments could very easily just get caught doing something and regret it later. • I think you kind of have to be on your toes and not take every single thing as fact. • I take what I have already learned and then I check that against what the AI is telling me.
Generative AI should be presented and viewed as a tool and not a replacement for learning.	• If you’re going to use it as a classroom tool, having it not be 100% right all the time is probably better because otherwise I wouldn’t have to do any work ever, which sounds great if you’re doing work, but if you’re learning, it kind of defeats the whole purpose. • I’ve realized that if I want to, outside of college, use AI to learn how to do something for a job, or if I use AI in school, I don’t really learn anything because I’m relying on this AI tool.

Student responses to this question aligned to a higher ranking than my initial ranking of 3–middle. These students were critical of the information received from ChatGPT and hesitant to exclusively rely on this generative AI when providing evidence of their learning.

To what degree does the AI-SA promote critical thinking about the benefits and drawbacks of using AI? [my initial framework ranking was 3–middle]

Emergent Themes	Supporting Focus Group Quotes
Students are typically skeptical of AI output and are often well versed in how this technology was designed.	• I asked for practice problems and had it [ChatGPT] figure one problem out. I asked probably seven or eight times, and I got a different answer every single time. So obviously not every single one is the right answer. I feel like that really makes me question the accuracy of what AI is putting out. • It only knows what the programmer gave during its training, all the data that it gave the AI during its training. • I think a common misconception with AI is that it knows everything. It doesn’t know everything.
Generative AI may be better suited for investigating topics or concepts rather than answering or solving specific problems.	• If they’re [students are] looking for how to understand a concept or something that they’re curious about or something that they want to expand on, that’s how AI could benefit, not just looking for the answer. • I believe it’s still working on knowing different coding languages. It knows Python, but it doesn’t really know SAS code yet.

Based on their focus group responses, students thought critically about how to appropriately integrate ChatGPT when completing an assessment for learning. Their responses gave some evidence for a higher ranking beyond 3–middle for this question on the AI-SA Framework. I was somewhat surprised by the level of skepticism these students expressed regarding the benefits of using AI.

To what degree does the AI-SA prepare students for using AI outside of academia? [my initial framework ranking was 4–high]

Emergent Themes

Supporting Focus Group Quotes

Requiring students to assess the quality of generative-AI output before using it in a product others will see or read helps prepare them for the realities of using AI outside of academia.

• I think that using it outside of school for your work, especially now that it’s not really fully correct all the time, might reflect badly on you.
• When I get to my job and I don’t really know how to do anything and I’m still using AI, I’ll end up just not being fully capable of doing everything that I could do if I just learned and did things on my own.
• The writing project helped me realize the dependency on AI is kind of a really bad thing that people have going on.

Student responses to this question aligned with my initial ranking, 4–high, and overlapped with some of the themes that emerged in the previous two questions. When analyzing focus group responses to this question, I was encouraged by students’ desire to provide accurate information when reporting on their learning and their awareness of how the quality of their work reflects on their character.

After analyzing part one of the student focus group responses, I found these students to be aware of how AI works and knowledgeable of potential benefits to their lives and potential pitfalls to overreliance. These traits may be partially attributed to the fact that students in this focus group volunteered to participate and likely have a high interest in AI integration both within and outside of academics. There was general alignment between my initial framework rankings and emergent themes, and students responded favorably to the distinction between AI-Active and AI-Inactive components to this AI-SA. A particularly interesting finding was that the order of AI-Active and AI-Inactive components affected the way students provided evidence of attaining the learning objective. Findings from part one of this focus group provide some evidence that the AI-SA Framework and related assessment task promoted student use of generative AI in ethical, productive, and educative ways when demonstrating learning.

In part two of the focus group, students considered summarized class survey results related to their experience completing the AI-SA, Using ChatGPT to Explore Calculus Applications, and spoke to generative AI use in broader contexts. Part two of the focus group was longer than part one as students expounded on their perceptions. In the next blog, I will share findings from the analysis of that portion of the focus group interview and provide a more comprehensive summary of focus group takeaways.

References

Creswell, John. 2013. Qualitative Inquiry and Research Design: Choosing Among Five Approaches, 3^rd ed.Thousand Oaks, CA: SAGE Publications, Inc.

Manor, Christopher, Stephen Bloch-Shulman, Kelly Flannery, and Peter Felten. 2010. “Foundations of Student-Faculty Partnerships in the Scholarship of Teaching and Learning.” In Engaging Students Voices in the Study of Teaching and Learning, edited by Carmen Werder and Megan Otis, 3–15. Sterling, VA: Stylus.

Aaron Trocki is an Associate Professor of Mathematics at Elon University. He is the CEL Scholar for 2023-2024 and is focusing on models of assessment and feedback outside of traditional grading assumptions and approaches.

How to Cite this Post

Trocki, Aaron. 2024. “Student-Reported Benefits and Tensions about Generative AI in Academics: Part 1.” Center for Engaged Learning (blog), Elon University. April 23, 2024. https://www.centerforengagedlearning.org/student-reported-benefits-and-tensions-about-generative-ai-in-academics-part-1.

Posted in:

Tagged:

Aaron Trocki