Since interviewing Rachel Forsyth a couple months ago, I have been writing and piloting assessments that incorporate artificial intelligence platforms like ChatGPT. My work was in response to her consideration of how AI could be used to help students and her comparison to how other technologies have become commonplace tools in teaching and learning (e.g. citation generators and calculators). She recognized that AI could do things that we might want students to do, but that the availability of AI platforms, such as ChatGPT, forces faculty to think about our instructional purposes.

Currently, I have been considering the many purposes threaded into my teaching of a Calculus I course and have been working on ways ChatGPT could help my students understand and achieve these purposes. During one class this semester, I asked students if they use ChatGPT in their academics, and my students unanimously expressed that they do. I had the impression that today’s college students are using this chatbot on a consistent basis.

In office hours this semester I have worked with a number of students on how ChatGPT could help them on various math and writing assessments. In these interactions, I took extensive notes about how students use ChatGPT and how artificial intelligence could support learning and assessment. In this work, I defined an artificial intelligence–supported assessment (AI-SA) as an assessment that includes student utilization of artificial intelligence technology such as a chatbot and provides the teacher with information about student progress towards achieving learning goals. I distinguished AI-SAs from assessments produced by faculty with the help of AI (e.g., a teacher uses a chatbot to generate quiz questions). My notes began to coalesce into guidelines for faculty to consider when planning how students should use AI technology in assessments.

This work was fresh on my mind when I attended the American Association of Colleges and Universities (AAC&U) annual Transforming STEM Higher Education Conference in November. I had the pleasure of talking with other faculty about how AI is affecting higher education and my work on developing guidelines for AI use in assessment. James Lang gave the keynote address, From Books to Bots?…The Role of AI and ChatGPT in Undergraduate STEM Teaching. His insightful presentation resonated with me, and I particularly absorbed his recommendations that faculty use AI to make connections among concepts and applications and to unbundle their assessments to promote student growth.

After the conference, I felt ready to implement my first AI-SA and finish my draft of a preliminary AI-SA framework. My work was influenced by the practice perspective of assessment (e.g., Boud et al. 2018) and its promotion of assessment for learning and the aim to unbundle assessments (e.g., Lang 2023) to better consider appropriate technology integration. In interactions with students using ChatGPT to respond to writing prompts about math, I realized that some assessment components lend themselves to the use of AI technology while others do not. It depends on the teacher’s purpose and/or learning goals. I began to label certain assessment components as AI-active and others as AI-inactive with the first indicating that students can interact (e.g., prompt engineer) with an AI technology such as ChatGPT and the second indicating that students cannot interact with AI technology. In this framework, I attempted to account for the student perspective, the purpose and/or learning goal(s), the AI technology, and implications for using the AI technology.

Artificial Intelligence–Supported Assessment Framework

Guiding Questions Response Scale: 1-very low; 2-low; 3-middle; 4-high; 5-very high	Ranking (1 – 5)
1) To what degree does the artificial intelligence–supported assessment (AI-SA) align with student learning objectives?
2) To what degree does the AI-SA give every student equitable access and opportunity to actively engage in learning?
3) To what degree does the AI-SA align with students’ current levels of AI literacy: ability to understand, use, monitor, and critically reflect on AI applications (e.g., Long, Blunt, and Magerko 2021)?
4) To what degree does the AI-SA encourage students to achieve the teacher’s purpose and/or learning goals in efficient and powerful ways, which may not be feasible without the use of AI?
5) To what degree does the AI-SA appropriately assign assessment components the designation of AI-active or AI-inactive?
6) To what degree does the AI-SA encourage students to evaluate the accuracy and usability of AI output?
7) To what degree does the AI-SA promote critical thinking about the benefits and drawbacks of using AI?
8) To what degree does the AI-SA prepare students for using AI outside of academia?
9) To what degree does the AI-SA align with the institution’s honor code?
10) To what degree does the AI-SA align with the institution’s position on AI?

My intent is to give faculty a lens through which to create and assess the quality of AI-SAs they employ with their students. Ranking framework question responses using a Likert scale from one to five may help faculty perceive assessment quality in individual framework question components and holistically by calculating the average, median, or mode of the framework rankings.

In conjunction with the writing of the AI-SA framework, I wrote an AI-SA for my current Calculus I students. This AI-SA adopts some of the structures from previous writing assignments I have used and studied in early college-level calculus (e.g., Trocki 2023) and folds in student use of the AI technology, ChatGPT.

---

Using ChatGPT to Explore Calculus Applications

Read all guidelines before you begin.

In this assignment, you will use ChatGPT to explore and document calculus applications. A major learning goal in our course is to “establish the connection between rates of change and accumulation.” In this writing project you will pick any area of interest you like. For example, you may be interested in physics, music, business, sports, etc. Once you have chosen your area of interest, please complete the following steps:

ChatGPT-Active

1. Make an account in ChatGPT
   • (https://chat.openai.com)
   • Choose version 3.5 (it’s free)
   • Your questions and output will be part of your report so be sure to not close ChatGPT until you are finished with this project.
2. Familiarize yourself with ChatGPT by asking it three random questions. For example, “How old am I?”
3. Ask ChatGPT some questions (i.e., prompt engineering) about calculus and the connection between rates of change and accumulation.
4. Ask ChatGPT some questions about the connection between rates of change and accumulation in relation to your area of interest.
5. Ask ChatGPT to produce an example problem connecting rates of change and accumulation with your area of interest.
6. Copy and paste the link to your ChatGPT output at the top of your writing project submission right below your name.

ChatGPT-Inactive

Make sure you’ve copied and pasted the link to your ChatGPT output at the top of your writing project submission right below your name.

You may reference the ChatGPT output in your written-by-you responses, but do not ask ChatGPT anything new.

The audience for your writing submission is a current student in our class who knows about Calculus I concepts and applications but does not know about how calculus applies to the area of interest you chose to use in ChatGPT. Your goal is to educate this student about calculus and the connection between rates of change and accumulation in your area of interest. Reference and use the questions (i.e., prompt engineering) and ChatGPT output you generated. Use the following prompts to guide your writing. You can screenshot and include any ChatGPT output you like. Your writing should be in the form of an email or letter, which is similar to previous writing assignments in our course.

1. Briefly introduce yourself to this student by giving your first name, major (or intended major), and something interesting about yourself (e.g., a hobby).
2. What area of interest did you choose and why is it important to you?
3. In the study of calculus, what is the connection between rates of change and accumulation?
4. How are rates of change and accumulation related or connected in the area of interest you chose? How are rates of change and accumulation applied in the area of interest you chose?
5. Report an example problem you found from ChatGPT. Use this problem to expound on your answers to #3 and #4.
6. Was ChatGPT helpful to you in responding to #3, #4, and #5? Give details in your response.
7. What are some ways you can critically evaluate the accuracy and quality of the ChatGPT output?
8. In general, do you think ChatGPT is a good resource for your learning? Give details in your response.

My grading and feedback will be focused on your use of ChatGPT and your treatment of these eight prompts in your email/letter to another student. The grade and feedback will be posted in our LMS just like the previous writing assignments.

---

When introducing this AI-SA to students we discussed how to access ChatGPT and the expectation of AI-active vs. AI-inactive.

Before giving this assessment to students, I used the AI-SA framework to write and respond to framework questions with rankings from one to five. For example, framework question one, “To what degree does the artificial intelligence-supported assessment (AI-SA) align with student learning objectives?” prompted me to look at the course syllabus to directly copy and paste the learning objective, “establish the connection between rates of change and accumulation” into the assessment guidelines. My response was a “5 – very high” to this framework question. In response to AI-SA framework question 10, my response was a “4 – high” as I feel the assessment aligns to a high degree with my institution’s position on AI. Readers are encouraged to consider my rankings against their own and practice ranking this AI-SA using the other eight (#s 2 – 9) framework questions.

I consider the AI-SA framework as a preliminary guide to assisting faculty with creating and considering the quality of AI-SAs. In the coming months I will test and refine the AI-SA framework by getting feedback from colleagues and using it to create and evaluate other AI-SAs. Some initial research questions to consider:

• How do we define quality levels of low, medium, and high according to the AI-SA framework rankings?
• How do assessments that rank high according to the AI-SA framework affect students’ learning?
• How do students perceive assessments that rank high according to the AI-SA framework?
• How do assessments that cycle students through AI-active and AI-inactive assessment components affect student learning?
• What revisions to the AI-SA framework are needed in response to research on its effectiveness for creating and judging the quality of assessments?
• How well does the AI-SA framework apply to developing in-class summative assessments?
• How well does the AI-SA framework align with Clark and Talbert’s (2023) four pillars of alternative grading?
• How does the instructional use of the AI-SA framework affect grading and feedback practices?
• Does instructional use of the AI-SA framework lead to normalized student expectations for using AI in academic settings?
• Does instructional use of the AI-SA framework promote the ethical use of AI within and outside of academia?

This list of research questions is not exhaustive, and it reflects our beginning understanding of how to utilize AI in academic settings. My hope is that the AI-SA framework is educative for faculty seeking to incorporate AI in their assessments. Try it out and use the Center for Engaged Learning’s email address to communicate feedback.

At the time of this writing, my students are currently working on this AI-SA in Calculus I. In the next blog post, I intend to follow up with student writing samples and my feedback on this AI-SA. We will also explore how the AI-SA framework rankings did (or did not) match what students report. I look forward to continuing our conversation about how to respond to and address the use of AI in assessment and feedback practices.

References

Boud, David, Phillip Dawson, Margaret Bearman, Sue Bennett, Gordon Joughin, and Elizabeth Molloy. 2018. “Reframing Assessment Research: Through a Practice Perspective.” Studies in Higher Education 43(7): 1107-1118.

Clark, David and Robert Talbert. 2023. Grading for Growth: A Guide to Alternative Grading Practices that Promote Authentic Learning and Student Engagement in Higher Education. Stylus Publishing, LLC.

Lang, James. “From Books to Bots?…The Role of AI and ChatGPT in Undergraduate STEM Teaching.” Presentation, American Association of Colleges and Universities (AAC&U) annual Transforming STEM Higher Education Conference, Arlington, VA, November 4, 2023.

Long, Duri, Takeria Blunt, and Brian Magerko. 2021. “Co-designing AI Literacy Exhibits for Informal Learning Spaces.” Proceedings of the ACM on Human-Computer Interaction 5, no. CSCW2 (2021): 1-35.

Trocki, Aaron. 2023. “Perceptions of Learning in a Calculus Course Infused with Multimodal Writing.” Proceedings of the Forty-Fifth Annual Meeting of the North American Chapter of the International Group for Psychology of Mathematics Education, 528-536. Reno, NV.

Aaron Trocki is an Associate Professor of Mathematics at Elon University. He is the CEL Scholar for 2023-2024 and is focusing on models of assessment and feedback outside of traditional grading assumptions and approaches.

How to Cite this Post

Trocki, Aaron. 2023. “A Preliminary Framework for Artificial Intelligence–Supported Assessments: Models of Assessment and Feedback. ” Center for Engaged Learning (blog), Elon University. December 5, 2023. https://www.centerforengagedlearning.org/a-preliminary-framework-for-artificial-intelligencesupported-assignments-models-of-assessment-and-feedback.