Reconsidering Writing Pedagogy in the Era of ChatGPT
Lee-Ann Kastman Breuch, Kathleen Bolander, Alison Obright, Asmita Ghimire, Stuart Deets, and Jessica Remcheck
Appendix A: Codebook—Qualitative comment “filters” reflecting student perspectives of ChatGPT
| Filter | Definition | Example |
|---|---|---|
| Low Order Concerns | A basic form of analysis; this theme captures how students used their understanding of sentence structure, grammar, voice, and flow to evaluate the text. Draws from the Writing Center language for "low order concerns." | "It's perfectly understandable. I didn’t notice any major issues in terms of grammar." |
| Ethics | A basic form of analysis students used. This theme category captures how students used external policies to evaluate the text, and how students expressed ethical concerns about using ChatGPT text instead of their own. Comments in this filter include worries about plagiarism, privacy, and getting "caught" using the technology. | "I wouldn’t use it because it's cheating. But if it was like an inspiration piece I would definitely use it." |
| Logic/Organization | A complex form of analysis students used. This theme captures the way students evaluated the arguments, organization, or logical inspirations of the text. This analysis theme includes both external systems for understanding information and a deeper, more personal evaluation of logic/organization that the student would expect based on their prompt. An example of this filter might occur in any kind of student comment that suggests how the text could be more thorough or detailed to provide a deeper or more meaningful statement or argument. | "I feel like the structure itself is really repetitive. And if you don't give it a specific enough prompt, it kind of just goes in circles repeating the same idea that you gave it." |
| Information Literacy | A complex form of analysis students used. Several codes indicate a theme of drawing on information literacy skills that students have been taught or have incorporated into intuitive evaluations of text. This filter would include student comments about a lack of citations, the presence or absence of correct or accurate information, and notes about their need to present a different prompt to the technology. These evaluations demonstrate how students leverage information literacy skills to evaluate a text and evaluate their own request for information. Comments may address questions about the credibility of ChatGPT texts and trustworthiness of sources used in ChatGPT texts as a form of information literacy. | "I think it provided the information accurately just a lack of citations." |
| Self and Experience | This advanced form of analysis includes students thinking through their own experiences and understandings of the text, the information requested, and their own experiences as a student. Comments in this theme capture a feeling of wanting to do things themselves, commenting on the increased or decreased effort using ChatGPT would require, and worries about how this technology could put their learning at risk. This is a large category with many different codes but generally, codes were grouped in this category if the student was primarily filtering their evaluation of the text through deeply personal knowledge or experience. This filter may also apply to social justice concerns, integrity of education, and concerns around desire for rigorous education. | "If a narrative is from the first person, which it seems like it is… because that's something that I could write very easily on my own, and that I don't really see why anyone would need to use ChatGPT for that." |
| Circumstance | This is a more advanced form of analysis that students used pertaining to the student's set of individual experiences, situations, or rhetorical choices. This theme speaks to the way students would filter information through their academic identity, their major, or content from a specific class or professor. The students evaluate the text for how well it matches class content, how much it would meet the requirements of the field, what information is included or missing, and how much analysis the text was performing. | "I guess part of it is because I’m a stats major and also a big sports fan and I am kind of like aware of most of this stuff…maybe my background is changing that, but I think it’s very solid overall." |
| Process | This filter is a way that students might evaluate how ChatGPT can impact their writing process as they are working on a paper. Students may comment on how ChatGPT texts may help them generate ideas for a paper topic, provide sample writings, generate outlines, offer citations, organize ideas, articulate topic sentences, etc. | "I would definitely need to write a lot more to make this like an academic paper. But it would be a good kind of first jumping off point." |
| Function | This filter would be applied to comments in which students express curiosity about the technological capabilities of ChatGPT. Comments might address ChatGPT's technological affordances such as speed of production, how ChatGPT is technically working, where and how it is pulling data and sources, and other technological capabilities. | "I'm just impressed that the robot can come up with this stuff." "It's, it's an AI, it's reading other things as well." |