Drafting a Policy for Critical Use of AI

Daniel Frank and Jennifer K. Johnson

Looking Ahead

As the technology continues to develop at break-neck speed, our own and our colleagues' experiences with it continue to evolve. In order to reveal some of the ranges of conversations, takes, and developing approaches, we include here a video of interview clips from some of our Writing Program colleagues detailing their thoughts, approaches, and experiences to date with Large Language Models such as ChatGPT in their classrooms. While it's clear that there is hardly a consensus among our department's faculty in regard to these tools, we are pleased to see the conversation continue to unfold and by the careful thought we see evident in their responses.

Video Transcript

James Donelan: James Donelan: I've been experiencing ChatGPT and other LLM tools in the classroom mainly as a negative influence. I teach a lot of skills at a lot of levels having to do with writing, from brainstorming and outlining to drafting and editing, and these are skills that I think need practice and exercise to develop. When I speak to students about what they do with ChatGPT, when they're trying to do something honest, what they tell me is that they're using it to get some ideas, or they're using it to help create an outline, or they're using it for proofreading, on the other end, but they're really doing the actual writing. Well, the actual writing is from start to finish, and skills that are unused do not develop. People make a lot of comparisons between ChatGPT and the introduction of calculators into math classes. And this is something, actually, the math teachers were pretty careful about. They managed to gradually introduce calculators into classrooms. They knew when to use them and when not to use them. They were pretty clear about what skills they were trying to develop at which stage, and whether those skills were forming.
James: This introduction of ChatGPT seems to me a little bit haphazard. What exactly the product is, whether it's of the right quality, whether it represents the thought of an actual human being, or represents a reflection of reality or genuine research, all these questions are up in the air, and yet it's everywhere. Students are using it, professors are using it. A more gradual and more thoughtful introduction would have been welcome, but that's not how things work these days.
James: As far as my own experience as an educator, I am trying to address the situation consciously, but so far, I haven't seen much upside. I can't think of a particular thing that the student can do with an LLM tool that they can't do better and more consciously under other circumstances. I realize that sounds very anti tech, which I'm not, but this particular tool I don't think is ready for prime time.
Paul Rogers: My experience with ChatGPT in the classroom has been positive. I've tried to have an open discussion with students about ChatGPT. I take the major assignments that I do, the prompts, and I push them through ChatGPT in front of them, so they can see what ChatGPT generates, and I certainly try to have open discussions with them about what they know and surface their background knowledge.
Paul: I'm also kind of looking for the professional guidance and sharing the professional guidance from various input sources like the APA, or journals, or the Screenwriters Guild, or other kinds of professional guidance that's being offered around how to use ChatGPT, and I think that's been interesting for the students.
Paul: As far as the ethical boundaries and guidelines, what I try to do is suggest to the students that we're not looking to try to get as close to the ethical line of plagiarism and see what we can get away with; what we're trying to do is actually create a big gap between that, so there's never a question that we would ever be accused, ever, of plagiarizing or presenting ChatGPT as if it was our own writing.
Paul: I think that finally I'll just say, I think that ChatGPT as a brainstorming partner has some real possibilities. They do an assignment in writing 107 WC where I have students brainstorm the types of content that they could create in order to meet the needs of individual users within particular communities, whether those communities are ultimate frisbee, or people studying abroad for the first time, or future doctors or whatever, and want them to see that there is some value as a brainstorming tool in ChatGPT.
Paul: But ultimately, I think that ChatGPT is here to stay, and I think it's really important that we're having this conversation. I find ChatGPT to be a little bit of a bully; kind of pedantic, it's so sure of itself, and offering me this material that, you know, I'm never going to be able to create text as fast as ChatGPT does, but I find the writing itself to be somewhat lifeless. So I think that it's a wave that we have to go through, and I really want to thank the leaders of the writing program for facilitating this discussion. I'm learning a lot. So thank you very much.
Christine d'Anca: So as more and more academics began incorporating ChatGPT into the classroom, I began thinking of how I would as well. Honestly, at first, I was hesitant, not because I didn't think it could work, but because I didn't know how. I didn't even know where to begin. Nevertheless, I was determined to show my students that using this technology was not going to get them what they desire.
Christine: Recalling Ben Rafoth's article, Why Visit Your Campus Writing Center" (Writing Spaces 1, 2010), in which he argues that students will ultimately do whatever it takes to get what they need in order to receive a better grade. So I thought if only I could demonstrate to them that ChatGPT is not going to provide them with the sufficient materials to earn good grades, they would stop using it and the problem would solve itself.
Christine: I was on the right track, but had not yet realized I was going about it wrong. As I began playing around with different potential lesson plans, I was inspired by Sydney Dobrin's Talking About Generative AI: A Guide For Educators" (Broadview Press, 2013). I came up with a mini lesson that would have my students deconstruct an AI generated response to a prompt, in order to reverse engineer the writing process that is not transparent in the final result, because a chatbot does not show its work, or produce rough drafts in the process of coming up with its answer. Then we could discuss the different rhetorical shortcomings. After which I'd have students revise the text, incorporating the different elements that would strengthen it while also having them conduct a meta cognitive reflection of their process. This would get them to learn and implement the various writing and literary conventions I teach, while also becoming better acquainted with the underlying mechanisms of Generative AI.
Christine: I was very pleased with the idea, but then decided to go a step further. So I decided, at first have my students use ChatGPT to generate an outline for a paper prompt and see what we could do with it. While the outline produced was seemingly comprehensive and very, very lengthy, my students were able to immediately identify a large problem with it: the outline provided only basic ideas about what the different sections of the paper should cover, without actually giving any details as to what those ideas should be. So for example, it would say something like, "identify the main arguments and evidence the author presents." OK. So what are those arguments? What are those main ideas? So I asked ChatGPT to get more specific. It generated a new outline. While it rearranged some of the points, the examples that provided were still very generic.
Christine: My students were initially upset but it doesn't actually give them any of the important information that they can incorporate into their papers. Yet it soon became clear that this is not in fact a problem but rather a solution. It allows them to use the technology to help organize the various components of the paper while doing all of the heavy lifting themselves. Thus, the outline becomes a guide for them to work with the text, and find the necessary information themselves, identify the arguments, the uses of evidence, et cetera. In other words, ChatGPT provided the perfect starting point, without doing any of the actual work for the students. It's no different than working with a template like the similar ones I've given them in class, which basically makes the process ethical, and also beneficial as from the multiple points created by the outline, they can pick and choose those that address the prompt before flushing it out with details from the text.
Christine: So, I'm still playing around with all of these ideas, and I look forward to actually see how my students interact with these processes over the academic year, and learn from these experience in in order to produce better assignments in the future.
Chris Dean: So the question is, what's your experience been with LLM tools such as ChatGPT in the classroom so far. So last year, particularly in Writing Two, we actually did a little bit of work with it around actually a prompt that we were using; I didn't use any student work because for me, that's a big no-go, don't want student work to be trained on by ChatGPT, but my prompt, that's fine, and the responses we got back were, predictably, very ChatGPT like; largely general, pulling from web sources, all the sorts of things that the prompt actually tells them not to do. What was very interesting though was my students asking them, based on the prompt, to try and create a 4 to 6 page paper. And this was an argumentative piece in Writing One at the time, and we got ChatGPT at the time to really kind of hallucinate like crazy when it was pushed on argumentative topics, which I thought was pretty interesting.
Chris: This quarter, what I'm doing is trying to get each and every class that I have to think about ChatGPT, first of all, and then we're gonna create language, and we're in the process of doing that next week, around the syllabus, and sort of how we agree that we should use ChatGPT right now. The big things are, you know, we're going to cite everything, we're not going to use it for final drafts, and we're going to agree to use it, both them and I, based upon some of the things that we want to say. We're also going to have, particularly in my Writing 109 ED class, since it's about the teaching of writing and education just more generally, some actual experiments with ChatGPT to see how it's used, because it's already being used, particularly in K through 12, by a lot of teachers, certainly a lot of students, and we have everything from blanket bans to people really sort of trying to push the technology and to see what it can do.
Chris: So it's gonna be definitely a part of the class, probably about three or four times in 109 ED, and a couple of times in Writing One and Writing Two, but the big thing is I think the co-construction of syllabus language and a talk around what it is that ChatGPT actually is and isn't, and how we sort of are going to agree to ethical standards for how we all use it.
Katie Baillargeon: So my experiences with ChatGPT were that once I started fiddling with it, I realized, in retrospect, that some of my students had clearly been using it, and had been using it for some of the larger assignments, assignments that I had kicked back, actually, as being incomplete, not responding to the prompt, and being problematic enough that they needed to rewrite them. And so, I then thought, OK, what am I going to do about this? And I had two ideas in mind with that "What am I going to do?". Number one was that it's a tool that's available to them, and that can save time when used responsibly and to create a rhetorically effective piece, so I can't really blame them for wanting to use that.
Katie: And then second of all, at the moment, back then, when I didn't know or think about that, they would have been using ChatGPT to I guess, draft their work, I just assumed that it was an early stage draft, and the student had run out of time, and submitted that instead of a final piece. So with those things in mind, and with knowing as well, my friends and family who work in non-academic sectors, and use ChatGPT to help them draft their own pieces of writing-- and it's always the draft stage, right? This isn't the final thing, it's something to start with and to kind of boost them to another level so that they can save time. So then I thought, well, is it not then my position as a writing instructor at a university to help students figure out, 'OK, how can I use this tool responsibly to create rhetorically effective pieces?' so that's what I do in my classes; in class on any given day with a particular genre, like a literature review, so it's OK, you've got your own list of 20 plus sources with abstracts that you've kind of already started to think about connections between these works, right now ask ChatGPT to write a literature review on your research topic. And all right, what do we have to discard? Because there's a lot to discard, we can't use much of it. And then what are the kernels of things that we can build upon and improve upon with what I know? So there's that human element to it. It does save a little bit of time, it kind of gives some help and some ideas for drafting, but it is not the end product at all. And I feel like that is, hopefully, the best way I can use it, for now, until it changes again. But yeah, so those are my experiences.
Victoria Houser: Hi, my name is Victoria Houser, and I'm an assistant teaching professor in the writing program at the University of California, Santa Barbara. I've been using ChatGPT in my classrooms in a few different ways, and I haven't used it yet this quarter, but I have used it in the past to help me do things like generate rubrics, and even revise project prompts, and do things like also generate questions for freewriting responses.
Victoria: And so everything that I've asked ChatGPT to do comes from a place of understanding what that tool is while we're using it, and before we use it, and after we use it. So, the discussion I have with my students is centered around how do we engage with this technology as human beings who also have nuance and all kinds of unique attributes to what we're bringing to our writing? What do we do with a tool like this? And so that was really the first step that I took was to generate a rubric for an assignment in class with my students, and we looked at what ChatGPT brought up and there were lots of things that my students said, ' I would like to see this thing not on there, or I would like to see this thing included.' And so we were able to ask ChatGPT to make those changes for us. And that discussion led us to think through together how ChatGPT is really a tool that requires a certain level of critical reflection and rhetorical thinking, right? Thinking about the ethics around it, and how we use it, and why we use it, and what it does for us as writers. And so, of course, this is still an evolving and ongoing conversation, but I look forward to seeing how my students pick this up. And I think one of the directions I would love to see this go is to kind of demystify what ChatGPT is for students, right? And for educators, of course, as well, that it isn't necessarily this giant evil plagiarism churning device, right?
Victoria: It is something that, like all technologies, we can engage with critically and reflect on what it means for us, what it does for us, and what we do in the context of the space itself. So I look forward to seeing the future conversations that come from reflection such as this.

There is still much to figure out. The conversation is by no means over and the answers are far from set in stone. Indeed, much of the approach heard in both this video and in our reflections relies on the fact that without hands-on mediation and reflection, the writing itself won’t be good enough to do the job. This might not always be the case. What we must promote to students, then, is the value and importance of their own reflection, voice, and the power and importance of taking ownership over the choices involved in interacting with the tools. Inherent in this—and this came up several times in faculty discussions—is the question of motivation. Why is a student writing? What are the goals? Does the student have a purpose in mind? Do they have a desire to develop knowledge, expertise, an individual and powerful voice? Or is the student going through the motions in order to get the grade and move on? Unfortunately, the grade-based, transactional structure of the traditional educational apparatus tends to promote the latter rather than the former. There very well may be a point down the line when these technologies are advanced enough to make this transactional structure untenable. If we get to that point, we will all have a lot of work to do. A lot to rethink. But forgive a hint of optimism here when we suggest that perhaps we might restructure for the better.