Composing with AI

Discussion

Implicit in any study of computers and writing pedagogy must be the question, “What will happen when my students use this tool to complete their work?” Setting aside the ethical and policy aspects of this question, this study attempts to answer in the most pragmatic and descriptive way possible. Any student could interact with ChatGPT basically as I've done here, asking it to complete the exercises in Williams' curriculum, without having trained CGPT or taken other, more expert preparations. What we see in the study above is the kind of results they'll mostly encounter (not exactly of course; responses will change from instance to instance). Overall, CGPT improves sentences in its revisions from their original. It is mostly successful at some tasks of diagnosis and revision. However, it also leaves problems in its revisions which require expertise to then diagnose and revise. Students who input the exercises into CGPT will only occasionally receive results that match the objectives of the curriculum. That CGPT’s failures tend to be those where interpretation, invention, and contextualization is required is not a surprise.

Interestingly, these are the same moments that students tend to struggle with, until they realize that the solutions to the stylistic problems they're finding often do not exist within the original sentence (e.g. finding appropriate characters, deciding what is a main idea). Fortunately, while CGPT’s horizontally intertextual focus may not improve on this point, students do. Using CGPT with students would present an opportunity to demonstrate limitations of these types, and promote discussions of possible remedies, as we do with student attempts in workshops together.

One potential explanation for CGPT's limitations also resides in its horizontal intertextuality and lack of contextual awareness. Being intertextual in this way, tending to reproduce what it has found to previously be most likely should create a descriptive bias rather than a prescriptive one. In the same ways that critics have discussed the bias of LLMs based on the biases of their corpuses (cf. Byrd, 2023), CGPT will revise sentences toward what other sentences have mostly tended to do, rather than what is “best” to do according to any particular stylistic paradigm, such as Williams'. Most sentences in the corpuses on which CGPT trains probably aren't clear, cohesive, coherent, concise, and shapely. Successive and amended prompts can improve CGPT's revisions, but its default is going to be to do what is descriptive of what people mostly do, not following the prescriptions of the curriculum.

This then leaves the pedagogical potential. How might using ChatGPT create or further opportunities for students' learning and my teaching of written style? When I'm teaching this curriculum, I use Williams' exercises as the jumping off point for students. Students read each Lesson and attempt to complete its exercises with minimal demonstration from me. This is low-stakes work that I've found makes them well and truly invested when I then throw their answers up on the board for everyone to see and discuss what they've improved in their revision and what is left to be done. Then we finish those revisions, live and together. I follow up with additional sentences, crafted to represent the problems Williams has given us to work on together in the classroom, and typically find the students almost competitively engaged in the game of revision. As a class, we might use LLMs in much the way we use the work of other students, to learn these skills of diagnosing and revising the problems in in first revisions of problematic sentences. While the greatest impact may come from workshopping student work, asking CGPT to provide initial diagnoses and revisions together with the class can allow us to lower the stakes of this practice even further, making CGPT a helpful baseline and stand-in for initial drafts of this revision work. When we show CGPT doing the exercises, the challenging work of diagnosing and improving these exercises will still need to be done.

CGPT seems to be best at subject-verb identification and revision and the work of revising for concision. For that percentage of my students who struggle with subject/verb identification, I'd welcome practice with the tool that might then help them begin to see the patterns that identify subjects and verbs when their classmates and I have largely moved on. When it comes to the tasks of finding the narrative implicit in sentences to serve as the basis for inventing characters and actions, determining what is given and what is new information, identifying main ideas to position them for emphasis, CGPT tends to struggle with what students struggle with. Revisions to style are not purely intra-textual activities, and the solutions to the problems in bad sentences are not often visible, evident, or explicit. We can then use CGPT together to stand in for initial draft work, to demonstrate steps toward diagnosis and revision that leave work for us to practice together.