Writing study finds AI works best when people retain creative control

Keeping people in charge of ideas and story structure produced stronger outcomes than letting an AI system run the writing process in a study of 285 students using ChatGPT 3.5. All three AI-assisted methods made participants write 36.2% faster than they did without AI, but the researchers found that quality, satisfaction and the diversity of the resulting stories depended on the division of labour.

The study, published in PNAS Nexus on Nov. 10, had students in an elective emerging-technologies course write a 1,000-word story without AI, then complete another under one of three collaboration designs. The study’s release described the Human Confirmation model as the least human-directed: the AI generated ideas, made an outline, drafted and edited the story, while the participant approved each stage.

That approach produced stories that human readers rated below the human-only work. It also left participants less satisfied and less proud of their stories than those using the other two models. Participants nevertheless valued AI’s efficiency, idea support and editorial polish, while also reporting that AI-assisted stories could lack emotional depth, originality and nuance, and could have inconsistent tone or plot.

In the Human Creativity condition, people supplied the ideas, chose the concept and built the outline before AI drafted and edited. In the Copilot condition, participants could call on AI throughout the process. Both groups reported greater satisfaction than the Human Confirmation and human-only groups, while human readers rated their stories above the no-AI baseline, according to the study release.

CNET’s account of the research described a narrower quality conclusion, saying AI use did not improve work quality in any of the three groups. It likewise found that the Human Creativity and Copilot methods outperformed Human Confirmation, generated more positive experiences and operated at the same faster pace. The shared result is that automation which relegated people to approval produced the weakest work, whereas continuing human involvement in the creative process changed the outcome.

The findings add a design-focused qualification to earlier evidence on AI-assisted fiction. In a separate online experiment involving 293 participants, access to one or five AI-generated story ideas improved rated novelty and usefulness, while gains in creativity, writing quality and enjoyment were especially pronounced for less creative writers; the AI-enabled stories were also more similar to one another than those written without AI. That experiment tested access to ideas rather than the full sequence of planning, drafting and editing examined in the newer study.

Kartik Hosanagar and Daehwan Ahn concluded that productivity gains need not require sacrificing creative diversity or worker satisfaction. Hosanagar said the goal should be to direct human effort towards stages where human judgement and creativity matter most, rather than eliminate it.