Are people more creative writers than ChatGPT?

In a recent study, UC Berkeley researcher and lecturer Nina Beguš explored how generative AI like ChatGPT measures up to human writers, especially in crafting creative narratives. While AI has made strides in many areas, it still falls short of human creativity, particularly in storytelling.

Her findings, published in Humanities and Social Sciences Communications, reveal the strengths and limitations of AI when it comes to creative writing and emphasize the importance of human values in AI's narrative capabilities.

Beguš's research is rooted in her background in comparative literature and extensive knowledge of AI. She calls her field of study "artificial humanities," a discipline that merges the humanities—history, literature, and other subjects—with AI, offering new perspectives on language and human expression within technology.

“The humanities can reveal a lot about the strengths and weaknesses of these new AI tools,” Beguš explained. “Fiction, in particular, offers a window into the collective cultural imaginary—the shared set of narratives, ideas, and symbols—that machines have inherited from us.”

To conduct her research, Beguš used a familiar storytelling motif: the ancient myth of Pygmalion. In this tale, an artist falls in love with a statue he created. This theme of creator and creation has been depicted in many modern narratives, including films like Her and Ex Machina.

Beguš provided both human participants and AI models like ChatGPT and Llama with simple prompts based on this myth. Each prompt involved an artificial human created by a protagonist who eventually falls in love with their creation.

This study generated 250 responses from human participants and 80 from AI models. Beguš then analyzed these responses, focusing on elements like gender representation, relationship dynamics, race, and cultural context, as well as overall narrative complexity.

“I was interested in that averageness,” Beguš said. “Most people are not professional writers.” She aimed to understand how average humans and AI handle basic story construction, hoping to uncover differences in narrative quality and originality.

The results showed significant differences between human and AI responses. While humans generally created more varied and nuanced stories, AI outputs tended to be repetitive and formulaic. According to Beguš, AI-generated narratives were often riddled with clichés, flat characterizations, and unmotivated actions.

These patterns highlighted the formulaic tendencies of AI, which often lack the depth and tension that make human stories compelling. “The characters were flat, generic, and unmotivated,” Beguš noted, summarizing her findings on AI-generated stories.

Despite AI’s limitations, some of its recent advancements in narrative diversity surprised Beguš. Early versions of ChatGPT often left gender and relationship types undefined, but newer models, like ChatGPT-4, showed a broader range of inclusive narratives. About one-quarter of AI-generated stories included same-sex relationships or polyamorous dynamics, a marked shift from early responses. Only 7% of human responses included similar themes, suggesting that AI models may mirror the evolving social values embedded in their training data.

"Large-language models mimic human values," Beguš explained, “and this paper shows that the values from training data can be overridden by technologists’ choices made during the process of value alignment.”

The integration of the humanities into AI research, especially in language and narrative generation, is a long-standing interest for Beguš. Although her academic journey with AI began in 2010, it wasn’t until 2020, when chatbots became more widely accessible, that she could actively test her theories.

She believes that scholars from the humanities—who have studied language, culture, and storytelling for centuries—have a crucial role to play in guiding AI development. “In the humanities, for centuries, we have been exploring and have become the experts on language, on writing, on what it means to be human,” Beguš said. “So this all just kind of naturally came together.”

Beguš’s research goes beyond academic curiosity. AI is changing how society interacts with language and writing. Universities are now incorporating tools like ChatGPT into the curriculum, helping students learn how to use these systems effectively in their work. Some professors even include Beguš’s research in their courses, underscoring the growing importance of understanding AI’s limitations and potential in education and beyond.

As a writer and researcher, Beguš frequently contemplates AI's impact on future generations and the role of writing in a world increasingly influenced by technology. She wonders if her grandchildren will marvel at the idea of writing from scratch, an activity that feels intrinsic to human culture and thought. “Writing is such an essential human activity,” she mused. “We have been taught to write since preschool. We connect our thought process with writing.”

Beguš underscores the need for writers and scholars in AI development, as quality narratives require quality creators. "We need quality writers to create quality stories,” she said. "I'm really curious about what insight writers will be able to get from machines, if there's something that is actually valuable, that is worthwhile. So far, I don't think there has been much."

However, Beguš acknowledges that AI technology is reshaping how people approach writing and storytelling. Despite its current limitations, AI has already transformed how students, writers, and professionals interact with language.

This technology, Beguš argues, is bound to continue evolving, raising questions about what it truly means to be a storyteller in an AI-driven world. Her research invites readers and writers alike to consider the cultural and ethical implications of generative AI, emphasizing that the humanities have a vital role to play in shaping the future of this technology.

Note: Materials provided above by The Brighter Side of News. Content may be edited for style and length.

Like these kind of feel good stories? Get The Brighter Side of News' newsletter.